Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To this end, GAPS (i) addresses the partial-annotation problem by leveraging copy-paste to generate fully-labeled data for training, (ii) helps augment the few images of novel objects by introducing a guided sampling process, and (iii) mitigates catastrophic forgetting by employing a diverse memory-replay buffer.

The review comment to be evaluated is:

Strengths:
- The paper introduces the copy-paste data augmentation strategy to the few-shot incremental semantic segmentation, which seems to be a novel combination.
- The proposed guided sample selection method for memory construction and data synthesis improves the performance of several baseline methods.
- The paper is most clearly written and easy to follow.

Weaknesses:
- While the proposed data augmentation strategy helps few-shot learning, its technical novelty in the incremental semantic segmentation (ISS) is rather limited, as it is an add-on module and has to rely on the base ISS methods.
- The construction of the memory-replay buffer has several limitations and will affect the quality of the data augmentation in ISS:
1) Only the base stage has fully-annotated images that can be used for the target images in the copy-paste operation, and all the images from the incremental stages are partially labeled, which are unusable for that purpose. As a result, when the incremental learning evolves, the memory has to add samples of new classes and remove some base-stage images due to the capacity limit. Eventually, the memory buffer would be less useful.
2) The diversity-guided selection procedure in Sec 3.2 considers each class separately in sampling. However, for semantic segmentation, it is common that each image has regions of multiple classes. It is unclear how such a method can balance different classes. This is also a problem for the class-frequency-guided selection, as each image can have multiple classes. Another issue is that this method did not take into account the region sizes as it only considers the image-level sampling factor.
- There are several concerns on the experimental evaluation, which make the conclusion less convincing:
1) Both benchmarks are mainly used in the few-shot segmentation literature, which are a bit small in terms of data set size and number of classes. Similar to SSUL and other ISS works, the method should be evaluated on more diverse ISS task settings and larger dataset, such as ADE.
2) For the few-shot side, while this work adopts a different evaluation protocol (as described in Sec. 4.2), it would be more informative if it also uses the protocol in Cermelli et al 2021 for a fair comparison. Moreover, unlike the standard ISS, the training process of the few-shot setting typically has a large variance due to different training samples annotated. Thus it should report both the mean and standard deviation in the final results.
3) The gaps in the ablation study are relatively small. It would more convincing if the standard deviation can be provided. It is unclear how significant such improvements are in the few-shot learning setting.

The evidence to be evaluated is:It is known that many learning-based models suffer from catastrophic forgetting (McCloskey and Cohen, 1989), a pheonomenon that causes models to perform significantly worse on old tasks when they are fine-tuned to adapt to new tasks.

In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

Copy-and-paste is an augmentation technique that copies a subset of objects from one image and pastes onto the other image using their segmentation masks. Many works (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018) have been done to investigate how copy-and-paste augmentation can help with various visual tasks. Dvornik _et al._ apply copy-paste augmentation in object detection by designing a neural network to consider context and guide copy-paste.

More recently, Ghiasi _et al._ conduct extensive experiments to demonstrate the effectiveness of simple copy-paste in the instance segmentation problem. We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting.

In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model. After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2). During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\). The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4).

For methods with memory-replaying (e.g., SSUL (Cha et al., 2021)), GAPS can work directly on top of their constructed buffers with minimal modification. For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi et al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.

Intuitively, we approximate the difficulty of every sample by their similarity between the estimated prototype with learned prototypes. Estimated prototypes that are close to the learned prototype are considered easy samples and vice versa. After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

number of exemplars \(n\) \(k\leftarrow\textsc{Floor}(n/|Y_{0}|)\)// Sample per class for\(c\) from \(1\) to \(|Y_{0}|\)do \(S_{c}\leftarrow\{(x_{i},y_{i})\in D_{0},c\in y_{i}\}\) for\((x_{i},y_{i})\in S_{c}\)do \(p_{i}\leftarrow\textsc{MAP}(x_{i},y_{i},c)\)// Pred. Proto. \(s_{i}\leftarrow\textsc{CosineSimilarity}(p_{ic},w_{c})\) endfor  Sort \(S_{c}\) by similarity score \(s_{i}\) \(ES_{c}\leftarrow\{\}\)// final exemplar set of class \(c\) for\(j=1,2,\dots,k\)do \(L_{idx}\gets j\cdot|S_{c}|/k\) \(U_{idx}\leftarrow\textsc{Min}(Lower+|S_{c}|/k,|S_{c}|)\) \((x,y)\leftarrow\textsc{Sample}(S_{c}[L_{idx}:U_{idx}])\) \(ES_{c}\gets ES_{c}\cup(x,y)\) endfor endfor \(D_{0}\leftarrow\textsc{UniformSample}(\bigcup_{i=1,\dots,|Y_{0}|}ES_{i},n)\) ```**Algorithm 1** Construct Memory-replay Buffer We hypothesize that synthesizing novel objects onto contextually consistent base images would result in an improved learning process. For example, a TV in an apartment should more likely be pasted onto an image of another apartment rather than an outdoor landscape. We design a context-guided sampling algorithm to select images from \(\hat{D_{0}}\) that are contextually similar to the provided partially-labeled images. GAPS uses a scene embedding network \(h:\mathcal{X}\rightarrow\mathbb{R}^{m}\) to estimate pairwise contextual similarity between any two given images. The embedding \(h\) maps an input image to a scene embedding vector in a metric space where pairwise similarity comparison between two embeddings are possible. In our system, we use an off-the-shelf VGGNet (Simonyan and Zisserman, 2015) and replace its last fully connected layer with a cosine-similarity-based classifier. The network \(h\) is trained using the Places365 (Zhou et al., 2017) dataset, which contains \(365\) scene categories and roughly 1.6 million images in its training split. During training, the scene prediction is done by comparing the predicted embedding with existing prototypes.

To find contextually similar base images to each novel image, we evaluate the cosine similarity of the novel image to each of the examples in \(\hat{D}_{0}\), and constuct a contextually similar subset \(\mathbb{S}\) with \(|\hat{D}_{0}|/10\) most contextually similar examples.

To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from \(\mathbb{S}\) with a probability of \(\alpha\), and sample from \(\hat{D}_{0}\) with a probability of \(1-\alpha\), where \(\alpha\) is a hyperparameter set to \(0.9\) in our implementation.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes.

Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis.

If \(c\in\mathcal{C}_{j}\) with \(j\geq 1\), we first compute its class frequency by \(f_{c}=nShot/|\hat{D}_{0}|\), where \(nShot\) denotes the number of images in \(D_{j}\) with at least one pixel of \(c\). Then, the category-wise sampling factor for \(c\) is given by \(r_{c}=\textsc{Max}(1,\sqrt{t/f_{c}})\).

However, we empirically found that setting \(t\) to be the multiplicative inverse of total number of classes, or \(t=1/|\cup_{1,...,j}Y_{j}|\), is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings.

We first decide if \(c_{o}\) should be pasted onto \(x_{B}\). To apply vRFS resampling, we hallucinate two _virtual_ samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of \(r_{i}=\textsc{Max}_{c\in i}r_{c}=r_{c_{o}}\). Thus, the probability to synthesize class \(c_{o}\) onto \(x_{B}\) is given by \(r_{c_{o}}/(1+r_{c_{o}})\). We then repeat the process for the novel class \(c_{N}\). Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of \(c_{N}\) in the final image.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now.

**Our diversity-guided exemplar selection method consistently increases performance on base categories**, which suggests that it is capable of choosing diverse samples to construct a representative memory-replay buffer and mitigate catastrophic forgetting. **Context-guided sampling steadily improves performance on novel classes**, which is consistent with findings in previous work (Dvornik et al., 2018) that background context is an important factor to consider in copy-paste synthesis. **Frequency-guided probabilistic synthesis boosts results on novel classes**.

We take a closer look at step-wise performance (whose visualization is available in Appendix A.11) and found that the reason is due to unguided copy-paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-replay buffer. Though the semantic background shift proposed by Cermelli et al. (2020) relaxes the requirement to provide full segmentation annotations, it still requires _all novel instances_ in images to be annotated, which can be time-consuming to obtain in cluttered scenes and hinder potential applications. Here we consider a more challenging task setting, which we term _single-instance incremental learning_. Namely, for training images provided in incremental learning stages, if there are multiple instances of a novel class in the image, we assume that only _one instance_ will be annotated. To simulate this setting, we use the instance-level segmentation annotation provided by the COCO dataset to enforce only annotation of one novel instance in every image is available to the model.

This highlights the potential of copy-paste synthesis as an alternative to the existing pseudo-labeling paradigm to cope with the more realistic single-instance setting.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We believe that copy-paste can serve as a promising alternative to pseudo-labeling and modified loss to enable learning on partially-annotated images. We also believe that further optimizing exemplar selection and sampling strategies can lead to better guidance and lead to even better performance.

In Table 6, we study how different memory buffer construction strategies influence the performance of the model. We experiment with multiple baseline implementations to construct the memory-replay buffer: _Full-set-random_ denotes samples that are uniformly sampled from the base training set. _Classwise-random_ denotes randomly selecting even number of samples from each base class. To consider multiple classes in images, we design two baseline methods. RFS (Repeat Factor Sampling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights to rare classes and has been shown to work well in long-tailed task settings. The other baseline, 'GreedyClass', is achieved by greedily picking samples in every base class with the most number of classes. To consider the region sizes in the image, we design a base-line method, 'BalancedRegion', which uses a similar algorithm as our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images. Intuitively, easy samples help model maintain learned prototypes during incremental learning, and hard samples help model distinguish decision boundaries. Hence, based on our sample difficulty ranking method based on prototypical distances, we also construct a few variants: _Easy_ denotes samples whose predicted embeddings are closest to learned prototypes; _Hard_ represents difficult samples whose predicted embedding are most distant to learned prototypes in the learned metric space; _Easy-Hard-Mix_ denotes equal mixtures of samples drawn from easiest samples and hardest samples. Lastly, _Diverse-Sampling_ is our proposed method, where samples of all difficulties are equally drawn. We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling.

We investigate how different probabilistic strategies to apply copy-and-paste augmentation affects GAPS.

In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes. For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Beyond confirming conclusion from previous work, such improvement alongside stronger augmentation demonstrates the effectiveness of augmentation technique even when there is few data presented, highlighting a possibility to further improve the performance by using even stronger augmentation technique. We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting.

We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS.

We explore an alternative memory-replay buffer implementation to add examples learned during the incremental learning stage to the memory-replay buffer, in an attempt to help performance on novel classes. Since examples observed during the incremental learning stage are only partially annotated, they can not be directly added to the memory-replay buffer.

We can observe that storing samples observed during the incremental learning stage leads to only moderate increase in novel IoU. Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column. We can see that PIFS and PIFS+NPS gives reasonably good predictions on classes that are adapted recently (e.g., the cow class in the last row). However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

On the other hand, PIFS(\(\mathcal{L}_{2}\))+GAPS is able to maintain good performance on learned classes as it adapts to new classes. \begin{table}
\begin{tabular}{c|c c c} Base Set & Base & Novel & HM \\ \hline Full-set-random & \(47.9\pm 0.2\) & \(24.6\pm 0.7\) & \(32.5\) \\ Classwise-random & \(48.3\pm 0.2\) & \(24.7\pm 0.8\) & \(32.7\) \\ RFS & \(47.6\pm 0.3\) & \(25.2\pm 0.7\) & \(33.0\) \\ GreedyClass & \(48.7\pm 0.2\) & \(25.0\pm 0.7\) & \(33.0\) \\ BalancedRegion & \(48.6\pm 0.2\) & \(24.9\pm 0.8\) & \(32.9\) \\ \hline Easy & \(48.0\pm 0.2\) & \(24.2\pm 0.8\) & \(32.2\) \\ Hard & \(47.7\pm 0.2\) & \(25.2\pm 0.8\) & \(33.0\) \\ Easy-Hard-Mix & \(48.4\pm 0.3\) & \(24.8\pm 0.7\) & \(32.8\) \\ Diverse Sampling (Ours) & \(\mathbf{48.9}\pm 0.2\) & \(\mathbf{25.3}\pm 0.9\) & \(\mathbf{33.3}\) \\ \end{tabular}
\end{table}
Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity-aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity-aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 runs are reported.

Input: Figure 4: Performance of PIFS(L2)+GAPS on the COCO-20-1 split under 5-shot setting with varying memory-replay buffer size. Best view in color.

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 6: Qualitative visualization of samples selected in the context-aware sampling process. Given an image of traffic lights in railway, our scene embedding network is able to select contextuallysimilar scenes (railway, street).

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input: Figure 5: Qualitative results on the PASCAL-5-1 split under 1-shot setting. Sequential adaptation to five classes are performed in the order indicated by the blue arrow on the left. Visualization is done after all adaptation steps are completed. Observe, particularly in rows 1, 3, and 4, that our method mitigates catastrophic forgetting and improves performance on both base and novel classes compared to PIFS and PIFS+NPS. (Best viewed in color)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence provided, I'll evaluate the claim's three main components:

1. Addressing partial-annotation: The evidence demonstrates GAPS effectively uses copy-paste to generate fully-labeled data, showing strong results even with single-instance annotation, supported by quantitative improvements in performance metrics.

2. Augmenting novel objects: The guided sampling process is well-documented through context-aware selection and frequency-guided synthesis, with ablation studies showing consistent improvements.

3. Mitigating catastrophic forgetting: The diversity-guided memory buffer selection shows statistically significant improvements over baselines (Table 6), though the gains are relatively modest (differences of 1-2 percentage points).

While all three claims are supported by evidence, the magnitude of improvement, particularly for catastrophic forgetting mitigation, is somewhat modest. The claim is generally accurate but slightly overstates the strength of the improvements.

<score>2</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Compared to existing state-of-the-art methods, GAPS dramatically boosts the novel IoU of baseline methods on established few-shot incremental segmentation benchmarks by up to 80%.

The evidence to be evaluated is:This problem has been studied extensively in image classification (Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Rebuffi et al., 2017; Mallya et al., 2018; Castro et al., 2018; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of semantic segmentation (Cermelli et al., 2020; Michieli and Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021; Zhang et al., 2022).

In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting. The results are given in Table 9. We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column. We can see that PIFS and PIFS+NPS gives reasonably good predictions on classes that are adapted recently (e.g., the cow class in the last row). However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Input: Table 9: Performance of PIFS(L2) and PIFS(L2)+GAPS on the ADE100-5 disjoint 5-shot setting. Mean and standard deviation over 5 runs are reported.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, the statement that "GAPS dramatically boosts the novel IoU by up to 80%" appears to be overstated. While the evidence shows GAPS does improve performance consistently, the actual improvements shown in Table 9 are more modest - novel IoU increases from 3.5% to 5.9%, which is about a 69% relative improvement. The qualitative results and other experiments demonstrate consistent benefits of GAPS, but the specific claim of "up to 80%" improvement isn't directly supported by the numerical evidence presented. The word "dramatically" also adds unnecessary hyperbole.

<score>4</score>

Justification:
1. The evidence shows clear improvements but not at the claimed 80% level
2. The strongest numerical evidence (Table 9) shows ~69% improvement in novel IoU
3. While GAPS shows consistent benefits across experiments, the use of "dramatically" overstates the magnitude
4. The claim lacks important context about specific experimental conditions
5. The evidence demonstrates meaningful improvements but the claim's specific numerical assertion isn't supported


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Compared to existing state-of-the-art methods, GAPS dramatically boosts the novel IoU of baseline methods on established few-shot incremental segmentation benchmarks by up to 80%.

The review comment to be evaluated is:

Strengths:
1. The paper investigates a novel and understudied scenario that is more realistic than earlier benchmarks.
2. The study gives a comprehensive summary of similar works as well as comparisons and contrasts between them.
3. The study expands the present benchmark with new experiments involving only one labeled instance of the new classes, demonstrating that the GAPS can address the issue of missing annotations.

Weaknesses:
1. The ablation study may be extended to other settings and may provide more insights about the paper choices. In particular, it may start from a baseline that uses exemplars without pasting the new classes on them, to show that pasting is important to improve performance and that the gain is not only given by the regularization effect of using exemplars (as proposed by GDumb [a]). Moreover, the paper should clarify if, while disabling the diversity guidance, the exemplars are sampled uniformly for the old classes or if they are randomly sampled on the full dataset.
2. From the ablation study, context sampling appears to marginally improve the results. Moreover, it is not clear how the performance will change by using a pre-training dataset different from Places-365, such as ImageNet or ADE20K. Furthermore, can it be trained directly on the base dataset of the considered setting or can the method use the segmentation network itself (for example, using an auxiliary classifier on the encoder features)?
3. The paper baselines appear to be very naive. In particular, the paper reports the results for Fine-Tuning, MiB, and PIFS without considering exemplars, introducing an unfair comparison with respect to GAPS. The baselines should be reported both without exemplars (as in the original paper) and with them (indeed, without the proposed copy-and-paste strategy).

[a] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. "Gdumb: A simple approach that questions our progress in continual learning." European conference on computer vision. Springer, Cham, 2020.

The evidence to be evaluated is:This problem has been studied extensively in image classification (Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Rebuffi et al., 2017; Mallya et al., 2018; Castro et al., 2018; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of semantic segmentation (Cermelli et al., 2020; Michieli and Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021; Zhang et al., 2022).

In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting. The results are given in Table 9. We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column. We can see that PIFS and PIFS+NPS gives reasonably good predictions on classes that are adapted recently (e.g., the cow class in the last row). However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Input: Table 9: Performance of PIFS(L2) and PIFS(L2)+GAPS on the ADE100-5 disjoint 5-shot setting. Mean and standard deviation over 5 runs are reported.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim significantly overstated for several reasons:

1. The claim states "up to 80% boost" but the evidence doesn't show such dramatic improvements. From Table 9, PIFS(L2)+GAPS shows an improvement in Novel IoU from 3.5% to 5.9%, which is about a 69% relative increase, not 80%.

2. The review points out that the baselines appear naive and the comparisons may be unfair since they don't consider exemplars, suggesting the improvements might be inflated.

3. The evidence shows consistent improvements but more modest ones than claimed. The qualitative results and experiments demonstrate benefits of GAPS but not the dramatic "80%" boost claimed.

4. The paper acknowledges limitations and variation across different settings, which the claim doesn't reflect.

5. The review notes that some components like context sampling only marginally improve results.

<score>4</score>

The claim would be better stated as "GAPS shows consistent improvements over baseline methods across few-shot incremental segmentation benchmarks, with relative improvements in novel IoU up to 69%."


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Compared to existing state-of-the-art methods, GAPS dramatically boosts the novel IoU of baseline methods on established few-shot incremental segmentation benchmarks by up to 80%.

The review comment to be evaluated is:

Strength:
 (1) This paper introduces copy-paste as a synthesis technique to few-shot incremental segmentation.
 (2) Experiments show clear improvement on performance with the proposed method.

Weaknesses:
(1)  My main concern is the novelty of the paper, copy-paste technique is more about a data augmentation methods, which has already been shown effective in many segmentation task, it is not novel enough to be accepted as an ICLR paper.
(2) The paper is not clear at least for me, especially figure 1 and figure, it's really hard for readers to understand the task, although the idea is clear.

The evidence to be evaluated is:This problem has been studied extensively in image classification (Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Rebuffi et al., 2017; Mallya et al., 2018; Castro et al., 2018; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of semantic segmentation (Cermelli et al., 2020; Michieli and Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021; Zhang et al., 2022).

In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting. The results are given in Table 9. We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column. We can see that PIFS and PIFS+NPS gives reasonably good predictions on classes that are adapted recently (e.g., the cow class in the last row). However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Input: Table 9: Performance of PIFS(L2) and PIFS(L2)+GAPS on the ADE100-5 disjoint 5-shot setting. Mean and standard deviation over 5 runs are reported.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim "GAPS dramatically boosts the novel IoU by up to 80%" to be significantly overstated. The evidence, particularly Table 9, shows PIFS(L2)+GAPS achieving a novel IoU of 5.9% compared to PIFS(L2)'s 3.5% - an improvement of about 69%, not 80%. While GAPS does show consistent improvements across different experimental settings and metrics, the specific claim of "80%" is not directly supported by the presented evidence. Additionally, the term "dramatically" may overstate the magnitude of improvement, especially given the relatively low absolute performance numbers.

<score>4</score>

Justification:
1. The claim uses exaggerated language ("dramatically") when describing modest absolute improvements
2. The stated 80% improvement is not supported by the numerical evidence presented
3. While GAPS does show consistent improvements across different scenarios (as shown in qualitative and quantitative results), the magnitude is overstated
4. The reviewer comments acknowledge improvements but don't specifically validate the 80% claim
5. The evidence shows more modest improvements (e.g., ~69% in Table 9 for novel IoU)

The claim would be more accurate if it stated something like "GAPS consistently improves the novel IoU of baseline methods by 40-70% across various few-shot incremental segmentation benchmarks."


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Compared to existing state-of-the-art methods, GAPS dramatically boosts the novel IoU of baseline methods on established few-shot incremental segmentation benchmarks by up to 80%.

The review comment to be evaluated is:

Strengths:
- The paper introduces the copy-paste data augmentation strategy to the few-shot incremental semantic segmentation, which seems to be a novel combination.
- The proposed guided sample selection method for memory construction and data synthesis improves the performance of several baseline methods.
- The paper is most clearly written and easy to follow.

Weaknesses:
- While the proposed data augmentation strategy helps few-shot learning, its technical novelty in the incremental semantic segmentation (ISS) is rather limited, as it is an add-on module and has to rely on the base ISS methods.
- The construction of the memory-replay buffer has several limitations and will affect the quality of the data augmentation in ISS:
1) Only the base stage has fully-annotated images that can be used for the target images in the copy-paste operation, and all the images from the incremental stages are partially labeled, which are unusable for that purpose. As a result, when the incremental learning evolves, the memory has to add samples of new classes and remove some base-stage images due to the capacity limit. Eventually, the memory buffer would be less useful.
2) The diversity-guided selection procedure in Sec 3.2 considers each class separately in sampling. However, for semantic segmentation, it is common that each image has regions of multiple classes. It is unclear how such a method can balance different classes. This is also a problem for the class-frequency-guided selection, as each image can have multiple classes. Another issue is that this method did not take into account the region sizes as it only considers the image-level sampling factor.
- There are several concerns on the experimental evaluation, which make the conclusion less convincing:
1) Both benchmarks are mainly used in the few-shot segmentation literature, which are a bit small in terms of data set size and number of classes. Similar to SSUL and other ISS works, the method should be evaluated on more diverse ISS task settings and larger dataset, such as ADE.
2) For the few-shot side, while this work adopts a different evaluation protocol (as described in Sec. 4.2), it would be more informative if it also uses the protocol in Cermelli et al 2021 for a fair comparison. Moreover, unlike the standard ISS, the training process of the few-shot setting typically has a large variance due to different training samples annotated. Thus it should report both the mean and standard deviation in the final results.
3) The gaps in the ablation study are relatively small. It would more convincing if the standard deviation can be provided. It is unclear how significant such improvements are in the few-shot learning setting.

The evidence to be evaluated is:This problem has been studied extensively in image classification (Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Rebuffi et al., 2017; Mallya et al., 2018; Castro et al., 2018; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of semantic segmentation (Cermelli et al., 2020; Michieli and Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021; Zhang et al., 2022).

In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting. The results are given in Table 9. We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column. We can see that PIFS and PIFS+NPS gives reasonably good predictions on classes that are adapted recently (e.g., the cow class in the last row). However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Input: Table 9: Performance of PIFS(L2) and PIFS(L2)+GAPS on the ADE100-5 disjoint 5-shot setting. Mean and standard deviation over 5 runs are reported.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim to be significantly overstated. The statement "dramatically boosts the novel IoU...by up to 80%" is not directly supported by the presented evidence. Looking at Table 9, which shows results on ADE100-5, PIFS(L2)+GAPS achieves 5.9% novel IoU compared to PIFS(L2)'s 3.5% - an improvement but far from the claimed 80%. The qualitative results and other evidence show consistent improvements, but not of the dramatic magnitude claimed. Additionally, the reviewer points out that the gaps in ablation studies are relatively small and calls for reporting standard deviations to establish statistical significance.

<score>4</score>

Justification:
1. The evidence shows consistent but modest improvements:
- Table 9 shows improvement from 3.5% to 5.9% for novel IoU
- Qualitative results show better performance but not "dramatic" improvements
- Reviewer notes "gaps in the ablation study are relatively small"

2. Exaggeration issues:
- The claim of "80%" improvement is not supported by any direct numerical evidence
- Use of words like "dramatically" overstates the actual demonstrated improvements
- The magnitude of improvement is not consistently demonstrated across different datasets and scenarios

3. Limitations not acknowledged:
- Reviewer points out several experimental limitations including dataset size and evaluation protocols
- Standard deviations and statistical significance of improvements are not consistently reported
- Technical novelty is limited as it's an "add-on module"

The claim would be better stated by specifically citing the actual improvement percentages shown in the results and avoiding dramatic language that isn't supported by the evidence.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

More notably, GAPS maintains good performance in even more impoverished annotation settings, where only single instances of novel objects are annotated.

The evidence to be evaluated is:However, existing work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting.

During the _incremental learning stage_, a sequence of tasks \(\{D_{1},D_{2},\dots\}\) with novel categories is presented to the model, where \(D_{j}=\{(x_{i},y_{i})|x_{i}\in\mathcal{X},y_{i}\in\mathcal{Y}^{\mathcal{C}_{j}}\}\) and \(\mathcal{C}_{j}\) is a set of classes for task \(D_{j}\). In few-shot learning, the size of the training sets for the novel tasks is small, _i.e_., \(|D_{j}|\ll|\mathcal{D}_{0}|\).

To enforce the partially-annotated image setting, we follow Cermelli _et al_. and assume that only novel classes are annotated, _i.e_., \(\mathcal{C}_{i}\cap C_{j}=\varnothing\) for all \(i\neq j\).

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model.

During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\).

In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

**Frequency-guided probabilistic synthesis boosts results on novel classes**.

Though the semantic background shift proposed by Cermelli et al. (2020) relaxes the requirement to provide full segmentation annotations, it still requires _all novel instances_ in images to be annotated, which can be time-consuming to obtain in cluttered scenes and hinder potential applications. Here we consider a more challenging task setting, which we term _single-instance incremental learning_. Namely, for training images provided in incremental learning stages, if there are multiple instances of a novel class in the image, we assume that only _one instance_ will be annotated. To simulate this setting, we use the instance-level segmentation annotation provided by the COCO dataset to enforce only annotation of one novel instance in every image is available to the model.

We can observe that the pseudo-labeling baseline, \(\text{PIFS}(\mathcal{L}_{2})^{\dagger}\), yields substantially worse performance when the model receives single-instance annotations.

On the contrary, \(\text{PIFS}(\mathcal{L}_{2})\)+GAPS shows only a minor performance decrease with single instances. This highlights the potential of copy-paste synthesis as an alternative to the existing pseudo-labeling paradigm to cope with the more realistic single-instance setting.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images.

Finally, the ability to learn with as few as one annotated instance in an image raises several intriguing possibilities. For example, integrating our work with learning-based interactive segmentation will enable human operators to continually and adaptively teach novel classes and correct failed predictions. This workflow has many interesting applications such as robot teleoperation where sparse annotations are preferable. Learning with weaker annotations, like bounding boxes or single clicks, and even self-supervision, is also an interesting direction to explore.

In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes.

In the result, we can observe that our method (vRFS) achieves the best result in terms of the novel IoU and harmonic mean of base and novel IoU.

We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting. The results are given in Table 9. We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

We conducted an additional experiment on the COCO-20-1 dataset under the 5-shot setting, where we vary the number of fully-annotated images in the memory-replay buffer to investigate the per formance of our method. As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column.

On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

On the other hand, PIFS(\(\mathcal{L}_{2}\))+GAPS is able to maintain good performance on learned classes as it adapts to new classes.

Input: Table 9: Performance of PIFS(L2) and PIFS(L2)+GAPS on the ADE100-5 disjoint 5-shot setting. Mean and standard deviation over 5 runs are reported.

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 7: Multi-step qualitative results of our method (GAPS) on the PASCAL-5-1 split under 1- shot setting. (Best view in color.)

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input: Figure 5: Qualitative results on the PASCAL-5-1 split under 1-shot setting. Sequential adaptation to five classes are performed in the order indicated by the blue arrow on the left. Visualization is done after all adaptation steps are completed. Observe, particularly in rows 1, 3, and 4, that our method mitigates catastrophic forgetting and improves performance on both base and novel classes compared to PIFS and PIFS+NPS. (Best viewed in color)

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence, I find the claim that "GAPS maintains good performance in even more impoverished annotation settings, where only single instances of novel objects are annotated" to be well-supported by the evidence. The paper demonstrates this through:

1. Direct experimental comparison showing only minor performance decrease with single-instance annotations compared to substantial drops in baseline methods
2. Quantitative results across multiple datasets (PASCAL, COCO, ADE) showing consistent performance gains
3. Qualitative results in figures showing effective learning from single instances
4. Clear experimental design specifying the single-instance setting and its implementation

The claim accurately reflects the evidence without exaggeration and acknowledges the specific setting (single-instance annotation) where the performance is maintained.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Our contribution in this work is threefold:

The review comment to be evaluated is:

This paper proposes a stochastic subset selection method for reducing the storage / transmission cost of datasets. The proposes method minimizes the expected loss over selected datasets. The data selection algorithm consists a candidate selection stage and an autoregressive selection stage, parameterized with neural networks, and are trainable by gradient methods. The authors formulate and tested their approach on four tasks. The problem formulation and methodology are technically sound. The proposed method also seems to be more general than competing methods, such as coreset.

However, I think the experimental results do not support the paper well.
1. In Table 1 and Table 2, SSS doesn't seem to have big advantage over RS. From the selected images (e.g., Fig. 10 and 11), I also don't gain an intuition on what does SSS actually select, and how do the selected instances differ from random subsampling.
2. The paper claims to address the dataset redundancy. From to my understanding, that is, removing similar instances. However, this claim is not explicitly supported by the experiments.

The evidence to be evaluated is:Firstly, our model can be applied to any type of sets (and instances represented as sets), while the aforementioned models mainly work for images represented in tensor form. Furthermore, our method can be applied both at the instance and dataset level.

In this work, we consider data of the type \(D=\{d_{1},\ldots,d_{n}\}\) where individual \(d_{i}\)'s are possibly represented as _input_\(x_{i}\) and _target_\(y_{i}\). \(D\) is the complete set and we assume that within \(D\), there exists a subset \(D_{s}=\{s_{i},\ldots,s_{k}\}\subset D\) such that \(k\ll n\) and that for an arbitrarily defined loss function \(\ell(.,D)\) that we are interested in optimizing over the full set \(D\), \(D_{s}\) can be used as a proxy for \(D\) such that \(\ell(.,D)\approx\ell(.,D_{s})\). In what follows, we present a method that learns the conditional distribution \(p(D_{s}|D)\) of the subset \(D_{s}\) via a two stage selection procedure dubbed _candidate selection_ and _autoregressive subset selection_. The overall objective is then to minimize the loss function with respect to the subset \(D_{s}\), \(\mathbb{E}_{p(D,|D)}[\ell(.,D_{s})]\). When the set \(D\) itself follows a distribution of sets as in the _meta-learning_ framework, then the objective becomes \(\mathbb{E}_{D}[\mathbb{E}_{p(D_{s}|D)}[\ell(.,D_{s})]]\). In essence, we seek to construct a subset \(D_{s}\) that is optimally representative of the full set \(D\) w.r.t \(\ell(.)\).

As such, we first present the _candidate selection_ procedure used to construct a smaller set, \(D_{c}\), without considering inter-sample dependencies. This is then followed by the _autoregressive subset selection_ procedure used to construct \(D_{s}\) from \(D_{c}\) by modeling inter-sample dependencies.

We model the task of candidate selection as a random Bernoulli process where the logits of the Bernoulli function are conditioned on the set representation of the full set \(D\) and the individual elements \(d_{i}\in D\). For a set \(D\) with cardinality \(n\) we define \(Z:=\{z_{i}\}_{i=1}^{n}\) such that \(z_{i}\in\{0,1\}\) and \(z_{i}=1\) implies that \(d_{i}\in D_{c}\) and for each \(d_{i}\), \(z_{i}\) is computed according to: \[p(z_{i}|d_{i},D)=\text{Ber}(z_{i};\rho(d_{i},r(D))), \tag{1}\] where \(r(D)\) is a permutation-invariant function that compresses \(D\) into a single vector set representation and \(\rho(d_{i},r(D))\) computes the logits used to calculate the probability of \(d_{i}\) belonging to \(D_{c}\). We implement both \(r(D)\) and \(\rho(d_{i},r(D))\) as neural networks and specifically for \(r(D)\), we use Deep Sets (Zaheer et al., 2017). Since _Ber_ is non-differentiable, we use the continuous relaxations of the Bernoulli distribution introduced in (Maddison et al., 2016; Jang et al., 2016; Gal et al., 2017). Specifically, to sample \(z_{i}\), we execute the following computational routine: \[z_{i}=\sigma\Big{(}\frac{1}{\tau}\Big{(}\log\frac{\pi_{i}}{1-\pi_{i}}+\log \frac{u}{1-u}\Big{)}\Big{)},\quad\pi_{i}=\rho(d_{i},r(D)),\quad u\sim\text{ Unif}(0,1), \tag{2}\] where \(\sigma\) is the Sigmoid function, \(\tau\) is the temperature for the continuous relaxation and \(u\) is sampled from the uniform distribution. \(\tau\) is set to \(0.05\) in all our experiments.

To alleviate this issue, we must first model the interactions between the elements of \(D_{c}\) and construct \(D_{s}\) based on the relative importance of individual elements. To construct a representative subset \(D_{s}\) with \(|D_{s}|=k\), \(k\) iterative steps are required and at step \(i\) the probability of an element in \(D_{c}\setminus D_{s}^{(i-1)}\) belonging to \(D_{s}\) is computed according to: \[p(s_{i}=d|D_{c},D_{s}^{(i-1)})=\frac{f(d,D_{c},D_{s}^{(i-1)})}{\sum_{d^{\prime }\in D_{c}\setminus D_{s}^{(i-1)}}f(d^{\prime},D_{c},D_{s}^{(i-1)})}\quad\forall d \in D_{c}\setminus D_{s}^{(i-1)}, \tag{3}\]

The key to avoiding samples with redundant information in \(D_{s}^{(k)}\) lies in the fact that for each element added to \(D_{s}\), it's selection is conditioned on both \(D_{c}\) and all elements in \(D_{s}^{(i-1)}\). We further propose a method that samples \(q\) elements from \(\text{Cat}(p_{1},\dots,p_{m})\) in a single pass for efficient training. Specifically, instead of sampling \(q\) times from the categorical distribution, we can sample the selection mask for element \(j\) from \(\text{Ber}(q*p_{j})\). In this routine, the probability of the element \(j\) being selected is \(q*p_{j}\) which is very close to the original distribution.

The inference complexity depends heavily on the choice of the function \(f\). If \(f\) considers the pairwise interactions between all candidate elements and the selected elements, the inference complexity is \(O(n)+O(k^{2}d/q)\) where \(n,d,k\) correspond to \(|D|,|D_{c}|\) and \(|D_{s}|\) respectively. In our experiments, for the choice of the function \(f\), we utilize either a Set Transformer(Lee et al., 2018) or DeepSets(Zaheer et al., 2017) to model the pairwise interactions between the elements of a given set.

We can also opt to train the network to predict a single target \(y_{D}\) for the set \(D\). For instance, the target could be the class of an image(classification) or the statistics of the set(regression problem). Here, \(p_{\theta}(y_{D}|D_{s})\) is a neural network that predicts the target \(y_{D}\). A set in this task, may be the features from a single example like an image and experimental results can be found in Section 4.2. The model for this task is depicted in Figure 2(b).

We describe a model capable of taking as input \(D_{i}\in\mathcal{D}\) to perform a task such as the reconstruction of all elements in the given dataset. For a single dataset \(D_{i}\in\mathcal{D}\), we apply the subset construction method already described to a \(D_{s}\) that can be used to reconstruct all the elements in \(D_{i}\). In essence, \(D_{i}\) is _distilled_ into a new dataset \(D_{s}\) with \(k<|D_{i}|\) elements.

As a first step, we represent \(D_{s}\) as a unique representative vector \(c\) for each element in the dataset akin to the statistics network used in the Neural Statistician (Edwards and Storkey, 2016) model. Specifically, to reconstruct an element \(d_{i}\in D_{i}\) given \(D_{s}\), \(c\) is computed by applying a stochastic cross-attention mechanism on \(D_{s}\) where the stochasticity is supplied by a query \(\alpha\) which is computed using \(d_{i}\). To obtain varying styles in the generated images, we additionally learn a latent variable \(w\) used to perturb \(c\) and both are combined to obtain a new element \(x\).

Additionally, to ensure that \(c\) is properly learnt, we add an informativity loss by reconstructing \(c\) from the generated samples from the given dataset. The objective for the model depicted in Figure 2(c) for a single dataset \(D\) is : \[\begin{split}\mathcal{L}(\theta,\phi,\psi)=\sum_{d_{i}\in D}[ \mathbb{E}_{q_{\phi}(w_{i}|d_{i})}[p_{\theta}(d_{i}|w_{i},c_{i})]-\text{KL}[q_ {\phi}(w_{i}|d_{i})||p_{\phi}(w)]\\ -\text{KL}[q_{\phi}(\alpha_{i}|d_{i})||p_{\phi}(\alpha)]-\text{ KL}[q_{\phi}(c_{i}|D_{s},\alpha_{i})||p_{\phi}(c)]\end{split} \tag{7}\] where \(p_{\psi}(\cdot)\) are priors on their respective latent variables and \(q_{\phi}(\cdot)\)'s are implemented with neural networks.

In addition to Equation 7 and the informativity loss, the model is optimized together with the subset selection model already described. When the model is fully optimized, it is applied to the instance selection task on the given dataset. In summary, the purpose of the generative model introduced is to train the subset selection module for the instance selection task.

Finally in the dataset distillation task, we consider the problem of selecting prototypes to be used for few-shot classification. Here, we adopt Prototypical Networks (Snell et al., 2017) and apply the subset selection model to the task of selecting representative prototypes from each class to be used for classifying new instances. By learning to select the prototypes, we can remove outliers that would otherwise change the class prediction boundaries in the classification task. The complete graphical model for this task is given in Figure 2(d) where again \(D_{s}\) corresponds to the selected prototypes and \(x_{*}\) and \(y_{*}\) correspond to query and class label respectively.

Our first experiment is on 1D function reconstruction. Suppose that we have a function \(f:[a,b]\rightarrow\mathbb{R}\). We first construct a set of data points of that function: \(D=\{(x_{1},y_{1}=f(x_{1})),(x_{2},y_{2}=f(x_{2})),\ldots,(x_{n},y_{n}=f(x_{n }))\}\) where \((x_{1},x_{2},\ldots,x_{n})\) are uniformly distributed along the x-axis within the interval \([a,b]\). Now if we have a family of functions \((f^{(1)},f^{(2)},\ldots,f^{(N)})\), this will lead to a family of sets \((D^{(1)},D^{(2)},\ldots,D^{(N)})\). We train our model which consists of the subset selection model \(p(D_{s}|D)\) and a task network \(p(Y|X,D_{s})\) (e.g. ANP), on this data set and report the reconstruction loss, which is the negative log-likelihood. SSS out-performs Random Select (RS), verifying that the subset selection model \(p(D_{s}|D)\) can learn a meaningful distribution for the selected elements. Our model also out-performs the Learning to Sample (LTS) baseline. Through the visualization of the selected points in Figure 5, we can see that out model tends to pick out more points (presented as red dots) in the drifting parts of the curve, which is reasonable since these parts are harder to reconstruct.

Given an image we learn to select a core subset of pixels that best reconstructs the original image.

An ANP is then used to reconstruct the remaining pixels from a set of context elements (selected subset in our case). We conduct this experiment on the CelebA dataset (Liu et al., 2018). Figure 3(b) shows that our model significantly outperforms ANP with RS (as in the original ANP paper) and the LTS baseline. Figure 6 shows the reconstruction samples of our model which are visually better than the reconstruction of the baselines for the same number of pixels.

We again use the CelebA dataset, but this time the selected pixels are used to give predictions for 40 attributes of a celebrity's face (in a multi-task learning setting). For our proposed model, only the selected pixels are used for prediction (other pixels' values are set to zeros). Table 1 shows that using only 500 pixels (\(\sim\)1.3% of total pixels in an image), we can achieve a mean AUC of 0.9093 (99.3% of the accuracy obtained with the full image).

The AUC with selected pixels learned from our SSS is significantly higher than that of the random pixels baseline, showing the effectiveness of our subset selection method.

This is our stochastic subset selection model trained for reconstruction, but then later used for classification. Our model outperforms this variant, showing the effectiveness of training with the target task.

Since our method is stochastic, the predictive distribution can be written as \(\mathbb{E}_{p(D_{s}|D)}\left[p_{\theta}(y_{D}|D_{s})\right]\), and we can use Monte Carlo sampling to get the prediction in practice. However, throughout the experiment section, we only reported the result with one sampled subset, since it gives the best reduction in memory and computational cost. This can be seen as MC sampling with one sample. We compare it against another variant: SSS(MC) with MC sampling (5 samples). It should be noted that by doing MC sampling with 5 samples, the computational cost (inference) is increased by 5 times, and the memory requirement can be increased by up to 5 times too. Table 1 shows that our model achieves comparable performance with that variant, thus justifying that it can achieve good performance for target tasks, while reducing memory and computation requirement. **Instance Selection** We present results on the instance selection task applied to a whole dataset. In this task, we use the CelebA dataset since it has an imbalance both in terms of gender and race. A dataset is constructed by sampling 200 random images from the full dataset. In this experiment, we seek to select only a few(5-30) representative images from these generated datasets. On this task, our subset selection module is trained via the procedure detailed in Section 3.6 on instance selection. To evaluate the effectiveness of the SSS model, we evaluate the model in terms of the diversity in the selected subset using the Frechet Inception Distance(FID Score) (Heusel et al., 2017) which measures the similarity and diversity between two datasets. We compare our model with the model that randomly samples instances from the full dataset. Additionally, we compare our method with the Farthest Point Sampling(FPS) algorithm which selects \(k\) points from a given set by computing distances on a metric space between all elements and selecting those elements that are furthest from each other. FPS in general seeks to obtain a wide coverage over a given set and hence is a suitable baseline. The results of this experiment is presented in Table 4 where our selection method achieves a lower FID score compared to FPS and Random Sampling. Additionally, given that the dataset is highly imbalanced, FPS performs worst since by selecting the furthest elements in the given set it cannot capture the true representation of the whole dataset even when compared with Random Sampling. Also for small sample selection, our method outperforms FPS and Random Sampling significantly since our method is able to model the interactions within the full dataset and hence can select the most representative subset.

The results for this experiment is shown in Table 3, where it can be observed that SSS can learn to select more representative prototypes compared to the other methods especially in the few-shot problems where the choice of prototypes matters more.

In this paper, we have proposed a stochastic subset selection method to reduce the size of an arbitrary set while preserving performance on a target task. Our selection method utilizes a Bernoulli mask to perform candidate selection, and a stack of Categorical distributions to iteratively select a core subset from the candidate set. As a result, the selection process does take the dependencies of the set's members into account. Hence, it can select a compact set that avoids samples with redundant information. By using the compact subset in place of the original set for a target task, we can save memory, communication and computational cost.

1:\(k\)(max subset size) 2:\(q\)(# elements selected at each iteration) 3:\(p(D)\) (distribution of sets) 4:\(\alpha\) (learning rate) 5: a target task with loss function \(\ell(\cdot,\cdot)\) 6:trained model with converged \(\theta\) and \(\phi\) 7:\(\theta,\phi\leftarrow\) initialization 8:while not converged do 9: Sample a minibatch with \(m\) sets \(D^{(1)},D^{(2)},\ldots,D^{(m)}\) from \(p(D)\)

Input: Figure 5: 1D Function Reconstruction.

Input: Figure 2: Overview. The subset Ds is sampled by a stochastic 2-stage subset selection process. Dark shaded boxes correspond to selected elements while lightly shaded boxes correspond to non-selected elements in the given set. Also, Ber and Cat correspond to the Bernoulli and Categorical distributions respectively. (a) Candidate(in blue) Selection. (b) Core Subset(in green) Selection using continuous relaxations of discrete distributions.

Input: Table 4: . FID Score for varying Instance Selection

Input: Figure 6: CelebA reconstruction samples with varying number of pixels.

Input: Figure 3: Graphical Models: (a) Feature Selection for Reconstruction. (b) Feature Selection for Prediction Task. (c) Instance Selection model. (d) Instance Selection model for Classification

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence carefully, I find that the stated contribution is not fully justified by the evidence presented. While the paper demonstrates applicability to different types of sets and shows both instance and dataset level applications, the review comment raises valid concerns about experimental validation, particularly regarding lack of significant advantage over random sampling and insufficient evidence for addressing dataset redundancy. The technical formulation appears sound but the experimental results don't fully validate the claimed advantages.

<score>3</score>

Justification:
1. The mathematical formulation shows the model can work with different set types and at different levels (instance/dataset), supporting part of the claim.
2. However, the review points out that:
   - Results in Tables 1-2 don't show significant advantages over random sampling
   - The claim about addressing dataset redundancy isn't explicitly supported by experiments
3. The evidence shows theoretical capability but practical advantages are not strongly demonstrated, suggesting moderate overstatement of the contribution's significance.
4. The paper provides solid mathematical foundations but experimental validation could be stronger to fully support the broad claims about the method's advantages.

The score of 3 reflects that while the core ideas are valid, there is some overstatement in how definitively the contributions are presented given the limitations in experimental validation noted in the review.