Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Copy-paste generates diverse training data to boost performance under few-shot settings, enables the model to learn with partially-annotated images with as few as one annotated novel instance out of many novel instances in an image (e.g., as illustrated at the lower left part of Fig. 1), which is a _stricter_ setting than semantic background shift (Cermelli et al., 2020).

The review comment to be evaluated is:

Strengths:
- The paper introduces the copy-paste data augmentation strategy to the few-shot incremental semantic segmentation, which seems to be a novel combination.
- The proposed guided sample selection method for memory construction and data synthesis improves the performance of several baseline methods.
- The paper is most clearly written and easy to follow.

Weaknesses:
- While the proposed data augmentation strategy helps few-shot learning, its technical novelty in the incremental semantic segmentation (ISS) is rather limited, as it is an add-on module and has to rely on the base ISS methods.
- The construction of the memory-replay buffer has several limitations and will affect the quality of the data augmentation in ISS:
1) Only the base stage has fully-annotated images that can be used for the target images in the copy-paste operation, and all the images from the incremental stages are partially labeled, which are unusable for that purpose. As a result, when the incremental learning evolves, the memory has to add samples of new classes and remove some base-stage images due to the capacity limit. Eventually, the memory buffer would be less useful.
2) The diversity-guided selection procedure in Sec 3.2 considers each class separately in sampling. However, for semantic segmentation, it is common that each image has regions of multiple classes. It is unclear how such a method can balance different classes. This is also a problem for the class-frequency-guided selection, as each image can have multiple classes. Another issue is that this method did not take into account the region sizes as it only considers the image-level sampling factor.
- There are several concerns on the experimental evaluation, which make the conclusion less convincing:
1) Both benchmarks are mainly used in the few-shot segmentation literature, which are a bit small in terms of data set size and number of classes. Similar to SSUL and other ISS works, the method should be evaluated on more diverse ISS task settings and larger dataset, such as ADE.
2) For the few-shot side, while this work adopts a different evaluation protocol (as described in Sec. 4.2), it would be more informative if it also uses the protocol in Cermelli et al 2021 for a fair comparison. Moreover, unlike the standard ISS, the training process of the few-shot setting typically has a large variance due to different training samples annotated. Thus it should report both the mean and standard deviation in the final results.
3) The gaps in the ablation study are relatively small. It would more convincing if the standard deviation can be provided. It is unclear how significant such improvements are in the few-shot learning setting.

The evidence to be evaluated is:Noticeably, a few attempts have been made by recent work to address the semantic background shift problem proposed by Cermelli et al. (2020) via either pseudo-labeling (Cha et al., 2021; Douillard et al., 2021; Yan et al., 2021) or modified loss (Cermelli et al., 2020; Zhang et al., 2022) to train on partially-annotated images of novel classes.

In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting.

Copy-and-paste is an augmentation technique that copies a subset of objects from one image and pastes onto the other image using their segmentation masks. Many works (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018) have been done to investigate how copy-and-paste augmentation can help with various visual tasks.

We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting.

Different from standard semantic segmentation, in few-shot incremental segmentation, \(\mathcal{C}\) is expanded over time through two stages.

During the _incremental learning stage_, a sequence of tasks \(\{D_{1},D_{2},\dots\}\) with novel categories is presented to the model, where \(D_{j}=\{(x_{i},y_{i})|x_{i}\in\mathcal{X},y_{i}\in\mathcal{Y}^{\mathcal{C}_{j}}\}\) and \(\mathcal{C}_{j}\) is a set of classes for task \(D_{j}\). In few-shot learning, the size of the training sets for the novel tasks is small, _i.e_., \(|D_{j}|\ll|\mathcal{D}_{0}|\).

To enforce the partially-annotated image setting, we follow Cermelli _et al_. and assume that only novel classes are annotated, _i.e_., \(\mathcal{C}_{i}\cap C_{j}=\varnothing\) for all \(i\neq j\).

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model.

During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\).

In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

We hypothesize that synthesizing novel objects onto contextually consistent base images would result in an improved learning process. For example, a TV in an apartment should more likely be pasted onto an image of another apartment rather than an outdoor landscape. We design a context-guided sampling algorithm to select images from \(\hat{D_{0}}\) that are contextually similar to the provided partially-labeled images. GAPS uses a scene embedding network \(h:\mathcal{X}\rightarrow\mathbb{R}^{m}\) to estimate pairwise contextual similarity between any two given images. The embedding \(h\) maps an input image to a scene embedding vector in a metric space where pairwise similarity comparison between two embeddings are possible. In our system, we use an off-the-shelf VGGNet (Simonyan and Zisserman, 2015) and replace its last fully connected layer with a cosine-similarity-based classifier. The network \(h\) is trained using the Places365 (Zhou et al., 2017) dataset, which contains \(365\) scene categories and roughly 1.6 million images in its training split. During training, the scene prediction is done by comparing the predicted embedding with existing prototypes.

Note that we only need to compute scene embedding once for every image in \(\hat{D}_{0}\) and incoming partially-annotated images.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

Though the semantic background shift proposed by Cermelli et al. (2020) relaxes the requirement to provide full segmentation annotations, it still requires _all novel instances_ in images to be annotated, which can be time-consuming to obtain in cluttered scenes and hinder potential applications. Here we consider a more challenging task setting, which we term _single-instance incremental learning_. Namely, for training images provided in incremental learning stages, if there are multiple instances of a novel class in the image, we assume that only _one instance_ will be annotated. To simulate this setting, we use the instance-level segmentation annotation provided by the COCO dataset to enforce only annotation of one novel instance in every image is available to the model.

This highlights the potential of copy-paste synthesis as an alternative to the existing pseudo-labeling paradigm to cope with the more realistic single-instance setting.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images.

We believe that copy-paste can serve as a promising alternative to pseudo-labeling and modified loss to enable learning on partially-annotated images.

Finally, the ability to learn with as few as one annotated instance in an image raises several intriguing possibilities.

For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Beyond confirming conclusion from previous work, such improvement alongside stronger augmentation demonstrates the effectiveness of augmentation technique even when there is few data presented, highlighting a possibility to further improve the performance by using even stronger augmentation technique. We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning.

Since examples observed during the incremental learning stage are only partially annotated, they can not be directly added to the memory-replay buffer.

Here, 'PIFS(\(\mathcal{L}_{2}\))+MEM+PL' means that we follow SSUL and perform pseudo-labeling on partially annotated samples before adding them to memory.

We conducted an additional experiment on the COCO-20-1 dataset under the 5-shot setting, where we vary the number of fully-annotated images in the memory-replay buffer to investigate the per formance of our method. As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 7: Multi-step qualitative results of our method (GAPS) on the PASCAL-5-1 split under 1- shot setting. (Best view in color.)

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be somewhat overstated. While the paper demonstrates that copy-paste can work with partially-annotated images containing one annotated instance, the claim that this is "stricter" than semantic background shift isn't fully substantiated. The evidence shows it's a "more challenging" setting but doesn't directly prove it's stricter. Additionally, while the method shows promise, there are significant limitations noted in the review about memory buffer constraints and evaluation protocols that temper the strength of the claim.

<score>3</score>

The claim's key elements are supported by experimental results showing the method can work with single-instance annotation. However, the language choice of "stricter" setting lacks rigorous comparison to semantic background shift. The review also points out important limitations in the experimental evaluation, including:
1. Small dataset sizes and limited class numbers
2. Lack of standard deviation reporting in few-shot results
3. Memory buffer limitations when scaling to more incremental stages
4. Relatively small gaps in ablation studies

The evidence demonstrates the approach works but doesn't fully justify characterizing it as a stricter setting than previous work.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To the best of our knowledge, we are the _first_ to introduce copy-paste as a synthesis technique to create a diverse data source for few-shot incremental segmentation.

The evidence to be evaluated is:This problem has been studied extensively in image classification (Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Rebuffi et al., 2017; Mallya et al., 2018; Castro et al., 2018; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of semantic segmentation (Cermelli et al., 2020; Michieli and Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021; Zhang et al., 2022).

However, existing work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

Copy-and-paste is an augmentation technique that copies a subset of objects from one image and pastes onto the other image using their segmentation masks. Many works (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018) have been done to investigate how copy-and-paste augmentation can help with various visual tasks.

We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting.

In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model.

During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\).

In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

This highlights the potential of copy-paste synthesis as an alternative to the existing pseudo-labeling paradigm to cope with the more realistic single-instance setting.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We investigate how different probabilistic strategies to apply copy-and-paste augmentation affects GAPS.

For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column.

On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the claim and evidence carefully, I find the claim to be moderately overstated. While the authors do present a novel application of copy-paste for few-shot incremental segmentation, the evidence shows that copy-paste augmentation has been previously studied extensively (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018). The authors even acknowledge building upon Ghiasi et al.'s (2021) augmentation strategy. Though they introduce new techniques to guide the copy-paste synthesizer specifically for few-shot incremental segmentation, claiming to be the "first" to use copy-paste as a synthesis technique overstates their contribution's novelty.

<score>4</score>

Justification:
1. The paper directly states: "Copy-and-paste is an augmentation technique... Many works (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018) have been done to investigate how copy-and-paste augmentation can help with various visual tasks."

2. They explicitly build on previous work: "We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline..."

3. While their specific application to few-shot incremental segmentation appears novel, the broad claim of being "first" to use copy-paste as a synthesis technique is not supported by the evidence.

4. A more accurate claim would have specified their novel contribution in applying guided copy-paste specifically to few-shot incremental segmentation, rather than claiming to be first to use copy-paste as a synthesis technique broadly.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To the best of our knowledge, we are the _first_ to introduce copy-paste as a synthesis technique to create a diverse data source for few-shot incremental segmentation.

The review comment to be evaluated is:

Strengths:
1. The paper investigates a novel and understudied scenario that is more realistic than earlier benchmarks.
2. The study gives a comprehensive summary of similar works as well as comparisons and contrasts between them.
3. The study expands the present benchmark with new experiments involving only one labeled instance of the new classes, demonstrating that the GAPS can address the issue of missing annotations.

Weaknesses:
1. The ablation study may be extended to other settings and may provide more insights about the paper choices. In particular, it may start from a baseline that uses exemplars without pasting the new classes on them, to show that pasting is important to improve performance and that the gain is not only given by the regularization effect of using exemplars (as proposed by GDumb [a]). Moreover, the paper should clarify if, while disabling the diversity guidance, the exemplars are sampled uniformly for the old classes or if they are randomly sampled on the full dataset.
2. From the ablation study, context sampling appears to marginally improve the results. Moreover, it is not clear how the performance will change by using a pre-training dataset different from Places-365, such as ImageNet or ADE20K. Furthermore, can it be trained directly on the base dataset of the considered setting or can the method use the segmentation network itself (for example, using an auxiliary classifier on the encoder features)?
3. The paper baselines appear to be very naive. In particular, the paper reports the results for Fine-Tuning, MiB, and PIFS without considering exemplars, introducing an unfair comparison with respect to GAPS. The baselines should be reported both without exemplars (as in the original paper) and with them (indeed, without the proposed copy-and-paste strategy).

[a] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. "Gdumb: A simple approach that questions our progress in continual learning." European conference on computer vision. Springer, Cham, 2020.

The evidence to be evaluated is:This problem has been studied extensively in image classification (Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Rebuffi et al., 2017; Mallya et al., 2018; Castro et al., 2018; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of semantic segmentation (Cermelli et al., 2020; Michieli and Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021; Zhang et al., 2022).

However, existing work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

Copy-and-paste is an augmentation technique that copies a subset of objects from one image and pastes onto the other image using their segmentation masks. Many works (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018) have been done to investigate how copy-and-paste augmentation can help with various visual tasks.

We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting.

In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model.

During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\).

In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

This highlights the potential of copy-paste synthesis as an alternative to the existing pseudo-labeling paradigm to cope with the more realistic single-instance setting.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We investigate how different probabilistic strategies to apply copy-and-paste augmentation affects GAPS.

For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column.

On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about being "first" to introduce copy-paste as a synthesis technique for few-shot incremental segmentation appears somewhat overstated. While the paper does present a novel application of copy-paste techniques in this specific context, the evidence shows they built upon and extended existing copy-paste augmentation work (citing Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018). The authors even state they "extend the augmentation strategy from Ghiasi et al. (2021)." While their application may be innovative, claiming to be definitively "first" without acknowledging the foundation of prior copy-paste work is too strong.

<score>4</score>

Justification:
1. The paper directly cites previous copy-paste augmentation work and acknowledges extending from Ghiasi et al. (2021)
2. Review comments recognize the novelty of the approach but do not verify the "first" claim
3. More precise wording could have acknowledged building upon existing copy-paste techniques while highlighting the novel application to few-shot incremental segmentation
4. The claim would be better stated as introducing a novel application or adaptation of copy-paste techniques rather than claiming to be first absolutely


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To the best of our knowledge, we are the _first_ to introduce copy-paste as a synthesis technique to create a diverse data source for few-shot incremental segmentation.

The review comment to be evaluated is:

Strength:
 (1) This paper introduces copy-paste as a synthesis technique to few-shot incremental segmentation.
 (2) Experiments show clear improvement on performance with the proposed method.

Weaknesses:
(1)  My main concern is the novelty of the paper, copy-paste technique is more about a data augmentation methods, which has already been shown effective in many segmentation task, it is not novel enough to be accepted as an ICLR paper.
(2) The paper is not clear at least for me, especially figure 1 and figure, it's really hard for readers to understand the task, although the idea is clear.

The evidence to be evaluated is:This problem has been studied extensively in image classification (Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Rebuffi et al., 2017; Mallya et al., 2018; Castro et al., 2018; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of semantic segmentation (Cermelli et al., 2020; Michieli and Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021; Zhang et al., 2022).

However, existing work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

Copy-and-paste is an augmentation technique that copies a subset of objects from one image and pastes onto the other image using their segmentation masks. Many works (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018) have been done to investigate how copy-and-paste augmentation can help with various visual tasks.

We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting.

In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model.

During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\).

In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

This highlights the potential of copy-paste synthesis as an alternative to the existing pseudo-labeling paradigm to cope with the more realistic single-instance setting.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We investigate how different probabilistic strategies to apply copy-and-paste augmentation affects GAPS.

For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column.

On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim about being "first" to introduce copy-paste as a synthesis technique for few-shot incremental segmentation appears overstated. The evidence shows:

1. Copy-paste augmentation has been extensively studied before (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018)
2. The authors directly state they "extend the augmentation strategy from Ghiasi et al. (2021)"
3. The reviewer points out that "copy-paste technique is more about a data augmentation methods, which has already been shown effective in many segmentation tasks"

While they present a novel application and adaptation of copy-paste for few-shot incremental segmentation, claiming to be "first" overstates the novelty given the clear prior work they build upon.

<score>4</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To the best of our knowledge, we are the _first_ to introduce copy-paste as a synthesis technique to create a diverse data source for few-shot incremental segmentation.

The review comment to be evaluated is:

Strengths:
- The paper introduces the copy-paste data augmentation strategy to the few-shot incremental semantic segmentation, which seems to be a novel combination.
- The proposed guided sample selection method for memory construction and data synthesis improves the performance of several baseline methods.
- The paper is most clearly written and easy to follow.

Weaknesses:
- While the proposed data augmentation strategy helps few-shot learning, its technical novelty in the incremental semantic segmentation (ISS) is rather limited, as it is an add-on module and has to rely on the base ISS methods.
- The construction of the memory-replay buffer has several limitations and will affect the quality of the data augmentation in ISS:
1) Only the base stage has fully-annotated images that can be used for the target images in the copy-paste operation, and all the images from the incremental stages are partially labeled, which are unusable for that purpose. As a result, when the incremental learning evolves, the memory has to add samples of new classes and remove some base-stage images due to the capacity limit. Eventually, the memory buffer would be less useful.
2) The diversity-guided selection procedure in Sec 3.2 considers each class separately in sampling. However, for semantic segmentation, it is common that each image has regions of multiple classes. It is unclear how such a method can balance different classes. This is also a problem for the class-frequency-guided selection, as each image can have multiple classes. Another issue is that this method did not take into account the region sizes as it only considers the image-level sampling factor.
- There are several concerns on the experimental evaluation, which make the conclusion less convincing:
1) Both benchmarks are mainly used in the few-shot segmentation literature, which are a bit small in terms of data set size and number of classes. Similar to SSUL and other ISS works, the method should be evaluated on more diverse ISS task settings and larger dataset, such as ADE.
2) For the few-shot side, while this work adopts a different evaluation protocol (as described in Sec. 4.2), it would be more informative if it also uses the protocol in Cermelli et al 2021 for a fair comparison. Moreover, unlike the standard ISS, the training process of the few-shot setting typically has a large variance due to different training samples annotated. Thus it should report both the mean and standard deviation in the final results.
3) The gaps in the ablation study are relatively small. It would more convincing if the standard deviation can be provided. It is unclear how significant such improvements are in the few-shot learning setting.

The evidence to be evaluated is:This problem has been studied extensively in image classification (Li and Hoiem, 2017; Lopez-Paz and Ranzato, 2017; Yoon et al., 2018; Rebuffi et al., 2017; Mallya et al., 2018; Castro et al., 2018; Bang et al., 2021); whilst relatively less work have been done to study incremental learning under the task setting of semantic segmentation (Cermelli et al., 2020; Michieli and Zanuttigh, 2019; Cha et al., 2021; Douillard et al., 2021; Zhang et al., 2022).

However, existing work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

Copy-and-paste is an augmentation technique that copies a subset of objects from one image and pastes onto the other image using their segmentation masks. Many works (Dwibedi et al., 2017; Ghiasi et al., 2021; Dvornik et al., 2018) have been done to investigate how copy-and-paste augmentation can help with various visual tasks.

We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting.

In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model.

During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\).

In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

This highlights the potential of copy-paste synthesis as an alternative to the existing pseudo-labeling paradigm to cope with the more realistic single-instance setting.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We investigate how different probabilistic strategies to apply copy-and-paste augmentation affects GAPS.

For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column.

On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence and reviewer comments, I find the claim that they are the "first" to introduce copy-paste as a synthesis technique for few-shot incremental segmentation requires careful evaluation.

The reviewer acknowledges this is a "novel combination" but notes it has "limited technical novelty" as it builds on existing copy-paste techniques. The evidence shows they extended Ghiasi et al. (2021)'s augmentation strategy and constructed a "Naive copy-Paste Synthesis (NPS)" baseline. While they may be the first to apply it specifically to few-shot incremental segmentation, claiming to be the "first" to introduce copy-paste as a synthesis technique overstates their contribution since they built upon existing copy-paste methods.

<score>4</score>

Justification:
1. The evidence shows they adapted and extended existing copy-paste techniques rather than introducing it as entirely new
2. The reviewer confirms it's a novel combination but with limited technical novelty
3. Their own text acknowledges building on prior work (e.g., "We extend the augmentation strategy from Ghiasi et al. (2021)")
4. The claim would be better stated by specifying they are the first to apply/adapt copy-paste specifically for few-shot incremental segmentation rather than broadly claiming to introduce copy-paste as a technique


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Third, after an informative image is selected, from what distribution should it sample current and previously learned novel objects to _balance_ sample frequency and avoid over-sampling or under-sampling?

The evidence to be evaluated is:After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2).

The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4).

For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi et al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.

After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

number of exemplars \(n\) \(k\leftarrow\textsc{Floor}(n/|Y_{0}|)\)// Sample per class for\(c\) from \(1\) to \(|Y_{0}|\)do \(S_{c}\leftarrow\{(x_{i},y_{i})\in D_{0},c\in y_{i}\}\) for\((x_{i},y_{i})\in S_{c}\)do \(p_{i}\leftarrow\textsc{MAP}(x_{i},y_{i},c)\)// Pred. Proto. \(s_{i}\leftarrow\textsc{CosineSimilarity}(p_{ic},w_{c})\) endfor  Sort \(S_{c}\) by similarity score \(s_{i}\) \(ES_{c}\leftarrow\{\}\)// final exemplar set of class \(c\) for\(j=1,2,\dots,k\)do \(L_{idx}\gets j\cdot|S_{c}|/k\) \(U_{idx}\leftarrow\textsc{Min}(Lower+|S_{c}|/k,|S_{c}|)\) \((x,y)\leftarrow\textsc{Sample}(S_{c}[L_{idx}:U_{idx}])\) \(ES_{c}\gets ES_{c}\cup(x,y)\) endfor endfor \(D_{0}\leftarrow\textsc{UniformSample}(\bigcup_{i=1,\dots,|Y_{0}|}ES_{i},n)\)

To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from \(\mathbb{S}\) with a probability of \(\alpha\), and sample from \(\hat{D}_{0}\) with a probability of \(1-\alpha\), where \(\alpha\) is a hyperparameter set to \(0.9\) in our implementation.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes.

Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis. To apply vRFS, we first need to compute category-wise sampling factor \(r_{c}\) for every \(c\) as in RFS. If \(c\in\mathcal{C}_{0}\), we set \(r_{c}=1\) as since during the construction of \(\hat{D}_{0}\) we already consider class balance by class-wise uniformly sampling. If \(c\in\mathcal{C}_{j}\) with \(j\geq 1\), we first compute its class frequency by \(f_{c}=nShot/|\hat{D}_{0}|\), where \(nShot\) denotes the number of images in \(D_{j}\) with at least one pixel of \(c\). Then, the category-wise sampling factor for \(c\) is given by \(r_{c}=\textsc{Max}(1,\sqrt{t/f_{c}})\).

However, we empirically found that setting \(t\) to be the multiplicative inverse of total number of classes, or \(t=1/|\cup_{1,...,j}Y_{j}|\), is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings. During the synthesis process, we first randomly select a novel class \(c_{N}\) from \(\mathcal{C}_{j}\), and another class \(c_{o}\) from \(\cup_{1,...,j}C_{j}\setminus\{c_{N}\}\).

To apply vRFS resampling, we hallucinate two _virtual_ samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of \(r_{i}=\textsc{Max}_{c\in i}r_{c}=r_{c_{o}}\). Thus, the probability to synthesize class \(c_{o}\) onto \(x_{B}\) is given by \(r_{c_{o}}/(1+r_{c_{o}})\). We then repeat the process for the novel class \(c_{N}\). Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of \(c_{N}\) in the final image.

**Frequency-guided probabilistic synthesis boosts results on novel classes**.

We take a closer look at step-wise performance (whose visualization is available in Appendix A.11) and found that the reason is due to unguided copy-paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-replay buffer.

_Full-set-random_ denotes samples that are uniformly sampled from the base training set. _Classwise-random_ denotes randomly selecting even number of samples from each base class.

RFS (Repeat Factor Sampling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights to rare classes and has been shown to work well in long-tailed task settings. The other baseline, 'GreedyClass', is achieved by greedily picking samples in every base class with the most number of classes. To consider the region sizes in the image, we design a base-line method, 'BalancedRegion', which uses a similar algorithm as our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images.

_Easy-Hard-Mix_ denotes equal mixtures of samples drawn from easiest samples and hardest samples. Lastly, _Diverse-Sampling_ is our proposed method, where samples of all difficulties are equally drawn. We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling.

_CAS_ (Class-aware-sampling) is a sampling strategy from (Gupta et al., 2019) which first uniformly samples a class and then selects a sample containing instances from the selected class. In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes.

Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me analyze the claim and evidence:

The claim asks about the distribution strategy for balancing sample frequency. The evidence provides extensive details about multiple sampling approaches:

1. Base exemplar selection using diversity-guided strategy
2. Memory replay buffer sampling with probability 1-α
3. vRFS (variant of Repeat Factor Sampling) for copy-paste frequency
4. Class-frequency guided sampling with category-wise sampling factors
5. Multiple baseline comparisons including CAS, GreedyClass, BalancedRegion etc.

The evidence thoroughly addresses the sampling distribution question through multiple complementary strategies, supported by implementation details and empirical findings. The results discuss both benefits and limitations (e.g., oversampling issues) of different approaches.

<score>1</score>

Justification:
The claim is well-stated as it simply asks "from what distribution should it sample" without making unsupported assertions. The evidence provides comprehensive sampling strategies including:

1. Mathematical formulations for sampling probabilities
2. Empirical validation through baseline comparisons
3. Clear acknowledgment of tradeoffs (e.g., "trade-off between oversampling and undersampling")
4. Implementation details (e.g., "α is a hyperparameter set to 0.9")
5. Experimental insights about what works and what doesn't (e.g., findings about oversampling hurting performance)

The evidence directly answers the question about sampling distribution without overstating capabilities or glossing over limitations.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Third, after an informative image is selected, from what distribution should it sample current and previously learned novel objects to _balance_ sample frequency and avoid over-sampling or under-sampling?

The review comment to be evaluated is:

Strengths:
1. The paper investigates a novel and understudied scenario that is more realistic than earlier benchmarks.
2. The study gives a comprehensive summary of similar works as well as comparisons and contrasts between them.
3. The study expands the present benchmark with new experiments involving only one labeled instance of the new classes, demonstrating that the GAPS can address the issue of missing annotations.

Weaknesses:
1. The ablation study may be extended to other settings and may provide more insights about the paper choices. In particular, it may start from a baseline that uses exemplars without pasting the new classes on them, to show that pasting is important to improve performance and that the gain is not only given by the regularization effect of using exemplars (as proposed by GDumb [a]). Moreover, the paper should clarify if, while disabling the diversity guidance, the exemplars are sampled uniformly for the old classes or if they are randomly sampled on the full dataset.
2. From the ablation study, context sampling appears to marginally improve the results. Moreover, it is not clear how the performance will change by using a pre-training dataset different from Places-365, such as ImageNet or ADE20K. Furthermore, can it be trained directly on the base dataset of the considered setting or can the method use the segmentation network itself (for example, using an auxiliary classifier on the encoder features)?
3. The paper baselines appear to be very naive. In particular, the paper reports the results for Fine-Tuning, MiB, and PIFS without considering exemplars, introducing an unfair comparison with respect to GAPS. The baselines should be reported both without exemplars (as in the original paper) and with them (indeed, without the proposed copy-and-paste strategy).

[a] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. "Gdumb: A simple approach that questions our progress in continual learning." European conference on computer vision. Springer, Cham, 2020.

The evidence to be evaluated is:After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2).

The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4).

For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi et al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.

After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

number of exemplars \(n\) \(k\leftarrow\textsc{Floor}(n/|Y_{0}|)\)// Sample per class for\(c\) from \(1\) to \(|Y_{0}|\)do \(S_{c}\leftarrow\{(x_{i},y_{i})\in D_{0},c\in y_{i}\}\) for\((x_{i},y_{i})\in S_{c}\)do \(p_{i}\leftarrow\textsc{MAP}(x_{i},y_{i},c)\)// Pred. Proto. \(s_{i}\leftarrow\textsc{CosineSimilarity}(p_{ic},w_{c})\) endfor  Sort \(S_{c}\) by similarity score \(s_{i}\) \(ES_{c}\leftarrow\{\}\)// final exemplar set of class \(c\) for\(j=1,2,\dots,k\)do \(L_{idx}\gets j\cdot|S_{c}|/k\) \(U_{idx}\leftarrow\textsc{Min}(Lower+|S_{c}|/k,|S_{c}|)\) \((x,y)\leftarrow\textsc{Sample}(S_{c}[L_{idx}:U_{idx}])\) \(ES_{c}\gets ES_{c}\cup(x,y)\) endfor endfor \(D_{0}\leftarrow\textsc{UniformSample}(\bigcup_{i=1,\dots,|Y_{0}|}ES_{i},n)\)

To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from \(\mathbb{S}\) with a probability of \(\alpha\), and sample from \(\hat{D}_{0}\) with a probability of \(1-\alpha\), where \(\alpha\) is a hyperparameter set to \(0.9\) in our implementation.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes.

Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis. To apply vRFS, we first need to compute category-wise sampling factor \(r_{c}\) for every \(c\) as in RFS. If \(c\in\mathcal{C}_{0}\), we set \(r_{c}=1\) as since during the construction of \(\hat{D}_{0}\) we already consider class balance by class-wise uniformly sampling. If \(c\in\mathcal{C}_{j}\) with \(j\geq 1\), we first compute its class frequency by \(f_{c}=nShot/|\hat{D}_{0}|\), where \(nShot\) denotes the number of images in \(D_{j}\) with at least one pixel of \(c\). Then, the category-wise sampling factor for \(c\) is given by \(r_{c}=\textsc{Max}(1,\sqrt{t/f_{c}})\).

However, we empirically found that setting \(t\) to be the multiplicative inverse of total number of classes, or \(t=1/|\cup_{1,...,j}Y_{j}|\), is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings. During the synthesis process, we first randomly select a novel class \(c_{N}\) from \(\mathcal{C}_{j}\), and another class \(c_{o}\) from \(\cup_{1,...,j}C_{j}\setminus\{c_{N}\}\).

To apply vRFS resampling, we hallucinate two _virtual_ samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of \(r_{i}=\textsc{Max}_{c\in i}r_{c}=r_{c_{o}}\). Thus, the probability to synthesize class \(c_{o}\) onto \(x_{B}\) is given by \(r_{c_{o}}/(1+r_{c_{o}})\). We then repeat the process for the novel class \(c_{N}\). Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of \(c_{N}\) in the final image.

**Frequency-guided probabilistic synthesis boosts results on novel classes**.

We take a closer look at step-wise performance (whose visualization is available in Appendix A.11) and found that the reason is due to unguided copy-paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-replay buffer.

_Full-set-random_ denotes samples that are uniformly sampled from the base training set. _Classwise-random_ denotes randomly selecting even number of samples from each base class.

RFS (Repeat Factor Sampling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights to rare classes and has been shown to work well in long-tailed task settings. The other baseline, 'GreedyClass', is achieved by greedily picking samples in every base class with the most number of classes. To consider the region sizes in the image, we design a base-line method, 'BalancedRegion', which uses a similar algorithm as our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images.

_Easy-Hard-Mix_ denotes equal mixtures of samples drawn from easiest samples and hardest samples. Lastly, _Diverse-Sampling_ is our proposed method, where samples of all difficulties are equally drawn. We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling.

_CAS_ (Class-aware-sampling) is a sampling strategy from (Gupta et al., 2019) which first uniformly samples a class and then selects a sample containing instances from the selected class. In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes.

Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll analyze whether the claim about balancing sample frequency is well-stated based on the evidence:

1. Summary and Evaluation:
The evidence provides detailed explanations of how sample frequency is balanced through multiple mechanisms:
- vRFS (variant of Repeat Factor Sampling) with category-wise sampling factors
- Probability-based synthesis approach (r_c_o/(1+r_c_o))
- Class-frequency-guided sampling with base examples
- Diversity-guided exemplar selection
- Empirical validation showing oversampling hurts performance

The claim questioning how to balance sampling is somewhat overstated since the evidence demonstrates clear solutions and implementation details for addressing this challenge. The evidence shows they have developed and tested specific approaches to handle this balancing problem.

<score>2</score>

Justification:
1. The evidence provides concrete solutions through:
- Detailed sampling algorithms and formulas (e.g., r_c=Max(1,√(t/f_c)))
- Clear probability calculations for synthesis
- Empirical validation showing their approach works better than alternatives

2. The reviewer comments don't raise specific concerns about the sampling frequency balancing approach.

3. While the claim poses this as an open question, the evidence shows they have developed and validated specific solutions, making the questioning tone slightly overstated but not severely so.

The claim would be better stated as an explanation of their solution rather than an open question, given the comprehensive evidence provided.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Third, after an informative image is selected, from what distribution should it sample current and previously learned novel objects to _balance_ sample frequency and avoid over-sampling or under-sampling?

The review comment to be evaluated is:

Strength:
 (1) This paper introduces copy-paste as a synthesis technique to few-shot incremental segmentation.
 (2) Experiments show clear improvement on performance with the proposed method.

Weaknesses:
(1)  My main concern is the novelty of the paper, copy-paste technique is more about a data augmentation methods, which has already been shown effective in many segmentation task, it is not novel enough to be accepted as an ICLR paper.
(2) The paper is not clear at least for me, especially figure 1 and figure, it's really hard for readers to understand the task, although the idea is clear.

The evidence to be evaluated is:After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2).

The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4).

For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi et al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.

After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

number of exemplars \(n\) \(k\leftarrow\textsc{Floor}(n/|Y_{0}|)\)// Sample per class for\(c\) from \(1\) to \(|Y_{0}|\)do \(S_{c}\leftarrow\{(x_{i},y_{i})\in D_{0},c\in y_{i}\}\) for\((x_{i},y_{i})\in S_{c}\)do \(p_{i}\leftarrow\textsc{MAP}(x_{i},y_{i},c)\)// Pred. Proto. \(s_{i}\leftarrow\textsc{CosineSimilarity}(p_{ic},w_{c})\) endfor  Sort \(S_{c}\) by similarity score \(s_{i}\) \(ES_{c}\leftarrow\{\}\)// final exemplar set of class \(c\) for\(j=1,2,\dots,k\)do \(L_{idx}\gets j\cdot|S_{c}|/k\) \(U_{idx}\leftarrow\textsc{Min}(Lower+|S_{c}|/k,|S_{c}|)\) \((x,y)\leftarrow\textsc{Sample}(S_{c}[L_{idx}:U_{idx}])\) \(ES_{c}\gets ES_{c}\cup(x,y)\) endfor endfor \(D_{0}\leftarrow\textsc{UniformSample}(\bigcup_{i=1,\dots,|Y_{0}|}ES_{i},n)\)

To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from \(\mathbb{S}\) with a probability of \(\alpha\), and sample from \(\hat{D}_{0}\) with a probability of \(1-\alpha\), where \(\alpha\) is a hyperparameter set to \(0.9\) in our implementation.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes.

Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis. To apply vRFS, we first need to compute category-wise sampling factor \(r_{c}\) for every \(c\) as in RFS. If \(c\in\mathcal{C}_{0}\), we set \(r_{c}=1\) as since during the construction of \(\hat{D}_{0}\) we already consider class balance by class-wise uniformly sampling. If \(c\in\mathcal{C}_{j}\) with \(j\geq 1\), we first compute its class frequency by \(f_{c}=nShot/|\hat{D}_{0}|\), where \(nShot\) denotes the number of images in \(D_{j}\) with at least one pixel of \(c\). Then, the category-wise sampling factor for \(c\) is given by \(r_{c}=\textsc{Max}(1,\sqrt{t/f_{c}})\).

However, we empirically found that setting \(t\) to be the multiplicative inverse of total number of classes, or \(t=1/|\cup_{1,...,j}Y_{j}|\), is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings. During the synthesis process, we first randomly select a novel class \(c_{N}\) from \(\mathcal{C}_{j}\), and another class \(c_{o}\) from \(\cup_{1,...,j}C_{j}\setminus\{c_{N}\}\).

To apply vRFS resampling, we hallucinate two _virtual_ samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of \(r_{i}=\textsc{Max}_{c\in i}r_{c}=r_{c_{o}}\). Thus, the probability to synthesize class \(c_{o}\) onto \(x_{B}\) is given by \(r_{c_{o}}/(1+r_{c_{o}})\). We then repeat the process for the novel class \(c_{N}\). Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of \(c_{N}\) in the final image.

**Frequency-guided probabilistic synthesis boosts results on novel classes**.

We take a closer look at step-wise performance (whose visualization is available in Appendix A.11) and found that the reason is due to unguided copy-paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-replay buffer.

_Full-set-random_ denotes samples that are uniformly sampled from the base training set. _Classwise-random_ denotes randomly selecting even number of samples from each base class.

RFS (Repeat Factor Sampling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights to rare classes and has been shown to work well in long-tailed task settings. The other baseline, 'GreedyClass', is achieved by greedily picking samples in every base class with the most number of classes. To consider the region sizes in the image, we design a base-line method, 'BalancedRegion', which uses a similar algorithm as our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images.

_Easy-Hard-Mix_ denotes equal mixtures of samples drawn from easiest samples and hardest samples. Lastly, _Diverse-Sampling_ is our proposed method, where samples of all difficulties are equally drawn. We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling.

_CAS_ (Class-aware-sampling) is a sampling strategy from (Gupta et al., 2019) which first uniformly samples a class and then selects a sample containing instances from the selected class. In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes.

Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Let me evaluate the claim about balancing sample frequency in relation to the provided evidence:

The claim questions how to balance sampling of current and previously learned objects to avoid over/under-sampling. The evidence thoroughly addresses this through:

1. A detailed vRFS (variant Repeat Factor Sampling) strategy that:
- Computes category-wise sampling factors
- Uses probabilistic synthesis based on class frequencies
- Handles both base and novel classes differently
- Automatically adjusts sampling rates using t=1/|∪Y_j|

2. Multiple sampling strategies are compared:
- CAS (Class-aware-sampling)
- Full-set-random
- Classwise-random
- Various diversity-based approaches

The evidence provides comprehensive mathematical formulations and experimental comparisons demonstrating the trade-offs and solutions for balancing sample frequencies. The claim's questioning nature is well-warranted given the complexity and importance of the sampling strategy, as supported by the detailed evidence.

<score>1</score>

The claim is well-stated (score 1) because it accurately reflects a genuine challenge that is thoroughly addressed in the evidence through multiple approaches and careful consideration of oversampling/undersampling issues. The review comments don't directly address this specific claim about sampling frequency balancing.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

Third, after an informative image is selected, from what distribution should it sample current and previously learned novel objects to _balance_ sample frequency and avoid over-sampling or under-sampling?

The review comment to be evaluated is:

Strengths:
- The paper introduces the copy-paste data augmentation strategy to the few-shot incremental semantic segmentation, which seems to be a novel combination.
- The proposed guided sample selection method for memory construction and data synthesis improves the performance of several baseline methods.
- The paper is most clearly written and easy to follow.

Weaknesses:
- While the proposed data augmentation strategy helps few-shot learning, its technical novelty in the incremental semantic segmentation (ISS) is rather limited, as it is an add-on module and has to rely on the base ISS methods.
- The construction of the memory-replay buffer has several limitations and will affect the quality of the data augmentation in ISS:
1) Only the base stage has fully-annotated images that can be used for the target images in the copy-paste operation, and all the images from the incremental stages are partially labeled, which are unusable for that purpose. As a result, when the incremental learning evolves, the memory has to add samples of new classes and remove some base-stage images due to the capacity limit. Eventually, the memory buffer would be less useful.
2) The diversity-guided selection procedure in Sec 3.2 considers each class separately in sampling. However, for semantic segmentation, it is common that each image has regions of multiple classes. It is unclear how such a method can balance different classes. This is also a problem for the class-frequency-guided selection, as each image can have multiple classes. Another issue is that this method did not take into account the region sizes as it only considers the image-level sampling factor.
- There are several concerns on the experimental evaluation, which make the conclusion less convincing:
1) Both benchmarks are mainly used in the few-shot segmentation literature, which are a bit small in terms of data set size and number of classes. Similar to SSUL and other ISS works, the method should be evaluated on more diverse ISS task settings and larger dataset, such as ADE.
2) For the few-shot side, while this work adopts a different evaluation protocol (as described in Sec. 4.2), it would be more informative if it also uses the protocol in Cermelli et al 2021 for a fair comparison. Moreover, unlike the standard ISS, the training process of the few-shot setting typically has a large variance due to different training samples annotated. Thus it should report both the mean and standard deviation in the final results.
3) The gaps in the ablation study are relatively small. It would more convincing if the standard deviation can be provided. It is unclear how significant such improvements are in the few-shot learning setting.

The evidence to be evaluated is:After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2).

The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4).

For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi et al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.

After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

number of exemplars \(n\) \(k\leftarrow\textsc{Floor}(n/|Y_{0}|)\)// Sample per class for\(c\) from \(1\) to \(|Y_{0}|\)do \(S_{c}\leftarrow\{(x_{i},y_{i})\in D_{0},c\in y_{i}\}\) for\((x_{i},y_{i})\in S_{c}\)do \(p_{i}\leftarrow\textsc{MAP}(x_{i},y_{i},c)\)// Pred. Proto. \(s_{i}\leftarrow\textsc{CosineSimilarity}(p_{ic},w_{c})\) endfor  Sort \(S_{c}\) by similarity score \(s_{i}\) \(ES_{c}\leftarrow\{\}\)// final exemplar set of class \(c\) for\(j=1,2,\dots,k\)do \(L_{idx}\gets j\cdot|S_{c}|/k\) \(U_{idx}\leftarrow\textsc{Min}(Lower+|S_{c}|/k,|S_{c}|)\) \((x,y)\leftarrow\textsc{Sample}(S_{c}[L_{idx}:U_{idx}])\) \(ES_{c}\gets ES_{c}\cup(x,y)\) endfor endfor \(D_{0}\leftarrow\textsc{UniformSample}(\bigcup_{i=1,\dots,|Y_{0}|}ES_{i},n)\)

To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from \(\mathbb{S}\) with a probability of \(\alpha\), and sample from \(\hat{D}_{0}\) with a probability of \(1-\alpha\), where \(\alpha\) is a hyperparameter set to \(0.9\) in our implementation.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes.

Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis. To apply vRFS, we first need to compute category-wise sampling factor \(r_{c}\) for every \(c\) as in RFS. If \(c\in\mathcal{C}_{0}\), we set \(r_{c}=1\) as since during the construction of \(\hat{D}_{0}\) we already consider class balance by class-wise uniformly sampling. If \(c\in\mathcal{C}_{j}\) with \(j\geq 1\), we first compute its class frequency by \(f_{c}=nShot/|\hat{D}_{0}|\), where \(nShot\) denotes the number of images in \(D_{j}\) with at least one pixel of \(c\). Then, the category-wise sampling factor for \(c\) is given by \(r_{c}=\textsc{Max}(1,\sqrt{t/f_{c}})\).

However, we empirically found that setting \(t\) to be the multiplicative inverse of total number of classes, or \(t=1/|\cup_{1,...,j}Y_{j}|\), is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings. During the synthesis process, we first randomly select a novel class \(c_{N}\) from \(\mathcal{C}_{j}\), and another class \(c_{o}\) from \(\cup_{1,...,j}C_{j}\setminus\{c_{N}\}\).

To apply vRFS resampling, we hallucinate two _virtual_ samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of \(r_{i}=\textsc{Max}_{c\in i}r_{c}=r_{c_{o}}\). Thus, the probability to synthesize class \(c_{o}\) onto \(x_{B}\) is given by \(r_{c_{o}}/(1+r_{c_{o}})\). We then repeat the process for the novel class \(c_{N}\). Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of \(c_{N}\) in the final image.

**Frequency-guided probabilistic synthesis boosts results on novel classes**.

We take a closer look at step-wise performance (whose visualization is available in Appendix A.11) and found that the reason is due to unguided copy-paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-replay buffer.

_Full-set-random_ denotes samples that are uniformly sampled from the base training set. _Classwise-random_ denotes randomly selecting even number of samples from each base class.

RFS (Repeat Factor Sampling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights to rare classes and has been shown to work well in long-tailed task settings. The other baseline, 'GreedyClass', is achieved by greedily picking samples in every base class with the most number of classes. To consider the region sizes in the image, we design a base-line method, 'BalancedRegion', which uses a similar algorithm as our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images.

_Easy-Hard-Mix_ denotes equal mixtures of samples drawn from easiest samples and hardest samples. Lastly, _Diverse-Sampling_ is our proposed method, where samples of all difficulties are equally drawn. We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling.

_CAS_ (Class-aware-sampling) is a sampling strategy from (Gupta et al., 2019) which first uniformly samples a class and then selects a sample containing instances from the selected class. In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes.

Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
I'll analyze the claim about balancing sample frequency for current and previously learned novel objects.

1. Summary of evaluation:
The claim appears to be well-stated as the evidence provides detailed technical explanations of multiple sampling strategies:
- vRFS-based synthesis strategy with clear formulas for sampling factors
- Class-frequency guided selection using nShot/|D̂0|
- Probability calculations for synthesis (rc0/(1+rc0))
- Empirical validation showing oversampling issues
- Multiple baseline comparisons (CAS, RFS, etc.)
The review comment raises valid concerns about class balancing with multiple classes per image, but the evidence demonstrates thorough consideration of sampling frequencies through mathematical formulations and comparative analysis.

2. <score>1</score>

Justification:
1. The evidence provides comprehensive mathematical formulation for sampling probabilities:
- Clear equations for category-wise sampling factors
- Defined probability calculations for synthesis
- Parameter t set as 1/|∪1,...,jYj| for stability

2. Multiple sampling strategies are evaluated:
- Class-aware sampling (CAS)
- Repeat Factor Sampling (RFS)
- Various baselines (Full-set-random, Classwise-random, etc.)

3. The evidence acknowledges and addresses limitations:
- Oversampling/undersampling trade-offs
- Class balance considerations in D̂0 construction
- Empirical validation of sampling effects

4. While the reviewer raises concerns about multiple classes per image, the evidence shows this is handled through:
- Maximum category-level factors
- Class-frequency guided selection
- vRFS resampling approach

The claim is well-stated as it accurately reflects the evidence's comprehensive treatment of sampling frequency balancing without exaggeration.