Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address the gaps between copy-paste under the offline setting as an augmentation technique and under the online setting as a synthesis technique, we design a guided copy-paste process that improves the distribution of synthesized images by enforcing diversity of the memory-replay buffer, exploiting contextual information, and balancing class frequencies.

The review comment to be evaluated is:

Strengths:
1. The paper investigates a novel and understudied scenario that is more realistic than earlier benchmarks.
2. The study gives a comprehensive summary of similar works as well as comparisons and contrasts between them.
3. The study expands the present benchmark with new experiments involving only one labeled instance of the new classes, demonstrating that the GAPS can address the issue of missing annotations.

Weaknesses:
1. The ablation study may be extended to other settings and may provide more insights about the paper choices. In particular, it may start from a baseline that uses exemplars without pasting the new classes on them, to show that pasting is important to improve performance and that the gain is not only given by the regularization effect of using exemplars (as proposed by GDumb [a]). Moreover, the paper should clarify if, while disabling the diversity guidance, the exemplars are sampled uniformly for the old classes or if they are randomly sampled on the full dataset.
2. From the ablation study, context sampling appears to marginally improve the results. Moreover, it is not clear how the performance will change by using a pre-training dataset different from Places-365, such as ImageNet or ADE20K. Furthermore, can it be trained directly on the base dataset of the considered setting or can the method use the segmentation network itself (for example, using an auxiliary classifier on the encoder features)?
3. The paper baselines appear to be very naive. In particular, the paper reports the results for Fine-Tuning, MiB, and PIFS without considering exemplars, introducing an unfair comparison with respect to GAPS. The baselines should be reported both without exemplars (as in the original paper) and with them (indeed, without the proposed copy-and-paste strategy).

[a] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. "Gdumb: A simple approach that questions our progress in continual learning." European conference on computer vision. Springer, Cham, 2020.

The evidence to be evaluated is:In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting. However, as we will demonstrate, such naive adaptation gives unsatisfactory performance in our task setting because of _gaps_ between the static offline learning and continual online learning. In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model. After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2). During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\). The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4).

We found experimentally that when a diverse data source is used (i.e., our proposed GAPS), \(\mathcal{L}_{2}\) regularization works better. To be more precise, we construct a penalization term \(\mathcal{L}_{\mathrm{REG}}\) to regularize the output before the classifier.

\[\mathcal{L}_{\mathrm{REG}}=||f_{j}(x)-f_{j-1}(x)||_{2}. \tag{1}\]

\[\mathcal{L}(x,y)=\mathcal{L}_{\mathrm{CE}}(\phi_{j}(x),y)+\lambda\mathcal{L}_{ \mathrm{REG}}, \tag{2}\]

For methods with memory-replaying (e.g., SSUL (Cha et al., 2021)), GAPS can work directly on top of their constructed buffers with minimal modification. For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi et al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.

Intuitively, we approximate the difficulty of every sample by their similarity between the estimated prototype with learned prototypes. Estimated prototypes that are close to the learned prototype are considered easy samples and vice versa. After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

number of exemplars \(n\) \(k\leftarrow\textsc{Floor}(n/|Y_{0}|)\)// Sample per class for\(c\) from \(1\) to \(|Y_{0}|\)do \(S_{c}\leftarrow\{(x_{i},y_{i})\in D_{0},c\in y_{i}\}\) for\((x_{i},y_{i})\in S_{c}\)do \(p_{i}\leftarrow\textsc{MAP}(x_{i},y_{i},c)\)// Pred. Proto. \(s_{i}\leftarrow\textsc{CosineSimilarity}(p_{ic},w_{c})\) endfor  Sort \(S_{c}\) by similarity score \(s_{i}\) \(ES_{c}\leftarrow\{\}\)// final exemplar set of class \(c\) for\(j=1,2,\dots,k\)do \(L_{idx}\gets j\cdot|S_{c}|/k\) \(U_{idx}\leftarrow\textsc{Min}(Lower+|S_{c}|/k,|S_{c}|)\) \((x,y)\leftarrow\textsc{Sample}(S_{c}[L_{idx}:U_{idx}])\) \(ES_{c}\gets ES_{c}\cup(x,y)\) endfor endfor \(D_{0}\leftarrow\textsc{UniformSample}(\bigcup_{i=1,\dots,|Y_{0}|}ES_{i},n)\)

We hypothesize that synthesizing novel objects onto contextually consistent base images would result in an improved learning process. For example, a TV in an apartment should more likely be pasted onto an image of another apartment rather than an outdoor landscape. We design a context-guided sampling algorithm to select images from \(\hat{D_{0}}\) that are contextually similar to the provided partially-labeled images. GAPS uses a scene embedding network \(h:\mathcal{X}\rightarrow\mathbb{R}^{m}\) to estimate pairwise contextual similarity between any two given images. The embedding \(h\) maps an input image to a scene embedding vector in a metric space where pairwise similarity comparison between two embeddings are possible. In our system, we use an off-the-shelf VGGNet (Simonyan and Zisserman, 2015) and replace its last fully connected layer with a cosine-similarity-based classifier. The network \(h\) is trained using the Places365 (Zhou et al., 2017) dataset, which contains \(365\) scene categories and roughly 1.6 million images in its training split. During training, the scene prediction is done by comparing the predicted embedding with existing prototypes.

To find contextually similar base images to each novel image, we evaluate the cosine similarity of the novel image to each of the examples in \(\hat{D}_{0}\), and constuct a contextually similar subset \(\mathbb{S}\) with \(|\hat{D}_{0}|/10\) most contextually similar examples. When there are multiple novel images, we take a union of selected examples. To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from \(\mathbb{S}\) with a probability of \(\alpha\), and sample from \(\hat{D}_{0}\) with a probability of \(1-\alpha\), where \(\alpha\) is a hyperparameter set to \(0.9\) in our implementation.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes. On the other hand, rarely pasting novel instances would lead to undersampling of the novel class. Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis. To apply vRFS, we first need to compute category-wise sampling factor \(r_{c}\) for every \(c\) as in RFS. If \(c\in\mathcal{C}_{0}\), we set \(r_{c}=1\) as since during the construction of \(\hat{D}_{0}\) we already consider class balance by class-wise uniformly sampling. If \(c\in\mathcal{C}_{j}\) with \(j\geq 1\), we first compute its class frequency by \(f_{c}=nShot/|\hat{D}_{0}|\), where \(nShot\) denotes the number of images in \(D_{j}\) with at least one pixel of \(c\). Then, the category-wise sampling factor for \(c\) is given by \(r_{c}=\textsc{Max}(1,\sqrt{t/f_{c}})\). Note that in (Gupta et al., 2019), \(t\) is chosen as a hyperparameter to be tuned. However, we empirically found that setting \(t\) to be the multiplicative inverse of total number of classes, or \(t=1/|\cup_{1,...,j}Y_{j}|\), is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings. During the synthesis process, we first randomly select a novel class \(c_{N}\) from \(\mathcal{C}_{j}\), and another class \(c_{o}\) from \(\cup_{1,...,j}C_{j}\setminus\{c_{N}\}\). We first decide if \(c_{o}\) should be pasted onto \(x_{B}\). To apply vRFS resampling, we hallucinate two _virtual_ samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of \(r_{i}=\textsc{Max}_{c\in i}r_{c}=r_{c_{o}}\). Thus, the probability to synthesize class \(c_{o}\) onto \(x_{B}\) is given by \(r_{c_{o}}/(1+r_{c_{o}})\). We then repeat the process for the novel class \(c_{N}\). Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of \(c_{N}\) in the final image.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories. In Table 2, we ablate guidance designs in GAPS to illustrate how different types of guidance contribute to the final incremental learning performance than naive copy-paste synthesis.

**Our diversity-guided exemplar selection method consistently increases performance on base categories**, which suggests that it is capable of choosing diverse samples to construct a representative memory-replay buffer and mitigate catastrophic forgetting. **Context-guided sampling steadily improves performance on novel classes**, which is consistent with findings in previous work (Dvornik et al., 2018) that background context is an important factor to consider in copy-paste synthesis. **Frequency-guided probabilistic synthesis boosts results on novel classes**.

We take a closer look at step-wise performance (whose visualization is available in Appendix A.11) and found that the reason is due to unguided copy-paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-replay buffer.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

In Table 6, we study how different memory buffer construction strategies influence the performance of the model. We experiment with multiple baseline implementations to construct the memory-replay buffer:

To consider multiple classes in images, we design two baseline methods. RFS (Repeat Factor Sampling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights to rare classes and has been shown to work well in long-tailed task settings. The other baseline, 'GreedyClass', is achieved by greedily picking samples in every base class with the most number of classes. To consider the region sizes in the image, we design a base-line method, 'BalancedRegion', which uses a similar algorithm as our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images.

We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling.

_CAS_ (Class-aware-sampling) is a sampling strategy from (Gupta et al., 2019) which first uniformly samples a class and then selects a sample containing instances from the selected class. In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes. For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

In the result, we can observe that our method (vRFS) achieves the best result in terms of the novel IoU and harmonic mean of base and novel IoU.

using more aggresive scale jittering helps increase the diversity of samples and increases performance. Beyond confirming conclusion from previous work, such improvement alongside stronger augmentation demonstrates the effectiveness of augmentation technique even when there is few data presented, highlighting a possibility to further improve the performance by using even stronger augmentation technique.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS.

We explore an alternative memory-replay buffer implementation to add examples learned during the incremental learning stage to the memory-replay buffer, in an attempt to help performance on novel classes.

Here, 'PIFS(\(\mathcal{L}_{2}\))+MEM+PL' means that we follow SSUL and perform pseudo-labeling on partially annotated samples before adding them to memory.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines. We experiment with several other pretraining models as the contextual guiding model. In particular, we use ImageNet-pretrained model and pretrained image encoder from CLIP (Radford et al., 2021) to compute scene embedding for incoming images.

We conducted an additional experiment on the COCO-20-1 dataset under the 5-shot setting, where we vary the number of fully-annotated images in the memory-replay buffer to investigate the per formance of our method. As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

We can see that given an image of traffic lights on a railway, our scene embedding model is able to select contextually-similar scenes such as railways and street scenes for subsequent synthesis.

On the other hand, PIFS(\(\mathcal{L}_{2}\))+GAPS is able to maintain good performance on learned classes as it adapts to new classes. \begin{table}
\begin{tabular}{c|c c c} Base Set & Base & Novel & HM \\ \hline Full-set-random & \(47.9\pm 0.2\) & \(24.6\pm 0.7\) & \(32.5\) \\ Classwise-random & \(48.3\pm 0.2\) & \(24.7\pm 0.8\) & \(32.7\) \\ RFS & \(47.6\pm 0.3\) & \(25.2\pm 0.7\) & \(33.0\) \\ GreedyClass & \(48.7\pm 0.2\) & \(25.0\pm 0.7\) & \(33.0\) \\ BalancedRegion & \(48.6\pm 0.2\) & \(24.9\pm 0.8\) & \(32.9\) \\ \hline Easy & \(48.0\pm 0.2\) & \(24.2\pm 0.8\) & \(32.2\) \\ Hard & \(47.7\pm 0.2\) & \(25.2\pm 0.8\) & \(33.0\) \\ Easy-Hard-Mix & \(48.4\pm 0.3\) & \(24.8\pm 0.7\) & \(32.8\) \\ Diverse Sampling (Ours) & \(\mathbf{48.9}\pm 0.2\) & \(\mathbf{25.3}\pm 0.9\) & \(\mathbf{33.3}\) \\ \end{tabular}
\end{table}
Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity-aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity-aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 2: Ablation study of components in GAPS on PIFS(L2) on the COCO-20i dataset under 5- shot setting. Note that when only combined with the memory-replay buffer, the base IoU is higher because model has access to additional full annotations. When diversity guidance (D-guide) is disabled, D̂0 consists of random examples from the base dataset, resulting in worse base performance. When context guidance (C-guide) is disabled, a base image is uniformly sampled. When frequency guidance (F-guide) is disabled, a novel instance is sampled uniformly and is always pasted onto

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 6: Qualitative visualization of samples selected in the context-aware sampling process. Given an image of traffic lights in railway, our scene embedding network is able to select contextuallysimilar scenes (railway, street).

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the evidence presented, the claim appears to be well-stated (score of 1) for these key reasons:

1. The evidence thoroughly demonstrates each component mentioned in the claim:
- Diversity guidance through exemplar selection is validated by ablation studies showing improved base category performance
- Context guidance is supported by scene embedding experiments and qualitative results showing appropriate scene matching
- Class frequency balancing is confirmed through the vRFS implementation details and results

2. The claim avoids overstatement by:
- Using measured language ("improves" rather than stronger claims)
- Acknowledging it as a "process" rather than claiming a complete solution
- Specifically scoping the improvements to "distribution of synthesized images"

3. The ablation studies in Table 2 and Table 6 provide direct empirical support for each claimed contribution showing incremental improvements.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address the gaps between copy-paste under the offline setting as an augmentation technique and under the online setting as a synthesis technique, we design a guided copy-paste process that improves the distribution of synthesized images by enforcing diversity of the memory-replay buffer, exploiting contextual information, and balancing class frequencies.

The review comment to be evaluated is:

Strength:
 (1) This paper introduces copy-paste as a synthesis technique to few-shot incremental segmentation.
 (2) Experiments show clear improvement on performance with the proposed method.

Weaknesses:
(1)  My main concern is the novelty of the paper, copy-paste technique is more about a data augmentation methods, which has already been shown effective in many segmentation task, it is not novel enough to be accepted as an ICLR paper.
(2) The paper is not clear at least for me, especially figure 1 and figure, it's really hard for readers to understand the task, although the idea is clear.

The evidence to be evaluated is:In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting. However, as we will demonstrate, such naive adaptation gives unsatisfactory performance in our task setting because of _gaps_ between the static offline learning and continual online learning. In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model. After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2). During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\). The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4).

We found experimentally that when a diverse data source is used (i.e., our proposed GAPS), \(\mathcal{L}_{2}\) regularization works better. To be more precise, we construct a penalization term \(\mathcal{L}_{\mathrm{REG}}\) to regularize the output before the classifier.

\[\mathcal{L}_{\mathrm{REG}}=||f_{j}(x)-f_{j-1}(x)||_{2}. \tag{1}\]

\[\mathcal{L}(x,y)=\mathcal{L}_{\mathrm{CE}}(\phi_{j}(x),y)+\lambda\mathcal{L}_{ \mathrm{REG}}, \tag{2}\]

For methods with memory-replaying (e.g., SSUL (Cha et al., 2021)), GAPS can work directly on top of their constructed buffers with minimal modification. For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi et al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.

Intuitively, we approximate the difficulty of every sample by their similarity between the estimated prototype with learned prototypes. Estimated prototypes that are close to the learned prototype are considered easy samples and vice versa. After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

number of exemplars \(n\) \(k\leftarrow\textsc{Floor}(n/|Y_{0}|)\)// Sample per class for\(c\) from \(1\) to \(|Y_{0}|\)do \(S_{c}\leftarrow\{(x_{i},y_{i})\in D_{0},c\in y_{i}\}\) for\((x_{i},y_{i})\in S_{c}\)do \(p_{i}\leftarrow\textsc{MAP}(x_{i},y_{i},c)\)// Pred. Proto. \(s_{i}\leftarrow\textsc{CosineSimilarity}(p_{ic},w_{c})\) endfor  Sort \(S_{c}\) by similarity score \(s_{i}\) \(ES_{c}\leftarrow\{\}\)// final exemplar set of class \(c\) for\(j=1,2,\dots,k\)do \(L_{idx}\gets j\cdot|S_{c}|/k\) \(U_{idx}\leftarrow\textsc{Min}(Lower+|S_{c}|/k,|S_{c}|)\) \((x,y)\leftarrow\textsc{Sample}(S_{c}[L_{idx}:U_{idx}])\) \(ES_{c}\gets ES_{c}\cup(x,y)\) endfor endfor \(D_{0}\leftarrow\textsc{UniformSample}(\bigcup_{i=1,\dots,|Y_{0}|}ES_{i},n)\)

We hypothesize that synthesizing novel objects onto contextually consistent base images would result in an improved learning process. For example, a TV in an apartment should more likely be pasted onto an image of another apartment rather than an outdoor landscape. We design a context-guided sampling algorithm to select images from \(\hat{D_{0}}\) that are contextually similar to the provided partially-labeled images. GAPS uses a scene embedding network \(h:\mathcal{X}\rightarrow\mathbb{R}^{m}\) to estimate pairwise contextual similarity between any two given images. The embedding \(h\) maps an input image to a scene embedding vector in a metric space where pairwise similarity comparison between two embeddings are possible. In our system, we use an off-the-shelf VGGNet (Simonyan and Zisserman, 2015) and replace its last fully connected layer with a cosine-similarity-based classifier. The network \(h\) is trained using the Places365 (Zhou et al., 2017) dataset, which contains \(365\) scene categories and roughly 1.6 million images in its training split. During training, the scene prediction is done by comparing the predicted embedding with existing prototypes.

To find contextually similar base images to each novel image, we evaluate the cosine similarity of the novel image to each of the examples in \(\hat{D}_{0}\), and constuct a contextually similar subset \(\mathbb{S}\) with \(|\hat{D}_{0}|/10\) most contextually similar examples. When there are multiple novel images, we take a union of selected examples. To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from \(\mathbb{S}\) with a probability of \(\alpha\), and sample from \(\hat{D}_{0}\) with a probability of \(1-\alpha\), where \(\alpha\) is a hyperparameter set to \(0.9\) in our implementation.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes. On the other hand, rarely pasting novel instances would lead to undersampling of the novel class. Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis. To apply vRFS, we first need to compute category-wise sampling factor \(r_{c}\) for every \(c\) as in RFS. If \(c\in\mathcal{C}_{0}\), we set \(r_{c}=1\) as since during the construction of \(\hat{D}_{0}\) we already consider class balance by class-wise uniformly sampling. If \(c\in\mathcal{C}_{j}\) with \(j\geq 1\), we first compute its class frequency by \(f_{c}=nShot/|\hat{D}_{0}|\), where \(nShot\) denotes the number of images in \(D_{j}\) with at least one pixel of \(c\). Then, the category-wise sampling factor for \(c\) is given by \(r_{c}=\textsc{Max}(1,\sqrt{t/f_{c}})\). Note that in (Gupta et al., 2019), \(t\) is chosen as a hyperparameter to be tuned. However, we empirically found that setting \(t\) to be the multiplicative inverse of total number of classes, or \(t=1/|\cup_{1,...,j}Y_{j}|\), is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings. During the synthesis process, we first randomly select a novel class \(c_{N}\) from \(\mathcal{C}_{j}\), and another class \(c_{o}\) from \(\cup_{1,...,j}C_{j}\setminus\{c_{N}\}\). We first decide if \(c_{o}\) should be pasted onto \(x_{B}\). To apply vRFS resampling, we hallucinate two _virtual_ samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of \(r_{i}=\textsc{Max}_{c\in i}r_{c}=r_{c_{o}}\). Thus, the probability to synthesize class \(c_{o}\) onto \(x_{B}\) is given by \(r_{c_{o}}/(1+r_{c_{o}})\). We then repeat the process for the novel class \(c_{N}\). Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of \(c_{N}\) in the final image.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories. In Table 2, we ablate guidance designs in GAPS to illustrate how different types of guidance contribute to the final incremental learning performance than naive copy-paste synthesis.

**Our diversity-guided exemplar selection method consistently increases performance on base categories**, which suggests that it is capable of choosing diverse samples to construct a representative memory-replay buffer and mitigate catastrophic forgetting. **Context-guided sampling steadily improves performance on novel classes**, which is consistent with findings in previous work (Dvornik et al., 2018) that background context is an important factor to consider in copy-paste synthesis. **Frequency-guided probabilistic synthesis boosts results on novel classes**.

We take a closer look at step-wise performance (whose visualization is available in Appendix A.11) and found that the reason is due to unguided copy-paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-replay buffer.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

In Table 6, we study how different memory buffer construction strategies influence the performance of the model. We experiment with multiple baseline implementations to construct the memory-replay buffer:

To consider multiple classes in images, we design two baseline methods. RFS (Repeat Factor Sampling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights to rare classes and has been shown to work well in long-tailed task settings. The other baseline, 'GreedyClass', is achieved by greedily picking samples in every base class with the most number of classes. To consider the region sizes in the image, we design a base-line method, 'BalancedRegion', which uses a similar algorithm as our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images.

We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling.

_CAS_ (Class-aware-sampling) is a sampling strategy from (Gupta et al., 2019) which first uniformly samples a class and then selects a sample containing instances from the selected class. In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes. For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

In the result, we can observe that our method (vRFS) achieves the best result in terms of the novel IoU and harmonic mean of base and novel IoU.

using more aggresive scale jittering helps increase the diversity of samples and increases performance. Beyond confirming conclusion from previous work, such improvement alongside stronger augmentation demonstrates the effectiveness of augmentation technique even when there is few data presented, highlighting a possibility to further improve the performance by using even stronger augmentation technique.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS.

We explore an alternative memory-replay buffer implementation to add examples learned during the incremental learning stage to the memory-replay buffer, in an attempt to help performance on novel classes.

Here, 'PIFS(\(\mathcal{L}_{2}\))+MEM+PL' means that we follow SSUL and perform pseudo-labeling on partially annotated samples before adding them to memory.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines. We experiment with several other pretraining models as the contextual guiding model. In particular, we use ImageNet-pretrained model and pretrained image encoder from CLIP (Radford et al., 2021) to compute scene embedding for incoming images.

We conducted an additional experiment on the COCO-20-1 dataset under the 5-shot setting, where we vary the number of fully-annotated images in the memory-replay buffer to investigate the per formance of our method. As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

We can see that given an image of traffic lights on a railway, our scene embedding model is able to select contextually-similar scenes such as railways and street scenes for subsequent synthesis.

On the other hand, PIFS(\(\mathcal{L}_{2}\))+GAPS is able to maintain good performance on learned classes as it adapts to new classes. \begin{table}
\begin{tabular}{c|c c c} Base Set & Base & Novel & HM \\ \hline Full-set-random & \(47.9\pm 0.2\) & \(24.6\pm 0.7\) & \(32.5\) \\ Classwise-random & \(48.3\pm 0.2\) & \(24.7\pm 0.8\) & \(32.7\) \\ RFS & \(47.6\pm 0.3\) & \(25.2\pm 0.7\) & \(33.0\) \\ GreedyClass & \(48.7\pm 0.2\) & \(25.0\pm 0.7\) & \(33.0\) \\ BalancedRegion & \(48.6\pm 0.2\) & \(24.9\pm 0.8\) & \(32.9\) \\ \hline Easy & \(48.0\pm 0.2\) & \(24.2\pm 0.8\) & \(32.2\) \\ Hard & \(47.7\pm 0.2\) & \(25.2\pm 0.8\) & \(33.0\) \\ Easy-Hard-Mix & \(48.4\pm 0.3\) & \(24.8\pm 0.7\) & \(32.8\) \\ Diverse Sampling (Ours) & \(\mathbf{48.9}\pm 0.2\) & \(\mathbf{25.3}\pm 0.9\) & \(\mathbf{33.3}\) \\ \end{tabular}
\end{table}
Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity-aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity-aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 2: Ablation study of components in GAPS on PIFS(L2) on the COCO-20i dataset under 5- shot setting. Note that when only combined with the memory-replay buffer, the base IoU is higher because model has access to additional full annotations. When diversity guidance (D-guide) is disabled, D̂0 consists of random examples from the base dataset, resulting in worse base performance. When context guidance (C-guide) is disabled, a base image is uniformly sampled. When frequency guidance (F-guide) is disabled, a novel instance is sampled uniformly and is always pasted onto

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 6: Qualitative visualization of samples selected in the context-aware sampling process. Given an image of traffic lights in railway, our scene embedding network is able to select contextuallysimilar scenes (railway, street).

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
Based on the extensive evidence provided, I'll evaluate the claim about the guided copy-paste process.

The evidence strongly supports the claim through:
- Clear demonstration of diversity-guided exemplar selection with detailed algorithms and ablation studies showing improved performance
- Concrete implementation of context-guided sampling using scene embedding networks, validated through experiments and visualizations
- Specific frequency-guided synthesis strategy (vRFS) with mathematical formulation and empirical validation
- Comprehensive ablation studies in Table 2 showing each component's contribution
- Reviewer comments acknowledge clear performance improvements

The claim appears well-stated as it accurately reflects the evidence without exaggeration. The three key components (diversity, context, class frequencies) are thoroughly validated, and the performance improvements are demonstrated through extensive experimentation.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

To address the gaps between copy-paste under the offline setting as an augmentation technique and under the online setting as a synthesis technique, we design a guided copy-paste process that improves the distribution of synthesized images by enforcing diversity of the memory-replay buffer, exploiting contextual information, and balancing class frequencies.

The review comment to be evaluated is:

Strengths:
- The paper introduces the copy-paste data augmentation strategy to the few-shot incremental semantic segmentation, which seems to be a novel combination.
- The proposed guided sample selection method for memory construction and data synthesis improves the performance of several baseline methods.
- The paper is most clearly written and easy to follow.

Weaknesses:
- While the proposed data augmentation strategy helps few-shot learning, its technical novelty in the incremental semantic segmentation (ISS) is rather limited, as it is an add-on module and has to rely on the base ISS methods.
- The construction of the memory-replay buffer has several limitations and will affect the quality of the data augmentation in ISS:
1) Only the base stage has fully-annotated images that can be used for the target images in the copy-paste operation, and all the images from the incremental stages are partially labeled, which are unusable for that purpose. As a result, when the incremental learning evolves, the memory has to add samples of new classes and remove some base-stage images due to the capacity limit. Eventually, the memory buffer would be less useful.
2) The diversity-guided selection procedure in Sec 3.2 considers each class separately in sampling. However, for semantic segmentation, it is common that each image has regions of multiple classes. It is unclear how such a method can balance different classes. This is also a problem for the class-frequency-guided selection, as each image can have multiple classes. Another issue is that this method did not take into account the region sizes as it only considers the image-level sampling factor.
- There are several concerns on the experimental evaluation, which make the conclusion less convincing:
1) Both benchmarks are mainly used in the few-shot segmentation literature, which are a bit small in terms of data set size and number of classes. Similar to SSUL and other ISS works, the method should be evaluated on more diverse ISS task settings and larger dataset, such as ADE.
2) For the few-shot side, while this work adopts a different evaluation protocol (as described in Sec. 4.2), it would be more informative if it also uses the protocol in Cermelli et al 2021 for a fair comparison. Moreover, unlike the standard ISS, the training process of the few-shot setting typically has a large variance due to different training samples annotated. Thus it should report both the mean and standard deviation in the final results.
3) The gaps in the ablation study are relatively small. It would more convincing if the standard deviation can be provided. It is unclear how significant such improvements are in the few-shot learning setting.

The evidence to be evaluated is:In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

We extend the augmentation strategy from Ghiasi et al. (2021) and construct an intuitive baseline called Naive copy-Paste Synthesis (NPS) to adapt it to our online task setting. However, as we will demonstrate, such naive adaptation gives unsatisfactory performance in our task setting because of _gaps_ between the static offline learning and continual online learning. In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model. After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2). During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\). The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4).

We found experimentally that when a diverse data source is used (i.e., our proposed GAPS), \(\mathcal{L}_{2}\) regularization works better. To be more precise, we construct a penalization term \(\mathcal{L}_{\mathrm{REG}}\) to regularize the output before the classifier.

\[\mathcal{L}_{\mathrm{REG}}=||f_{j}(x)-f_{j-1}(x)||_{2}. \tag{1}\]

\[\mathcal{L}(x,y)=\mathcal{L}_{\mathrm{CE}}(\phi_{j}(x),y)+\lambda\mathcal{L}_{ \mathrm{REG}}, \tag{2}\]

For methods with memory-replaying (e.g., SSUL (Cha et al., 2021)), GAPS can work directly on top of their constructed buffers with minimal modification. For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting. Selecting diverse examples that are representative of the base dataset helps mitigate catastrophic forgetting, as suggested by Rebuffi et al. (2017). Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity.

Intuitively, we approximate the difficulty of every sample by their similarity between the estimated prototype with learned prototypes. Estimated prototypes that are close to the learned prototype are considered easy samples and vice versa. After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

number of exemplars \(n\) \(k\leftarrow\textsc{Floor}(n/|Y_{0}|)\)// Sample per class for\(c\) from \(1\) to \(|Y_{0}|\)do \(S_{c}\leftarrow\{(x_{i},y_{i})\in D_{0},c\in y_{i}\}\) for\((x_{i},y_{i})\in S_{c}\)do \(p_{i}\leftarrow\textsc{MAP}(x_{i},y_{i},c)\)// Pred. Proto. \(s_{i}\leftarrow\textsc{CosineSimilarity}(p_{ic},w_{c})\) endfor  Sort \(S_{c}\) by similarity score \(s_{i}\) \(ES_{c}\leftarrow\{\}\)// final exemplar set of class \(c\) for\(j=1,2,\dots,k\)do \(L_{idx}\gets j\cdot|S_{c}|/k\) \(U_{idx}\leftarrow\textsc{Min}(Lower+|S_{c}|/k,|S_{c}|)\) \((x,y)\leftarrow\textsc{Sample}(S_{c}[L_{idx}:U_{idx}])\) \(ES_{c}\gets ES_{c}\cup(x,y)\) endfor endfor \(D_{0}\leftarrow\textsc{UniformSample}(\bigcup_{i=1,\dots,|Y_{0}|}ES_{i},n)\)

We hypothesize that synthesizing novel objects onto contextually consistent base images would result in an improved learning process. For example, a TV in an apartment should more likely be pasted onto an image of another apartment rather than an outdoor landscape. We design a context-guided sampling algorithm to select images from \(\hat{D_{0}}\) that are contextually similar to the provided partially-labeled images. GAPS uses a scene embedding network \(h:\mathcal{X}\rightarrow\mathbb{R}^{m}\) to estimate pairwise contextual similarity between any two given images. The embedding \(h\) maps an input image to a scene embedding vector in a metric space where pairwise similarity comparison between two embeddings are possible. In our system, we use an off-the-shelf VGGNet (Simonyan and Zisserman, 2015) and replace its last fully connected layer with a cosine-similarity-based classifier. The network \(h\) is trained using the Places365 (Zhou et al., 2017) dataset, which contains \(365\) scene categories and roughly 1.6 million images in its training split. During training, the scene prediction is done by comparing the predicted embedding with existing prototypes.

To find contextually similar base images to each novel image, we evaluate the cosine similarity of the novel image to each of the examples in \(\hat{D}_{0}\), and constuct a contextually similar subset \(\mathbb{S}\) with \(|\hat{D}_{0}|/10\) most contextually similar examples. When there are multiple novel images, we take a union of selected examples. To allow other base scenes to be sampled to mitigate catastrophic forgetting, we sample from \(\mathbb{S}\) with a probability of \(\alpha\), and sample from \(\hat{D}_{0}\) with a probability of \(1-\alpha\), where \(\alpha\) is a hyperparameter set to \(0.9\) in our implementation.

Now the final question is, given a fully-annotated image \(x_{B}\) and an image of a novel object \(x_{N}\), how frequent should we apply copy-paste? There is a trade-off between oversampling and undersampling. As one extreme, one can follow Ghiasi et al. (2021) and always apply copy-paste augmentation to paste novel objects onto every base image. However, this will lead to oversampling of novel categories in the current task, which we found to hurt the performance of existing classes. On the other hand, rarely pasting novel instances would lead to undersampling of the novel class. Therefore, to guide copy-paste in the online setting, we design a synthesis strategy called vRFS based on RFS (Repeat Factor Sampling) described by Gupta et al. (2019) to perform synthesis. To apply vRFS, we first need to compute category-wise sampling factor \(r_{c}\) for every \(c\) as in RFS. If \(c\in\mathcal{C}_{0}\), we set \(r_{c}=1\) as since during the construction of \(\hat{D}_{0}\) we already consider class balance by class-wise uniformly sampling. If \(c\in\mathcal{C}_{j}\) with \(j\geq 1\), we first compute its class frequency by \(f_{c}=nShot/|\hat{D}_{0}|\), where \(nShot\) denotes the number of images in \(D_{j}\) with at least one pixel of \(c\). Then, the category-wise sampling factor for \(c\) is given by \(r_{c}=\textsc{Max}(1,\sqrt{t/f_{c}})\). Note that in (Gupta et al., 2019), \(t\) is chosen as a hyperparameter to be tuned. However, we empirically found that setting \(t\) to be the multiplicative inverse of total number of classes, or \(t=1/|\cup_{1,...,j}Y_{j}|\), is enough to yield stable results across different datasets and under different few-shot settings. This eliminates the need to search a hyperparameter for different settings and make our proposed method more robust towards different task settings. During the synthesis process, we first randomly select a novel class \(c_{N}\) from \(\mathcal{C}_{j}\), and another class \(c_{o}\) from \(\cup_{1,...,j}C_{j}\setminus\{c_{N}\}\). We first decide if \(c_{o}\) should be pasted onto \(x_{B}\). To apply vRFS resampling, we hallucinate two _virtual_ samples: in the first sample where copy-paste would not be applied, the image-level sampling factor is given by 1. In the second sample where copy-paste synthesis were to be performed, we would obtain a sample with image-level sampling factor of \(r_{i}=\textsc{Max}_{c\in i}r_{c}=r_{c_{o}}\). Thus, the probability to synthesize class \(c_{o}\) onto \(x_{B}\) is given by \(r_{c_{o}}/(1+r_{c_{o}})\). We then repeat the process for the novel class \(c_{N}\). Note that vRFS synthesis is applied twice for every class, resulting in up to two pasted instances of \(c_{N}\) in the final image.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories. In Table 2, we ablate guidance designs in GAPS to illustrate how different types of guidance contribute to the final incremental learning performance than naive copy-paste synthesis.

**Our diversity-guided exemplar selection method consistently increases performance on base categories**, which suggests that it is capable of choosing diverse samples to construct a representative memory-replay buffer and mitigate catastrophic forgetting. **Context-guided sampling steadily improves performance on novel classes**, which is consistent with findings in previous work (Dvornik et al., 2018) that background context is an important factor to consider in copy-paste synthesis. **Frequency-guided probabilistic synthesis boosts results on novel classes**.

We take a closer look at step-wise performance (whose visualization is available in Appendix A.11) and found that the reason is due to unguided copy-paste's oversampling of novel classes that are being adapted, and forgetting of classes learned in the previous incremental learning stage and not in the memory-replay buffer.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

In Table 6, we study how different memory buffer construction strategies influence the performance of the model. We experiment with multiple baseline implementations to construct the memory-replay buffer:

To consider multiple classes in images, we design two baseline methods. RFS (Repeat Factor Sampling) is done by using the weighted sampling technique described in LVIS (Gupta et al., 2019), which considers multiple classes in samples by computing category-level re-sampling factor and creates image-level re-sampling factor based on the maximum category-level factor in the image. RFS assigns higher weights to rare classes and has been shown to work well in long-tailed task settings. The other baseline, 'GreedyClass', is achieved by greedily picking samples in every base class with the most number of classes. To consider the region sizes in the image, we design a base-line method, 'BalancedRegion', which uses a similar algorithm as our proposed diversity sampling method, but we replace the difficulty estimation with the number of pixels the selected class occupies in images.

We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling.

_CAS_ (Class-aware-sampling) is a sampling strategy from (Gupta et al., 2019) which first uniformly samples a class and then selects a sample containing instances from the selected class. In our implementation, the probability to synthesize novel objects is simply given by \(1/M\), where \(M\) is the sum of total number of base classes and seen novel classes. For every selected base image, the _Aways Paste_ method randomly picks a novel category and always synthesizes an instance of the selected novel category onto the given image. Finally, the _50-50-Paste_ method is an extension of the naive copy-paste method to leverage past novel samples like vRFS. Essentially, it replays samples from learned novel classes with 50% probability, and uses novel class from the current incremental learning task also with 50% probability.

In the result, we can observe that our method (vRFS) achieves the best result in terms of the novel IoU and harmonic mean of base and novel IoU.

using more aggresive scale jittering helps increase the diversity of samples and increases performance. Beyond confirming conclusion from previous work, such improvement alongside stronger augmentation demonstrates the effectiveness of augmentation technique even when there is few data presented, highlighting a possibility to further improve the performance by using even stronger augmentation technique.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS.

We explore an alternative memory-replay buffer implementation to add examples learned during the incremental learning stage to the memory-replay buffer, in an attempt to help performance on novel classes.

Here, 'PIFS(\(\mathcal{L}_{2}\))+MEM+PL' means that we follow SSUL and perform pseudo-labeling on partially annotated samples before adding them to memory.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines. We experiment with several other pretraining models as the contextual guiding model. In particular, we use ImageNet-pretrained model and pretrained image encoder from CLIP (Radford et al., 2021) to compute scene embedding for incoming images.

We conducted an additional experiment on the COCO-20-1 dataset under the 5-shot setting, where we vary the number of fully-annotated images in the memory-replay buffer to investigate the per formance of our method. As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

We can see that given an image of traffic lights on a railway, our scene embedding model is able to select contextually-similar scenes such as railways and street scenes for subsequent synthesis.

On the other hand, PIFS(\(\mathcal{L}_{2}\))+GAPS is able to maintain good performance on learned classes as it adapts to new classes. \begin{table}
\begin{tabular}{c|c c c} Base Set & Base & Novel & HM \\ \hline Full-set-random & \(47.9\pm 0.2\) & \(24.6\pm 0.7\) & \(32.5\) \\ Classwise-random & \(48.3\pm 0.2\) & \(24.7\pm 0.8\) & \(32.7\) \\ RFS & \(47.6\pm 0.3\) & \(25.2\pm 0.7\) & \(33.0\) \\ GreedyClass & \(48.7\pm 0.2\) & \(25.0\pm 0.7\) & \(33.0\) \\ BalancedRegion & \(48.6\pm 0.2\) & \(24.9\pm 0.8\) & \(32.9\) \\ \hline Easy & \(48.0\pm 0.2\) & \(24.2\pm 0.8\) & \(32.2\) \\ Hard & \(47.7\pm 0.2\) & \(25.2\pm 0.8\) & \(33.0\) \\ Easy-Hard-Mix & \(48.4\pm 0.3\) & \(24.8\pm 0.7\) & \(32.8\) \\ Diverse Sampling (Ours) & \(\mathbf{48.9}\pm 0.2\) & \(\mathbf{25.3}\pm 0.9\) & \(\mathbf{33.3}\) \\ \end{tabular}
\end{table}
Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity-aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 6: Ablation study of different memory-replay buffer construction strategies on COCO-20-1. We found that the performance is maximized when using our diversity-aware memory-replay buffer reconstruction strategy. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 2: Ablation study of components in GAPS on PIFS(L2) on the COCO-20i dataset under 5- shot setting. Note that when only combined with the memory-replay buffer, the base IoU is higher because model has access to additional full annotations. When diversity guidance (D-guide) is disabled, D̂0 consists of random examples from the base dataset, resulting in worse base performance. When context guidance (C-guide) is disabled, a base image is uniformly sampled. When frequency guidance (F-guide) is disabled, a novel instance is sampled uniformly and is always pasted onto

Input: Figure 1: Our proposed method utilizes guided copy-paste augmentation to synthesize diverse training data, using as few as one single novel instance for training. For example, the model encounters an image of many motorcycles, which is novel to the model. As a result, the model incorrectly assigns learned bicycle labels to these pixels and therefore needs to be updated. Our proposed method can adapt to the novel motorcycle class with an annotation of a single motorcycle, which can be efficiently annotated; whereas previous work (Cermelli et al., 2020; 2021) require time-consuming annotation of all instances of motorcycles or even the entire image. Best view in color.

Input: Figure 6: Qualitative visualization of samples selected in the context-aware sampling process. Given an image of traffic lights in railway, our scene embedding network is able to select contextuallysimilar scenes (railway, street).

Input: Figure 2: Overview of GAPS. During the incremental learning stage, GAPS takes in as few as one annotated instance of a single image. It is more probable for GAPS to select a scene contextually similar to the provided image from memory-replay buffer D̂0. The image is then probabilistically pasted to generate synthetic fully-labeled scenes. Note that GAPS is model-agnostic, and here we use PIFS (Cermelli et al., 2021) as an example for the underlying segmentation model to illustrate how GAPS is applied as a pre-processor. Best seen in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, the claim appears to be well-substantiated and accurately reflects the proposed guided copy-paste process. The evidence demonstrates:

- Clear technical details of the diversity-guided exemplar selection, context-guided sampling, and frequency-guided synthesis components
- Ablation studies showing each component's contribution (Table 2)
- Quantitative results validating improved performance compared to baselines
- Visualization examples showing contextual similarity working as claimed (Fig 6)
- Detailed algorithms and mathematical formulations backing the claimed techniques
- Statistical significance with confidence intervals reported

The claim avoids exaggeration and precisely describes what the method does, matching the extensive evidence provided. The claim's scope aligns well with the demonstrated capabilities and limitations discussed in the paper.

<score>1</score>


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

3. The proposed GAPS technique consistently boosts the performance of a variety of incremental learning algorithms from simple fine-tuning to sophisticated state-of-the-arts under the few-shot setting.

The evidence to be evaluated is:_Incremental learning_ studies how to enable models to adapt to new classes while mitigating catastrophic forgetting without accessing the old dataset or full-scale re-training.

However, existing work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model. After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2). During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\). The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4). In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

We found experimentally that when a diverse data source is used (i.e., our proposed GAPS), \(\mathcal{L}_{2}\) regularization works better. To be more precise, we construct a penalization term \(\mathcal{L}_{\mathrm{REG}}\) to regularize the output before the classifier. For incremental learning task \(D_{j}\) with image-mask pairs \((x,y)\), we have \[\mathcal{L}_{\mathrm{REG}}=||f_{j}(x)-f_{j-1}(x)||_{2}. \tag{1}\] The final training loss is given by \[\mathcal{L}(x,y)=\mathcal{L}_{\mathrm{CE}}(\phi_{j}(x),y)+\lambda\mathcal{L}_{ \mathrm{REG}}, \tag{2}\] where \(\mathcal{L}_{\mathrm{CE}}\) is either the standard cross-entropy loss or the modified cross-entropy loss from Cermelli et al. (2020). \(\lambda\) is a hyper-parameter used to weight the regularization loss.

For methods with memory-replaying (e.g., SSUL (Cha et al., 2021)), GAPS can work directly on top of their constructed buffers with minimal modification. For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting.

Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity. Here, we present an algorithm (Algorithm 1) to construct \(\hat{D_{0}}\) by exploiting the Masked Average Pooling (Map) function from Cermelli et al. (2021). Intuitively, we approximate the difficulty of every sample by their similarity between the estimated prototype with learned prototypes. Estimated prototypes that are close to the learned prototype are considered easy samples and vice versa. After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

We hypothesize that synthesizing novel objects onto contextually consistent base images would result in an improved learning process.

GAPS uses a scene embedding network \(h:\mathcal{X}\rightarrow\mathbb{R}^{m}\) to estimate pairwise contextual similarity between any two given images. The embedding \(h\) maps an input image to a scene embedding vector in a metric space where pairwise similarity comparison between two embeddings are possible. In our system, we use an off-the-shelf VGGNet (Simonyan and Zisserman, 2015) and replace its last fully connected layer with a cosine-similarity-based classifier. The network \(h\) is trained using the Places365 (Zhou et al., 2017) dataset, which contains \(365\) scene categories and roughly 1.6 million images in its training split. During training, the scene prediction is done by comparing the predicted embedding with existing prototypes.

We follow literature in few-shot segmentation and few-shot incremental segmentation (Shaban et al., 2017; Nguyen and Todorovic, 2019; Tian et al., 2020; Cermelli et al., 2021) and evaluate our model on the PASCAL-5\({}^{\text{i}}\) dataset (Shaban et al., 2017) and the COCO-20\({}^{\text{i}}\) dataset (Nguyen and Todorovic, 2019). PASCAL-5\({}^{\text{i}}\) is artificially built from the PASCAL VOC 2012 Semantic Segmentation dataset (Eveningham et al., 2010) with additional annotations from the SBD (Hariharan et al., 2011) dataset. The original VOC segmentation dataset provides segmentation annotations for 20 object categories.

In Table 1, we evaluate various incremental segmentation methods on the PASCAL-5\({}^{\text{i}}\) dataset and the COCO-20\({}^{\text{i}}\) dataset, and combine them with GAPS where appropriate.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

**Context-guided sampling steadily improves performance on novel classes**, which is consistent with findings in previous work (Dvornik et al., 2018) that background context is an important factor to consider in copy-paste synthesis. **Frequency-guided probabilistic synthesis boosts results on novel classes**.

On the contrary, \(\text{PIFS}(\mathcal{L}_{2})\)+GAPS shows only a minor performance decrease with single instances.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling. It is also worth noting that both the 'GreedyClass' method and the 'BalancedRegion' method achieve almost comparable results to our methods, but still slightly underperform our diversity guidance.

Beyond confirming conclusion from previous work, such improvement alongside stronger augmentation demonstrates the effectiveness of augmentation technique even when there is few data presented, highlighting a possibility to further improve the performance by using even stronger augmentation technique. We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting. The results are given in Table 9. We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS. This further validates the effectiveness of our approach.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column. We can see that PIFS and PIFS+NPS gives reasonably good predictions on classes that are adapted recently (e.g., the cow class in the last row). However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Fig. 7 and Fig. 8 demonstrate how PIFS(\(\mathcal{L}_{2}\))+GAPS compare to the baseline PIFS(\(\mathcal{L}_{2}\)) across multiple step.

These visualizations demonstrate how PIFS(\(\mathcal{L}_{2}\)) forgets learned classes earlier as more incremental learning steps occur. On the other hand, PIFS(\(\mathcal{L}_{2}\))+GAPS is able to maintain good performance on learned classes as it adapts to new classes.

Input: Table 9: Performance of PIFS(L2) and PIFS(L2)+GAPS on the ADE100-5 disjoint 5-shot setting. Mean and standard deviation over 5 runs are reported.

Input: Table 1: Methods augmented with our proposed GAPS consistently outperform their un-augmented counterparts across different few-shot settings on COCO-20i and PASCAL-5i. Methods noted with* use fully-annotated images, others use same sets of images with novel-class-only partial annotation. 1: our re-implementation using L2 regularization. Highest results are colored red and the second highest results are colored blue. HM stands for harmonic mean. (Best view in color).

Input: Table 11: Performance different pretrained models for contextual guidance under the COCO-20-1 5-shot setting. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 10: Comparisons of GAPS and various exemplar-only methods with PIFS(L2). Mean and 95% confidence intervals over 10 runs are reported.

Input: Figure 3: Long-term continual learning performance of PIFS(L2) and PIFS(L2)+GAPS on the COCO-20-1 split under 5-shot setting. Best view in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After reviewing the evidence, this claim is moderately overstated. While the evidence shows GAPS does improve performance across different incremental learning algorithms (PIFS, SSUL, Fine-tuning) in few-shot settings, the improvements vary in magnitude and consistency. For example, Table 1 shows SSUL+GAPS achieves modest gains in some metrics while PIFS(L2)+GAPS shows larger improvements. The long-term performance in Figure 3 indicates GAPS helps reduce forgetting but doesn't eliminate it. The claim's use of "consistently boosts" overstates the uniformity of improvements across all methods and scenarios.

<score>3</score>

The evidence supports that GAPS generally improves performance, but the claim could be better stated to acknowledge:
1. Varying degrees of improvement across different methods
2. Some cases with only minor gains
3. Performance still degrades over time even with GAPS
4. Specific conditions where it works best

A more precise claim would specify the varying levels of improvement rather than suggesting uniform consistency across all cases.


Input:
Your role is to assess whether a claim is well-stated or overstated based on the available evidence.

The Input Information will include:
1. Original Claim: The claim under evaluation.
2. Evidence: Research findings, including figures, tables, or other relevant data supporting the claim.

Optional. review comment: Reviewer feedback relevant to the claim's validity.

The claim should be assesed with the following:
Score: Evaluate the claim on a scale of 1 to 5, where:
- 1: Well-stated (accurately reflects the evidence without exaggeration)
- 5: Overstated (exaggerates findings' significance, magnitude, or certainty)

Justification: Provide a detailed explanation that includes:
- Instances of exaggerated wording, insufficient experiments, lack of experimental details, gaps in knowledge, weak grounding in evidence, or lack of acknowledgment of limitations.
- Direct citations from relevant evidence that substantiate your reasoning.
- If review comment is included only take into account the relevant comments towards the claim.

The claim to be assessed is:

3. The proposed GAPS technique consistently boosts the performance of a variety of incremental learning algorithms from simple fine-tuning to sophisticated state-of-the-arts under the few-shot setting.

The review comment to be evaluated is:

Strengths:
1. The paper investigates a novel and understudied scenario that is more realistic than earlier benchmarks.
2. The study gives a comprehensive summary of similar works as well as comparisons and contrasts between them.
3. The study expands the present benchmark with new experiments involving only one labeled instance of the new classes, demonstrating that the GAPS can address the issue of missing annotations.

Weaknesses:
1. The ablation study may be extended to other settings and may provide more insights about the paper choices. In particular, it may start from a baseline that uses exemplars without pasting the new classes on them, to show that pasting is important to improve performance and that the gain is not only given by the regularization effect of using exemplars (as proposed by GDumb [a]). Moreover, the paper should clarify if, while disabling the diversity guidance, the exemplars are sampled uniformly for the old classes or if they are randomly sampled on the full dataset.
2. From the ablation study, context sampling appears to marginally improve the results. Moreover, it is not clear how the performance will change by using a pre-training dataset different from Places-365, such as ImageNet or ADE20K. Furthermore, can it be trained directly on the base dataset of the considered setting or can the method use the segmentation network itself (for example, using an auxiliary classifier on the encoder features)?
3. The paper baselines appear to be very naive. In particular, the paper reports the results for Fine-Tuning, MiB, and PIFS without considering exemplars, introducing an unfair comparison with respect to GAPS. The baselines should be reported both without exemplars (as in the original paper) and with them (indeed, without the proposed copy-and-paste strategy).

[a] Prabhu, Ameya, Philip HS Torr, and Puneet K. Dokania. "Gdumb: A simple approach that questions our progress in continual learning." European conference on computer vision. Springer, Cham, 2020.

The evidence to be evaluated is:_Incremental learning_ studies how to enable models to adapt to new classes while mitigating catastrophic forgetting without accessing the old dataset or full-scale re-training.

However, existing work relies on rich annotations and tends to fail when only a limited amount of data is available. In contrast, our work enables incremental segmentation learning with _few data_ via a guided copy-paste process, which demonstrates promising performance under the few-shot and more impoverished single-instance setting. Furthermore, GAPS is a _model-agnostic data pre-processor_, which is orthogonal to incremental learning techniques such as regularization (Li and Hoiem, 2017).

In our work, we propose a series of techniques to guide the copy-paste synthesizer to address these gaps, whose effectiveness is evident from the significant improvement from NPS.

It is a _generic and model-agnostic_ data synthesis framework that generates fully-labeled scenes from partially-annotated images of novel objects as a preprocessor to the underlying segmentation model. After the standard base learning stage with base dataset \(D_{0}\), we build a memory-replay buffer \(\hat{D}_{0}\) using an _diversity-guided exemplar selection strategy_ (Section 3.2). During the incremental learning stage, fully-labeled samples are synthesized by copying from the masked novel objects in \(D_{1},\dots,D_{j}\) and pasting onto base exemplars from the replay buffer \(\hat{D}_{0}\). The strategy by which we choose base exemplars and novel segments is _context-guided_ (Section 3.3) and _class-frequency-guided_ (Section 3.4). In principle, GAPS is model-agnostic, which means that it can work with many incremental segmentation models as a diverse data source to improve their performance.

We found experimentally that when a diverse data source is used (i.e., our proposed GAPS), \(\mathcal{L}_{2}\) regularization works better. To be more precise, we construct a penalization term \(\mathcal{L}_{\mathrm{REG}}\) to regularize the output before the classifier. For incremental learning task \(D_{j}\) with image-mask pairs \((x,y)\), we have \[\mathcal{L}_{\mathrm{REG}}=||f_{j}(x)-f_{j-1}(x)||_{2}. \tag{1}\] The final training loss is given by \[\mathcal{L}(x,y)=\mathcal{L}_{\mathrm{CE}}(\phi_{j}(x),y)+\lambda\mathcal{L}_{ \mathrm{REG}}, \tag{2}\] where \(\mathcal{L}_{\mathrm{CE}}\) is either the standard cross-entropy loss or the modified cross-entropy loss from Cermelli et al. (2020). \(\lambda\) is a hyper-parameter used to weight the regularization loss.

For methods with memory-replaying (e.g., SSUL (Cha et al., 2021)), GAPS can work directly on top of their constructed buffers with minimal modification. For other methods such as PIFS (Cermelli et al., 2021), we propose a diversity-guided exemplar selection process that builds a small yet diverse memory-replay buffer \(\hat{D_{0}}\) from \(\mathcal{D}_{0}\) to mitigate catastrophic forgetting.

Inspired by Bang et al. (2021), we select samples distributed uniformly along a spectrum from easy to hard for diversity. Here, we present an algorithm (Algorithm 1) to construct \(\hat{D_{0}}\) by exploiting the Masked Average Pooling (Map) function from Cermelli et al. (2021). Intuitively, we approximate the difficulty of every sample by their similarity between the estimated prototype with learned prototypes. Estimated prototypes that are close to the learned prototype are considered easy samples and vice versa. After building a list of base samples sorted by difficulties, we select samples from equally-spaced intervals to ensure samples of all difficulties are selected for diversity.

We hypothesize that synthesizing novel objects onto contextually consistent base images would result in an improved learning process.

GAPS uses a scene embedding network \(h:\mathcal{X}\rightarrow\mathbb{R}^{m}\) to estimate pairwise contextual similarity between any two given images. The embedding \(h\) maps an input image to a scene embedding vector in a metric space where pairwise similarity comparison between two embeddings are possible. In our system, we use an off-the-shelf VGGNet (Simonyan and Zisserman, 2015) and replace its last fully connected layer with a cosine-similarity-based classifier. The network \(h\) is trained using the Places365 (Zhou et al., 2017) dataset, which contains \(365\) scene categories and roughly 1.6 million images in its training split. During training, the scene prediction is done by comparing the predicted embedding with existing prototypes.

We follow literature in few-shot segmentation and few-shot incremental segmentation (Shaban et al., 2017; Nguyen and Todorovic, 2019; Tian et al., 2020; Cermelli et al., 2021) and evaluate our model on the PASCAL-5\({}^{\text{i}}\) dataset (Shaban et al., 2017) and the COCO-20\({}^{\text{i}}\) dataset (Nguyen and Todorovic, 2019). PASCAL-5\({}^{\text{i}}\) is artificially built from the PASCAL VOC 2012 Semantic Segmentation dataset (Eveningham et al., 2010) with additional annotations from the SBD (Hariharan et al., 2011) dataset. The original VOC segmentation dataset provides segmentation annotations for 20 object categories.

In Table 1, we evaluate various incremental segmentation methods on the PASCAL-5\({}^{\text{i}}\) dataset and the COCO-20\({}^{\text{i}}\) dataset, and combine them with GAPS where appropriate.

**GAPS consistently increases performance under few-shot settings.** Methods combined with our proposed data source, GAPS, consistently outperform their un-augmented counterpart on both the base and novel categories performance. It is worth noting that GAPS substantially boosts the performance of methods that originally require fully-annotated training images (i.e., fine-tuning and PIFS), despite using only partially-annotated images now. Even for methods that do not carry out end-to-end training and update only the classifier (i.e., SSUL), GAPS still steadily increases performance on novel categories.

**Context-guided sampling steadily improves performance on novel classes**, which is consistent with findings in previous work (Dvornik et al., 2018) that background context is an important factor to consider in copy-paste synthesis. **Frequency-guided probabilistic synthesis boosts results on novel classes**.

On the contrary, \(\text{PIFS}(\mathcal{L}_{2})\)+GAPS shows only a minor performance decrease with single instances.

In this paper, we demonstrate how judicious use of copy-paste dramatically boosts the performance of incremental segmentation methods under the few-shot setting and enables learning with partially-annotated images. Our proposed GAPS technique selects representative exemplars in the memory-replay buffer and addresses the problems of class imbalance and contextual mismatch in synthesis.

We experimented with all of these variants and found that the diversity-aware construction yields the best result in all metrics. Compared to randomly selected samples or easy samples, using hard samples result in better performance on novel categories. On the other hand, we found that using a mixture of easiest samples and hardest sample is also a good strategy, which achieves superior performance in base categories compared to other baselines, and comparable performance to our diversity-aware sampling. It is also worth noting that both the 'GreedyClass' method and the 'BalancedRegion' method achieve almost comparable results to our methods, but still slightly underperform our diversity guidance.

Beyond confirming conclusion from previous work, such improvement alongside stronger augmentation demonstrates the effectiveness of augmentation technique even when there is few data presented, highlighting a possibility to further improve the performance by using even stronger augmentation technique. We perform an additional experiment on COCO-20-1 where a long few-shot incremental learning sequence (60 base; 20 novel classes with 20 incremental steps; 5-shot) is used.

PIFS(\(\mathcal{L}_{2}\)), augmented with GAPS, demonstrates more robustness to forgetting with many incremental learning steps than standard PIFS. To validate our approach on the ADE20k dataset (Zhou et al., 2017, 2019), we ran a suite of experiments where we follow the ADE100-5 (11 steps) setup discussed in (Cha et al., 2021) and modify it as 5-shot learning. We tested PIFS(\(\mathcal{L}_{2}\)) and PIFS(\(\mathcal{L}_{2}\))+GAPS on this setting. The results are given in Table 9. We observe a steady increase in both the base IoU and novel IoU in this challenging task setting on methods augmented with GAPS. This further validates the effectiveness of our approach.

Our PIFS(\(\mathcal{L}_{2}\))+GAPS significantly outperforms both baselines.

As can be seen from Fig. 4, even with as few as 100 fully-annotated base images, GAPS still consistently boosts the performance of the underlying PIFS(\(\mathcal{L}_{2}\)) method.

Fig. 5 shows qualitative results of PIFS(\(\mathcal{L}_{2}\)), PIFS(\(\mathcal{L}_{2}\))+NPS (naive copy-paste), and PIFS(\(\mathcal{L}_{2}\))+GAPS on the PASCAL-5-1 split dataset under the 1-shot setting using images on the left column. We can see that PIFS and PIFS+NPS gives reasonably good predictions on classes that are adapted recently (e.g., the cow class in the last row). However, they tend to exhibit catastrophic forgetting for classes learned earlier (e.g., the bus example in PIFS). On the other hand, the model trained using GAPS not only yields good result on recently learned classes, but also maintains promising performance on classes adapted earlier.

Fig. 7 and Fig. 8 demonstrate how PIFS(\(\mathcal{L}_{2}\))+GAPS compare to the baseline PIFS(\(\mathcal{L}_{2}\)) across multiple step.

These visualizations demonstrate how PIFS(\(\mathcal{L}_{2}\)) forgets learned classes earlier as more incremental learning steps occur. On the other hand, PIFS(\(\mathcal{L}_{2}\))+GAPS is able to maintain good performance on learned classes as it adapts to new classes.

Input: Table 9: Performance of PIFS(L2) and PIFS(L2)+GAPS on the ADE100-5 disjoint 5-shot setting. Mean and standard deviation over 5 runs are reported.

Input: Table 1: Methods augmented with our proposed GAPS consistently outperform their un-augmented counterparts across different few-shot settings on COCO-20i and PASCAL-5i. Methods noted with* use fully-annotated images, others use same sets of images with novel-class-only partial annotation. 1: our re-implementation using L2 regularization. Highest results are colored red and the second highest results are colored blue. HM stands for harmonic mean. (Best view in color).

Input: Table 11: Performance different pretrained models for contextual guidance under the COCO-20-1 5-shot setting. Mean and 95% confidence intervals over 10 runs are reported.

Input: Table 10: Comparisons of GAPS and various exemplar-only methods with PIFS(L2). Mean and 95% confidence intervals over 10 runs are reported.

Input: Figure 3: Long-term continual learning performance of PIFS(L2) and PIFS(L2)+GAPS on the COCO-20-1 split under 5-shot setting. Best view in color.

Input:
You should:
1. Carefully review the claim and the text and image evidence. Summarize how the evidence influenced your evaluation of the claim's validity and strength and briefly explain whether the claim is wellstated or overstated on a scale of 1 to 5 (up to 100 words).
2. Provide the final score (choose from 1 to 5), in the format: <score>{score}</score>.

Output:
After careful review, the claim "GAPS consistently boosts performance of incremental learning algorithms" appears moderately overstated based on the evidence. While Table 1 shows GAPS generally improves performance across methods, there are some inconsistencies and limitations:

1. The improvements vary significantly in magnitude
2. For some metrics (e.g., Base accuracy in COCO-20i), GAPS occasionally shows slight decreases
3. The reviewer points out that baseline comparisons may be unfair without considering exemplars
4. Limited ablation studies and dependence on pre-training datasets (per reviewer comments) suggest the consistency claim needs more validation
5. The term "consistently" implies universal improvement, but the evidence shows more nuanced results

<score>3</score>