Evaluation Study on SAM 2 for Class-agnostic Instance-level Segmentation
Tiantian Zhang Zhangjun Zhou Jialun Pei

top=0.7in, bottom=0.7in, left=0.6in, right=0.6in



i.e.
e.g.
etc
et al.








Pei et al.





§ ABSTRACT 
Segment Anything Model (SAM) has demonstrated powerful zero-shot segmentation performance in natural scenes. The recently released Segment Anything Model 2 (SAM2) has further heightened researchers' expectations towards image segmentation capabilities. To evaluate the performance of SAM2 on class-agnostic instance-level segmentation tasks, we adopt different prompt strategies for SAM2 to cope with instance-level tasks for three relevant scenarios: Salient Instance Segmentation (SIS), Camouflaged Instance Segmentation (CIS), and Shadow Instance Detection (SID). In addition, to further explore the effectiveness of SAM2 in segmenting granular object structures, we also conduct detailed tests on the high-resolution Dichotomous Image Segmentation (DIS) benchmark to assess the fine-grained segmentation capability. Qualitative and quantitative experimental results indicate that the performance of SAM2 varies significantly across different scenarios. 

Besides, SAM2 is not particularly sensitive to segmenting high-resolution fine details.

We hope this technique report can drive the emergence of SAM2-based adapters, aiming to enhance the performance ceiling of large vision models on class-agnostic instance segmentation tasks. 

Project link: <https://github.com/PJLallen/InstanceSAM2Eval>.



Foundation Model, Large Vision Model, SAM 2, Instance-level Segmentation, Dichotomous Image Segmentation.


Evaluation Study on SAM 2 for Class-agnostic
Instance-level Segmentation
    Tiantian Zhang, Zhangjun Zhou,
        Jialun Pei^†


Jialun Pei is with the Department of Computer Science and Engineering, The Chinese University of Hong Kong, HKSAR, China.
Tiantian Zhang is with the Department of Computer Science and Engineering, The Chinese University of Hong Kong, HKSAR, China.
Zhangjun Zhou is with the School of Software Engineering, Huazhong University of Science and Technology, Wuhan, China.
^† Corresponding author: Jialun Pei (Email: peijialun@gmail.com).

    September 9, 2024
===========================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================










§ INTRODUCTION



The advent of large foundation models, including ChatGPT, GPT-4, and LLaMA, has revolutionized the artificial intelligence (AI) landscape. Powered by extensive datasets, these models excel in multi-modal processing, , language, image, video, and audio, showcasing substantial progress in AI capabilities. Building on these developments, the Segment Anything Model (SAM) <cit.> stands out breakthrough in scene segmentation with large vision models.
The generality and Adaptability of SAM highlight its potential for understanding complex scenarios and targets, further expanding the frontiers of image segmentation tasks.


SAM allows users to input custom prompts, such as points or bounding boxes, resulting in highly accurate segmentation masks. This adaptability enables SAM to perform a wide range of image segmentation tasks. More recently, the release of SAM2 <cit.> further overcomes the limitation that SAM does not handle video content well. In the field of image segmentation, SAM2 has shown improvement in segmentation accuracy and inference efficiency [<https://sam2.metademolab.com/demo>]. A variety of evaluations have recently emerged to examine the segmentation performance of SAM2 in different scenarios <cit.>. For instance, Lian  <cit.> assessed its instance segmentation performance in underwater environments, while Yan  <cit.> explored its effectiveness in endoscopic and microscopic images. Additionally, Ma  <cit.> conducted a comprehensive benchmark of SAM2 across 11 medical image modalities and videos, highlighting its strengths and weaknesses compared to SAM and MedSAM. Furthermore, Tang  <cit.> compared SAM2 and SAM on the camouflage object detection benchmark. This research found that SAM2 significantly degrades the performance of SAM2 compared to SAM for detecting camouflaged objects when no prompts are provided, and significantly improves its performance over SAM when segmentation prompts are available. These interesting findings raise curiosity about SAM2's performance in class-agnostic instance-level segmentation tasks.


In this paper, we evaluate the performance of SAM2 in class-agnostic instance-level segmentation tasks, focusing on three distinct scenarios: Salient Instance Segmentation (SIS) <cit.>, Camouflaged Instance Segmentation (CIS) <cit.>, and Shadow Instance Detection (SID) <cit.>. Moreover, We also thoroughly evaluate SAM2 on the high-resolution dichotomous image segmentation (DIS) benchmark <cit.> to analyze its ability to segment granular target structures. We compare SAM2 with SAM and well-known specific models on multiple benchmarks. Based on extensive experimental results, we summarize the following conclusions:

    
  * SAM2 outperforms task-specific methods for CIS and SIS when using bounding boxes as prompt inputs. However, the performance of SAM2 drops remarkably without box prompts, especially for camouflaged instances.
    
  * SAM2 performs poorly on the DIS task, whether or not it uses bounding boxes as prompts. It indicates that SAM2 is not well suited for fine-grained segmentation of complex object structures.
    
  * For the SID task, while SAM2 performs well in segmenting instances, it struggles with shadow matching.
    
  * SAM2 with fewer parameters achieves superior results compared to SAM across four tasks when using bounding boxes as prompts. In contrast, SAM2 without box prompts performs inferior to SAM for SIS, CIS, and SID.
































§ EXPERIMENTS


This section provides the guidelines and details of our basic and extensive experiments, , datasets, the evaluation protocol, implementation settings, and the quantitative and qualitative results of SAM2 on four tasks.



 §.§ Datasets


In line with <cit.>, we utilize the ILSO <cit.>, SOCK <cit.>, SIS10K <cit.>, and SIP <cit.> datasets for the SIS task. For the CIS task, we employ COD10K <cit.> and NC4K <cit.> to evaluate the performance. For the SID task, we use the SOBA-challenge and SOBA-test datasets <cit.>. For DIS, We conduct experiments on DIS5K <cit.>, including DIS-VD and DIS-TE. DIS-TE is further divided
into four subsets , DIS-TE1, DIS-TE2, DIS-TE3, and DIS-TE4, representing four levels of testing difficulty.
The number of test samples for datasets in each task is summarised below:

    
  * SIS: ILSO: 300; SOC: 600; SIS10K: 1,170; SIP: 929.
    
  * CIS: COD10K: 2,026; NC4K: 4,121.
    
  * SID: SOBA-challenge: 100; SOBA-test: 160.
    
  * DIS: DIS-VD: 470; DIS-TE1: 500; DIS-TE2: 500; DIS-TE3: 500; DIS-TE4: 500; Overall DIS-TE: 2,000.








 §.§ Evaluation Protocol


To evaluate camouflaged instance segmentation, we employ COCO-style evaluation metrics, including AP_50, AP_75, and AP values. For salient instance segmentation, we adopt the AP_70 metric, which is commonly used in related literatures <cit.>, instead of the AP_75 metric. In shadow instance segmentation, while task-specific methods employ the SOAP metric to assess object and shadow matching, SAM2 do not involve this matching mechanism. In this regard, we focus only on performance with the instance AP metric.

To assess high-accuracy DIS, we employ six evaluation metrics to evaluate SAM2, SAM, and DIS-specific models, including maximal F-measure (F_β^max↑) <cit.>, weighted F-measure (F_β^ω↑) <cit.>, Mean Absolute Error (MAE, M↓) <cit.>, Structural measure (S-measure, S_α↑) <cit.>, mean Enhanced alignment measure (E-measure, E_ϕ^m↑) <cit.>, and Human Correction Efforts (HCE_γ↓) <cit.>. 































 §.§ Implementation Details


To ensure a fair comparison, we use the original official code of SAM2 and SAM to test on different datasets. Both SAM2 and SAM are evaluated under two settings: automatic mode and ground truth bounding box (GT-Bbox) mode. In automatic mode, we use the default setting of a 32×32 point prompt for both. In GT-Bbox mode, the ground truth bounding box serves as the box prompt input. All parameters remain at their default settings, and the input images are resized to 1024×1024. Furthermore, we use different backbones for SAM and SAM2. For SAM, we use ViT-Base, ViT-Large, and ViT-Huge. For SAM2, we use Hiera-Tiny, Hiera-Base+, and Hiera-Large.
All experiments are implemented with a single Tesla A40 GPU.



 §.§ Results










  §.§.§ Salient Instance Segmentation


Quantitative Results.
The quantitative results of salient instance segmentation are presented in table:SIS. On ILSO and SIS10K datasets, SAM2 models generally outperform in the GT-Bbox setting. For example, SAM2-L achieves an AP score of 82.2 on ILSO, slightly higher than 79.2 of SAM-H. 

However, in the automatic setting, SAM2-L scores lower, with an AP of 49.1 versus SAM-H's 72.2. 

A similar trend is observed on SIS10K, with SAM2-L reaching 45.2 compared to SAM-H's 68.4.
On SOC and SIP datasets, SAM2 models also excel in the GT-Bbox setting, with SAM2-L scoring 83.1 on SOC and 93.4 on SIP. 

Compared to specific methods like SCNet, S4Net, RDPNet, and OQTR, SAM2 models often achieve higher AP scores in the GT-Bbox setting. 

This indicates that SAM2 models show significant improvements with ground truth boxes, outperforming traditional methods in certain scenarios. 
However, it should be noted that SAM2's bounding box mode relies on inputting the ground truth instance locations, which might introduce a slight unfair compared to other frameworks.

Qualitative Results.
As shown in fig:SIS, in the qualitative analysis of salient instance segmentation, both SAM-Auto and SAM2-Auto perform global segmentation because they do not specify particular objects to segment. 
The segmentation quality of SAM is slightly better, likely attributed to SAM using a larger version (huge) compared to large version of SAM2. 
This difference in model size may account for the finer details in segmentation masks of SAM, which, though they still appear somewhat fragmented. 
Nonetheless, when bounding box prompts are adopted, both SAM-bbox and SAM2-bbox achieve significantly improved and precise segmentation, highlighting the value of guided segmentation.






  §.§.§ Camouflaged Instance Segmentation


Quantitative Results.
table:CIS shows the segmentation performance of SAM2 on camouflaged instances, which are more difficult to segment than salient instances in table:SIS. 

In automatic mode, SAM2's performance is comparable to the unsupervised methods of task-specific algorithms and falls short of SAM, likely due to differences in parameter counts.
However, with box prompts, the performance of SAM2 improves dramatically. 
Specifically, on COD10K test set, the AP jumps from 10.6 to 68.8 with a large backbone, surpassing all other models. 
This suggests that the primary challenge of SAM2 in CIS is locating objects, but once the position is identified, it can produce precise segmentation.



Qualitative Results.
For qualitative analysis of the CIS task, as shown in fig:CIS, SAM-Auto can partially segment certain lightly concealed targets such as fish and giraffes, whereas SAM2-Auto has difficulty detecting camouflages. 
However, when provided with bounding box prompts, both SAM and SAM2 effectively segment camouflaged instances. Notably, SAM2 excels in capturing fine details, showcasing its strength in producing intricate features.






















  §.§.§ Shadow Instance Segmentation


Quantitative Results.
It is important to note that in shadow instance detection tasks, the matching degree between the shadow and the object needs to be measured. However, SAM2 lacks this functionality, so our comparison does not involve measuring this aspect. 

In our experiments, we treat instances and shadows as separate entities, rather than pairs of instances and corresponding shadows.

Based on the table:shadow, SAM2 models perform exceptionally well in the GT-Bbox setting, with SAM2-T achieving AP scores of 51.9 on the SOBA-challenge and 58.9 on the SOBA-test, surpassing all SAM models and task-specific approaches. 
However, an interesting phenomenon is observed: using different backbones with SAM2 does not lead to significant performance differences. 
In fact, SAM2 with larger backbones has the potential to decreases segmentation performance, and the same phenomenon exists for SAM models. 

Switching to automatic mode results in a significant drop in performance for both SAM2 and SAM, but the changes in backbone parameter size do not greatly impact segmentation results. 

Therefore, to improve the performance of SAM2 on the SID task, it is not appropriate to simply increase the depth and parameter size of the model.

Qualitative Results.
As shown in fig:shadow, both SAM and SAM2 are effective in segmenting instances in automatic mode, but face challenges in accurately identifying shadows. 

This may be caused by the lack of instance-shadow IoU matching operations in SAM2.
When provided with box prompts, both models show significant improvement in shadow segmentation, with SAM2 having a slight edge in capturing shadow details. 
Despite these improvements, the overall quality of shadow segmentation by SAM2 still falls short compared to corresponding instances.




  §.§.§ Dichotomous Image Segmentation

Dichotomous image segmentation focuses on identifying class-agnostic foreground objects in natural scenes. In automatic prediction mode, SAM generates multiple binary masks for each sample. To select the most suitable foreground mask, we use a maximum Intersection over Union (IoU) strategy, choosing the mask with the highest IoU score.
Quantitative Results.
table:DIS1 and table:DIS2 present the qualitative comparison results of SAM and SAM2 against task-specific methods. In the automatic setting, SAM2 models, particularly SAM2-T, show marked improvement over SAM models. 
Concretely, SAM2-T achieves an F_β^max of 0.306 on DIS-VD, compared to 0.215 for SAM-B. SAM2-B+ and SAM2-L further enhance performance, with SAM2-B+ reaching 0.428 in F_β^max on DIS-VD, surpassing all SAM variants. 

These improvements indicate better segmentation quality and alignment, as evidenced by higher S_α and E_m values. In the GT-Bbox setting, SAM2 models demonstrate significant gains across all metrics, except for HCE. For instance, SAM2-B+ achieves F_β^max of 0.765 on DIS-VD, surpassing SAM-L's 0.739, although the HCE metric increases by 100 points. 

Generally, the HCE metric is more sensitive to the structural refinement of the segmentation map compared to traditional accuracy metrics like weighted F-measure, mean absolute error, and mean enhanced alignment measure. 
It indicates that SAM2 enhances the overall perception of the target, but it still struggles to identify the dominant area while accurately segmenting detailed object structures.

Overall, SAM2 models offer substantial improvements over SAM across the vast majority of metrics in all datasets, nearly approaching the performance of the fully supervised method IS-Net. 
However, the HCE scores illustrate that both SAM and SAM2 have limited potential in representing detailed structures.






Qualitative Results. 
Further analysis of the qualitative results, particularly in fig:DIS, reveals that both SAM and SAM2 encounter difficulties in identifying foreground objects, whether in automatic mode or with box prompts. 
Notably, when receiving a bounding box prompt, SAM can roughly outline the main body of objects, while SAM2 enhances this capability by improving segmentation completeness. 
For example, SAM can segment the body of a ship with the aid of a bounding box prompt (see the seventh column of fig:DIS), but it tends to miss smaller details such as the mast and thin lines. 
Thus, although SAM2 improves the accuracy for locating foreground objects in natural scenes, it still falls short in accurately capturing the full extent of dominant areas of targets and rendering intricate structural details.
























§ DISCUSSION


We conducted extensive quantitative and qualitative evaluations of SAM2 on various class-agnostic instance-level segmentation benchmarks.
In CIS, SIS, and SID tasks, SAM2 outperforms task-specific methods in the GT-Bbox mode. 
For the relatively straightforward SIS task, SAM2 attains an impressive AP score of 93.4 on SIP test set. 
For the more challenging CIS task, SAM2 reaches a significant AP score of 73.5 on the NC4K test set, far exceeding the performance of specific methods. 
In the SID task, SAM2 segments instances effectively but struggle with shadow matching. 

As observed in the qualitative comparison, SAM2 without the GT-Bbox is almost impossible to segment out shadows.

In the DIS task, SAM2 performs poorly, even in settings with box prompts it is not able to perform granular segmentation for complex structured objects.
Overall, SAM2 with GT-Bbox achieves better results for both salient and camouflaged objects, especially for salient instances. However, its ability to handle very delicate objects requires improvement.

In comparison with SAM, we found that SAM2 falls short of the performance of SAM in automatic mode across the CIS, SIS, and SID tasks. 


Interestingly, in GT-Bbox mode, SAM2 significantly outperforms SAM.


Besides, we observe that SAM2 models with larger backbones does not always enhance the performance, and even degrades it, which is especially noticeable in the automatic mode.


These findings provide valuable insights for future applications of SAM2 in instance-level segmentation tasks.




§ CONCLUSION


In this study, we evaluate the zero-shot performance of SAM2 in class-agnostic instance-level segmentation tasks across four scenarios: Salient Instance Segmentation (SIS), Camouflaged Instance Segmentation (CIS), Shadow Instance Detection (SID), and Dichotomous Image Segmentation (DIS). 
In the automatic setting, SAM2 underperforms compared to SAM and task-specific methods in the SIS, CIS, and SID tasks. 
When provided with bounding box prompts, especially in the DIS task, SAM2 demonstrates its capability to generate more refined masks. 
The experimental results demonstrate that SAM2 excels in class-agnostic instance-level segmentation when guided by prompts, as well as its potential capabilities in diverse scenarios. 
In future work, we aim to fine-tune SAM2 and develop adapters to boost its performance across various instance-level segmentation tasks.














IEEEtran





