FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentation
Xi Chen Haosen Yang Sheng Jin Xiatian Zhu Hongxun Yao








FrozenSeg: Harmonizing Frozen Foundation Models for Open-Vocabulary Segmentation
    
  Xi Chen 1 Haosen Yang 2 Sheng Jin 3 Xiatian Zhu 2 Hongxun Yao 1  

  1Harbin Institute of Technology  2 University of Surrey  3 Nanyang Technological University 


    
==========================================================================================================================================================================



§ ABSTRACT 
Open-vocabulary segmentation poses significant challenges, as it requires segmenting and recognizing objects across an open set of categories in unconstrained environments. Building on the success of powerful vision-language (ViL) foundation models, such as CLIP, recent efforts sought to harness their zero-short capabilities to recognize unseen categories. 
Despite notable performance improvements, these models still encounter the critical issue of generating precise mask proposals for unseen categories and scenarios,
resulting in inferior segmentation performance eventually.
To address this challenge, we introduce a novel approach, , designed to integrate spatial knowledge from a localization foundation model (SAM) and semantic knowledge extracted from a ViL model (CLIP), in a synergistic framework. 
Taking the ViL model's visual encoder as the feature backbone, we inject the space-aware feature into the learnable queries and CLIP features within the transformer decoder.
In addition, we devise a mask proposal ensemble strategy for further improving the recall rate and mask quality.
To fully exploit pre-trained knowledge while minimizing training overhead, we freeze both foundation models, focusing optimization efforts solely on 
a lightweight transformer decoder for mask proposal generation – the performance bottleneck.
Extensive experiments demonstrate that  advances state-of-the-art
results across various segmentation benchmarks, trained exclusively on COCO panoptic data, and tested in a zero-shot manner. Code is available at <https://github.com/chenxi52/FrozenSeg>.



§ INTRODUCTION

Image segmentation is a fundamental task in computer vision, enabling a wide range of applications
such as object recognition <cit.>, scene understanding <cit.>, and image manipulation <cit.>. However, traditional techniques are often tailored to specific datasets and segmentation tasks, resulting in a significant gap compared to human visual intelligence, which can perceive diverse visual concepts in the open world. To bridge this disparity, the concept of open-vocabulary segmentation has emerged. In this task, the segmenter is trained to recognize and segment instances and scene elements from any category, mirroring the broad capabilities of human perception.



Parallel to these efforts, significant advancements have been made in the field of purpose-generic image-level large-dataset pretrained Vision Language (ViL) representation learning, exemplified by foundational models such as CLIP <cit.> and ALIGN <cit.>. 
These models are pivotal in understanding open scenes, as they leverage rich, descriptive language cues to enhance models’ ability to generalize across a wide array of unseen categories. 
However, the absence of sufficient pixel-level annotations often leads to challenges in dense-level image-text alignment.
Recent studies have utilized these pre-trained ViL models for region classification <cit.>,
necessitating further training of a segmentation model <cit.> for precise pixel-level alignment, often resulting in inefficiencies and reduced effectiveness. 
Alternatively, mask proposals generated with the CLIP visual encoder <cit.> are still suboptimal due to their limited fine-grained pixel-level understanding, which becomes a performance bottleneck as the mask proposal generation may overfit to the training classes, undermining the model’s generalizability to unseen classes. As shown in Fig. <ref>, existing methods such as FC-CLIP <cit.> struggle to generalize to unseen categories under different IoU thresholds, significantly limiting their practical utility.



In this paper, to overcome the above limitation, we introduce , a system that harnesses the capabilities of localization foundation model SAM to synergistically and efficiently enhance the coarse semantic features extracted from CLIP by incorporating generalized fine space-aware features.
 has three key modules: (1) , which aggregate local space-aware features from SAM to serve as the spatial query for the corresponding mask region, enhancing the learnability of queries in a transformer decoder.
(2) ,
designed to enrich each pixel's CLIP feature by incorporating comprehensive global spatial information from SAM.
(3) , designed to further boost the quality of mask predictions based on the spatial information injection of SAM during training by ensembling with zero-shot mask proposals from SAM.
Building upon these modules, as shown in Fig. <ref>, the recall metrics of unseen categories on the challenging CityScapes dataset <cit.> showed significant improvement, consequently boosting PQ from 44.3 to 45.8. This upward trend is further supported by the results in PC-459 <cit.>, with mIoU increase from 17.3 to 19.7, validating the observed enhancement. 

Our contributions can be summarized as follows:
(1) Addressing an acknowledged limitation in mask proposal quality, we introduce , a framework that incorporates foundational models to tackle the open-vocabulary segmentation task effectively.
(2) We propose three critical components: the , the , and the . These components are designed to enhance the integration of SAM features into the transformer decoder, facilitating generalized mask generation.
(3) Extensive experiments on various segmentation tasks demonstrate the superiority of our  in generating mask proposals and achieving enhanced final performance, surpassing previous approaches.



§ RELATED WORKS



 §.§ Open-vocabulary Segmentation

Open-vocabulary segmentation aims to segment objects even without seeing those classes during training. Previous approaches <cit.> typically employ a two-stage process, where an additional segmentation model generates class-agnostic mask proposals, which are then interacted with CLIP features. 
In the context of open-vocabulary panoptic segmentation, which necessitates instance segmentation and interaction with multiple mask proposals <cit.>, methods such as OPSNet <cit.> combine query embeddings with the last-layer CLIP embeddings and applies an IoU branch to filter out less informative proposals. MaskCLIP <cit.> integrates learnable mask tokens with CLIP embeddings and class-agnostic masks.
Despite these advancements, challenges remain in effectively aligning segmenters with CLIP.

Alternatively, one-stage open-vocabulary segmentation faces challenges in extending vision-language models without dedicated segmentation models and addressing overfitting in an end-to-end format. CLIP’s pre-training on image-text pairs necessitates reconciling the region-level biases of the vision-language model. Research such as FC-CLIP and F-VLM <cit.> indicates that convolutional CLIP models generally exhibit superior generalization capabilities compared to ViT-based <cit.> counterparts, primarily due to their capability to handle larger input resolutions effectively. This finding highlights a promising direction for adapting CLIP for improved performance in segmentation tasks.
Despite these advancements, a fundamental issue persists: accurately generating mask proposals for unseen categories and scenarios. This challenge is compounded by the methods' dependence on a static Vision and ViL model, which is not equipped to discern intricate pixel-level details, thereby limiting its effectiveness in mask proposal generation.



 §.§ Large-scale Foundation Models

Recent advances in large-scale foundation models, pre-trained on extensive datasets, have showcased exceptional zero-shot capabilities. Multi-modal foundation models, such as CLIP and ALIGN, exhibit strong generalization across various downstream tasks. Although these models are trained on image-level data with inherent noise, they can be effectively fine-tuned for various applications. Common strategies include prompt learning <cit.> and the use of adapters <cit.>, with CLIP often remaining frozen to preserve its broad generalization.

In the realm of the segmentation foundation models,  significant progress is exemplified by the SAM model <cit.>, which leverages the extensive SA-1B dataset to achieve notable zero-shot generalization. SAM can adapt to new datasets without additional training by using input prompts. 
Subsequent models, such as HQ-SAM <cit.> and GenSAM <cit.> have built upon this foundation by optimizing output tokens and integrating textual semantic reasoning, respectively. Despite these advancements, these methods often rely on manually crafted prompts, which constrains their wider applicability and scalability.

Recent research <cit.> has explored the use of bounding boxes generated through open-vocabulary detection methods as prompts, combining SAM and CLIP to exploit their complementary strengths in open-vocabulary segmentation. These approaches aim to combine SAM’s zero-shot generalization capabilities with CLIP’s robust feature representations. Despite these efforts, significant challenges remain in achieving fully automatic open-vocabulary segmentation and transitioning from instance segmentation to semantic and panoptic segmentation.









§ METHOD

Our objective is to achieve efficient open-vocabulary segmentation using frozen foundation models. In this section, we start by defining the problem. Subsequently, we present our method, , which integrates frozen foundation models for open-vocabulary segmentation through two key components: the  and the , as illustrated in Fig. <ref>. Finally, we detail our inference strategy, the .



 §.§ Problem Definition

Open-vocabulary segmentation involves training with ground-truth masks corresponding to a predefined set of class labels, 𝐂_train. 
During testing, the model encounters a different set of class labels, 𝐂_test, which includes novel classes not seen during training. This process requires segmenting images in an open-world context, where the model must categorize pixels into semantic classes for semantic segmentation, identify individual instances for instance segmentation, or combine both for panoptic segmentation. The notation 𝐂 represents either 𝐂_train or 𝐂_test, depending on whether the phase is training or testing.





 §.§ Our Approach 




  
Overall Architecture
Following the approach of <cit.>, we adopt Mask2Former <cit.> as our framework. A set of N learnable queries that represent all things and stuff in the input image is processed through the transformer decoder to get mask predictions m. 
To adapt the framework for open vocabulary segmentation, we replace the original classification layer with the text embeddings derived from the CLIP text encoder, resulting in class prediction p_d, where d denotes the mask detector.
Post-training, the embeddings for each mask and its corresponding category text are projected into a shared embedding space, facilitating effective categorization within the open-vocabulary framework.
In line with <cit.>, we adopt the convolution-based CLIP visual encoder as our image feature extractor, leveraging its pre-trained, frozen weights to obtain high-resolution semantic information.

To address the limitations of CLIP’s coarse features, we introduce two key modules: the  and the . These modules integrate the spatial features from SAM into the mask proposal generation process, as depicted in Fig. <ref>.
Unlike  <cit.>, which incorporates multi-level spatial information into the vision transformer, our injectors focus on infusing spatial information directly into mask queries.
Additionally, we propose the  to further enhance segmentation performance during inference.
We detail our approach below: 



  
Query Injector To improve local spatial understanding, we introduce the Query Injector, which enhances the learnable query with space-aware features derived from SAM. 
The transformer decoder uses masked multi-head attention to bolster cross-attention between the image's foreground region and the learnable queries. 
This mechanism facilitates the integration of both content and spatial information within the query, a concept supported by prior studies such as <cit.>.
However, capturing detailed spatial information remains challenging when the backbone is frozen.

To address this challenge, we devise the Query Injector, which leverages newly generated masks at each decoder layer to pool and transform SAM visual features into a spatial query. The process for generating the spatial query is defined as follows:

    x_l = f(pool(M_l, ℱ_sam))


Here, l represents the layer index in the transformer decoder, f denotes a linear projection function, and pool refers to the mask pooling operation.  ℱ_sam represents the SAM-derived image features. 
This spatial query is specifically designed to concentrate on a region encompassing the mask region. Subsequently, the spatial query is integrated with the learnable query via element-wise addition.



  
Feature Injector
To refine the CLIP features for mask generation on a global scale, we introduce the Feature Injector, which uses the multi-head cross-attention mechanism (MHCA) as detailed in <cit.>. This mechanism is renowned for its effectiveness in amalgamating diverse information. In our approach, we extend MHCA to enhance the coarse semantic features from CLIP.
Specifically, the Feature Injector integrates semantic content from CLIP with pixel-level spatial awareness from SAM, providing a more nuanced understanding at the pixel scale.
The mathematical formulation of this feature integration is as follows:

    ℱ = SoftMax( f_q (ℱ_clip) · f_k( ℱ_sam)/√(D)) · f_v(ℱ_sam)


Here, f_q, f_k, and f_v are linear projection functions in MHCA. ℱ_clip and ℱ_sam represent the features extracted from CLIP and SAM, respectively, while D denotes the dimensionality of the projected features.




  
Inference Strategy

Previous works such as <cit.> have validated the efficacy of mask pooling on CLIP features within class ensemble methodologies to enhance open-vocabulary segmentation capabilities. Building on these techniques, our approach introduces a novel mask ensemble strategy. As illustrated in Fig. <ref>, our OpenSeg Ensemble Module initiates with the class ensemble process:

    p_i(j) =
        
         (p_i,d(j))^(1-α)· (p_i,cl(j))^α,    if j∈𝐂_train
    
        (p_i,d(j))^(1-β)· (p_i,cl(j))^β,    else


Here, p_i(j) denotes the combined probability for class j in proposal i, integrating inputs from both the detector (p_i,d) and CLIP (p_i,cl).
The mask predictions r for N queries are then generated by aggregating the products of these probability-mask pairs:
∑_i=1^N p_i(c)·  m_i[x,y] = r ∈ℝ^C× HW.

Drawing inspiration from the class ensemble, we utilize zero-shot mask predictions from SAM to perform a mask ensemble on r. The SAM masks, denoted as M_sam={m̂_i}_i=1^N, are generated by uniformly sampling points prompts across the image. 
These masks are used to pool CLIP features and derive classification scores P_sam = {p̂_i}_i=1^N by aligning with CLIP text features. A threshold ξ=0.5 is applied to filter these masks based on the maximum probability, resulting in the selected probability-mask pairs {(m̂_i,p̂_i) |argmax_cp̂_i > ξ}_i=1^N'.

In the context of semantic segmentation, the SAM mask predictions, denoted as r̂, are computed similarly as follows: r̂ = ∑_i=1^N'p̂_i(c)·m̂_i[x,y]. The final mask prediction, r^', is obtained by integrating the predictions r and r̂ through a mask ensemble approach:

    r'[x,y](j) =
        
         r[x,y](j), if j∈𝐂_train
    
        (1-ϵ)*r[x,y](j) + ϵ * r̂[x,y](j), else


Subsequently, the final semantic segmentation results are determined by assigning each pixel [x,y] a class based on 
argmax_c∈{1,...,|𝐂|} r'[x,y].

In the context of panoptic segmentation, the efficacy of the results significantly depends on the performance of individual queries. This dependency reduces the effectiveness of integrating class-agnostic mask predictions. Therefore, the final results are determined by assigning each pixel to one of the N predicted probability pairs. This assignment is performed through the following expression: argmax_i:c_i ≠∅p_i(c_i)· m_i[x,y]. In this expression, c_i represents the most likely class label, which is determined by c_i=argmax_c∈{1,...,|𝐂|,∅} p_i(c). Here, ∅ denotes the class of 'no object'.




§ EXPERIMENTS






 §.§ Datasets and Evaluation Protocal

For training, we use the COCO panoptic <cit.> dataset, which includes 133 classes. Our evaluation covers open-vocabulary panoptic, semantic, and instance segmentation tasks in a zero-shot setting spanning several test datasets. In semantic segmentation, we access performance on ADE20K dataset <cit.>, which includes both a subset with 150 classes (A-150) and a full version with 847 classes (A-847). Additionally, we evaluate on PASCAL VOC <cit.>(PAS-21), which has 20 object classes and one background class, and PASCAL-Context <cit.>, an extension of PASCAL VOC with 459 classes (PC-459). For panoptic segmentation, the datasets used are ADE20K <cit.>, Cityscapes <cit.>, Mapillary Vistas <cit.>, and BDD100K <cit.>, alongside the closed-set COCO validation dataset. 
For instance segmentation, we choose to evaluate LVIS v1.0 <cit.>, which features 337 rare categories.
The evaluation metrics include mean intersection-over-union (mIoU) and frequency-weighted IoU (FWIoU) that offer a comprehensive evaluation of overall performance for semantic segmentation, panoptic quality (PQ), average precision (AP), and mIoU for panoptic segmentation, as well as AP for instance segmentation.



 §.§ Baselines
 
We compare with multiple state-of-art approaches as follows: OPSNet <cit.>,
MaskCLIP <cit.>, MasQCLIP <cit.>, ODISE <cit.>, CLIPSelf <cit.>, FC-CLIP <cit.>, Ovseg <cit.>, SAN <cit.> , RegionSpot <cit.> and Open-Vocabulary SAM <cit.>.




 §.§ Implement Details


We use 250 queries for both training and testing, with CLIP serving as the backbone for open-vocabulary text-image alignment. Specifically, we employ the RN50x64 and ConvNext-Large <cit.> versions of CLIP. Additionally, we validate our approach using the ViT-Base <cit.> model from SAM, with the selection rationale detailed in the .
To obtain multi-level semantic features, we apply feature pyramid networks (FPN) after CLIP. SAM processes input images 𝐈∈ℝ^H, W, where H = W = 1024. As demonstrated by PlainViT <cit.>, the deepest feature of ViT contains sufficient information for multi-scale object recognition, and given that SAM is frozen, we do not use FPN for SAM. Instead, we utilize a single convolution layer to project the features to the necessary resolution and then feed them into a single-scale deformable attention transformer <cit.> as the pixel decoder in the SAM branch.
Our transformer decoder comprises L=9 layers. Feature maps with resolutions of 1/8, 1/16, and 1/32 are processed by successive decoder layers in a round-robin fashion. During training, we follow the strategy and losses outlined in FC-CLIP, selecting the model from the final iteration for our primary results. Training is conducted on 4 Tesla A100 GPUs with a batch size of 16.




 §.§ Inference Details

During inference, we adhere to the FC-CLIP by resizing images such that the shortest side is 800 pixels for general datasets and 1024 for the Cityscapes and Mapillary Vistas datasets.
We employ a 32x32 grid of points research to generate masks from the SAM ViT-Huge model. The default parameters are set as follows: α=0.4 and β=0.8 in Eq.(<ref>), and a mask ensemble parameter ϵ=0.2 in Eq. (<ref>).













 §.§ Evaluation on Open-vocabulary Segmentation




  §.§.§ Open-vocabulary Panoptic Segmentation

Tab. <ref> presents a comparison of  with leading methods in zero-shot open-vocabulary panoptic segmentation. Our approach,  with RN50x64, notably surpasses other works and the baseline FC-CLIP, achieving improvements of +1.8 PQ, +0.3 AP, and +2.0 mIoU on ADE20K; +2.6 PQ, +1.6 AP, and +0.9 mIoU on Cityscapes; and +4.8 mIoU on BDD100K.
Additionally, the  configuration with ConvNeXt-L delivers enhanced performance on open-set datasets without compromising results on the closed-set COCO validation dataset. Significant improvements include +0.8 PQ and +1.6 mIoU on ADE20K, +1.5 PQ, +0.5 AP, and +0.8 mIoU on Cityscapes, +0.4 PQ on Mapillary Vistas, and +1.4 PQ and +2.9 mIoU on BDD10K. 
Qualitative results of panoptic segmentation on Cityscapes, depicted in Fig. <ref>, show improvements in segmentation, particularly for small objects, entity recognition, and novel class recognition. Additional details and additional results are available in the .




  §.§.§ Open-vocabulary Semantic Segmentation

Tab. <ref> presents a comparative analysis of  in cross-dataset open-vocabulary semantic segmentation. Using the RN50x64 backbone,  significantly outperforms the baselines. Compared to FC-CLIP,  achieves gains of +3.1 mIoU and +10.1 FWIoU on PC-459, +1.0 mIoU and +8.2 FWIoU on A-847, and +2.0 mIoU and +4.1 FWIoU on A-150. These improvements are also reflected in the ConvNeXt-L configuration. Overall,  sets a new benchmark in performance across the datasets PC-459, A-847, and A-150. It is important to note that PAS-21’s categories fully overlap with those of the training dataset, which suggests that FC-CLIP may overfit to base classes and thus limit its generalization.
For qualitative insights, refer to Fig. <ref>, where  delivers segmentations that are contextually more accurate compared to both the baseline and ground truth annotations, demonstrating its effective handling of complex scenes. Additional results are available in the .




  §.§.§ Open-vocabulary Instance Segmentation

Tab. <ref> presents results for rare categories in the LVIS dataset. We compare  with approaches that integrate SAM with CLIP for open-vocabulary segmentation tasks, specifically RegionSpot and Open-Vocabulary SAM. Both of these methods rely on proposals as location prompts and are trained on datasets beyond the COCO panoptic to align the models. Our method achieves the highest performance, with an improvement of +0.6 AP over FC-CLIP.



 §.§ Ablation Studies

We perform a series of ablation studies on our method. All findings are presented using the ConvNext-L version of CLIP and the ViT-B version of SAM.



  §.§.§ The Effectiveness of Each Component

We perform ablation studies to assess the effectiveness of each component of our method. Tab. <ref> presents the results of these ablations on three challenging out-of-vocabulary datasets, with rows 2-6 highlighting the contribution of each component to overall performance.
Specifically, row 1 illustrates the scenario where only proposals from SAM are utilized. In this setup, SAM masks are used to pool CLIP features, providing basic semantic understanding without explicit semantic guidance. This configuration achieves approximately 6.5 mIoU on PC-459 and A-847, and 25.4 mIoU on ADE20K, demonstrating the fundamental generalization capability of SAM masks.
Therefore, we integrate  to address the limitation of unseen mask proposals. This enhancement is evident in the comparison between ablation cases (2) and (3), and also between cases (5) and (6).  Fig. <ref> provides a clearer visual comparison, showing improved segmentation accuracy for objects such as ‘people’ and ‘fences’ in the PC-459(2) example, particularly in columns 3 and 4.




  §.§.§ Where to Inject

Tab. <ref> presents an ablation examining the impact of layer insertion for the  within the transformer decoder, which consists of a total of 9 layers. Since SAM Vision Transformers provide the final layer features as the most relevant feature maps, we explore the optimal layer for query injection based on its interaction with corresponding CLIP feature maps. The results indicate that injecting SAM query features at layers l=3, 6, 9 yields the most significant improvement, with layer l+1 leveraging the newly introduced queries for further refinement.
For the , due to the exponential increase in computational complexity associated with cross-attention computations as feature size expands, we restrict the application of the Feature Injector to 1/32-sized features, specifically at layers l=1, 4, 7.




  §.§.§ Speed and Model Size

As shown in Tab. <ref>, incorporating SAM along with two custom injectors results in a slight reduction in inference speed, manifesting as a 0.56 and 0.09 decrement in frames per second (FPS) during single-image processing. Despite this, the adjustment leads to a notable improvement of 1.8 PQ on the Cityscapes datasets, with minimal impact on COCO. This reflects a well-balanced trade-off between enhanced performance and computational efficiency. Compared to FC-CLIP, our model requires a modest increase of only 5.5M training parameters and 93.5M frozen parameters, demonstrating its effectiveness.



§ CONCLUSION

In this study, we introduced , a method designed to enhance mask proposal quality in open-vocabulary segmentation by leveraging SAM’s dense-prediction capabilities.  employs the  and  modules to integrate SAM visual features with learned queries and CLIP visual features, thereby refining mask proposals through multiple transformer decoder layers. Additionally, we introduce the  for inference, which aggregates zero-shot SAM masks to improve out-vocabulary predictions further. Our experiments demonstrated that  significantly enhances the mask proposal quality in open-vocabulary scenarios, highlighting its versatility.













§ APPENDIX

Our supplementary material begins with the in-depth analysis of the FC-CLIP baseline's performance. Next, we explore more numerical results across various datasets, including evaluating mask recalls and an ablation study focusing on the co-training size of SAM. Finally, we present further qualitative visualization findings, featuring mask attention maps and segmentation results for two challenging datasets, A-847 <cit.> and PC-459 <cit.>.



 §.§ Further discussion of FC-CLIP baseline

FC-CLIP adopts a checkpoint selection strategy based on the PQ accuracy within the ADE20K benchmark <cit.>, a dataset known for its complexity with 150 diverse classes. Upon executing the FC-CLIP code and analyzing the final-round results, marked by * in Tab. <ref>, we observed tendencies towards overfitting and a subsequent decline in generalizability, This was accompanied by reduced effectiveness across various other open-vocabulary evaluation datasets, although there was improved performance on the COCO validation dataset.
Despite FC-CLIP's strategies to mitigate overfitting, the method's effectiveness in open-vocabulary scenarios, especially in the context of ADE20K's datasets, remains questionable. This raises concerns about the transparency of its model selection methodology. In contrast, our proposed framework, , which leverages the last iteration's checkpoints for inference, performs comparably on both the ADE20K and A-847 datasets. It demonstrates consistent and robust performance across all tested scenarios, thus eliminating the need for selective model evaluation.



 §.§ More numerical results 



  §.§.§ Comparative recall across datasets

In Tab. <ref>, we present the recall rates for our method and FC-CLIP across four datasets. This comparison is between the predicted mask proposals and class-agnostic semantic ground truth. We detail recall rates at IoU thresholds of 0.5, 0.75, and 0.9. The results demonstrate that  generally outperforms in generalizing to mask proposals for unseen classes.




  §.§.§ Comparison with different SAMs

In Tab. <ref>, we provide detailed results of  (w/o. mask ensemble), using ConvNeXt-L CLIP <cit.> alongside different size of co-trained SAM <cit.>: ViT-T (Tiny) <cit.>, ViT-B (Base), ViT-L (Large) and ViT-H (Huge). Across the board, the ViT-B configuration stands out, delivering better performance in our evaluations. We also provide visualizations of the k-means clustering results for feature embeddings from the SAM image encoders. The visualization demonstrates that ViT-B balances segmentation accuracy and connectivity, offering precise segmentation with good instance connectivity. ViT-T provides coarse boundaries, while ViT-L and ViT-H, though more precise, have reduced instance connectivity and may be less effective for panoptic segmentation with CLIP. Thus, ViT-B’s balanced performance makes it a robust choice.




 §.§ More qualitative visualizations






  §.§.§ Attention maps

To illustrate the refinement of query features facilitated by injectors, we identify the query with the highest confidence and present its corresponding attention map from the final cross-attention layer within the transformer decoder. We map the attention map back to the original image for visualization purposes. The results are depicted in Fig. <ref>. It is evident that our queries exhibit heightened attention towards both the object boundaries and intra-content regions, indicative of the effectiveness of our approach in mask proposal generation.




  §.§.§ More results

We have expanded our visual comparisons in PC-459 dataset shown in Fig. <ref>, and the A-847 dataset, depicted in Fig. <ref>. In Fig. <ref>, it can be seen that our method generates more precise masks which are highlighted by red boxes.
Notably, we draw attention to the areas enclosed by white boxes, which exhibit coarse or imprecise annotations. For instance, the 'door' is overlooked in the first column, and the 'chair' annotations fail to precisely demarcate the chair legs. Meanwhile, in the second column, although the ground truth predominantly annotates the background as 'grass', a closer inspection reveals a composite of 'soil' and 'grass', with 'sidewalks' situated in the lower left quadrant. 
Fig. <ref> exemplifies the efficacy of our proposed method in producing high-quality masks, extending across a diverse array of novel classes. These include but are not limited to, 'toys', 'painted pictures', 'baptismal fonts', 'altars', 'decorative elements', 'columns', 'pipes', and 'fluorescent lighting', among others.







ieee_fullname



