\begin{table}[t!]
\centering
\scriptsize
\begin{tabular}{c|c|c|ccc}
\cellcolor{lightgray!30} Model & \cellcolor{lightgray!30} Backbone & \cellcolor{lightgray!30} Pretrain & \cellcolor{lightgray!30} PQ & \cellcolor{lightgray!30} SQ & \cellcolor{lightgray!30} RQ \\ \hline
\multirow{2}*{PanopticFPN~\cite{kirillov2019panopticfpn}} &
\multirow{2}*{R-50} & ImageNet & 3.6 & 25.8 & 5.5   \\
& & COCO-P & 6.7 & 36.4 & 10.1  \\ \hline
\multirow{7}*{Mask2Former~\cite{cheng2022masked}} & \multirow{2}*{R-50} & ImageNet & 5.5 & 26.3 & 8.2 \\ 
& & COCO-I & 9.6 & 39.0 & 14.1 \\ \cline{2-6}
& \multirow{2}*{Swin-T} & COCO-E & 7.4 & 32.6 & 11.0  \\ 
& & COCO-P & 9.8 & 38.5 & 14.6 \\ \cline{2-6}
& \multirow{2}*{Swin-L} & COCO-E & 11.7 & 43.2 & 17.3 \\ 
& & COCO-P & 13.4 & 48.7 & 19.9 \\ \hline
\end{tabular}
\caption{Benchmark on class-aware panoptic segmentation in Entity Dataset.}
\vspace{-0.2in}
\label{Tab:aba_benchmark_classaware_panop}
\end{table}
\begin{abstract}
In dense image segmentation tasks (\textit{e.g.,} semantic, panoptic), existing methods can hardly generalize well to unseen image domains, predefined classes, and image resolution \& quality variations.
Motivated by these observations, we construct a large-scale entity segmentation dataset to explore fine-grained entity segmentation, with a strong focus on open-world and high-quality dense segmentation. 
The dataset contains images spanning diverse image domains and resolutions, along with high-quality mask annotations for training and testing. 
Given the high-quality and -resolution nature of the dataset, we propose CropFormer for high-quality segmentation, which can improve mask prediction using high-res image crops that provide more fine-grained image details than the full image. CropFormer is the first query-based Transformer architecture that can effectively ensemble mask predictions from multiple image crops, by learning queries that can associate the same entities across the full image and its crop. With CropFormer, we achieve a significant AP gain of $1.9$ on the challenging fine-grained entity segmentation task. The dataset and code will be released at \href{http://luqi.info/entityv2.github.io/}{http://luqi.info/entityv2.github.io/}.
\end{abstract}

\section{Introduction}
Dense image segmentation is an important topic in computer vision with a wide range of applications such as autonomous driving~\cite{qi2019amodal}, robotics~\cite{shu2014human}, video surveillance~\cite{morrison2018cartman}, and image editing tools. 
Recently, entity segmentation~\cite{qi2021open} has been introduced as a promising dense instance-aware approach for tackling generalization issues caused by narrow image domains and/or predefined class sets in existing panoptic~\cite{kirillov2019panoptic,kirillov2019panopticfpn,xiong2019upsnet,li2020fully,carion2020end} 
and semantic~\cite{long2015fully,zhao2017pyramid,chen2017rethinking,chen2018encoder,zhao2018psanet,huang2019ccnet,liu2019auto} segmentation tasks.
Although the early method~\cite{qi2021open} shows some promising results, it merely borrows existing panoptic segmentation datasets to demonstrate the feasibility and benefits of the entity segmentation. 
However, existing panoptic segmentation datasets (\textit{e.g,} COCO~\cite{lin2014microsoft}, Cityscapes~\cite{cordts2016cityscapes}, ADE20K~\cite{zhou2017scene}) 
suffer from the same issues that entity segmentation~\cite{qi2021open} aims to avoid, since those datasets are collected and annotated with 
panoptic segmentation task requirements in mind.

\vspace{-4mm}
\paragraph{Crop Dataloader} In CropFormer, the crop dataloader is designed to generate a batch of images that simultaneously include the full images and their corresponding crops. There are two steps in our dataloader: \textit{crop} and \textit{resize}. At first, we augment the full image $I^o$ with a crop that is part of the full image. The 
crop is randomly extracted from one of the fixed image corners: upper-left, upper-right, bottom-left, bottom-right. The crop size is controlled by a fixed ratio hyperparameter $\delta \in \mathbb{R}$ relative to the full image size. Second, we resize both the full image and crop to the same size. In this way, we construct an input batch $\mathbf{I} \in \mathbb{R}^{N \times 2 \times H_\mathbf{I} \times W_\mathbf{I} \times 3}$ with 2 elements (full image $\&$ 1 crop) along the temporal dimension. Compared to the full image, corner crops preserve more image details and local information useful for fine-grained segmentation.
Given dataloader $\Gamma$, we define the data preprocessing as

\vspace{-4mm}
\paragraph{Image Encoder and Decoder} Based on learnable image-level queries $\mathbf{Q}_{i} \in \mathbb{R}^{N\times K}$,  we use image encoder $\Theta$ and decoder $\Phi_{i}$ to generate image-level embeddings $\mathbf{E_i} \in \mathbb{R}^{N \times 2 \times 1 \times 1 \times K}$ for the full input image and its crop. 
For the image encoder and decoder, we directly adopt the ones in standard image-level Mask2Former~\cite{cheng2022masked} and do not modify their structures or designs. $\mathbf{E_i}$ is obtained as follows:

Table~\ref{Tab:aba_benchmark_entity} shows the benchmark of our dataset with Mask-RCNN~\cite{he2017mask}, EntityFramework~\cite{qi2021open}, Mask2Former~\cite{cheng2022masked}. The first two methods are convolution-based dense prediction methods, whereas the last is a transformer-based query prediction method. In this table, the transformer-based Mask2Former is better than the other two convolution-based methods, demonstrating the advantage of transform-based methods on high-quality mask generation, Moreover, the COCO-E pretrained weights can further boost entity segmentation performance. We also explored the optimal training iterations needed by Mask2Former and found that it performs the best with 3$\times$ training schedule.

\vspace{-4mm}
\paragraph{Semantic Segmentation} We choose 150 categories with the highest pixel-level frequency as EntitySem for semantic segmentation. EntitySem has 9,729 and 1,444 images for training and testing, respectively. In Table~\ref{Tab:aba_benchmark_classaware_sem}, we evaluate the two popular semantic segmentation methods DeeplabV3~\cite{chen2017deeplab} and Mask2Former~\cite{cheng2022masked} on EntitySem. Semantic segmentation performance is still related to the pretraining weights and network structure. Mask2Former with Swin-L backbone and COCO-E pretrained weights obtains the best mIoU of 50.5 on EntitySem. 

\vspace{-4mm}
\paragraph{Instance Segmentation}
We select 206 thing categories with the highest object-level frequency in the EntityClass dataset to benchmark instance segmentation. In the EntityIns, 8,993 and 1,498 images for training and testing, respectively. In Table~\ref{Tab:aba_benchmark_classaware_ins}, we ablate two popular instance segmentation methods including Mask-RCNN~\cite{he2017mask} and Mask2Former~\cite{cheng2022masked} on EntityIns. Mask2Former with Swin-L backbone and COCO-P pretrained weights the best AP of 30.3 on EntityIns. 

\vspace{-4mm}
\paragraph{Panoptic Segmentation} Similar to EntityIns, we select 345 categories, including 274 things and 71 stuff, with the highest entity-level frequency to construct EntityPan. In EntityPan, there are 9,968 and 1,481 images for training and testing. Table~\ref{Tab:aba_benchmark_classaware_panop} shows the performance of two popular panoptic segmentation methods, including PanopticFPN~\cite{kirillov2019panopticfpn} and Mask2Former~\cite{cheng2022masked}. In this table, we see that the PQs are much lower than those of existing panoptic datasets, due to the greater variety of class labels which makes the task more challenging.
In addition, current panoptic methods perform worse on EntityPan compared to existing datasets, particularly on the recognition quality (RQ) side which adversely impacts PQ.
{"value": "3.6", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "PQ", "experimental settings": {"xx": "yy"}, "model": "PanopticFPN", "model settings": {"backbone": "R-50", "pretrain": "ImageNet"}} {"value": "25.8", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "SQ", "experimental settings": {"xx": "yy"}, "model": "PanopticFPN", "model settings": {"backbone": "R-50", "pretrain": "ImageNet"}} {"value": "5.5", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "RQ", "experimental settings": {"xx": "yy"}, "model": "PanopticFPN", "model settings": {"backbone": "R-50", "pretrain": "ImageNet"}} {"value": "6.7", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "PQ", "experimental settings": {"xx": "yy"}, "model": "PanopticFPN", "model settings": {"backbone": "R-50", "pretrain": "COCO-P"}} {"value": "36.4", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "SQ", "experimental settings": {"xx": "yy"}, "model": "PanopticFPN", "model settings": {"backbone": "R-50", "pretrain": "COCO-P"}} {"value": "10.1", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "RQ", "experimental settings": {"xx": "yy"}, "model": "PanopticFPN", "model settings": {"backbone": "R-50", "pretrain": "COCO-P"}} {"value": "5.5", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "PQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "R-50", "pretrain": "ImageNet"}} {"value": "26.3", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "SQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "R-50", "pretrain": "ImageNet"}} {"value": "8.2", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "RQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "R-50", "pretrain": "ImageNet"}} {"value": "9.6", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "PQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "R-50", "pretrain": "COCO-I"}} {"value": "39.0", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "SQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "R-50", "pretrain": "COCO-I"}} {"value": "14.1", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "RQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "R-50", "pretrain": "COCO-I"}} {"value": "7.4", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "PQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrain": "COCO-E"}} {"value": "32.6", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "SQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrain": "COCO-E"}} {"value": "11.0", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "RQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrain": "COCO-E"}} {"value": "9.8", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "PQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrain": "COCO-P"}} {"value": "38.5", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "SQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrain": "COCO-P"}} {"value": "14.6", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "RQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrain": "COCO-P"}} {"value": "11.7", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "PQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "pretrain": "COCO-E"}} {"value": "43.2", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "SQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "pretrain": "COCO-E"}} {"value": "17.3", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "RQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "pretrain": "COCO-E"}} {"value": "13.4", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "PQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "pretrain": "COCO-P"}} {"value": "48.7", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "SQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "pretrain": "COCO-P"}} {"value": "19.9", "type": "Result", "training data/set": "EntityPan", "test data/set": "EntityPan", "task": "panoptic segmentation", "metric": "RQ", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "pretrain": "COCO-P"}}