\begin{table}[t!]
\addtolength{\tabcolsep}{-2pt}
\centering
\scriptsize
\begin{tabular}{c|c|c|c|ccc}
\cellcolor{lightgray!30} Model & \cellcolor{lightgray!30} Backbone & \cellcolor{lightgray!30} Iteration & \cellcolor{lightgray!30} Pretrain  & \cellcolor{lightgray!30} AP$^{e}$ & \cellcolor{lightgray!30} AP$^{e}_{50}$ & \cellcolor{lightgray!30} AP$^{e}_{75}$ \\ \hline
\multirow{2}*{Mask-RCNN~\cite{he2017mask}} & \multirow{2}*{Swin-T} & \multirow{2}*{1$\times$} & ImageNet & 24.9 & 45.8 & 24.1 \\
& & & COCO-E & 28.4 & 49.2 & 28.1 \\ \hline
\multirow{2}*{EntityFramework~\cite{qi2021open}} & \multirow{2}*{Swin-T} &\multirow{2}*{1$\times$} & ImageNet & 26.0 & 42.8 & 25.7 \\
& & & COCO-E & 29.9 & 47.6 & 30.1 \\ \hline
\multirow{6}*{Mask2Former~\cite{cheng2022masked}} & \multirow{5}*{Swin-T} & \multirow{2}*{1$\times$} & ImageNet & 33.2 & 50.2 & 33.1 \\ \cline{4-4}
& & & \multirow{6}*{COCO-E} & 39.5 & 56.9 & 40.2 \\ \cline{3-3}
& & 2$\times$ & & 40.2 & 57.6 & 41.1 \\
& & 3$\times$ & & 40.9 & 58.1 & 41.6 \\
& & 4$\times$ & & 40.9 & 57.9  & 41.9 \\ \cline{2-3}
& Swin-L & 3$\times$ & & 46.2 & 63.7 & 47.5 \\ \hline
\end{tabular}
\caption{Entity segmentation benchmark in Entity Dataset. The column of `Pretrain' indicates the pretraining weights we used where the `ImageNet' is ImageNet pretraining and `COCO-E' refers to pretraining on COCO dataset that has been converted to class-agnostic entity segmentation format \cite{qi2021open}.}
\vspace{-0.2in}
\label{Tab:aba_benchmark_entity}
\end{table}
\begin{abstract}
In dense image segmentation tasks (\textit{e.g.,} semantic, panoptic), existing methods can hardly generalize well to unseen image domains, predefined classes, and image resolution \& quality variations.
Motivated by these observations, we construct a large-scale entity segmentation dataset to explore fine-grained entity segmentation, with a strong focus on open-world and high-quality dense segmentation. 
The dataset contains images spanning diverse image domains and resolutions, along with high-quality mask annotations for training and testing. 
Given the high-quality and -resolution nature of the dataset, we propose CropFormer for high-quality segmentation, which can improve mask prediction using high-res image crops that provide more fine-grained image details than the full image. CropFormer is the first query-based Transformer architecture that can effectively ensemble mask predictions from multiple image crops, by learning queries that can associate the same entities across the full image and its crop. With CropFormer, we achieve a significant AP gain of $1.9$ on the challenging fine-grained entity segmentation task. The dataset and code will be released at \href{http://luqi.info/entityv2.github.io/}{http://luqi.info/entityv2.github.io/}.
\end{abstract}

\vspace{-4mm}
\paragraph{Scene Parsing Methods}
Scene parsing methods are mainly developed by convolution-based dense prediction or transformer-based query prediction. 
The method based on convolution-based dense prediction often uses explicit localization information like bounding box proposals or pixel position for pixel-level mask prediction.
These approaches include
FCN~\cite{long2015fully}, DeepLab~\cite{chen2017deeplab}, PSPNet~\cite{zhao2017pyramid}, Mask R-CNN~\cite{he2017mask}, PANet~\cite{liu2018path}, SOLO~\cite{wang2019solo}, CondInst~\cite{tian2020conditional}, PanopticFPN~\cite{kirillov2019panopticfpn}, and PanopticFCN~\cite{li2022fully}. 
Motivated by the success of transformers DETR~\cite{carion2020end} for detection tasks~\cite{lin2017feature,lin2017focal,dai2017deformable,qi2021multi}, numerous methods, e.g., Max-Deeplab~\cite{wang2021max} and Mask2Former~\cite{cheng2022masked}, directly predict segmentation masks without the guidance of bounding boxes. 
The key to these methods is to use learnable queries to model the implicit location of the potential instance area.
Although query-based methods are effective, they cannot benefit from TTA which is commonly used to boost the accuracy of convolution-based methods.
Specifically, it is challenging for a query-based method to ensemble multiple results from augmented image views, due to the lack of exact location for queries.
Motivated by video-level Mask2Former, we endow the queries with ability to connect the same entities among different image views. 
As such, our approach can enhance high-resolution image segmentation by combining results from full image and local crops.

\subsection{Mask2Former Preliminaries}
Mask2Former~\cite{cheng2022masked} is a universal Transformer-based framework for image or video instance segmentation. Given an input, Mask2Former uses a Transformer decoder to decode $N$ number of $K$-dimensional queries $\mathbf{Q}\in \mathbb{R}^{N\times K}$ to generate embeddings $\mathbf{E} \in \mathbb{R}^{N \times 1 \times 1 \times 1 \times K}$ to predict $N$ segmentation masks, where the \nth{2}, \nth{3}, and \nth{4} dimensions of $\mathbf{E}$ correspond to temporal ($T$), output height ($H$), and output width  ($W$) dimensions respectively. When applying the image-level Mask2Former to an image input, $\mathbf{E}$ is broadcasted along spatial dimensions to the shape of $N \times 1 \times H \times W \times K$. Alternatively, when applying video-level Mask2Former to a video input, $\mathbf{E}$ is broadcasted to the shape of $N \times T \times H \times W \times K$. Note that in video-level Mask2Former, $\mathbf{E}$ is further broadcasted along the both temporal and spatial dimensions, enabling Mask2Former to utilize the same embeddings to represent the same visual instances across different frames consistently.

\vspace{-4mm}
\paragraph{Crop Dataloader} In CropFormer, the crop dataloader is designed to generate a batch of images that simultaneously include the full images and their corresponding crops. There are two steps in our dataloader: \textit{crop} and \textit{resize}. At first, we augment the full image $I^o$ with a crop that is part of the full image. The 
crop is randomly extracted from one of the fixed image corners: upper-left, upper-right, bottom-left, bottom-right. The crop size is controlled by a fixed ratio hyperparameter $\delta \in \mathbb{R}$ relative to the full image size. Second, we resize both the full image and crop to the same size. In this way, we construct an input batch $\mathbf{I} \in \mathbb{R}^{N \times 2 \times H_\mathbf{I} \times W_\mathbf{I} \times 3}$ with 2 elements (full image $\&$ 1 crop) along the temporal dimension. Compared to the full image, corner crops preserve more image details and local information useful for fine-grained segmentation.
Given dataloader $\Gamma$, we define the data preprocessing as

\vspace{-4mm}
\paragraph{Image Encoder and Decoder} Based on learnable image-level queries $\mathbf{Q}_{i} \in \mathbb{R}^{N\times K}$,  we use image encoder $\Theta$ and decoder $\Phi_{i}$ to generate image-level embeddings $\mathbf{E_i} \in \mathbb{R}^{N \times 2 \times 1 \times 1 \times K}$ for the full input image and its crop. 
For the image encoder and decoder, we directly adopt the ones in standard image-level Mask2Former~\cite{cheng2022masked} and do not modify their structures or designs. $\mathbf{E_i}$ is obtained as follows:

Similar to video-level Mask2Former, the full image and its crop can be seen as 2 video frames. Thus, we broadcast $\mathbf{E_b}$ to the shape of $N \times 2 \times 1 \times 1 \times K$ to share it between the full image and its crop. $\mathbf{E_b}$ and batch-level pyramid features $\mathbf{P_b^h}\in\mathbb{R}^{1\times2\times H^h\times W^h}$ (reshaped from $\mathbf{P_i^h}\in\mathbb{R}^{2\times1\times H^h\times W^h}$) are used for batch-level mask classification and pixel-wise prediction:

\subsection{Entity Dataset Benchmark}
\label{subsec:entity_dataset_benchmark}
\paragraph{Entity Segmentation} We divide the EntitySeg dataset into train and test sets, which have 31,913 and 1,314 images. To obtain a less-biased dataset split, we constructed 20 random dataset splits. We selected the dataset split that is statistically closest to the 
all 20 splits in terms of entity segmentation accuracy, as detailed in the supplementary file.Furthermore, we use class-agnostic metric AP$^\text{e}$~\cite{qi2021open} with a strict non-overlapping mask constraint to evaluate the entity segmentation task.

Table~\ref{Tab:aba_benchmark_entity} shows the benchmark of our dataset with Mask-RCNN~\cite{he2017mask}, EntityFramework~\cite{qi2021open}, Mask2Former~\cite{cheng2022masked}. The first two methods are convolution-based dense prediction methods, whereas the last is a transformer-based query prediction method. In this table, the transformer-based Mask2Former is better than the other two convolution-based methods, demonstrating the advantage of transform-based methods on high-quality mask generation, Moreover, the COCO-E pretrained weights can further boost entity segmentation performance. We also explored the optimal training iterations needed by Mask2Former and found that it performs the best with 3$\times$ training schedule.
{"value": "24.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "Mask-RCNN", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "45.8", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "Mask-RCNN", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "24.1", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "Mask-RCNN", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "28.4", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "Mask-RCNN", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "49.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "Mask-RCNN", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "28.1", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "Mask-RCNN", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "26.0", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "EntityFramework", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "42.8", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "EntityFramework", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "25.7", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "EntityFramework", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "29.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "EntityFramework", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "47.6", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "EntityFramework", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "30.1", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "EntityFramework", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "33.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "50.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "33.1", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "ImageNet"}} {"value": "39.5", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "56.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "40.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "1$\\times$", "pretrain": "COCO-E"}} {"value": "40.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "2$\\times$", "pretrain": "COCO-E"}} {"value": "57.6", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "2$\\times$", "pretrain": "COCO-E"}} {"value": "41.1", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "2$\\times$", "pretrain": "COCO-E"}} {"value": "40.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "3$\\times$", "pretrain": "COCO-E"}} {"value": "58.1", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "3$\\times$", "pretrain": "COCO-E"}} {"value": "41.6", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "3$\\times$", "pretrain": "COCO-E"}} {"value": "40.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "4$\\times$", "pretrain": "COCO-E"}} {"value": "57.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "4$\\times$", "pretrain": "COCO-E"}} {"value": "41.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "iteration": "4$\\times$", "pretrain": "COCO-E"}} {"value": "46.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "iteration": "3$\\times$", "pretrain": "COCO-E"}} {"value": "63.7", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "iteration": "3$\\times$", "pretrain": "COCO-E"}} {"value": "47.5", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-L", "iteration": "3$\\times$", "pretrain": "COCO-E"}}