\begin{table}[t!]
\centering
\scriptsize
\begin{tabular}{c|c|ccc|c}
\cellcolor{lightgray!30} Method & \cellcolor{lightgray!30} Decoder & \cellcolor{lightgray!30} AP$^{e}$ & \cellcolor{lightgray!30} AP$^{e}_{50}$ & A\cellcolor{lightgray!30} P$^{e}_{75}$ & \cellcolor{lightgray!30} RT (ms) \\ \hline
SS-Mask2Former & Image-O & 39.5 & 56.9 & 40.2 & 637 \\ 
SS-Mask2Former($\times\delta$) & Image-O & 39.9 & 57.4 & 40.3 & 876 \\ 
MS-Mask2Former & Image-O & 39.2 & 56.3 & 39.5 & 1324 \\ 
MS-Mask2Former & Batch-OC & 39.3 & 56.4 & 39.7 & 2783 \\ \hline
\multirow{4}*{CropFormer} & Image-O & 39.3 & 56.7 & 39.8 & 637 \\ 
 & Batch-O & 39.1 & 56.6 & 39.7 & 1514 \\ 
 & Batch-C & 40.2 & 57.5 & 40.8 & 1507 \\
 & Batch-OC & \textbf{41.0} & \textbf{58.4} & \textbf{41.9} & 1545 \\ \hline
\end{tabular}
\caption{Ablation study on the ensemble strategy on full image and four crops. The `Decoder' column indicates whether we use the inference result of the full image (`O'), four cropped patches (`C'), or both of them (`OC') from the `Image' or `Batch' decoder. Here, the run-time (RT) is the time of network forward except the data processing and calculated on A100 GPU.}
\vspace{-0.1in}
\label{Tab:aba_ivresults}
\end{table}
\begin{abstract}
In dense image segmentation tasks (\textit{e.g.,} semantic, panoptic), existing methods can hardly generalize well to unseen image domains, predefined classes, and image resolution \& quality variations.
Motivated by these observations, we construct a large-scale entity segmentation dataset to explore fine-grained entity segmentation, with a strong focus on open-world and high-quality dense segmentation. 
The dataset contains images spanning diverse image domains and resolutions, along with high-quality mask annotations for training and testing. 
Given the high-quality and -resolution nature of the dataset, we propose CropFormer for high-quality segmentation, which can improve mask prediction using high-res image crops that provide more fine-grained image details than the full image. CropFormer is the first query-based Transformer architecture that can effectively ensemble mask predictions from multiple image crops, by learning queries that can associate the same entities across the full image and its crop. With CropFormer, we achieve a significant AP gain of $1.9$ on the challenging fine-grained entity segmentation task. The dataset and code will be released at \href{http://luqi.info/entityv2.github.io/}{http://luqi.info/entityv2.github.io/}.
\end{abstract}

\subsection{Mask2Former Preliminaries}
Mask2Former~\cite{cheng2022masked} is a universal Transformer-based framework for image or video instance segmentation. Given an input, Mask2Former uses a Transformer decoder to decode $N$ number of $K$-dimensional queries $\mathbf{Q}\in \mathbb{R}^{N\times K}$ to generate embeddings $\mathbf{E} \in \mathbb{R}^{N \times 1 \times 1 \times 1 \times K}$ to predict $N$ segmentation masks, where the \nth{2}, \nth{3}, and \nth{4} dimensions of $\mathbf{E}$ correspond to temporal ($T$), output height ($H$), and output width  ($W$) dimensions respectively. When applying the image-level Mask2Former to an image input, $\mathbf{E}$ is broadcasted along spatial dimensions to the shape of $N \times 1 \times H \times W \times K$. Alternatively, when applying video-level Mask2Former to a video input, $\mathbf{E}$ is broadcasted to the shape of $N \times T \times H \times W \times K$. Note that in video-level Mask2Former, $\mathbf{E}$ is further broadcasted along the both temporal and spatial dimensions, enabling Mask2Former to utilize the same embeddings to represent the same visual instances across different frames consistently.

\vspace{-4mm}
\paragraph{Crop Dataloader} In CropFormer, the crop dataloader is designed to generate a batch of images that simultaneously include the full images and their corresponding crops. There are two steps in our dataloader: \textit{crop} and \textit{resize}. At first, we augment the full image $I^o$ with a crop that is part of the full image. The 
crop is randomly extracted from one of the fixed image corners: upper-left, upper-right, bottom-left, bottom-right. The crop size is controlled by a fixed ratio hyperparameter $\delta \in \mathbb{R}$ relative to the full image size. Second, we resize both the full image and crop to the same size. In this way, we construct an input batch $\mathbf{I} \in \mathbb{R}^{N \times 2 \times H_\mathbf{I} \times W_\mathbf{I} \times 3}$ with 2 elements (full image $\&$ 1 crop) along the temporal dimension. Compared to the full image, corner crops preserve more image details and local information useful for fine-grained segmentation.
Given dataloader $\Gamma$, we define the data preprocessing as

\vspace{-4mm}
\paragraph{Image Encoder and Decoder} Based on learnable image-level queries $\mathbf{Q}_{i} \in \mathbb{R}^{N\times K}$,  we use image encoder $\Theta$ and decoder $\Phi_{i}$ to generate image-level embeddings $\mathbf{E_i} \in \mathbb{R}^{N \times 2 \times 1 \times 1 \times K}$ for the full input image and its crop. 
For the image encoder and decoder, we directly adopt the ones in standard image-level Mask2Former~\cite{cheng2022masked} and do not modify their structures or designs. $\mathbf{E_i}$ is obtained as follows:

\vspace{-4mm}
\paragraph{Association Module} Similar to video-level Mask2Former~\cite{cheng2021mask2former}, the association module aims to generate batch queries $\mathbf{Q_b}$ that are fully shared by the full image and its crop to represent the same entities consistently. Instead of treating $\mathbf{Q_b}$ as learnable network parameters, here we generate $\mathbf{Q_b}$ directly from $\mathbf{E_i}=\{\mathbf{E}_{I^o}, \mathbf{E}_{I^c})\}$, considering that $\mathbf{E_i}$ already contains strong segmentation features. 
In particular, we use a Transformer architecture with cross attention (XAtt), self-attention (SAtt), and feedforward network (FFN) to obtain $\mathbf{Q_b}$:

where $f_{\{\text{q},\text{k},\text{v}\}}$ are linear transformations. Since the full image is more crucial to entity segmentation than its crop, we take the image-level embeddings of the full image $\mathbf{E}_{I^o}$ as the \textit{query}, while treating all the image-level embeddings $\mathbf{E_i}$ as and \textit{key} and \textit{value}. This design is similar to the cross-attention 
of original 
Transformer \cite{vaswani2017attention} and SelfDoc \cite{li2021selfdoc}.

Similar to video-level Mask2Former, the full image and its crop can be seen as 2 video frames. Thus, we broadcast $\mathbf{E_b}$ to the shape of $N \times 2 \times 1 \times 1 \times K$ to share it between the full image and its crop. $\mathbf{E_b}$ and batch-level pyramid features $\mathbf{P_b^h}\in\mathbb{R}^{1\times2\times H^h\times W^h}$ (reshaped from $\mathbf{P_i^h}\in\mathbb{R}^{2\times1\times H^h\times W^h}$) are used for batch-level mask classification and pixel-wise prediction:

Table~\ref{Tab:aba_ivresults} shows the overall improvement of CropFromer to the Mask2Former~\cite{cheng2022masked} under 1$\times$ training setting. The first and second row is our baseline Mask2Former with the single-scale inference (800 and 1040 shortest side) and (1333 and 1732 longest side) for the full images, where 1040=800$\times\delta$ and 1732=1333$\times\delta$ with $\delta$ 0.7 in default. For the results in the third and fourth rows, we use test-time bipartite-matching to associate the same entities obtained with multi-scale images and crops. We find no improvements with such inference strategies. Whereas, using the crop output from CropFormer's batch decoder achieves a significant AP$^\text{e}$ gain as indicated by the second last row. By combining the full image and four crop outputs from CropFormer (final row), we obtain even stronger 1.5 AP$^\text{e}$ and 1.7 AP$^\text{e}_{75}$ gain compared to the baseline (first row).
{"value": "39.5", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}", "experimental settings": {"iteration": "1$\\times$"}, "model": "SS-Mask2Former", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "56.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}", "experimental settings": {"iteration": "1$\\times$"}, "model": "SS-Mask2Former", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "40.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}", "experimental settings": {"iteration": "1$\\times$"}, "model": "SS-Mask2Former", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "637", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "RT (ms)", "experimental settings": {"iteration": "1$\\times$"}, "model": "SS-Mask2Former", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}", "experimental settings": {"iteration": "1$\\times$"}, "model": "SS-Mask2Former($\\times\\delta$)", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "57.4", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}", "experimental settings": {"iteration": "1$\\times$"}, "model": "SS-Mask2Former($\\times\\delta$)", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "40.3", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}", "experimental settings": {"iteration": "1$\\times$"}, "model": "SS-Mask2Former($\\times\\delta$)", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "876", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "RT (ms)", "experimental settings": {"iteration": "1$\\times$"}, "model": "SS-Mask2Former($\\times\\delta$)", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}", "experimental settings": {"iteration": "1$\\times$"}, "model": "MS-Mask2Former", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "56.3", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}", "experimental settings": {"iteration": "1$\\times$"}, "model": "MS-Mask2Former", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.5", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}", "experimental settings": {"iteration": "1$\\times$"}, "model": "MS-Mask2Former", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "1324", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "RT (ms)", "experimental settings": {"iteration": "1$\\times$"}, "model": "MS-Mask2Former", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.3", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}", "experimental settings": {"iteration": "1$\\times$"}, "model": "MS-Mask2Former", "model settings": {"Decoder": "Batch-OC", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "56.4", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}", "experimental settings": {"iteration": "1$\\times$"}, "model": "MS-Mask2Former", "model settings": {"Decoder": "Batch-OC", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.7", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}", "experimental settings": {"iteration": "1$\\times$"}, "model": "MS-Mask2Former", "model settings": {"Decoder": "Batch-OC", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "2783", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "RT (ms)", "experimental settings": {"iteration": "1$\\times$"}, "model": "MS-Mask2Former", "model settings": {"Decoder": "Batch-OC", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.3", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "56.7", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.8", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "637", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "RT (ms)", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Image-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.1", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "56.6", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.7", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "1514", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "RT (ms)", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-O", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "40.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-C", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "57.5", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-C", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "40.8", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-C", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "1507", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "RT (ms)", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-C", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "41.0", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-OC", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "58.4", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-OC", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "41.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-OC", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "1545", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "RT (ms)", "experimental settings": {"iteration": "1$\\times$"}, "model": "CropFormer", "model settings": {"Decoder": "Batch-OC", "backbone": "Swin-L", "pretrained weights": "COCO-E"}}