\begin{table}[t!]
\centering
\scriptsize
\begin{tabular}{c|c|ccc}
\cellcolor{lightgray!30} Fusion-Style & \cellcolor{lightgray!30} Type & \cellcolor{lightgray!30} AP$^{e}$ & \cellcolor{lightgray!30} AP$^{e}_{50}$ & \cellcolor{lightgray!30} AP$^{e}_{75}$ \\ \hline
VI-Mask2Former~\cite{cheng2022masked,cheng2021mask2former} & Parallel & 39.7 & 57.3 & 40.6 \\ 
VITA~\cite{heo2022vita} & Cascade & 39.9 & 57.8 & 40.7  \\
Ours & Cascade & \textbf{41.0} & \textbf{58.4} & \textbf{41.9} \\ \hline
\end{tabular}
\caption{Ablation study on the fusion style between image decoder and batch decoder. `Type' indicates the structure relationship between two decoders.}
\vspace{-0.2in}
\label{Tab:aba_fusion_style}
\end{table}
\begin{abstract}
In dense image segmentation tasks (\textit{e.g.,} semantic, panoptic), existing methods can hardly generalize well to unseen image domains, predefined classes, and image resolution \& quality variations.
Motivated by these observations, we construct a large-scale entity segmentation dataset to explore fine-grained entity segmentation, with a strong focus on open-world and high-quality dense segmentation. 
The dataset contains images spanning diverse image domains and resolutions, along with high-quality mask annotations for training and testing. 
Given the high-quality and -resolution nature of the dataset, we propose CropFormer for high-quality segmentation, which can improve mask prediction using high-res image crops that provide more fine-grained image details than the full image. CropFormer is the first query-based Transformer architecture that can effectively ensemble mask predictions from multiple image crops, by learning queries that can associate the same entities across the full image and its crop. With CropFormer, we achieve a significant AP gain of $1.9$ on the challenging fine-grained entity segmentation task. The dataset and code will be released at \href{http://luqi.info/entityv2.github.io/}{http://luqi.info/entityv2.github.io/}.
\end{abstract}

\vspace{-4mm}
\paragraph{Crop Dataloader} In CropFormer, the crop dataloader is designed to generate a batch of images that simultaneously include the full images and their corresponding crops. There are two steps in our dataloader: \textit{crop} and \textit{resize}. At first, we augment the full image $I^o$ with a crop that is part of the full image. The 
crop is randomly extracted from one of the fixed image corners: upper-left, upper-right, bottom-left, bottom-right. The crop size is controlled by a fixed ratio hyperparameter $\delta \in \mathbb{R}$ relative to the full image size. Second, we resize both the full image and crop to the same size. In this way, we construct an input batch $\mathbf{I} \in \mathbb{R}^{N \times 2 \times H_\mathbf{I} \times W_\mathbf{I} \times 3}$ with 2 elements (full image $\&$ 1 crop) along the temporal dimension. Compared to the full image, corner crops preserve more image details and local information useful for fine-grained segmentation.
Given dataloader $\Gamma$, we define the data preprocessing as

\vspace{-4mm}
\paragraph{Image Encoder and Decoder} Based on learnable image-level queries $\mathbf{Q}_{i} \in \mathbb{R}^{N\times K}$,  we use image encoder $\Theta$ and decoder $\Phi_{i}$ to generate image-level embeddings $\mathbf{E_i} \in \mathbb{R}^{N \times 2 \times 1 \times 1 \times K}$ for the full input image and its crop. 
For the image encoder and decoder, we directly adopt the ones in standard image-level Mask2Former~\cite{cheng2022masked} and do not modify their structures or designs. $\mathbf{E_i}$ is obtained as follows:

\vspace{-4mm}
\paragraph{Association Module} Similar to video-level Mask2Former~\cite{cheng2021mask2former}, the association module aims to generate batch queries $\mathbf{Q_b}$ that are fully shared by the full image and its crop to represent the same entities consistently. Instead of treating $\mathbf{Q_b}$ as learnable network parameters, here we generate $\mathbf{Q_b}$ directly from $\mathbf{E_i}=\{\mathbf{E}_{I^o}, \mathbf{E}_{I^c})\}$, considering that $\mathbf{E_i}$ already contains strong segmentation features. 
In particular, we use a Transformer architecture with cross attention (XAtt), self-attention (SAtt), and feedforward network (FFN) to obtain $\mathbf{Q_b}$:

\vspace{-4mm}
\paragraph{Batch Decoder}
Given $\mathbf{Q_b} \in \mathbb{R}^{N \times 1 \times 1 \times 1 \times K}$, we obtain batch embeddings $\mathbf{E_b} \in \mathbb{R}^{N \times 1 \times 1 \times 1 \times K}$:

\vspace{-4mm}
\paragraph{Instance Segmentation}
We select 206 thing categories with the highest object-level frequency in the EntityClass dataset to benchmark instance segmentation. In the EntityIns, 8,993 and 1,498 images for training and testing, respectively. In Table~\ref{Tab:aba_benchmark_classaware_ins}, we ablate two popular instance segmentation methods including Mask-RCNN~\cite{he2017mask} and Mask2Former~\cite{cheng2022masked} on EntityIns. Mask2Former with Swin-L backbone and COCO-P pretrained weights the best AP of 30.3 on EntityIns. 

Table~\ref{Tab:aba_ivresults} shows the overall improvement of CropFromer to the Mask2Former~\cite{cheng2022masked} under 1$\times$ training setting. The first and second row is our baseline Mask2Former with the single-scale inference (800 and 1040 shortest side) and (1333 and 1732 longest side) for the full images, where 1040=800$\times\delta$ and 1732=1333$\times\delta$ with $\delta$ 0.7 in default. For the results in the third and fourth rows, we use test-time bipartite-matching to associate the same entities obtained with multi-scale images and crops. We find no improvements with such inference strategies. Whereas, using the crop output from CropFormer's batch decoder achieves a significant AP$^\text{e}$ gain as indicated by the second last row. By combining the full image and four crop outputs from CropFormer (final row), we obtain even stronger 1.5 AP$^\text{e}$ and 1.7 AP$^\text{e}_{75}$ gain compared to the baseline (first row).

\vspace{-4mm}
\paragraph{Other Batch Fusion Design} Table~\ref{Tab:aba_fusion_style} shows the ablation study with different fusion designs between original images and cropped patches. Directly using video-level Mask2former or VITA fusion method merely brings marginal performance gains.
{"value": "39.7", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "VI-Mask2Former", "model settings": {"Type": "Parallel", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "57.3", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "VI-Mask2Former", "model settings": {"Type": "Parallel", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "40.6", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "VI-Mask2Former", "model settings": {"Type": "Parallel", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "39.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "VITA", "model settings": {"Type": "Cascade", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "57.8", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "VITA", "model settings": {"Type": "Cascade", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "40.7", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "VITA", "model settings": {"Type": "Cascade", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "41.0", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "Ours", "model settings": {"Type": "Cascade", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "58.4", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{50}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "Ours", "model settings": {"Type": "Cascade", "backbone": "Swin-L", "pretrained weights": "COCO-E"}} {"value": "41.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^{e}_{75}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "Ours", "model settings": {"Type": "Cascade", "backbone": "Swin-L", "pretrained weights": "COCO-E"}}