\begin{table}[t!]
\centering
\scriptsize
\begin{tabular}{c|c|c|c}
\cellcolor{lightgray!30} Model & \cellcolor{lightgray!30} Backbone & \cellcolor{lightgray!30} Pretrain & \cellcolor{lightgray!30} mIoU \\ \hline
DeeplabV3~\cite{chen2017rethinking} & R-50 & ImageNet & 27.9  \\ \hline
\multirow{6}*{Mask2Former~\cite{cheng2022masked}} & \multirow{2}*{R-50} & ImageNet & 37.8 \\ 
& & COCO-P & 43.3  \\ \cline{2-4}
& \multirow{2}*{Swin-T} & COCO-E & 43.0  \\ 
& & COCO-P & 45.0 \\ \cline{2-4}
& \multirow{2}*{Swin-L} & COCO-E & 50.7  \\ 
& & COCO-P & 50.5 \\ \hline
\end{tabular}
\caption{Benchmark on class-aware semantic segmentation in EntitySem Dataset. The `COCO-P' and `COCO-E' indicate weights trained in
the COCO datasets with panoptic and entity
segmentation tasks. }
\vspace{-0.1in}
\label{Tab:aba_benchmark_classaware_sem}
\end{table}
\begin{abstract}
In dense image segmentation tasks (\textit{e.g.,} semantic, panoptic), existing methods can hardly generalize well to unseen image domains, predefined classes, and image resolution \& quality variations.
Motivated by these observations, we construct a large-scale entity segmentation dataset to explore fine-grained entity segmentation, with a strong focus on open-world and high-quality dense segmentation. 
The dataset contains images spanning diverse image domains and resolutions, along with high-quality mask annotations for training and testing. 
Given the high-quality and -resolution nature of the dataset, we propose CropFormer for high-quality segmentation, which can improve mask prediction using high-res image crops that provide more fine-grained image details than the full image. CropFormer is the first query-based Transformer architecture that can effectively ensemble mask predictions from multiple image crops, by learning queries that can associate the same entities across the full image and its crop. With CropFormer, we achieve a significant AP gain of $1.9$ on the challenging fine-grained entity segmentation task. The dataset and code will be released at \href{http://luqi.info/entityv2.github.io/}{http://luqi.info/entityv2.github.io/}.
\end{abstract}

\section{Introduction}
Dense image segmentation is an important topic in computer vision with a wide range of applications such as autonomous driving~\cite{qi2019amodal}, robotics~\cite{shu2014human}, video surveillance~\cite{morrison2018cartman}, and image editing tools. 
Recently, entity segmentation~\cite{qi2021open} has been introduced as a promising dense instance-aware approach for tackling generalization issues caused by narrow image domains and/or predefined class sets in existing panoptic~\cite{kirillov2019panoptic,kirillov2019panopticfpn,xiong2019upsnet,li2020fully,carion2020end} 
and semantic~\cite{long2015fully,zhao2017pyramid,chen2017rethinking,chen2018encoder,zhao2018psanet,huang2019ccnet,liu2019auto} segmentation tasks.
Although the early method~\cite{qi2021open} shows some promising results, it merely borrows existing panoptic segmentation datasets to demonstrate the feasibility and benefits of the entity segmentation. 
However, existing panoptic segmentation datasets (\textit{e.g,} COCO~\cite{lin2014microsoft}, Cityscapes~\cite{cordts2016cityscapes}, ADE20K~\cite{zhou2017scene}) 
suffer from the same issues that entity segmentation~\cite{qi2021open} aims to avoid, since those datasets are collected and annotated with 
panoptic segmentation task requirements in mind.

\subsection{Image Collection}
Inspired by the collection criterion of COCO~\cite{lin2014microsoft}, we collect most non-iconic images with more contextual information and non-canonical viewpoints. 
In addition, the image domains should be as diversified as possible to guarantee substantial domain diversity. Therefore, the image sources of our dataset include several public datasets and the Internet where the images permitted for academic research use. For public dataset sources, we select part of the images from
COCO~\cite{lin2014microsoft}, ADE20K~\cite{zhou2017scene}, 
Pascal VOC~\cite{everingham2010pascal}, Cityscapes~\cite{cordts2016cityscapes}, 
Mapillary Vistas~\cite{neuhold2017mapillary}, ImageNet~\cite{krizhevsky2017imagenet}, 
Open Images~\cite{kuznetsova2020open}, DIV2K~\cite{agustsson2017ntire}, Flick2K~\cite{wang2019flickr1024}, Clipart~\cite{inoue2018cross},
Comic~\cite{inoue2018cross}, DOTA~\cite{xia2018dota} and some computer game screenshots from Synscapes~\cite{wrenninge2018synscapes} and GTA5~\cite{richter2016playing}. From the Internet, we crawled mainly high-resolution images from Pixabay~\cite{pixabay}, Unsplash~\cite{Unsplash}, and LAION-400M~\cite{schuhmann2021laion}. In Figure~\ref{fig:statistics_01}(c), we show the distribution of image sources in our dataset and see that most of the images are from high-resolution sources like Pixabay, Unsplash, and LAION-400M. In Figure~\ref{fig:statistics_01}(a), our dataset has more high-resolution images whose resolutions are normally distributed in the resolution range between 0px to 15,000px, whereas all the images of ADE20K~\cite{zhou2017scene} and COCO~\cite{lin2014microsoft} are under 1000px.

\vspace{-4mm}
\paragraph{Crop Dataloader} In CropFormer, the crop dataloader is designed to generate a batch of images that simultaneously include the full images and their corresponding crops. There are two steps in our dataloader: \textit{crop} and \textit{resize}. At first, we augment the full image $I^o$ with a crop that is part of the full image. The 
crop is randomly extracted from one of the fixed image corners: upper-left, upper-right, bottom-left, bottom-right. The crop size is controlled by a fixed ratio hyperparameter $\delta \in \mathbb{R}$ relative to the full image size. Second, we resize both the full image and crop to the same size. In this way, we construct an input batch $\mathbf{I} \in \mathbb{R}^{N \times 2 \times H_\mathbf{I} \times W_\mathbf{I} \times 3}$ with 2 elements (full image $\&$ 1 crop) along the temporal dimension. Compared to the full image, corner crops preserve more image details and local information useful for fine-grained segmentation.
Given dataloader $\Gamma$, we define the data preprocessing as

\vspace{-4mm}
\paragraph{Image Encoder and Decoder} Based on learnable image-level queries $\mathbf{Q}_{i} \in \mathbb{R}^{N\times K}$,  we use image encoder $\Theta$ and decoder $\Phi_{i}$ to generate image-level embeddings $\mathbf{E_i} \in \mathbb{R}^{N \times 2 \times 1 \times 1 \times K}$ for the full input image and its crop. 
For the image encoder and decoder, we directly adopt the ones in standard image-level Mask2Former~\cite{cheng2022masked} and do not modify their structures or designs. $\mathbf{E_i}$ is obtained as follows:

\vspace{-4mm}
\paragraph{Association Module} Similar to video-level Mask2Former~\cite{cheng2021mask2former}, the association module aims to generate batch queries $\mathbf{Q_b}$ that are fully shared by the full image and its crop to represent the same entities consistently. Instead of treating $\mathbf{Q_b}$ as learnable network parameters, here we generate $\mathbf{Q_b}$ directly from $\mathbf{E_i}=\{\mathbf{E}_{I^o}, \mathbf{E}_{I^c})\}$, considering that $\mathbf{E_i}$ already contains strong segmentation features. 
In particular, we use a Transformer architecture with cross attention (XAtt), self-attention (SAtt), and feedforward network (FFN) to obtain $\mathbf{Q_b}$:

Table~\ref{Tab:aba_benchmark_entity} shows the benchmark of our dataset with Mask-RCNN~\cite{he2017mask}, EntityFramework~\cite{qi2021open}, Mask2Former~\cite{cheng2022masked}. The first two methods are convolution-based dense prediction methods, whereas the last is a transformer-based query prediction method. In this table, the transformer-based Mask2Former is better than the other two convolution-based methods, demonstrating the advantage of transform-based methods on high-quality mask generation, Moreover, the COCO-E pretrained weights can further boost entity segmentation performance. We also explored the optimal training iterations needed by Mask2Former and found that it performs the best with 3$\times$ training schedule.

\vspace{-4mm}
\paragraph{Semantic Segmentation} We choose 150 categories with the highest pixel-level frequency as EntitySem for semantic segmentation. EntitySem has 9,729 and 1,444 images for training and testing, respectively. In Table~\ref{Tab:aba_benchmark_classaware_sem}, we evaluate the two popular semantic segmentation methods DeeplabV3~\cite{chen2017deeplab} and Mask2Former~\cite{cheng2022masked} on EntitySem. Semantic segmentation performance is still related to the pretraining weights and network structure. Mask2Former with Swin-L backbone and COCO-E pretrained weights obtains the best mIoU of 50.5 on EntitySem. 
{"value": "27.9", "type": "Result", "training data/set": "EntitySem", "test data/set": "EntitySem", "task": "Semantic Segmentation", "metric": "mIoU", "experimental settings": {"xx": "yy"}, "model": "DeeplabV3", "model settings": {"Backbone": "R-50", "Pretrain": "ImageNet"}} {"value": "37.8", "type": "Result", "training data/set": "EntitySem", "test data/set": "EntitySem", "task": "Semantic Segmentation", "metric": "mIoU", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"Backbone": "R-50", "Pretrain": "ImageNet"}} {"value": "43.3", "type": "Result", "training data/set": "EntitySem", "test data/set": "EntitySem", "task": "Semantic Segmentation", "metric": "mIoU", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"Backbone": "R-50", "Pretrain": "COCO-P"}} {"value": "43.0", "type": "Result", "training data/set": "EntitySem", "test data/set": "EntitySem", "task": "Semantic Segmentation", "metric": "mIoU", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"Backbone": "Swin-T", "Pretrain": "COCO-E"}} {"value": "45.0", "type": "Result", "training data/set": "EntitySem", "test data/set": "EntitySem", "task": "Semantic Segmentation", "metric": "mIoU", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"Backbone": "Swin-T", "Pretrain": "COCO-P"}} {"value": "50.7", "type": "Result", "training data/set": "EntitySem", "test data/set": "EntitySem", "task": "Semantic Segmentation", "metric": "mIoU", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"Backbone": "Swin-L", "Pretrain": "COCO-E"}} {"value": "50.5", "type": "Result", "training data/set": "EntitySem", "test data/set": "EntitySem", "task": "Semantic Segmentation", "metric": "mIoU", "experimental settings": {"xx": "yy"}, "model": "Mask2Former", "model settings": {"Backbone": "Swin-L", "Pretrain": "COCO-P"}}