\begin{tabular}{c|c}
            \cellcolor{lightgray!30} $\delta$ & \cellcolor{lightgray!30} AP$^\text{e}$ \\ \hline
            0.5 & 38.5 \\ 
            0.6 & 40.2  \\ 
            0.7 & \textbf{41.0}  \\ 
            0.8 & 40.9 \\ \hline
            \end{tabular}
\vspace{-4mm}
\paragraph{Crop Dataloader} In CropFormer, the crop dataloader is designed to generate a batch of images that simultaneously include the full images and their corresponding crops. There are two steps in our dataloader: \textit{crop} and \textit{resize}. At first, we augment the full image $I^o$ with a crop that is part of the full image. The 
crop is randomly extracted from one of the fixed image corners: upper-left, upper-right, bottom-left, bottom-right. The crop size is controlled by a fixed ratio hyperparameter $\delta \in \mathbb{R}$ relative to the full image size. Second, we resize both the full image and crop to the same size. In this way, we construct an input batch $\mathbf{I} \in \mathbb{R}^{N \times 2 \times H_\mathbf{I} \times W_\mathbf{I} \times 3}$ with 2 elements (full image $\&$ 1 crop) along the temporal dimension. Compared to the full image, corner crops preserve more image details and local information useful for fine-grained segmentation.
Given dataloader $\Gamma$, we define the data preprocessing as

\vspace{-4mm}
\paragraph{Association Module} Similar to video-level Mask2Former~\cite{cheng2021mask2former}, the association module aims to generate batch queries $\mathbf{Q_b}$ that are fully shared by the full image and its crop to represent the same entities consistently. Instead of treating $\mathbf{Q_b}$ as learnable network parameters, here we generate $\mathbf{Q_b}$ directly from $\mathbf{E_i}=\{\mathbf{E}_{I^o}, \mathbf{E}_{I^c})\}$, considering that $\mathbf{E_i}$ already contains strong segmentation features. 
In particular, we use a Transformer architecture with cross attention (XAtt), self-attention (SAtt), and feedforward network (FFN) to obtain $\mathbf{Q_b}$:

\vspace{-4mm}
\paragraph{Semantic Segmentation} We choose 150 categories with the highest pixel-level frequency as EntitySem for semantic segmentation. EntitySem has 9,729 and 1,444 images for training and testing, respectively. In Table~\ref{Tab:aba_benchmark_classaware_sem}, we evaluate the two popular semantic segmentation methods DeeplabV3~\cite{chen2017deeplab} and Mask2Former~\cite{cheng2022masked} on EntitySem. Semantic segmentation performance is still related to the pretraining weights and network structure. Mask2Former with Swin-L backbone and COCO-E pretrained weights obtains the best mIoU of 50.5 on EntitySem. 

\vspace{-4mm}
\paragraph{Panoptic Segmentation} Similar to EntityIns, we select 345 categories, including 274 things and 71 stuff, with the highest entity-level frequency to construct EntityPan. In EntityPan, there are 9,968 and 1,481 images for training and testing. Table~\ref{Tab:aba_benchmark_classaware_panop} shows the performance of two popular panoptic segmentation methods, including PanopticFPN~\cite{kirillov2019panopticfpn} and Mask2Former~\cite{cheng2022masked}. In this table, we see that the PQs are much lower than those of existing panoptic datasets, due to the greater variety of class labels which makes the task more challenging.
In addition, current panoptic methods perform worse on EntityPan compared to existing datasets, particularly on the recognition quality (RQ) side which adversely impacts PQ.

Table~\ref{Tab:aba_ivresults} shows the overall improvement of CropFromer to the Mask2Former~\cite{cheng2022masked} under 1$\times$ training setting. The first and second row is our baseline Mask2Former with the single-scale inference (800 and 1040 shortest side) and (1333 and 1732 longest side) for the full images, where 1040=800$\times\delta$ and 1732=1333$\times\delta$ with $\delta$ 0.7 in default. For the results in the third and fourth rows, we use test-time bipartite-matching to associate the same entities obtained with multi-scale images and crops. We find no improvements with such inference strategies. Whereas, using the crop output from CropFormer's batch decoder achieves a significant AP$^\text{e}$ gain as indicated by the second last row. By combining the full image and four crop outputs from CropFormer (final row), we obtain even stronger 1.5 AP$^\text{e}$ and 1.7 AP$^\text{e}_{75}$ gain compared to the baseline (first row).

\vspace{-4mm}
\paragraph{Other Batch Fusion Design} Table~\ref{Tab:aba_fusion_style} shows the ablation study with different fusion designs between original images and cropped patches. Directly using video-level Mask2former or VITA fusion method merely brings marginal performance gains.

\vspace{-4mm}
\paragraph{CropDataloader} Table~\ref{abl_1}(a) and (b) show the ablation study on the usage of crop ratio and crop type. As indicated by Table~\ref{abl_1}(a), CropFormer obtains the best performance with a crop ratio $\delta$ of 0.7. Smaller crop ratios perform worse and it might be a problem caused by queries that are not robust against drastic image transformation, which we leave for future work. Table~\ref{abl_1}(b) shows the impacts of the type of crop and the number of crops. 4 and 8 fixed crops during training and testing performs the best.

\vspace{-4mm}
\paragraph{Association Module} Table~\ref{abl_1}(c) shows the ablation study on the structure of the association module. There is a slight performance difference between using self-attention or a feedforward module following cross-attention.
{"value": "0.5", "type": "Hyper-parameter/Architecture", "model": "CropFormer", "parameter/architecture name": "crop ratio", "dataset": "EntitySeg"} {"value": "38.5", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"xx": "yy"}, "model": "CropFormer", "model settings": {"crop ratio": "0.5"}} {"value": "0.6", "type": "Hyper-parameter/Architecture", "model": "CropFormer", "parameter/architecture name": "crop ratio", "dataset": "EntitySeg"} {"value": "40.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"xx": "yy"}, "model": "CropFormer", "model settings": {"crop ratio": "0.6"}} {"value": "0.7", "type": "Hyper-parameter/Architecture", "model": "CropFormer", "parameter/architecture name": "crop ratio", "dataset": "EntitySeg"} {"value": "41.0", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"xx": "yy"}, "model": "CropFormer", "model settings": {"crop ratio": "0.7"}} {"value": "0.8", "type": "Hyper-parameter/Architecture", "model": "CropFormer", "parameter/architecture name": "crop ratio", "dataset": "EntitySeg"} {"value": "40.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"xx": "yy"}, "model": "CropFormer", "model settings": {"crop ratio": "0.8"}}