\begin{tabular}{c|c|c}
            \cellcolor{lightgray!30} Training & \cellcolor{lightgray!30} Baseline &  \cellcolor{lightgray!30} Ours  \\  \hline
            1$\times$ & 39.5 & 41.0 \textcolor[RGB]{34,139,34}{(+1.5)}  \\ 
            2$\times$ & 40.2 & 42.1 \textcolor[RGB]{34,139,34}{(+1.9)}  \\ 
            3$\times$ & 40.9 & 42.8 \textcolor[RGB]{34,139,34}{(+1.9)}\\ 
            4$\times$ & 40.9 & 42.7 \textcolor[RGB]{34,139,34}{(+1.8)} \\ \hline
            \end{tabular}
\subsection{Mask2Former Preliminaries}
Mask2Former~\cite{cheng2022masked} is a universal Transformer-based framework for image or video instance segmentation. Given an input, Mask2Former uses a Transformer decoder to decode $N$ number of $K$-dimensional queries $\mathbf{Q}\in \mathbb{R}^{N\times K}$ to generate embeddings $\mathbf{E} \in \mathbb{R}^{N \times 1 \times 1 \times 1 \times K}$ to predict $N$ segmentation masks, where the \nth{2}, \nth{3}, and \nth{4} dimensions of $\mathbf{E}$ correspond to temporal ($T$), output height ($H$), and output width  ($W$) dimensions respectively. When applying the image-level Mask2Former to an image input, $\mathbf{E}$ is broadcasted along spatial dimensions to the shape of $N \times 1 \times H \times W \times K$. Alternatively, when applying video-level Mask2Former to a video input, $\mathbf{E}$ is broadcasted to the shape of $N \times T \times H \times W \times K$. Note that in video-level Mask2Former, $\mathbf{E}$ is further broadcasted along the both temporal and spatial dimensions, enabling Mask2Former to utilize the same embeddings to represent the same visual instances across different frames consistently.

\vspace{-4mm}
\paragraph{Crop Dataloader} In CropFormer, the crop dataloader is designed to generate a batch of images that simultaneously include the full images and their corresponding crops. There are two steps in our dataloader: \textit{crop} and \textit{resize}. At first, we augment the full image $I^o$ with a crop that is part of the full image. The 
crop is randomly extracted from one of the fixed image corners: upper-left, upper-right, bottom-left, bottom-right. The crop size is controlled by a fixed ratio hyperparameter $\delta \in \mathbb{R}$ relative to the full image size. Second, we resize both the full image and crop to the same size. In this way, we construct an input batch $\mathbf{I} \in \mathbb{R}^{N \times 2 \times H_\mathbf{I} \times W_\mathbf{I} \times 3}$ with 2 elements (full image $\&$ 1 crop) along the temporal dimension. Compared to the full image, corner crops preserve more image details and local information useful for fine-grained segmentation.
Given dataloader $\Gamma$, we define the data preprocessing as

\vspace{-4mm}
\paragraph{Image Encoder and Decoder} Based on learnable image-level queries $\mathbf{Q}_{i} \in \mathbb{R}^{N\times K}$,  we use image encoder $\Theta$ and decoder $\Phi_{i}$ to generate image-level embeddings $\mathbf{E_i} \in \mathbb{R}^{N \times 2 \times 1 \times 1 \times K}$ for the full input image and its crop. 
For the image encoder and decoder, we directly adopt the ones in standard image-level Mask2Former~\cite{cheng2022masked} and do not modify their structures or designs. $\mathbf{E_i}$ is obtained as follows:

Similar to video-level Mask2Former, the full image and its crop can be seen as 2 video frames. Thus, we broadcast $\mathbf{E_b}$ to the shape of $N \times 2 \times 1 \times 1 \times K$ to share it between the full image and its crop. $\mathbf{E_b}$ and batch-level pyramid features $\mathbf{P_b^h}\in\mathbb{R}^{1\times2\times H^h\times W^h}$ (reshaped from $\mathbf{P_i^h}\in\mathbb{R}^{2\times1\times H^h\times W^h}$) are used for batch-level mask classification and pixel-wise prediction:

Table~\ref{Tab:aba_ivresults} shows the overall improvement of CropFromer to the Mask2Former~\cite{cheng2022masked} under 1$\times$ training setting. The first and second row is our baseline Mask2Former with the single-scale inference (800 and 1040 shortest side) and (1333 and 1732 longest side) for the full images, where 1040=800$\times\delta$ and 1732=1333$\times\delta$ with $\delta$ 0.7 in default. For the results in the third and fourth rows, we use test-time bipartite-matching to associate the same entities obtained with multi-scale images and crops. We find no improvements with such inference strategies. Whereas, using the crop output from CropFormer's batch decoder achieves a significant AP$^\text{e}$ gain as indicated by the second last row. By combining the full image and four crop outputs from CropFormer (final row), we obtain even stronger 1.5 AP$^\text{e}$ and 1.7 AP$^\text{e}_{75}$ gain compared to the baseline (first row).

\vspace{-4mm}
\paragraph{CropDataloader} Table~\ref{abl_1}(a) and (b) show the ablation study on the usage of crop ratio and crop type. As indicated by Table~\ref{abl_1}(a), CropFormer obtains the best performance with a crop ratio $\delta$ of 0.7. Smaller crop ratios perform worse and it might be a problem caused by queries that are not robust against drastic image transformation, which we leave for future work. Table~\ref{abl_1}(b) shows the impacts of the type of crop and the number of crops. 4 and 8 fixed crops during training and testing performs the best.

\vspace{-4mm}
\paragraph{Longer Training Schedules} 
Table~\ref{abl_2}(a) indicates that CropFormer consistently improves over the baseline Mask2Former when we adopt 2$\times$/3$\times$ 
training schedules. A similar trend is observed when using Swin-L~\cite{liu2021swin} backbone, which is provided in our supplementary file.

\vspace{-4mm}
\paragraph{Improvement on class-aware segmentation tasks} 
Table~\ref{abl_2}(b) shows the 
class-aware task performance of CropFromer with Swin-T~\cite{liu2021swin} backbone and 3$\times$ training schedule. Compared to the class-agnostic entity segmentation tasks, CropFormer provides smaller gains on class-aware segmentation task, suggesting that CropFormer mainly benefits mask prediction rather than class prediction.
{"value": "39.5", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrained weights": "COCO-E"}} {"value": "41.0", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"training schedule": "1$\\times$"}, "model": "CropFormer", "model settings": {"backbone": "Swin-T", "pretrained weights": "COCO-E"}} {"value": "40.2", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"training schedule": "2$\\times$"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrained weights": "COCO-E"}} {"value": "42.1", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"training schedule": "2$\\times$"}, "model": "CropFormer", "model settings": {"backbone": "Swin-T", "pretrained weights": "COCO-E"}} {"value": "40.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"training schedule": "3$\\times$"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrained weights": "COCO-E"}} {"value": "42.8", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"training schedule": "3$\\times$"}, "model": "CropFormer", "model settings": {"backbone": "Swin-T", "pretrained weights": "COCO-E"}} {"value": "40.9", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"training schedule": "4$\\times$"}, "model": "Mask2Former", "model settings": {"backbone": "Swin-T", "pretrained weights": "COCO-E"}} {"value": "42.7", "type": "Result", "training data/set": "EntitySeg", "test data/set": "EntitySeg", "task": "Entity Segmentation", "metric": "AP$^\\text{e}$", "experimental settings": {"training schedule": "4$\\times$"}, "model": "CropFormer", "model settings": {"backbone": "Swin-T", "pretrained weights": "COCO-E"}}