\begin{table}[t]\small
	\centering
	\renewcommand{\arraystretch}{0.93}
	\footnotesize
    \setlength\tabcolsep{2.4mm}{
    \begin{tabular}{l|cc|cccccc|cc}
        Method & \#Params & \#FLOPs & $\rm AP^b$ & $\rm AP^m$ \\
        \hline
        Swin-L$^\ddagger$~\cite{liu2021swin} & 253M & 1382G & 51.8 & 44.9  \\
        ConvNeXt-L$^\ddagger$~\cite{liu2022convnet} & 255M & 1354G & 53.5 & 46.4 \\
        \rowcolor{gray!20}
        InternImage-L$^\ddagger$ (ours) & 277M & 1399G & 54.9 & 47.7  \\
        \hline
        ConvNeXt-XL$^\ddagger$~\cite{liu2022convnet} & 407M & 1898G & 53.6 & 46.5 \\
        \rowcolor{gray!20}
        InternImage-XL$^\ddagger$ (ours) & 387M & 1782G & 55.3 & 48.0  \\
    \end{tabular}}
    
    \caption{
    \textbf{Object detection and instance segmentation with Cascade Mask R-CNN 1$\times$ schedule on COCO \texttt{val2017}.}
    The FLOPs are measured with 1280$\times$800 inputs.
	}
    \label{tab:results_detection_cascade}
    \label{tab:mask}
\end{table}
\author{
    Wenhai Wang$^{1*}$, 
    Jifeng Dai$^{2,1*}$,
    Zhe Chen$^{3,1*}$,
    Zhenhang Huang$^{1*}$,
    Zhiqi Li$^{3,1*}$,
    Xizhou Zhu$^{4*}$,\\
    Xiaowei Hu$^{1}$,
    Tong Lu$^{3}$,
    Lewei Lu$^{4}$,
    Hongsheng Li$^{5}$,
    Xiaogang Wang$^{3,5}$,
    Yu Qiao$^{1}$\textsuperscript{\Letter}\\
    $^1$ Shanghai AI Laboratory~~~
    $^2$Tsinghua University~~~\\
    $^3$Nanjing University~~~
    $^4$SenseTime Research~~~
    $^5$The Chinese University of Hong Kong\\
    {\small \url{https://github.com/OpenGVLab/InternImage}}
}

To bridge the gap between CNNs and ViTs, we first summarize their differences from two aspects:
(1) From the operator level~\cite{dosovitskiy2020image,liu2022convnet,ding2022replknet}, the multi-head self-attention (MHSA) of ViTs has long-range dependencies and adaptive spatial aggregation (see Fig. \ref{fig:coreop}(a)).
Benefiting from the flexible MHSA,
ViTs can learn more powerful and robust representations than CNNs from massive data.
(2) From the architecture view~\cite{dosovitskiy2020image,ding2022replknet,yu2022metaformer}, besides MHSA, ViTs contain a series of advanced components that are not included in standard CNNs, such as Layer Normalization (LN)~\cite{ba2016layernorm}, feed-forward network (FFN)~\cite{vaswani2017attention}, GELU~\cite{hendrycks2016gelu}, etc.
Although recent works~\cite{liu2022convnet,ding2022replknet} have made meaningful attempts to introduce long-range dependencies into CNNs by using dense convolutions with very large kernels (\emph{e.g.}, 31$\times$31) as shown in Fig. \ref{fig:coreop} (c),
there is still a considerable gap with the state-of-the-art large-scale ViTs~\cite{zhai2022scalingvit,dai2021coatnet,liu2021swinv2,riquelme2021vmoe,wei2022fdswin} in terms of performance and model scale.

where $K$ represents the total number of sampling points, and $k$ enumerates the sampling point. 
$\mathbf{w}_k\!\in\!\mathbb{R}^{C\times C}$ denotes the projection weights of the $k$-th sampling point,
and $\mathbf{m}_k\!\in\!\mathbb{R}$ represents the modulation scalar of the $k$-th sampling point, which is normalized by sigmoid function.
$p_k$ denotes the $k$-th location of the pre-defined grid sampling $\{(-1, -1), (-1, 0), ..., (0, +1), ..., (+1, +1)\}$ as in regular convolutions,
and $\Delta p_k$ is the offset corresponding to the $k$-th grid sampling location.
We see from the equation that (1) for long-range dependencies, the sampling offset $\Delta p_k$ is flexible and able to interact with short- or long-range features; and (2) for adaptive spatial aggregation, both the sampling offset $\Delta p_k$ and modulation scalar $\mathbf{m}_k$ are learnable and conditioned by input $\mathbf{x}$. 
So it can be found that \emph{DCNv2 shares similar favorable properties with MHSA}, which motivated us to develop large-scale CNN-based foundation models on the basis of this operator.

where $G$ denotes the total number of aggregation groups.
For the $g$-th group,
$\mathbf{w}_g\!\in\!\mathbb{R}^{C\times C'}$, $\mathbf{m}_{gk}\!\in\!\mathbb{R}$ denote the location-irrelevant projection weights of the group, where $C'\!=\!C/G$ represents the group dimension.
$\mathbf{m}_{gk}\!\in\!\mathbb{R}$ denotes the modulation scalar of the $k$-th sampling point in the $g$-th group, normalized by the softmax function along the dimension $K$.
$\mathbf{x}_g\!\in\!\mathbb{R}^{C'\times H \times W}$ represents the sliced input feature map.
$\Delta p_{gk}$ is the offset corresponding to the grid sampling location $p_k$ in the $g$-th group.

\textbf{Basic block.}
Unlike the widely used bottlenecks in traditional CNNs~\cite{he2016deep}, the design of our basic block is closer to ViTs, which is equipped with more advanced components including LN~\cite{ba2016layernorm}, feed-forward networks (FFN)~\cite{vaswani2017attention}, and GELU~\cite{hendrycks2016gelu}.
This design is proved to be efficient~\cite{wang2021pyramid,wang2021pvtv2,liu2021swin,liu2022convnet,ding2022replknet} in various vision tasks.
The details of our basic block are illustrated in Fig. \ref{fig:arch}, where the core operator is DCNv3,
and the sampling offsets and modulation scales are predicted by passing input feature $\mathbf{x}$ through a separable convolution (a 3$\times$3 depth-wise convolution followed by a linear projection).
For other components, we use the post-normalization setting~\cite{xiong2020layer} by default and follow the same design as that of the plain transformer~\cite{vaswani2017attention,dosovitskiy2020image}.

where $\alpha\!\geq\!1$, $\beta\!\geq\!1$, and $\alpha\beta^{1.99}\!\approx\!2$. Here, 1.99 is specific for InternImage and calculated by doubling the model width and keeping the depth constant.
We experimentally find out that the best scaling setting is $\alpha\!=\!1.09$ and $\beta\!=\!1.36$, and then we base on it to construct InternImage\ variants with different parameter scales, namely InternImage-T/S/B/L/XL, whose complexity is similar to those of ConvNeXt~\cite{liu2022convnet}. 
To further test the capability, we built a larger InternImage-H with 1 billion parameters.
The detailed configurations are summarized in Table~\ref{tab:model_size}.

\textbf{Settings.}
We verify the effectiveness of our InternImage\ on the COCO benchmark~\cite{lin2014microsoft}, on top of two representative object detection frameworks: Mask R-CNN~\cite{he2017mask}, and Cascade Mask R-CNN~\cite{cai2018cascade}.
We follow common practices~\cite{liu2021swin,wang2021pvtv2} to initialize the backbone with pre-trained classification weights, and by default use a 1$\times$ (12 epochs) or 3$\times$ (36 epochs) training schedule to train detection models.

\textbf{Results.}
As shown in Table~\ref{tab:results_detection_mask}, when using Mask R-CNN for object detection, 
we find that under a comparable number of parameters, our models significantly surpass their counterparts. 
For example, with the $1\times$ training schedule, the box AP (AP$^{\rm b}$) of InternImage-T is 4.5 points better than Swin-T~\cite{liu2021swin} (47.2 \vs 42.7),
and 3.0 points higher than ConvNeXt-T~\cite{liu2022convnet} (47.2 \vs 44.2).
As reported in Table~\ref{tab:results_detection_cascade}, with more parameters and more advanced Cascade Mask R-CNN, InternImage-XL achieves AP$^{\rm b}$ of 55.3, surpassing ConvNeXt-XL by 2.7 points (55.3 \vs 53.6).
{"value": "253M", "type": "Hyper-parameter/Architecture", "model": "Swin-L", "parameter/architecture name": ["# Params", "number of parameters"], "dataset": "COCO"} {"value": "1382G", "type": "Hyper-parameter/Architecture", "model": "Swin-L", "parameter/architecture name": ["# FLOPs", "number of FLOPs"], "dataset": "COCO"} {"value": "51.8", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "object detection", "metric": ["AP^b", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "Swin-L", "model settings": {"backbone": "Mask R-CNN"}} {"value": "44.9", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "instance segmentation", "metric": ["AP^m", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "Swin-L", "model settings": {"backbone": "Mask R-CNN"}} {"value": "255M", "type": "Hyper-parameter/Architecture", "model": "ConvNeXt-L", "parameter/architecture name": ["# Params", "number of parameters"], "dataset": "COCO"} {"value": "1354G", "type": "Hyper-parameter/Architecture", "model": "ConvNeXt-L", "parameter/architecture name": ["# FLOPs", "number of FLOPs"], "dataset": "COCO"} {"value": "53.5", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "object detection", "metric": ["AP^b", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "ConvNeXt-L", "model settings": {"backbone": "Mask R-CNN"}} {"value": "46.4", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "instance segmentation", "metric": ["AP^m", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "ConvNeXt-L", "model settings": {"backbone": "Mask R-CNN"}} {"value": "277M", "type": "Hyper-parameter/Architecture", "model": "InternImage-L", "parameter/architecture name": ["# Params", "number of parameters"], "dataset": "COCO"} {"value": "1399G", "type": "Hyper-parameter/Architecture", "model": "InternImage-L", "parameter/architecture name": ["# FLOPs", "number of FLOPs"], "dataset": "COCO"} {"value": "54.9", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "object detection", "metric": ["AP^b", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "InternImage-L", "model settings": {"backbone": "Mask R-CNN"}} {"value": "47.7", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "instance segmentation", "metric": ["AP^m", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "InternImage-L", "model settings": {"backbone": "Mask R-CNN"}} {"value": "407M", "type": "Hyper-parameter/Architecture", "model": "ConvNeXt-XL", "parameter/architecture name": ["# Params", "number of parameters"], "dataset": "COCO"} {"value": "1898G", "type": "Hyper-parameter/Architecture", "model": "ConvNeXt-XL", "parameter/architecture name": ["# FLOPs", "number of FLOPs"], "dataset": "COCO"} {"value": "53.6", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "object detection", "metric": ["AP^b", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "ConvNeXt-XL", "model settings": {"backbone": "Mask R-CNN"}} {"value": "46.5", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "instance segmentation", "metric": ["AP^m", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "ConvNeXt-XL", "model settings": {"backbone": "Mask R-CNN"}} {"value": "387M", "type": "Hyper-parameter/Architecture", "model": "InternImage-XL", "parameter/architecture name": ["# Params", "number of parameters"], "dataset": "COCO"} {"value": "1782G", "type": "Hyper-parameter/Architecture", "model": "InternImage-XL", "parameter/architecture name": ["# FLOPs", "number of FLOPs"], "dataset": "COCO"} {"value": "55.3", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "object detection", "metric": ["AP^b", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "InternImage-XL", "model settings": {"backbone": "Mask R-CNN"}} {"value": "48.0", "type": "Result", "training data/set": "COCO", "test data/set": "COCO val2017", "task": "instance segmentation", "metric": ["AP^m", "AP"], "experimental settings": {"schedule": "1$\\times$"}, "model": "InternImage-XL", "model settings": {"backbone": "Mask R-CNN"}}