Title: MaskConver: A Universal Panoptic and Semantic Segmentation Model with Pure Convolutions. Abstract: Universal panoptic segmentation models have achieved state-of-the-art quality by using transformers for predicting masks.
However, in mobile applications, transformer models are not computation-friendly due to the quadratic complexity with respect to the input length. In this work, we present MaskConver, a unified panoptic and semantic segmentation model with pure convolutions, which is optimized for mobile devices. We propose a novel lightweight mask embedding decoder to predict mask embeddings. These mask embeddings are used to predict a set of binary masks for both things and stuff classes. MaskConver achieves \textbf{37.2\%} panoptic quality score on COCO validation set, which is \textbf{6.4\%} better than Panoptic DeepLab with the same MobileNet backbone.
After mobile-specific optimizations, MaskConver runs at \textbf{30} FPS and delivers 29.7\% panoptic quality score on a Pixel 6, making it a real-time model, which is 10$\times$ faster than Panoptic DeepLab using the same backbone.