Title: Building compact representations for image-language learning. Abstract: We propose a method to learn compact vision and language representations, which adaptively and iteratively fuses the multi-modal features. It greatly lowers the FLOPs of the model by effectively combining and reducing the number of tokens used for both text and images. This allows the model to scale without a large increase in FLOPs or memory and leads to a data efficient training. In addition, we propose adaptive pre-training data sampling which further improves the data efficiency. We achieve competitive performance compared to much larger models, and do so with significantly less data and FLOPs. With only 40M training examples and with 39 GFLOPs our model of 350M parameters outperforms all methods that have used less than 1B examples for pre-training. Code will be released.
