Given the importance of remote sensing, surprisingly little attention has been paid to it by the representation learning community. To address it and to speed up innovation in this domain, we provide simplified access to 5 diverse remote sensing datasets in a standardized form. We specifically explore in-domain representation learning and address the question of "what characteristics should a dataset have to be a good source for remote sensing representation learning". The established baselines achieve state-of-the-art performance on these datasets. 
 Remote sensing via computer vision and transfer learning is an important domain to address climate change as outlined by Rolnick et al. (2019) . Among others, research in remote sensing promises to help in solving challenges in food security (precision farming), water sustainability, disaster prevention (floods/landslides/earthquake forecasting), deforestation or wild fire detection, urban planning, and monitoring of carbon stocks and fluxes or the air quality. The number of Earth observing satellites is constantly increasing, with currently over 700 satellites monitoring many aspects of the Earth's surface and atmosphere from space, generating terabytes of imagery data every day. However the ground truth data acquisition is costly, usually requiring extensive campaign preparation, people and equipment transportation, and in-field gathering of the characteristics under question. While there are remote sensing communities working on applying general deep learning methods to remote sensing problems, this domain has received relatively little attention from the representation learning community. Given its importance, it is still in an early development stage in comparison to the progress made in representation learning on natural and medical images (eg. by Raghu et al. (2019) ). Some reasons for this are the diversity of data sources (satellite types, data acquisition modes, resolutions), the need of domain knowledge and special data processing, and the wide and scattered field of applications. The scarcity of standard recognized benchmark datasets and evaluation frameworks is another. For a long time there were only small labeled remote sensing datasets available. Only recenctly new large-scale datasets (eg. by ; Sumbul et al. (2019) ) have been generated for remote sensing problems. However, a consistent evaluation framework is still missing and the performance is usually reported on non-standard splits and with varying metrics, making reproduction and quick research iteration difficult. To address this, we provide five representative and diverse remote sensing datasets in a standardized form for easy reuse. In particular, we explore the importance of in-domain representation learning for remote sensing at various data sizes and establish new state-of-the-art baseline results. The main goal of this work is to develop general remote sensing representations that can be applied by researchers to other unseen remote sensing tasks. By providing these standardized datasets, common problem definition and baselines, we hope this work will simplify and enable faster iteration of research on remote sensing and inspire general representation learning experts to test their newest methods in this critical domain. In summary, the main contributions of this work are as follows: 1. Exploring in-domain representation learning to train generic remote sensing representations. 2. Generating 5 existing remote sensing datasets in a standardized format and establishing a common evaluation protocol. Publishing the best trained representations for easy reuse in transfer learning applications 1 . 3. Establishing state-of-the-art baselines for the BigEarthNet, EuroSAT, RESISC-45, So2Sat, and UC Merced datasets. We present a common evaluation benchmark for remote sensing representation learning based on five diverse datasets. The results demonstrate the enhanced performance of in-domain representations, especially for tasks with limited number of training samples, and achieve state-of-the-art performance. The five analyzed datasets and the best trained in-domain representations are published for easy reuse by the public. We investigate dataset characteristics to be a good source for remote sensing representation learning. As the experimental results indicate, having a multi-resolution dataset helps to train more generalizable representations. Other factors seem to be label quality, number of classes, visual similarity across the classes and visual diversity within the classes. Surprisingly, we observed that representations trained on the large weakly-supervised datasets were not as successful as that of a smaller and more diverse human-curated dataset. However, some results were inconclusive and require more investigation. Understanding the main factors of a good remote sensing dataset for representation learning is a major challenge, solving which could improve performance across a wide range of remote sensing tasks and applications.
Exploration of in-domain representation learning for remote sensing datasets. This paper provided several standardized remote sensing data sets and showed that in-domain representation could produce better baseline results for remote sensing compared to fine-tuning on ImageNet or learning from scratch.