Title: Rényi Supervised Contrastive Learning for Transferable Representation. Abstract: A mighty goal of representation learning is to train a feature that can transfer to various tasks or datasets. A conventional approach is to pre-train a neural network on a large-scale labeled dataset, e.g., ImageNet, and use its feature for downstream tasks. However, the feature often lacks transferability due to the class-collapse issue; existing supervised losses (such as cross-entropy) restrain the intra-class variation and limit the capability of learning rich representations. This issue becomes more severe when pre-training datasets are class-imbalanced or coarse-labeled. To address the problem, we propose a new representation learning method, named R\'enyi supervised contrastive learning~(R\'enyiSCL), which can effectively learn transferable representation using a labeled dataset. Our main idea is to use the recently proposed self-supervised R\'enyi contrastive learning in the supervised setup. We show that R\'enyiSCL can mitigate the class-collapse problem by contrasting features with both instance-wise and class-wise information.
Through experiments on the ImageNet dataset, we show that R\'enyiSCL outperforms all supervised and self-supervised methods under various transfer learning tasks. In particular, we also validate the effectiveness of R\'enyiSCL under class-imbalanced or coarse-labeled datasets.