Title: Deep Contrastive Learning Approximates Ensembles of One-Class SVMs with Neural Tangent Kernels. Abstract: To demystify the (self-supervised) contrastive learning in representation learning, in the paper we show that a model learned by deep contrastive learning with a family of loss functions such as InfoNCE essentially approximates an ensemble of one-class support vector machines (SVMs) with neural tangent kernels (NTKs). This result comes from the fact that each gradient for network weight update in contrastive learning can be interpreted approximately as the primal solution for a one-class SVM with contrastive gradients as input. From the dual perspective, the Lagrange multipliers provide unique insights into the importance of the anchor-positive-negative triplet samples. In this way, we further propose a novel sequential convex programming (SCP) algorithm for contrastive learning, where each sub-problem is a one-class SVM. Empirically we demonstrate that our approach can learn better gradients than conventional contrastive learning approaches that significantly improve performance.