Title: On the Importance of Contrastive Loss in Multimodal Learning. Abstract: Recently, contrastive learning approaches (e.g., CLIP (Radford et al., 2021)) have received huge success in multimodal learning, where the model tries to minimize the distance between the representations of diﬀerent views (e.g., image and its caption) of the same data point, while keep the representations of diﬀerent data points away from each other. However, from a theoretical perspective, it is unclear how contrastive learning can learn to align the representations from diﬀerent views eﬃciently, especially in cases where the data is not isotropic. In this work, we analyze the training dynamics of a simple multimodal contrastive learning model, and show that contrastive pairs are important for the model to eﬃciently balance the learned representations. In particular, we reveal a stage-wise behavior of the learning process: In the ﬁrst stage, the model aligns the feature representations using positive pairs and the condition number grows in this stage. Then, in the second stage, the model reduces the condition number of the learned representations using negative pairs.