Title: CLIPPING: Distilling CLIP-based Models for Video-Language Understanding. Abstract: Pre-training a vison-language model and then fine-tuning it on downstream tasks have become a popular paradigm. However, pre-trained vison-language models with the Transformer architecture usually have a large number of parameters and take long inference time. Knowledge distillation has been an efficient technique to transfer the capability of a large model to a small one while maintaining the accuracy, which has achieved remarkable success in natural language processing. However, the collection of the pre-training data for the pre-training knowledge distillation costs huge manpower in multi-modality applications. In this paper, we propose a novel knowledge distillation method, named CLIPPING, where the plentiful knowledge of a large teacher model that has been fine-tuned for video-language tasks with the powerful pre-trained CLIP can be effectively transferred to a small student only at the fine-tuning stage. Especially, a new layer-wise alignment is proposed for knowledge distillation of the intermediate layers from the Transformer to the CNN in CLIPPING, which enables the student model to well absorb the knowledge of the teacher. Besides, we present an effective cross-modality knowledge distillation, which includes both the knowledge of the global video-caption distributions from the teacher model and the knowledge of the local video-caption distributions from the pre-training model (CLIP). Finally, CLIPPING with MobileViT-v2 as the vison encoder without any vison-language pre-training achieves 91.5%-95.3% of the performance of its teacher on three video-language retrieval benchmarks, with its vison encoder being 19.5x smaller. CLIPPING also significantly outperforms a state-of-the-art small baseline (ALL-in-one-B) on the MSR-VTT dataset, obtaining relatively 7.4% performance gain, with 29% fewer parameters and 86.9% fewer flops. Moreover, CLIPPING is comparable or even superior to many large pre-training models.