Self-supervised learning;knowledge distillation;lightweight models;contrastive learning;consistency learning
Retro: Reusing teacher projection head for efficient embedding distillation on Lightweight Models via Self-supervised Learning