Rethinking Contrastive Loss in CLIP Post-training: A Complementary Framework with Frozen Text Encoder
arXiv:2610.11374v2 Announce Type: replace-cross Abstract: CLIP serves as a foundational vision-language model and the de facto vision encoder for downstream VLMs such as LLaVA. Post-training offers a lightweight route to refine CLIP, but recent work argues that the standard contrastive loss is unsuitable for post-training due to catastrophic forgetting under small batches, motivating designs that abandon the contrastive objective in favor of distillation. We revisit this premise and find that,…