CoCa

CoCa (Contrastive Captioners) is an image-text foundation model from Google Research, published in May 2022. Its minimalist encoder-decoder design jointly trains a contrastive loss and a captioning loss in a single forward-backward pass, subsuming the strengths of CLIP-style dual encoders and SimVLM-style generative models.

The architecture skips cross-attention in the first half of the text decoder to produce unimodal embeddings for contrastive learning, then cascades cross-attending layers for multimodal captioning. It reached 86.3% zero-shot top-1 on ImageNet and a then-record 91.0% fine-tuned, plus state-of-the-art results on retrieval, VQA, and captioning benchmarks.

Google published the paper but not the weights. CoCa's joint-loss design influenced later unified vision-language training recipes.