PaLM-E

PaLM-E (Pathways Language Model, Embodied), published by Google and TU Berlin in March 2023, was the first model to inject continuous robot sensor observations directly into a pretrained language model. The largest variant combined the 540B PaLM with a 22B Vision Transformer into a single 562B network taking interleaved text, images, and robot state as input — producing natural-language plans, VQA answers, and robot commands as output.

At release it was the largest vision-language model ever publicly described. It set a new state of the art on OK-VQA (66.1%) without task-specific fine-tuning while simultaneously serving as the high-level planner for physical robot platforms — demonstrating positive transfer from internet-scale vision-language training to embodied tasks. It directly informed the VLA lineage (RT-2, OpenVLA, pi0).

PaLM-E was never released publicly — no weights, no API. It remains a research reference, with its ideas living on in open successors.