CogAgent

CogAgent is an 18-billion-parameter visual language model from Tsinghua University and Zhipu AI, published in December 2023. Built on CogVLM, it uses both low- and high-resolution image encoders to read screenshots at 1120x1120, recognizing tiny interface elements and text that defeat text-only agents.

It set state-of-the-art results on GUI navigation benchmarks including Mind2Web (PC) and AITW (Android), using only screenshots rather than extracted HTML, while staying competitive on general VQA benchmarks like TextVQA, ChartQA, and DocVQA.

CogAgent was an early proof that vision-language models could operate real interfaces like a human, and it seeded the wave of computer-use agent research. The model and code were released through the CogVLM project.