AgentBench
Multi-dimensional benchmark evaluating LLMs as agents across 8 distinct environments.
About
AgentBench, from Tsinghua's THUDM group (Liu et al., arXiv 2308.03688, August 2023), was one of the first benchmarks to evaluate language models specifically as agents rather than as question-answerers. It drops the model into eight very different environments — an operating system shell, a database, a knowledge graph, a digital card game, lateral-thinking puzzles, a household simulator, web shopping, and web browsing — and measures whether it can act, observe results, and adapt.
The headline finding still gets cited: there was a sharp capability gap between commercial models and open-weight models on agentic tasks, much larger than on standard NLP benchmarks. That framing shaped how the field talks about agent evaluation. The benchmark is Apache 2.0 licensed and the environments remain a useful stress test, though newer agent benchmarks with simulated users and business policies have since gone further.
What it can do
- 8 agent environments: OS, database, knowledge graph, card game, puzzles, household, web shopping, web browsing
- Standardized agent interaction loop with observation and action
- Per-environment scoring and aggregate reporting
- Apache 2.0 licensed harness
Setup
- Clone https://github.com/THUDM/AgentBench.
- Install dependencies and set your model API key.
- Run the evaluation harness against one or more of the 8 environments.
- Aggregate per-environment scores for comparison.
Prerequisites
Python 3.8+, API access to the model under test.
Use cases
Works with
Trust score — 8/10
Repo THUDM/AgentBench verified (3.8k stars), Apache 2.0 confirmed via GitHub API, paper arXiv 2308.03688 verified live. Eight environments confirmed via paper. No official benchmark logo exists, so logoUrl is null. Some findings about model gaps reflect 2023-era models and should not be read as current.
★ 3,800 GitHub stars
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.