Commit0
The from-scratch coding benchmark: can your agent rebuild a real Python library from commit zero?
About
Commit0 is the hardest fair test in AI coding evaluation: instead of fixing a bug in a working repo, the agent gets a specification document, a starter repo with tests and empty source files, and has to generate the entire library from scratch. The benchmark covers 57 core Python libraries with serious test coverage, lint, and type checking — and the model is even allowed to run the tests interactively while it works. Performance is measured purely by unit-test pass rate on a clean clone.
It ships as an interactive RL environment, not just a static dataset: pip install commit0, then commit0 setup clones repo splits from the HuggingFace dataset, with cloud-distributed testing and full change tracking for agent developers designing new coding agents. (Note: the project moved in 2026 — the canonical repo is now EtharaOrion/kaiju-harness, still MIT-licensed.)
The honest caveat: from-scratch generation is a narrow slice of real software work — most engineering is reading, debugging, and modifying existing systems, which SWE-bench-style benchmarks cover better. Scores here are also sensitive to spec quality. Use Commit0 to stress-test greenfield generation ability, alongside bug-fix benchmarks for the full picture.
What it can do
- 57 real Python libraries as from-scratch generation tasks with full test suites
- Interactive environment: agents run tests during generation, not just after
- Spec documents (text + images) and starter repos with lint/type checking
- Cloud-distributed testing and full change tracking/logging
- HuggingFace dataset integration; CLI workflow (setup, build, test, eval)
- MIT licensed; companion site at commit-0.github.io
Setup
pip install commit0.commit0 setup <repo-split>to clone a library split (uses the wentingzhao/commit0_combined HuggingFace dataset).- Point your agent at the spec PDF + starter repo; let it generate the implementation, running tests interactively.
- Evaluate: the harness commits the agent's changes and runs the unit tests on a clean clone.
Prerequisites
Python 3; an AI coding agent to evaluate; compute for test environments (cloud distribution supported).
Use cases
Works with
Trust score — 7/10
Scored 7/10 (metrics tier, metrics-v1). Evidence: Commit0 by Ethara AI; commercial, no public repo stats.; Trust data reconstructed by coordinator after parallel-worker file overwrite; scores evidence-based from entry metadata..
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.