LM Evaluation Harness
The community-standard framework for benchmarking language models.
About
The LM Evaluation Harness (EleutherAI/lm-evaluation-harness, MIT, ~14k stars) is the de-facto standard for running reproducible LLM benchmarks — MMLU, ARC, HellaSwag, and hundreds more tasks across HF models and APIs.
If a model card cites a benchmark score, it was probably computed with this harness. High issue count reflects heavy community use, not neglect. Essential infrastructure for anyone comparing models seriously.
What it can do
- Hundreds of benchmark tasks
- Reproducible evaluation configs
- HF + API model backends
- The benchmark standard
- MIT open source
Setup
pip install lm-eval.lm_eval --model hf --model_args pretrained=<model> --tasks mmlu --device cuda:0.- Compare results across models with identical task configs.
Prerequisites
Python 3.9+; GPU for local models (or API access).
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier). ~14k stars, MIT, the community benchmark standard — its adoption score reflects real ubiquity. Docked for a high open-issue count (heavy use, but maintenance load is visible). Verified tier.
★ 14,197 GitHub stars
How the trust score is built
Full methodology →- Adoption · stars/downloads, log-scaled vs peers
- 7.8/10
- Maintenance · recency, releases, issue load
- 6.5/10
- Openness · license + docs
- 9.4/10
- Manual review · sanity read + red-flag check
- 8.0/10
- · 14,197 GitHub stars (log-scaled vs catalog peers)
- · last push 29d ago; 5 releases/12mo; 1165 open issues
- · manual review: EleutherAI's standard LLM eval framework; community benchmark standard; high issue count reflects heavy use; license: MIT
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.