llama-cpp-python
Python bindings for llama.cpp with an OpenAI-compatible server.
About
llama-cpp-python is the long-running community Python binding for llama.cpp, maintained by Andrei Betlen. It wraps the C API through ctypes, exposes a high-level completion API modeled on OpenAI's, and ships a web server mode that turns a local model into an OpenAI-compatible endpoint with function calling, vision support, and code completion modes.\n\nIt became the default way many Python developers ran quantized local models before dedicated servers matured: pip install pulls and builds llama.cpp from source (with prebuilt CPU wheels available), hardware acceleration backends are configured through CMAKE_ARGS at install time, and the server mode has been widely used as a local Copilot replacement. Documentation lives on Read the Docs.
What it can do
- High-level Python API for completion with OpenAI-like calling conventions
- OpenAI-compatible web server with function calling and vision support
- Code-completion endpoint usable as a local Copilot replacement
- Configurable hardware backends via CMAKE_ARGS at install time
- Compatibility integrations with LangChain and LlamaIndex
Setup
Run pip install llama-cpp-python (set CMAKE_ARGS for GPU backends), then import the package or launch its server mode.
Prerequisites
Python 3.8+, a C compiler, and optionally CUDA for GPU acceleration.
Use cases
Works with
Trust score — 8/10
Scored 8/10 (verified tier, metrics-v1). Evidence: Verified via vendor site/docs/GitHub and reputable coverage, wave 100; Trending score 80/100 from researcher assessment.
Sources
Every fact above was checked against these sources. Where a claim comes from the provider and couldn't be independently verified, the text says so.