SGLang

SGLang is an open-source inference framework for large language, vision-language, and diffusion models, hosted by the LMSYS non-profit (Berkeley). Its signature technology is RadixAttention — automatic KV-cache reuse across requests that share a prompt prefix — which makes it especially strong for RAG, multi-turn chat, and agentic tool-calling loops.

It ships an OpenAI-compatible server, structured generation (JSON schema, function calling) as a first-class feature, built-in image/video generation via SGLang Diffusion, and TTS/ASR via SGLang Omni. Independent 2026 benchmarks give it roughly a 29% throughput edge over vLLM on prefix-heavy workloads, with the two engines tied on unique-prompt batches.

1. Choose the Docker path (easiest): docker pull lmsysorg/sglang:latest — or the pip path in a fresh venv: uv pip install --prerelease=allow sglang
2. Launch a model (auto-downloads from Hugging Face): python -m sglang.launch_server --model-path Qwen/Qwen2.5-0.5B-Instruct --port 30000
3. Wait for readiness: curl -f http://127.0.0.1:30000/health
4. Send an OpenAI-format request: curl http://localhost:30000/v1/chat/completions -H "Content-Type: application/json" -d '{"model": "Qwen/Qwen2.5-0.5B-Instruct", "messages": [{"role": "user", "content": "Hello"}]}'
5. Tune for your hardware: add --tp N for multi-GPU tensor parallelism, --host 0.0.0.0 to expose beyond localhost, --api-key <key> to require auth, or --config config.yaml for reproducible launches
6. Note: a CUDA-capable GPU (or other supported accelerator: AMD, Intel, TPU, Apple Silicon, Ascend) is required for real workloads