Zamba2-7B

Zamba2-7B is Zyphra's second-generation hybrid state-space model, released November 2024. It interleaves a Mamba2 backbone with two shared transformer attention blocks in an ABAB pattern, plus LoRA projectors that let each shared block specialize per layer at minimal parameter cost. The result is a 7B model that competes with Mistral-7B, Gemma-7B, and Llama-3-8B on benchmarks while delivering substantially faster time-to-first-token and much smaller KV-cache memory.

Trained on up to 3 trillion tokens including a carefully curated annealing phase, the model is released under Apache 2.0 with base and instruction-tuned variants. Its efficiency profile makes it a strong fit for on-device, edge, and cost-sensitive deployments where transformer latency is the bottleneck.