RecurrentGemma

RecurrentGemma-2B is an open language model from Google DeepMind built on the novel Griffin architecture, which replaces global attention with a mixture of linear recurrences and local attention. Because it compresses input into a fixed-size state instead of a growing KV cache, it needs substantially less memory than a comparable transformer and runs inference on long sequences much faster — while matching Gemma-2B's downstream performance despite being trained on fewer tokens.

Released with both pre-trained and instruction-tuned checkpoints plus efficient JAX training code (including a specialized Pallas kernel for TPUs), RecurrentGemma is a landmark proof that attention-free architectures can compete with transformers at matched budgets. It shipped under Gemma-style terms, and its release accelerated industry interest in linear-time sequence models as an alternative to the KV-cache bottleneck.