Llama 3 Explained

What it is

Llama 3 is Meta's family of open-weight large language models, released on April 18, 2024 in 8-billion and 70-billion parameter sizes, each in pretrained and instruction-tuned variants. It was the third generation of Llama and a major quality jump over Llama 2, trained on more than 15 trillion tokens.

The release mattered because of the license. The weights were free to download and use commercially, which made Llama 3 the default foundation for startups, researchers, and hobbyists who wanted a capable model without paying API bills or sending data to a third party. Within months it was everywhere: fine-tuned into thousands of variants on Hugging Face, deployed inside companies that would never touch a closed API, and running on local machines through tools like Ollama.

How it works

Llama 3 is a dense decoder-only transformer, not a mixture-of-experts design. Meta's model card documents the specifics: an optimized transformer architecture with grouped-query attention for inference efficiency, a 128,000-token vocabulary tokenizer that handles multilingual text more gracefully than its predecessor, and instruction-tuned versions trained with supervised fine-tuning plus reinforcement learning from human feedback. Training the two models took 7.7 million H100 GPU-hours in total, about 1.3 million for the 8B and 6.4 million for the 70B, with carbon emissions fully offset through Meta's sustainability program.

Meta paired the release with its Purple Llama safeguards, an open toolkit including Llama Guard 2 for input and output filtering and Code Shield for flagging risky generated code. The message was that openness and safety tooling could ship together.

It was text-only at launch: text in, text and code out, with no image or audio understanding.

Key details

The context window was 8,192 tokens, which was short for 2024; Llama 3.1 raised it to 128K that July. Knowledge cutoffs were March 2023 for the 8B model and December 2023 for the 70B. Meta's published benchmarks for the instruction-tuned models: the 8B scored 68.4 on MMLU and 62.2 on HumanEval, while the 70B scored 82.0 on MMLU and 81.7 on HumanEval, with strong results on grade-school math and competition math as well. These are Meta's figures.

The license deserves a careful read. It permits commercial use, but organizations with more than 700 million monthly active users must request a separate license from Meta, a clause aimed at the largest tech companies. Attribution is required, phrased as "Built with Llama," and an acceptable use policy restricts certain applications. Meta never released the training data, training code, or data pipeline, so Llama 3 is open-weight rather than open-source in the OSI sense. That distinction matters to purists and mostly does not matter to practitioners. The weights are downloadable from Meta and Hugging Face; cloud providers charge their own hosting fees.

Limitations

Meta's own model card is candid about the gaps. The model was text-only. The 8K context window limited long-document work, and competitors were already shipping far larger windows. It was intended primarily for English, and safety testing was conducted in English only, so performance and safety in other languages were less assured.

The card warns that the model may produce inaccurate, biased, or otherwise objectionable responses, and that developers must do their own safety testing and tuning before deployment. Open weights shift that responsibility to whoever deploys the model, which is both the point and the risk. Later versions, 3.1 through 3.3, superseded it with longer context, multilingual support, and multimodal capabilities, but Llama 3 remains the release that proved open-weight models could compete with the closed frontier.

The ecosystem it seeded

Llama 3's largest impact was not the model itself but what thousands of other people built on it. Within months of release, Hugging Face hosted an enormous number of fine-tuned variants: instruction-tuned versions for specific languages, domain-specialized models for medicine and law, uncensored versions, quantized versions for every hardware tier, and merges that combined the strengths of multiple fine-tunes. No closed model has ever generated that kind of derivative ecosystem, because no closed model can be downloaded.

Local tooling made the ecosystem accessible. Ollama packaged Llama 3 into a two-command install, and it became the default model people tried first. LM Studio offered a graphical alternative. Enterprises that would never send data to a third-party API deployed Llama 3 inside their own infrastructure, which opened the regulated industries, finance, healthcare, government, to language models years earlier than closed APIs could have.

Meta kept the line moving. Llama 3.1 arrived in July 2024 with a 128K context window and a 405-billion-parameter flagship, followed by 3.2 with multimodal and small-device models and 3.3 with a refined 70B. Each release extended the same bargain: frontier-adjacent capability, free weights, and a license with the large-company carve-out. The succession also illustrated the pace of open-weight progress. Llama 3's 8K context window looked cramped within months, which is both a criticism of the release and a measure of how fast the field was moving.