Training vs Inference

Two phases, two cost profiles

Every neural network lives in two phases, and confusing them is the source of half the misunderstandings about AI costs. Training is the phase where the model learns: data flows through the network, errors are measured, and weights are adjusted, millions or billions of times. Inference is the phase where the finished model is used: new input goes in, a prediction comes out, and nothing is learned.

The economics could hardly be more different. Training a large model is a one-time (or occasional) expense measured in weeks of GPU time and large budgets. Inference is a per-use expense measured in milliseconds and fractions of a cent. When people say AI is expensive, they usually mean training. When they say it is cheap, they usually mean inference.

What training actually does

A training step has three parts. The forward pass runs a batch of examples through the network and produces predictions. The loss function scores those predictions against the right answers. The backward pass uses backpropagation to compute how each weight contributed to the error, and the optimizer nudges every weight to reduce it. Then the next batch arrives and the loop repeats.

This loop needs three things that inference does not. It needs labeled data, or at least data with a training signal, in large quantities. It needs to store gradients, one per weight, plus whatever state the optimizer keeps. The Adam optimizer, the most common choice, stores two extra numbers per weight, which triples the memory beyond the weights themselves. And it needs to keep intermediate activations around for the backward pass, which adds more memory on top.

To put numbers on it: fine-tuning a 1.2 billion parameter model in 16-bit precision needs about 2.4 GB for the weights, another 2.4 GB for the gradients, and roughly 9.6 GB for Adam's state, plus a few gigabytes for activations depending on batch size. That is 16 to 18 GB before the data even arrives, which is why even modest training needs a serious GPU. Full pretraining of frontier models multiplies this by thousands.

Training is also where all the design decisions live: the architecture, the dataset mix, the learning rate schedule, the regularization. Inference inherits all of those choices silently.

What inference actually does

Inference is the forward pass alone. Input goes in, activations flow through the layers, output comes out. No gradients are computed, no weights change, no optimizer state exists. The memory needed is roughly the weights plus the activations for a single batch, a fraction of training's footprint.

Because nothing is stored for a backward pass, inference can use tricks that training cannot. Weights can be quantized, stored in 8-bit or 4-bit precision instead of 16 or 32, which cuts memory dramatically with a small accuracy cost. Batches of independent requests can be processed together for throughput. Parts of the computation can be cached: in language models, the keys and values computed for earlier tokens are reused when generating later ones, avoiding redundant work.

Latency is the metric that matters here, not throughput alone. A chatbot that takes thirty seconds to answer is useless no matter how cheap each answer is. Inference engineering is largely about this tradeoff: how to serve the model fast enough and cheap enough for the product built on top of it.

Why the distinction matters

It matters first for cost planning. Training is capital expenditure: big, lumpy, and infrequent. Inference is operational: small per unit, scaling with usage. A product with a million users spends almost everything on inference. A research lab pushing the frontier spends almost everything on training.

It matters for who can do what. Training a frontier model takes resources only large organizations have. Running inference on an already-trained model is within reach of a laptop for small models, which is why open-weight releases matter: they move capability from the training side of the divide to the inference side.

It matters for correctness too. A model cannot learn at inference time. If the world changes after training, if new facts appear or the data distribution shifts, the model's behavior does not update on its own. Techniques like retrieval-augmented generation and fine-tuning exist precisely to bridge this gap, giving a frozen model access to fresh information or new behavior without full retraining.

The blurry edges

The line between the phases is not absolute. Fine-tuning is training, but on a small scale, starting from existing weights. In-context learning looks like learning, the model adapts to examples in the prompt, but no weights change, so it is inference with a clever input. Some research systems update weights during deployment, which is genuine learning at inference time, but it remains rare in production because it is hard to control.

For most practical purposes, though, the split holds: training builds the model, inference runs it. Everything about cost, hardware, and engineering follows from which side of that line you are on.