What is training vs. inference?

At a glance

• Training happens once; inference happens every time someone uses the model and never stops.
• You almost never pay to train a frontier model. You rent inference per token, or fine-tune a hosted model cheaply.
• Inference is roughly 80-90% of an AI system’s lifetime cost, because it scales with usage.
• The live model does not learn from your prompts. Customizing it is a separate step.

How it works

Training shows the model huge amounts of data and adjusts billions of internal numbers until it captures useful patterns^[1]. It is expensive, slow, and done once before shipping. Inference runs that fixed model on each request to generate an answer^[4]. Training builds the engine; inference is the fuel you burn every time you drive.

Why your bill is an inference bill

You pay per token through a vendor API, or for the GPUs hosting an open model. Either way, cost scales with usage, so inference is 80-90% of lifetime cost^[2]. Per-token prices fell about 280x in two years^[3], yet total spend often still rose because adoption outpaced the price cuts^[2]. Budget for the running cost, not the setup.

Customizing and trusting AI

The live model applies fixed knowledge and forgets each conversation; it does not “learn from us.” Teaching it your business is a deliberate, separate step. In rising order of cost: better prompting, retrieval (RAG, looking up your documents at inference time), then fine-tuning. Start with prompting and RAG; reserve fine-tuning for when behavior stays wrong^[5].

Bottom line

Training is a one-time cost you rarely pay directly; inference is the recurring bill that grows with every customer, so budget for the stream, not the spike.