Cost, Latency & Quality

The levers

Four controls move you around the triangle. Learn what each one trades.

• Model choice. The biggest lever by far. A smaller model can be 10-20x cheaper and several times faster than a frontier model. The skill is knowing which tasks a small model handles well, often more than teams assume.
• Context length. You pay per token, and input tokens are part of the bill. Long prompts, stuffed retrieval context, and full chat histories cost money and add latency on every call. Trim ruthlessly: retrieve fewer, better chunks; summarize old turns; drop boilerplate.
• Caching. Reuse work instead of paying for it twice. Prompt caching stores a long, stable prefix (system instructions, a big document) so repeated calls skip re-processing it. Response caching returns a stored answer for an identical or near-identical request without calling the model at all.
• Routing. Send each request to the cheapest model that can handle it. A small model takes the easy cases; only the hard ones escalate to the expensive model.

How cost per request is built

Providers bill separately for tokens you send (input) and tokens you get back (output), usually priced per million tokens, with output several times pricier than input. Symbolically:

$\text{cost per request} = \frac{p_{\text{in}} \cdot t_{\text{in}} + p_{\text{out}} \cdot t_{\text{out}}}{10^6}$

where $p_{\text{in}}$ and $p_{\text{out}}$ are prices per million tokens and $t_{\text{in}}$, $t_{\text{out}}$ are the token counts. Two things fall out of this immediately: long context inflates $t_{\text{in}}$ on every call, and verbose output is expensive because $p_{\text{out}}$ dominates. Asking the model to "be concise" is a real cost control, not just a style note.

A worked comparison

Take one feature, drafting a short reply from a customer message, at 1,500 input tokens and 500 output tokens per request. Compare a small model against a large one using illustrative per-million-token prices (check your provider's current rates; the shape of the result is what matters).

Assumed prices (per 1,000,000 tokens):
  Small model:  input 0.15 dollars,  output 0.60 dollars
  Large model:  input 3.00 dollars,  output 15.00 dollars

Per request = (input_tokens x input_price + output_tokens x output_price) / 1,000,000

Small model:
  = (1,500 x 0.15 + 500 x 0.60) / 1,000,000 dollars
  = (225 + 300) / 1,000,000
  = 0.000525 dollars  (about 0.05 cents per request)

Large model:
  = (1,500 x 3.00 + 500 x 15.00) / 1,000,000 dollars
  = (4,500 + 7,500) / 1,000,000
  = 0.012 dollars     (about 1.2 cents per request)

The large model costs roughly 23x more per request here. That gap is invisible in a demo and decisive at scale:

At 1,000,000 requests per month:
  Small model:  ~525 dollars / month
  Large model:  ~12,000 dollars / month

Same feature, a difference of over 11,000 dollars a month, driven entirely by model choice. If a small model answers 90 percent of these requests well and routes only the hard 10 percent to the large model, you capture most of the quality at a fraction of the cost.

Putting the levers together with routing

Routing is where the levers combine into a real strategy. A pragmatic pattern:

1. 1Classify the request cheaplyUse a small, fast model (or even plain heuristics) to judge difficulty. Simple, common, well-scoped requests are the majority.
2. 2Serve easy cases with the small modelFast and cheap. Cache responses for identical or near-identical inputs so repeats cost nothing.
3. 3Escalate hard cases to the large modelMulti-step reasoning, ambiguity, or high stakes justify the expensive call. You are paying for quality only where it is needed.
4. 4Cache the stable prefix everywherePrompt-cache long shared instructions or documents so every call skips re-processing them, cutting both latency and input cost.
5. 5Measure the blendTrack the mix of cheap vs expensive calls and the blended cost per request. That number, not the sticker price of one model, is your real unit economics.

Further reading

• OpenAI, Latency optimization
• OpenAI, Prompt caching
• Anthropic, Prompt caching
• Anthropic, Reducing latency
• OpenAI, Production best practices