Cutting AI Inference Costs Without Cutting Quality

If you charge a monthly subscription, every conversation has a budget. Heavy users can cost more than they pay, and the naive fix — degrade the model for everyone — damages the product.

These are the levers that worked, ordered by return.

1. Cache aggressively

The highest-return change by a wide margin, and the least glamorous.

In a bounded domain, questions repeat. Not word-for-word, but semantically: a great many electricians want to know about wire sizing for a 200 amp service.

Two layers:

• Exact-match cache on normalised queries. Cheap, and hits more than you would guess.
• Retrieval cache. Even when the final answer differs, the retrieved passages are often identical. Caching retrieval saves the embedding and search cost on every repeat.

Cache invalidation is the real work. When the corpus updates, dependent entries must go. Version the corpus and key the cache on that version.

2. Route by difficulty

Not every request needs the strongest model. Query rewriting, classification, and extraction run well on a smaller model at a fraction of the cost and latency.

The rule we use: route conservatively. A misrouted hard question produces a confident wrong answer, which costs far more than the tokens saved. When the classifier is uncertain, escalate.

3. Stop sending the whole conversation

The lazy pattern is to resend full history on every turn. Cost grows quadratically with conversation length, and long contexts often produce worse answers, not better ones.

Summarise older turns and keep recent ones verbatim. Quality improved when we did this — the model was no longer distracted by six exchanges of irrelevant preamble.

4. Retrieve less, better

Stuffing twenty chunks into context to be safe is expensive and counterproductive. Precision beats volume: better ranking with fewer passages consistently outperformed more passages with worse ranking, at a fraction of the token cost.

5. Move deterministic work out of the model

Calculators should be code. Unit conversion should be code. Table lookups should be code.

This is not primarily a cost decision — deterministic code is exactly right every time, which a model is not — but the cost savings are substantial because these are high-frequency operations.

What we did not do

Degrade the model for everyone. Tempting and wrong. The expensive interactions are usually the valuable ones.

Hard usage caps. A tech on a job site hitting a wall mid-diagnosis is a cancelled subscription.

Silently shorten answers. Users notice, and it reads as the product getting worse.

Measure per-conversation, not per-request

The metric that changed our decisions was cost per resolved conversation, not cost per API call.

A single expensive call that answers the question completely is better than four cheap ones that leave the user re-asking. Optimising per-request pushed us toward exactly that failure, and the aggregate got worse while the per-call number looked better.

The general lesson: cost work should follow the same principle as quality work. Optimise the outcome the user cares about, not the number that is easiest to measure.