Small Models, Sharp Domains: When Smaller Wins

The reflex is to reach for the largest available model. Sometimes that is right. Often, in a narrow domain, it is an expensive way to solve a problem that had a better solution.

Where the work actually goes

In a grounded domain assistant, a surprising amount of the quality comes from things that are not the model:

• Retrieval quality. If you hand the model the right passage, a mid-sized model answers correctly. If you hand it the wrong passage, the largest model available answers confidently and wrongly.
• Corpus curation. Nothing recovers from missing or outdated source material.
• Deterministic tools. A calculator is not a model problem. It is arithmetic, and it should be exactly right every time.

Improving any of those three usually beats upgrading the model.

What smaller models are good enough for

• Extraction and structuring. Pulling equipment details out of a photographed nameplate.
• Query rewriting. Turning "what wire for a 200 amp service" into terms the corpus uses.
• Classification and routing. Deciding whether a question needs the code corpus, a calculator, or a refusal.
• Summarising retrieved passages when the passage is already correct.

These are high-volume, latency-sensitive operations. Running them on a smaller model is faster, cheaper, and no less accurate.

What still needs the strong model

• Multi-step reasoning across several references. When the answer depends on an article, its exception, and a table interacting.
• Ambiguity handling. Recognising that a question cannot be answered as asked and articulating precisely what is missing.
• Explaining the reasoning. The apprentice-teaching use case genuinely benefits from a stronger model.

The pattern: use the strong model where judgement is required, and smaller models everywhere judgement is not.

The economics, honestly

Cost matters at consumer subscription prices. A field app doing many small operations per session cannot route all of them to the most expensive model and remain viable.

But the argument is not primarily about cost. It is that a routed architecture is better, not just cheaper. Deterministic work should be deterministic. Extraction should be a small fast call. Reserving the expensive reasoning for actual reasoning produces a more predictable product.

Where this breaks

Two honest failure modes.

Routing is itself a hard problem. A classifier that sends a genuinely complex question to a small model produces exactly the confident-wrong-answer we most want to avoid. Bias the router toward escalation.

Complexity has a cost. A multi-model pipeline has more failure surface than one call to one model. If your volume is low, that complexity is not worth it — use the strong model and move on.

The takeaway

Model size is one lever among several, and rarely the most effective one. In a bounded domain, the ranking is usually: fix retrieval, curate the corpus, build real tools, then consider a bigger model.

Teams that start at the end of that list spend a lot and improve little.