How We Evaluate AI That Cites Code

Public benchmarks measure whether a model is smart. They do not measure whether a working professional can rely on your product at 7am with an inspector on the way.

Those are different questions and they need different tests.

The failure we optimise against

Before any metric, you need a clear picture of the worst outcome. Ours is specific:

A confident answer, with a real-looking citation, that is wrong.

Not a refusal. Not an obviously garbled response. Those are safe — the user discards them. The dangerous output is the one that manufactures trust and then misleads. It gets acted on.

Everything downstream is designed to catch that case.

Four layers

1. Retrieval correctness. Given a question, did we surface the passage that actually governs the answer? Scored against questions where a domain expert has labelled the controlling reference. This catches the "close but wrong section" class of error.

2. Citation faithfulness. Does the cited reference genuinely support the claim? This cannot be automated away. It requires someone who knows the trade reading the answer next to the source. It is slow, and it is the highest-value thing we do.

3. Numeric fidelity. Any value that should come from a table must match that table exactly. This one is automatable and unforgiving — an off-by-one row in an ampacity table is a total failure regardless of how good the prose was.

4. Appropriate refusal. A deliberately adversarial set of questions the corpus cannot answer: out of scope, equipment-specific, ambiguous without a measurement. We measure how often the system correctly declines and names what is needed instead.

The calculators are tested differently

Calculators are not AI and should not be evaluated like it. They are deterministic functions with known-correct answers.

We test them against worked examples from the manuals themselves. If the codebook contains an example calculation, that example becomes a test case. Results must match what a competent professional gets by hand, because that is the standard they will be checked against.

What we do not measure

We stopped tracking answer length, helpfulness ratings from non-experts, and generic quality scores. They all reward fluency, and fluency is precisely what makes a wrong answer dangerous.

A short answer that says "this depends on the data plate, here is how to read it" beats three confident paragraphs. Our evaluation has to reflect that, or it pushes the product the wrong way.

Who does the reviewing

Domain experts, not engineers. An engineer can tell you the retrieval worked. Only a tradesperson can tell you the answer would have been useful, or that it was technically true and practically misleading.

That review loop is the most expensive part of building these products. It is also the reason they get used.