Skip to content

Evaluation before features: shipping AI you can defend

AI1 minOmid Manoochehri
Steel calipers measuring a faceted glass piece beside identical test pieces

Model choice is the easy part. The hard part is designing for an answer that might be wrong, and being able to say how often it is.

Week one of an AI engagement produces an evaluation set, not a prototype. A hundred real inputs, the answers a domain expert accepts, and a script that scores a run in under a minute. Everything after that is a comparison rather than a debate.

Design for the wrong answer

If a feature cannot show its evidence, a reviewer cannot correct it, and an uncorrectable feature quietly stops being used. Retrieval spans, confidence and a one-click override are interface work, and they belong in the first design review.

A model-backed feature that nobody can audit is a feature nobody will trust twice.
All postsRead a case study

Bring us aHarder problem.

Tell us what you are building. We will tell you what we would do first.