Most AI features fail before a single prompt is written, because nobody defined what good means, what wrong costs, or what happens when the model is unsure. This is the framework I run before engineering starts, and the same one I use to review AI work other teams bring to me.
If the input space is enumerable and the rules are stable, deterministic code wins on cost, latency, and testability. A model is the wrong tool for a lookup table.
If I cannot write the rubric, I cannot write the evaluation set, and without an evaluation set I have no way to tell iteration from noise. Rubric first, prompt second.
Reversible, low-cost errors tolerate autonomy. Irreversible or regulated actions require confirmation, validation, or a human reviewer in the path.
I price the feature at 10x current volume before building. If token and retrieval cost per successful task does not fall with scale, the design is wrong, not the pricing.
Every AI surface ships with a defined degraded mode: deterministic path, cached answer, narrowed scope, or escalation. Never a blank screen and never a confident guess.
The default is deterministic. A model earns its place only where the input is genuinely open-ended and the output can be validated.
I set a p95 rather than an average, because the tail is what users remember. Retrieval, model call, and validation each get an allocation, and any step that cannot justify its share of the tail gets cut or moved off the critical path.
The metric is cost per successful task, not cost per call. Retries, failed runs, and re-prompts belong in the numerator. Context trimming, caching stable prefixes, and routing easy cases to smaller models are the levers I reach for first.
More context is not more accuracy. I cap retrieved context, rank before packing, and keep instructions and schema at stable positions so caching stays effective and behavior stays reproducible.
Data classification decides architecture. Sensitive fields are redacted or tokenized before they leave the trust boundary, retention is explicit, and any vendor path is documented before the first prototype, not after.
Output passes schema validation and grounding checks. The system acts and logs the decision for sampling review.
Output is presented as a draft with sources shown. A person confirms before anything irreversible happens.
The system declines to answer, narrows scope, or routes to a human path. Refusal is a successful outcome here.
The thresholds are set from the evaluation runs described in the evaluation dashboard, not chosen by feel.