Seher C.Contact
04 · Self-directed AI productBack to work

Can an AI agent be trusted to do the job?

Designing an evaluation framework for reliable multi-step AI workflows - where the question is not whether the system can produce a good answer, but when it is good enough to trust.

6
Eval dimensions
Multi-step
Agent workflows
Failure-first
Design approach
Human-in-loop
Fallback path
01

The product question

Personal build · Synthetic data · Not client-attributed

A compelling AI demo is not necessarily a useful product. I explored how to evaluate an agent performing multi-step tasks where the system has to retrieve information, synthesize it, use tools, and produce structured outputs.

The challenge wasn't getting a good answer. It was deciding: when is the system good enough to trust?

02

What I evaluated

Dimension
What I measured
Quality
Accuracy, relevance, completeness
Consistency
Stability across repeated runs
Reliability
Frequency and severity of failures
Latency
End-to-end response time
Tool use
Correctness of tool selection and execution
Edge cases
Behavior under ambiguous or unexpected inputs
03

Product decisions

I prioritized predictability over peak performance.

A system that produces an exceptional result 80% of the time can be less useful than one that produces a consistently good result and appropriately escalates the rest.

  • Define representative and adversarial evaluation scenarios before optimizing prompts
  • Treat failure modes as product signals rather than edge cases
  • Add guardrails where deterministic behavior matters
  • Remove workflow steps that add latency without improving quality
  • Use fallback and human-review paths when confidence is insufficient
04

Architecture trade-offs

Every architectural choice here is a trade. The matrix is how I make the trade explicit rather than discovering it in production.

Accuracy vs latency
Bounded p95 over best-case answer quality
A few points of peak accuracy
Each additional reasoning or retrieval hop widened the tail more than it improved the rubric score. I capped hops and moved optional enrichment off the critical path.
Retrieval breadth vs grounding
Ranked, capped context
Recall on rare queries
Packing more passages raised token cost and made silent drift harder to detect. Rare queries route to a broader, slower path instead of taxing every request.
Model size vs unit cost
Tiered routing, small model first
Uniform behavior across cases
Most requests are easy. Routing them to a smaller model and escalating only on validation failure keeps cost per successful task falling as volume grows.
Autonomy vs trust
Confidence-banded human review
Full end-to-end automation
An unreviewed wrong action costs more than a reviewed slow one. Autonomy expands as evaluation evidence accumulates, not on launch day.
05

Token economics and fallback design

Unit economics

The number I track is cost per successful task, with retries, failed runs, and re-prompts in the numerator. Cost per call flatters a system that fails often.

The levers that actually move it: trimming and ranking context before packing, keeping instruction and schema prefixes stable so caching holds, and routing easy cases to smaller models.

Human-in-the-loop fallback

When confidence or validation drops below threshold, the system does not guess. It narrows scope, returns a grounded partial answer with sources, or escalates to a human path.

Refusal is a measured outcome, not a failure. Overreach, where the system answers instead of escalating, is the failure class I weight most heavily.

06

The evaluation loop

Define task→Build eval set→Run→Measure→Diagnose failure→Change system→Re-run

The important shift was from “does the demo look good?” to “can we measure whether the system is improving?”

07

System view and ownership boundary

User task
↓
Orchestrator
↓
RetrievalToolsMemory
↓
Output validation
↓
Evaluation layer → Human review / fallback → Telemetry
PM ownership

Problem definition · Product requirements · Evaluation framework · Tradeoffs · Metrics · UX · Risk · Prioritization

Engineering partnership

Architecture · Model integration · Instrumentation · Implementation

08

What I learned

AI product management requires a different definition of quality. Traditional software can be evaluated against deterministic acceptance criteria; AI systems require product teams to manage probability, variance, failure modes, latency, cost, and trust together.

The PM's job is not to make the model look impressive. It is to determine where the system can be trusted, where it cannot, and what product experience should exist around that boundary.

The running evidence for this work lives in the evaluation dashboard.