AI Lab · Artifact

AI Agent Evaluation Dashboard

This is the instrument I use to decide whether a multi-step agent is good enough to trust: the metrics that matter, the target I hold each one to, and the failure classes I diagnose against. The workbench below is a self-directed 0 to 1 build using a public synthetic dataset.

Status: Working prototype · Synthetic data
Metric
Baseline
Validated run
Target
Task success
-
-
90%
Factual accuracy
-
-
92%
Schema compliance
-
-
95%
Failure rate
-
-
<5%
Median latency
-
-
<5s

Targets are product hypotheses, not measured claims. Baseline and validated-run fields stay empty until a controlled model run is completed. Task success and accuracy govern whether the workflow ships; failure rate and latency govern whether it ships without a human in the loop.

0 to 1 AI product · Live evaluation workbench

Edit a candidate response and watch four deterministic checks run instantly. The prototype includes a synthetic dataset, expected outputs, suite-level status, failure diagnosis, and a downloadable artifact.

Suite result
1/4 cases pass
Grounded pass

Extract the settlement window and currency.

Source

Partner notice: settlements for EU merchants complete within 45 seconds and are denominated in EUR.

Deterministic scorer
4/4
Schema
Required fields are present
Grounding
Claim is grounded or safely withheld
Numeric fidelity
Numeric behavior is supported
Escalation
Abstention policy is respected
Expected
{"settlement_seconds":45,"currency":"EUR","source_quote":"complete within 45 seconds"}
Diagnosis

Well-formed and grounded.

Failure taxonomy
Retrieval miss
Right task, wrong or missing source material.
Tool misuse
Correct intent, incorrect tool selection or arguments.
Format break
Valid content in an unusable structure.
Silent drift
Plausible output that quietly diverges from the source.
Overreach
System answers where it should have escalated.