AI evaluation and golden-set harness
Duckbill™ gates material prompt, model, and taxonomy changes with a checked-in golden set. This is Stream B of Sprint 05 (MAIL-36, MAIL-57, MAIL-114). See ADR 0020.
Rules
- The runner uses fixtures only. It must not call mail provider adapters.
- Corpus messages are synthetic or redacted. Live mailbox tokens and production mail stay out of git.
- Email bodies are untrusted. Prompt-injection fixtures must not be followed as system instructions.
- Invalid JSON and schema-invalid structured output fail the case. The runner does not coerce them into business data.
- Results are reproducible from schema, taxonomy, prompt, model, and corpus version metadata.
- A material AI change without a golden-set run cannot be called done.
Code
| Module | Role |
|---|---|
packages/ai/src/eval-port.ts | Typed classify/extract port and schema validation |
packages/ai/src/fixture-classifier.ts | Deterministic double for the harness |
packages/application/src/ai-eval.ts | Regression runner and report |
tests/ai/golden/corpus.json | Checked-in golden corpus |
scripts/run-ai-eval.ts | CLI used by npm run ai:eval |
Metrics
The report includes schema validity, critical-field regressions (intent, attention, priority, deadlines, action items), high-impact false positives (newsletter/receipt/notification marked urgent or actionable; injection followed or secrets echoed), latency, and cost. The fixture classifier reports zero tokens and zero cost.
Running
npm test -- tests/ai/golden-set.test.ts
npm run ai:eval
The CLI prints a JSON report and exits non-zero on unexplained critical regression.
Classify, extract, persist, drafts, and security policy stay on their own issues.