Agent Evaluation & Safety Harness
A flight recorder that turns agent failures into replayable evidence
Dependency-free harness that runs normal and adversarial tasks against tool-using agents, scores quality and safety, routes uncertain cases to a human review console, and diffs runs to find the first divergent step.
Visual evidence
Trace comparison: baseline vs. guardrailed agent
Supplementary evidence
Run console with per-case traces
Case study
Problem, Build, Outcome
Problem
Agent demos end at a pass/fail score, which says nothing about which step broke, whether a fix regressed something else, or what a new safety policy would have prevented.
Build
Built a Python harness with typed tool contracts, sandboxed state, canary egress checks and prompt-injection scanning, plus SQLite run storage, an authenticated review console, trace comparison and offline policy replay.
Outcome
Every failure becomes an evidence capsule with a fingerprint, root cause and suggested control, and a candidate agent can be compared to its baseline case by case before release.
Architecture
System Flow
Architecture Notes
- Adapters cover an offline workflow agent, OpenAI, Anthropic, Ollama, recorded runs and any HTTP agent.
- Every step is captured with provenance, arguments, timing, cost and mitigations, then redacted before storage.
- JSON, HTML, SQLite and OpenTelemetry outputs share one versioned report contract.
Engineering choices
Technical Decisions
Kept the core dependency-free so it runs in any CI job without installing a framework.
Treated retrieved email and file content as untrusted input and scanned it for indirect injection.
Made policy replay offline so safety rules can be iterated without re-running state-changing tools.
Evidence
Proof Points
Adversarial suites: direct and indirect injection, exfiltration canaries, hallucination traps
First-divergence trace diffing with signed per-metric deltas
Human review queue with concurrency-safe decisions
Next read