Score agent runs with judges, not scorers.
Point a rule at any trace, conversation, agent, or dataset. Get back a pass, a score, and the reason why. No hand-rolled scorers, no benchmark project.
The three signals every agent owner asks for.
Pick the one that hurts most. Add the others when the first starts paying off.
Track task completion.
The rule reads the trace and decides if the agent actually finished the work, versus claiming it did. Catches stub responses, partial loops, and silent skips.
Track CSAT.
An LLM judge scores tone, helpfulness, and resolution from the conversation. Surfaces the runs your users would have rated low, without waiting for the survey.
Track output quality.
Agent Judge opens the files the run produced and checks them against your rubric. Catches the failures a transcript never shows: a wrong number in a report, a missing field, a contract clause that should not be there.
Judge the work, not just the transcript.
Some failures never show up in a chat log. The report has the wrong total. A field is missing. A refund went out without approval. Agent Judge spins a fresh sandboxed agent for each run, opens the files it produced, and scores them against your rubric.
Deterministic gate first.
Structural checks (files present, JSON valid, fields counted) run before any sandbox. Format failures come back as an instant fail, with zero tokens spent.
A fresh agent per row.
Each dataset entry gets its own isolated, sandboxed agent. It reads the bundle the way a reviewer would, then commits to pass or fail with a score.
An auditable trail.
Every file the judge opened and every check it ran is recorded on the verdict. No black-box numbers, no guessing what it looked at.
A $4,200 refund was issued with no manager approval. policy.md section 3 requires sign-off above $1,000, and the approval field in decision.json is empty.
- 01read_filedecision.json
- 02read_filepolicy.md
- 03grepapproval
- 04read_fileaudit.log
Eval is the on-ramp to enforcement.
Every verdict is one click from a guard rule. Every guard rule is one click from re-evaluation. Same data, same instruction, compounding — that's the loop.
Run lands
Spans, tool calls, and conversation, captured automatically by the SDK or CLI.
Block in real time
Static and LLM judges enforce at the tool boundary, in under 50ms, before a bad action ships.
Score after the fact
Async judges read each run and return a pass, a fail, a score, and the reason.
Diagnose root cause
A sandbox replays the failing run and returns a root-cause diagnosis.
On demand, sampled, or backfilled.
Run it now.
Point a run at a trace, a conversation, an agent, or a dataset and get verdicts back. Useful for spot checks before you trust a rule.
A slice of traffic.
Scope a run to an agent or a time window and sample N% of its traces. Continuous quality signal without scoring every single run.
Over history.
Replay a past time window for an agent and score traces that already happened. The fastest way to validate a rule you just wrote.
Three lines from agent run to verdict.
staso.init wires tracing. conversation() groups the run. Verdicts come from the dashboard or from staso.evals.run — your call.
01import staso02staso.init(agent_name="my-agent", agent_version="v1")04with staso.conversation("run-001"):06run_my_agent()07# Trigger an eval explicitly09verdicts = staso.evals.run({10"session_id": "run-001",11"zero_config_names": ["task_completion_rate", "hallucination"],12}).verdicts
Stop spot-checking. Start scoring.
Wire the SDK in a minute. Attach a rule. Watch verdicts arrive on every run — no benchmark project, no scorer code, no engineer yak-shaving.