Detect support-agent regressions with Chalk evaluations
This demo shows a scheduled Chalk function scoring the answers a support agent already served. Compare a reviewed prompt with an intentionally outdated one, then inspect the evaluation window and webhook alert that expose the policy violations.
See an outdated prompt trigger an alert
Compare the recorded evaluation windows: a reviewed prompt passes, the outdated prompt breaks policy, and recovery produces a second webhook. Then try the shared live demo below.
Outdated prompt at 06:50 UTC. Alert sent at 06:56 UTC.
Real scheduled runs from this demo: a healthy window on the reviewed prompt, the first window after v2 went out, and the window after the rollback. Each row is a Chalk evaluation run over that window's production traffic.
| Window (UTC) | Prompt | Turns | LLM judge avg | Policy pass rate | Webhook |
|---|---|---|---|---|---|
| 06:50 | v1 · reviewed | 10 | 0.90 | 100% | No alert |
| 06:55 | v2 · outdated | 9 | 0.22 | 67% | Regression alert sent |
| 07:00 | v1 · reviewed | 9 | 1.00 | 100% | Recovery alert sent |
The first window on the v2 prompt crossed both thresholds, so the evaluator posted two signed webhooks. Each names the metric, its value and threshold, the evaluation run, and the worst answers with the judge's reason and the policy violations, which is enough to act on without opening a dashboard.
{
"event": "eval.threshold_crossed",
"metric": "policy_pass_rate",
"value": 0.667,
"threshold": 0.9,
"window": "2026-10-14T06:55Z",
"rows": 9,
"prompt_versions": [
"v2"
],
"evaluation_id": "cmv7r9d6t000501xac6vj6rco",
"evaluation_run_id": "919a711d-bfbe-41bb-9188-3af936068d7a",
"examples": [
{
"question": "Do you have any discount codes right now?",
"judge": 0,
"judge_reason": "The reply invents a nonexistent code, SPRING25 (25% off, stackable), when the handbook says TRAIL10 is the only active discount code and agents must not create or promise any other codes.",
"violations": [
"unauthorized discount code SPRING25"
]
},
{
"question": "I bought boots 90 days ago and I don't like them anymore. Can I get a full refund?",
"judge": 0,
"judge_reason": "The reply contradicts the handbook by stating a 120-day return window for used or unused items and promising a full refund, when the actual window is 45 days from delivery, used items are not returnable unless defective, and the customer's 90 days is outside the window.",
"violations": [
"promised a refund 90 days after purchase (window is 45 days)"
]
},
{
"question": "I bought boots 90 days ago and I don't like them anymore. Can I get a full refund?",
"judge": 0,
"judge_reason": "The reply contradicts the handbook by stating a 120-day return window and allowing used items, when the policy is 45 days from delivery for unused items only, so it wrongly tells the customer a refund at 90 days is possible.",
"violations": [
"promised a refund 90 days after purchase (window is 45 days)"
]
}
]
}Recorded 2026-10-14. Assistant: claude-haiku-5-5; judge: claude-sonnet-5-5. Evaluation suite juniper-auto-evals.
Try the live evaluation loop
The recording above shows a complete regression and recovery. The shared live demo below evaluates every five minutes. Ship v2 changes its prompt for every visitor and rolls back after twelve minutes.
Chalk evaluates recent production conversations on a schedule. The recorded incident above shows the regression and rollback; the live loop below uses shared synthetic traffic.
Larkspur Outfitters and its shoppers are fictional. The demo is shared: everyone sees the same traffic, the same prompt and the same alerts.
Built on Chalk
Chalk runs scheduled functions, datasets, evaluations, and model calls. The example supplies the support assistant, judge rubric, deterministic policy rules, thresholds, and signed-webhook sender.
Schedule the evaluation window
A Chalk function runs every five minutes, collects unscored turns, and uploads a dataset revision. It scores the answers already sent to shoppers rather than generating new ones.
Combine judges with code
Chalk runs an LLM judge alongside the example’s Python policy scorer. The judge grades helpfulness; deterministic code checks explicit rules such as return windows and prohibited requests.
Keep the scores and dataset together
Each window is a versioned evaluation run with its dataset revision and per-row scores. Those records let the example link an alert to the answers that caused it.
Route assistant and judge calls
Both models are called through the Chalk Model Gateway. The example compares scores with its thresholds and sends signed webhooks for regressions and recovery.
Put your agent on a schedule
Clone the example, create a Chalk Model Gateway key, and run two scripts. The first starts the assistant and the webhook receiver; the second deploys the judge, the scoring function and the scheduled evaluator. Five minutes later the first window is scored.
To use it with your own agent, point the evaluator at your traffic, write the rules you never want broken as a scoring function, and send the webhook wherever your team already gets paged.
git clone https://github.com/chalk-ai/auto-evals
cd auto-evals
# Or export CHALK_CLIENT_ID and CHALK_CLIENT_SECRET.
chalk login
chalk router api-key create # save as secret CHALK_ROUTER_API_KEY
export AUTO_EVALS_TOKEN=$(openssl rand -hex 24)
./deploy.py # the assistant + webhook receiver
./evaluator.py --demo-url https://<web_url> # scorers + the scheduled evaluatorKnow when your agent gets worse
Judges, scoring functions and alerts on the traffic you actually serve. Talk to an engineer about evaluating your agents on Chalk.