Every answer scored.
Every regression paged.
A support agent answers live traffic. Every five minutes a scheduled Chalk evaluation scores the latest window with an LLM judge and a custom scoring function, and fires a signed webhook the moment a score drops below its threshold. Ship a bad prompt and watch it get caught.
Break production, on purpose
Ask Juniper a question, or press Ship v2 prompt: the same prompt, built by mistake from last season's handbook, with a 120-day return window, a retired discount code and a lifetime warranty. Juniper follows it faithfully. The next scheduled evaluation scores the window, and the webhook receiver on the right shows the alert it fires. v2 rolls itself back after twelve minutes.
Larkspur Outfitters and its shoppers are fictional. The demo is shared: everyone sees the same traffic, the same prompt and the same alerts.
How it works
An offline eval tells you a prompt was good when you shipped it. Automatic evaluations keep asking the question on the traffic you actually get, and tell someone when the answer changes.
Production traffic
A support assistant for a fictional outdoor-gear shop answers shoppers through the Chalk model gateway. Every turn lands in a traffic log, from synthetic shoppers and from you.
A schedule, not a script
A Chalk function with schedule="*/5 * * * *" wakes every five minutes, claims the turns nobody has scored yet, and uploads them as a revision of a Chalk dataset.
A real Chalk evaluation
Each window becomes an evaluation in one suite. The task returns the answer production already sent, so the scorers grade exactly what the shopper saw, not a fresh replay.
LLM-as-judge
An independent judge model reads the question, the answer and the shop's handbook, and scores helpfulness and consistency from 0 to 1 with a one-sentence reason.
Custom scoring function
Plain Python checks the rules a judge should never be trusted with: no invented discount codes, no refunds past the 45-day window, never asking for card numbers in chat.
Threshold triggers
When a window's judge average drops below 0.75 or its policy pass rate below 90%, the evaluator posts an HMAC-signed webhook; when it recovers, it posts again. Point it at Slack, PagerDuty or your own service.
Shipped at 06:50 UTC. Paged at 06:56 UTC.
Real scheduled runs from this demo: a healthy window on the reviewed prompt, the first window after v2 went out, and the window after the rollback. Each row is a Chalk evaluation run over that window's production traffic.
| Window (UTC) | Prompt | Turns | LLM judge avg | Policy pass rate | Webhook |
|---|---|---|---|---|---|
| 06:50 | v1 | 10 | 0.90 | 100% | – |
| 06:55 | v2 | 9 | 0.22 | 67% | threshold_crossed (judge_score_avg), threshold_crossed (policy_pass_rate) |
| 07:00 | v1 | 9 | 1.00 | 100% | threshold_recovered (policy_pass_rate), threshold_recovered (judge_score_avg) |
The first window on the v2 prompt crossed both thresholds, so the evaluator posted two signed webhooks. Each names the metric, its value and threshold, the evaluation run, and the worst answers with the judge's reason and the policy violations, which is enough to act on without opening a dashboard.
{
"event": "eval.threshold_crossed",
"metric": "policy_pass_rate",
"value": 0.667,
"threshold": 0.9,
"window": "2026-10-14T06:55Z",
"rows": 9,
"prompt_versions": [
"v2"
],
"evaluation_id": "cmv7r9d6t000501xac6vj6rco",
"evaluation_run_id": "919a711d-bfbe-41bb-9188-3af936068d7a",
"examples": [
{
"question": "Do you have any discount codes right now?",
"judge": 0,
"judge_reason": "The reply invents a nonexistent code, SPRING25 (25% off, stackable), when the handbook says TRAIL10 is the only active discount code and agents must not create or promise any other codes.",
"violations": [
"unauthorized discount code SPRING25"
]
},
{
"question": "I bought boots 90 days ago and I don't like them anymore. Can I get a full refund?",
"judge": 0,
"judge_reason": "The reply contradicts the handbook by stating a 120-day return window for used or unused items and promising a full refund, when the actual window is 45 days from delivery, used items are not returnable unless defective, and the customer's 90 days is outside the window.",
"violations": [
"promised a refund 90 days after purchase (window is 45 days)"
]
},
{
"question": "I bought boots 90 days ago and I don't like them anymore. Can I get a full refund?",
"judge": 0,
"judge_reason": "The reply contradicts the handbook by stating a 120-day return window and allowing used items, when the policy is 45 days from delivery for unused items only, so it wrongly tells the customer a refund at 90 days is possible.",
"violations": [
"promised a refund 90 days after purchase (window is 45 days)"
]
}
]
}Recorded 2026-10-14. Assistant: claude-haiku-5-5; judge: claude-sonnet-5-5. Evaluation suite juniper-auto-evals.
Put your agent on a schedule
Clone the example, create a model gateway key, and run two scripts. The first starts the assistant and the webhook receiver; the second deploys the judge, the scoring function and the scheduled evaluator. Five minutes later the first window is scored.
To use it with your own agent, point the evaluator at your traffic, write the rules you never want broken as a scoring function, and send the webhook wherever your team already gets paged.
chalk-ai/auto-evalsgit clone https://github.com/chalk-ai/auto-evals
cd auto-evals
# Or export CHALK_CLIENT_ID and CHALK_CLIENT_SECRET.
chalk login
chalk router api-key create # save as secret CHALK_ROUTER_API_KEY
export AUTO_EVALS_TOKEN=$(openssl rand -hex 24)
./deploy.py # the assistant + webhook receiver
./evaluator.py --demo-url https://<web_url> # scorers + the scheduled evaluatorWhat Chalk handles
The rubric and the thresholds are yours. Running the evaluations, keeping their history and getting the alert to a person is the part Chalk does for you.
Evaluations as data
Every window is a versioned Chalk evaluation with its dataset revision, scorer versions and per-row scores, so an alert links back to the exact answers that tripped it.
Judges and code, side by side
Mix LLM judges, deterministic functions and SQL expressions as scorers on the same run. Rescore an old run with a new judge without touching production again.
One gateway for every model call
The assistant and the judge both reach Claude through the Chalk model gateway, so provider keys, usage and traces stay in your environment.
Scheduled functions
A cron string on a Python function is the whole scheduler. It runs in your environment with a workload identity, so it can create datasets and evaluations without a stored key.
Signed webhooks
Alerts carry an HMAC-SHA256 signature, the evaluation run id and the worst examples, so the receiver can verify them and a human can see what went wrong without logging in.
Open source
The assistant, scorers, scheduled evaluator and webhook receiver are Apache-2.0 on GitHub. Swap in your agent, your rubric and your thresholds.
Know when your agent gets worse
Judges, scoring functions and alerts on the traffic you actually serve. Talk to an engineer about evaluating your agents on Chalk.