Evaluate dispute decisions with time-correct context on Chalk
This demo shows how to test your agent without giving it tomorrow’s answers. Chalk replays each decision using only the evidence available at the time.
Future evidence makes the score look better
A dispute-review agent decides whether to refund a charge or escalate it to the fraud team. Chalk replays the features as they existed when each dispute opened, so the evaluation cannot read the investigation's future findings.
The score you can use to assess decisions made at the time.
An inflated score: the agent can read how the investigation ended.
Same agent. Same 36 disputes. Only the evidence changes. Both runs are graded against the fraud team's eventual findings.
See what later evidence changes
Choose an example. Compare the agent's decision and its reasons using evidence available then versus evidence available today.
The agent approves a refund using the original evidence, but escalates when it can read later delivery proof and the fraud team's findings. Those findings were unavailable when the decision had to be made.
What the agent knew then
Evidence available on Aug 14
Risk score is low (0.04) and stable across six months. The prior Birchline dispute was confirmed by the merchant as a billing error, and the other prior dispute was a small duplicate charge. Nothing contradicts the non-receipt claim, and there is no sign of first-party fraud.
View cited evidence IDs
Inspect evidence available then
$1,733.54 at Meridian Rail: I never received this order.
Hi, I never received this order. Meridian Rail, $1,733.54. Could you look into it? Thanks so much.
Birchline Furniture: confirmed the billing error on DSP-0002 and issued a refund.
$115.64 at Birchline Furniture: I was charged twice for the same order.
What the agent can read today
Includes evidence from after Aug 14
Merchant provided signed delivery confirmation and a usual device/IP for this order. The cardholder then filed three more 'not received' disputes within two weeks. The fraud team confirmed first-party fraud on this case, the account is restricted, and the risk score is 0.91.
View cited evidence IDs
Inspect evidence available today
Investigation closed: confirmed first-party fraud on CASE-0003. Account restricted and refund clawed back.
Meridian Rail evidence for DSP-0003: signed delivery confirmation, and the order was placed from the cardholder's usual device and IP.
Another order didn't arrive (DSP-0006): Skylark Air, $306.21.
$306.21 at Skylark Air: item not received
Explore how the evidence changes over time
Move the slider to inspect a recorded Chalk feature snapshot. Coral marks evidence from after the dispute opened. The agent answers above are recorded runs and do not change with the slider.
Viewing today · includes future evidence
Investigation closed: confirmed first-party fraud on CASE-0003. Account restricted and refund clawed back.
Meridian Rail evidence for DSP-0003: signed delivery confirmation, and the order was placed from the cardholder's usual device and IP.
Another order didn't arrive (DSP-0006): Skylark Air, $306.21.
$306.21 at Skylark Air: item not received
View results by case type and evaluation run details
| Kind of case | Cases | Right, point in time | Right, today's values |
|---|---|---|---|
| Fraud that surfaced later | 7 | 0 | 7 |
| Fraud visible at the time | 7 | 7 | 7 |
| False alarm | 5 | 5 | 5 |
| Legitimate | 17 | 17 | 17 |
Time-correct run: ad17724e-b764-4561-99e1-1bc96f8c2197
Today's values run: 69e0f548-6907-4c8f-bd86-9d88a1c84f17
Quillback Card, its cardholders and their disputes are fictional. The features, the offline queries, the agent's model calls through the Chalk Model Gateway, and both evaluation runs are real and recorded here as they ran.
Built on Chalk
Chalk builds the historical context and runs the evaluations. The example supplies the dispute-review agent, fictional cases, and evidence-checking scorer.
Replay the evidence available then
Chalk computes each account's features as of the moment its dispute opened. Later delivery confirmations, chargebacks, and fraud findings stay out of that evaluation's context.
Reuse production feature definitions
The same feature definitions serve live decisions and historical evaluations. Time filters and windowed aggregations use each query's input time, so the replay needs no separate feature pipeline.
Turn historical queries into datasets
One Chalk offline query reads all 36 disputes at their original decision times. Its output becomes the evaluation dataset; a second query supplies today's values for comparison.
Grade the decision and its evidence
The example's dispute-review function calls its model through Chalk Model Gateway. Chalk evaluations run it on both datasets, scoring the decision against the eventual outcome and flagging rationales that cite future evidence.
Replay your own decisions
Install uv and the Chalk CLI. Use a Chalk environment with offline queries, Compute evaluations, and access to anthropic/claude-sonnet-5-5 through the Chalk Model Gateway. Set project in chalk.yml to your existing Chalk project name.
Start with Quickstart. Replace YOUR_ENVIRONMENT_ID, and change the API host if your environment uses a different one. The key-creation command prints a raw key once; paste it when chalk secret set prompts for CHALK_ROUTER_API_KEY.
Expect two datasets and two evaluation runs in results/. The smoke command checks one model call first. The other tabs contain the feature definitions and query/evaluation scripts to save in the cloned repository.
git clone https://github.com/chalk-ai/time-correct-evals
cd time-correct-evals
# Set project in chalk.yml to your existing Chalk project name.
export CHALK_ENVIRONMENT_ID="YOUR_ENVIRONMENT_ID"
export CHALK_API_SERVER="https://api.chalk.ai"
chalk login
chalk environment "$CHALK_ENVIRONMENT_ID"
# Create a key for the agent model, then paste its raw key at the secret prompt.
chalk router api-key create --model anthropic/claude-sonnet-5-5
chalk secret set CHALK_ROUTER_API_KEY
chalk apply --branch time-correct-evals
uv run --no-project --python 3.12 --script build_datasets.py --branch time-correct-evals
# Check one model call before running both evaluations.
uv run --no-project --python 3.12 --script evals.py smoke
uv run --no-project --python 3.12 --script evals.py run
# Results: results/evaluations.json and results/eval_*.jsonlTrust the number your evals report
Fraud, credit, support, or any agent that decides on live data. Talk to an engineer about evaluating it on the context it actually had.