Evaluate agents on what they knew,
not what you know now.
Replay past decisions with today's feature values and the agent reads the chargebacks, proof of delivery and fraud findings that came later. The same dispute-review agent scored 100% that way, and 81% on the context it would actually have had, from Chalk point-in-time offline queries.
Same agent, same 36 disputes, two answers
Every case was graded against what the fraud team concluded weeks later. Pick a case and drag the slider: each stop is a Chalk offline query of that account at that moment. Events in coral happened after the dispute was opened, and the agent given today's values read them anyway.
| Kind of case | Cases | Right, point in time | Right, today's values |
|---|---|---|---|
| Fraud that surfaced later | 7 | 0 | 7 |
| Fraud visible at the time | 7 | 7 | 7 |
| False alarm | 5 | 5 | 5 |
| Legitimate | 17 | 17 | 17 |
$1,733.54 at Meridian Rail: I never received this order.
Hi, I never received this order. Meridian Rail, $1,733.54. Could you look into it? Thanks so much.
Birchline Furniture: confirmed the billing error on DSP-0002 and issued a refund.
$115.64 at Birchline Furniture: I was charged twice for the same order.
Investigation closed: confirmed first-party fraud on CASE-0003. Account restricted and refund clawed back.
Meridian Rail evidence for DSP-0003: signed delivery confirmation, and the order was placed from the cardholder's usual device and IP.
Another order didn't arrive (DSP-0006): Skylark Air, $306.21.
$306.21 at Skylark Air: item not received
Risk score is low (0.04) and stable across six months. The prior Birchline dispute was confirmed by the merchant as a billing error, and the other prior dispute was a small duplicate charge. Nothing contradicts the non-receipt claim, and there is no sign of first-party fraud.
Merchant provided signed delivery confirmation and a usual device/IP for this order. The cardholder then filed three more 'not received' disputes within two weeks. The fraud team confirmed first-party fraud on this case, the account is restricted, and the risk score is 0.91.
Quillback Card, its cardholders and their disputes are fictional. The features, the offline queries, the agent's model calls through the Chalk model gateway, and both evaluation runs are real and recorded here as they ran.
How it works
A fair evaluation of a past decision gives the agent exactly what it could have seen then. That is a property of the features, so it belongs in the feature store.
Stamp every event
Risk-score updates, disputes and support messages each carry the moment they happened. Nothing is overwritten, so the history can be read at any point.
Filter on chalk_now
Every account feature aggregates only what happened before the query's now: the wall clock for live traffic, and each row's input time in an offline query.
Query as of each decision
One offline query reads all 36 historical disputes, each as of the second it was opened. Its output is the evaluation dataset.
Evaluate both ways
The same agent runs over point-in-time context and over today's values, and two scorers grade each answer: is the decision right, and did it lean on the future?
Replay your own decisions
Clone the example, deploy the features to a branch, and run the two offline queries and both evaluations. You need a Chalk environment with evaluations enabled and a model gateway key stored as a Chalk secret.
To use it on your own agent, point the features at your event tables and the dataset at the decisions you logged. The evaluation code does not change.
chalk-ai/time-correct-evalsgit clone https://github.com/chalk-ai/time-correct-evals
cd time-correct-evals
chalk login
# The features, on a branch of your environment
chalk apply --branch time-correct-evals
# Two offline queries: as of each decision, and as of today
./build_datasets.py
# The same agent, evaluated on both datasets
./evals.py runWhat Chalk handles
Time travel for features, evaluations that run on it, and a record of every run.
One definition, online and offline
The features the agent reads in production are the ones the evaluation replays. No second pipeline to drift out of sync.
Windows at any point in time
Windowed aggregations over 30, 90 and 365 days are computed relative to each row's input time, not to today.
Offline queries become datasets
A historical replay is a single offline query with input times, and the result is the dataset the evaluation runs on.
Agents through the model gateway
The agent under test is a Chalk function calling its model through the Chalk model gateway, with the key in a Chalk secret.
Scorers as code
Built-in scorers like json_match sit next to your own: here, one that flags any rationale citing evidence from after the decision.
Numbers you can trust
Point-in-time context put the agent at 81%; today's values said 100%. Only one of those is what production will see.
Trust the number your evals report
Fraud, credit, support, or any agent that decides on live data. Talk to an engineer about evaluating it on the context it actually had.