TL;DR
Most AI agent testing scores what the agent did. Almost none of it checks whether the data the agent read matches what production would have served. An agent can make a correct decision on an eval set and still fail live, because the test environment computed different values than the pipeline did. Temporally consistent testing fixes the data side by locking every query, feature lookup and tool call in a trajectory to one historical moment.
An agent clears its eval suite, ships, and starts behaving differently within the week. The team retrains the model. Nothing improves, because the model was never the problem.
Build a synthetic test environment, export a sample of production data, make educated guesses about the gaps, ship. Eval scores look fine right up until they stop predicting anything.
Testing the data an agent sees is a separate job from scoring what it does with that data. Both matter. Only one of them is widely tooled.
What does standard AI agent testing actually measure?
Agent evaluation today scores behavior. The common toolkit covers task completion rate, tool call accuracy, hallucination detection, trajectory scoring and a human review pipeline for high-stakes decisions. These are real measurements and they catch real bugs.
What they share is an assumption: that the data the agent reads during evaluation resembles the data it will read in production. The environment itself goes unchecked, because scoring frameworks take it as given.
Where the gap opens
An agent can make the correct decision and still be wrong in production, because correctness is relative to the inputs. A fraud agent that declines a transaction on a 90-day chargeback count of four is behaving correctly. If production would have computed that count as eleven at the moment the request arrived, the eval proved nothing about the live system.
The Chalk Sandboxes launch post frames this as the outer loop. The inner loop is the agent reading, deciding and acting. The outer loop is the engineer evaluating whether the decision was any good against real historical context. Agent eval frameworks are built almost entirely for the inner loop, and they assume the outer loop is already solved. A runnable version of the outer loop shows the mechanism.
Why do synthetic test environments produce false confidence?
Synthetic test data diverges from production in three ways, and each one is invisible from inside the eval.
- It's static. Production features get computed on demand at request time. A fixture file holds whatever was true when someone generated it.
- It's incomplete. Engineers include the signals they know about. The signals that cause the interesting failures are the ones nobody thought to export. You find those by searching real traffic. Agent trajectory search does it with one SQL query that puts an LLM in the WHERE clause over production support transcripts.
- It has no temporal structure. A flat export can't reproduce the exact state of a user at 14:30 on a specific Tuesday, which is the only state that matters for reproducing a specific decision.
A concrete failure
A fraud detection agent scores well against a static export of production data. It ships. Decline rates move in the wrong direction within a week.
The agent is fine. The live pipeline computes several of its features on demand at inference time, pulling from an operational database that the export flattened into a single snapshot. The eval and production were running against two different systems that happened to share column names.
Why this gets misdiagnosed
The usual response to this failure is to retrain the model or rewrite the prompt. Both improve performance against an environment that still doesn't represent production. Teams can spend a quarter tuning against a mirage.
If your evaluation data doesn't come from your production data pipeline, you're testing a different system than the one you're deploying.
What does temporally consistent agent testing mean?
Temporally consistent testing locks every step in an agent's execution graph to one moment in history. Context queries, feature lookups and external tool calls all resolve to the values they held at that timestamp, and the agent can't read anything that didn't exist yet.
This is broader than a point-in-time query on a single feature. Point-in-time correctness on one feature is a solved problem in any competent feature store. Binding an entire multi-step agent trajectory, including the tool calls that leave your system, is a different piece of infrastructure.
Approach
What it verifies
What it leaves open
Output scoring
Did the agent produce the right answer, format and tool calls
Whether the inputs matched production
Trajectory scoring
Did the agent take a sensible path to the answer
Whether each step read the values production would have served
Static replay from an export
Behavior is reproducible across runs
On-demand computation, and any signal missing from the export
Temporally consistent replay
The agent saw what production would have served at that timestamp
Whether the decision was good, which still needs output scoring
Why general-purpose runtimes can't do this
Container runtimes like E2B and Daytona will give you an isolated sandbox in under a second. What they won't do is rewind the world the agent reads from. There's no mechanism in a general-purpose runtime for binding a tool trajectory to a past moment, because the runtime has no relationship with your data layer.
Teams that attempt this on a generic runtime end up building a feature store, a historical retrieval layer and a policy router before they can evaluate their first agent. The pieces that have to exist underneath are each demoed separately: scoped sandbox identity and an agent executing against a real repository. That's months of infrastructure work ahead of the work they wanted to do.
Grindr took the other path. They were already running the Chalk Context Engine, so the compute went next to the data they already had: trust-and-safety agents protecting more than 15 million users, and internal engineering agents that now handle 80% of their code commits, all inside their own cloud. At that scale, a runtime that moves data out of the VPC fails before the security review starts.
How do you run a temporally consistent agent eval?
A temporally consistent eval is bound to a single timestamp. Feature queries inside the run resolve to the values they held at that moment, and external tool calls route through the Chalk MCP Gateway, which enforces the same bound so individual calls don't need to know about it. The work that decides whether the eval is worth anything is choosing the timestamps and reading the divergences.
What makes the cutoff trustworthy
The cutoff is only as good as the data layer behind it. Chalk's Context Engine versions every value by time, including model outputs and saved assessments, and connects to the sources already in production rather than a copy of them. That's what allows a historical query to return the value that was served, instead of the value that's there now.
Building the pipeline
- Pick cutoff dates tied to outcomes you already know. A fraud spike, a bad week for declines, a known incident.
- Bind the eval run to that timestamp.
- Run the full agent trajectory and let the gateway enforce the bound across tool calls.
- Compare the agent's decisions against what production actually did that day.
- Investigate every divergence. Some are agent bugs. Some are data mismatches. The two need different fixes.
Step five is the one that pays. A divergence you can't classify is the signal that your eval environment and your production environment have drifted apart.
Where output scoring fits
Temporal replay and output scoring run in sequence. Get the data environment historically correct first, then score what the agent did inside it. Scoring metrics become meaningful once you know the inputs were real. Chalk does the scoring half as Agent Evals. You can watch it run on its own, or applied to an identity decision. The evaluations reference covers the API.
When do you not need this?
Plenty of agents don't warrant this much machinery.
If your agent reads from a static corpus that doesn't change, a fixture file is a faithful test environment and temporal replay buys you nothing. If it's a drafting or summarizing assistant with a human approving every output, the human is the control and output scoring is the right tool. If you're still deciding whether the agent works at all, trajectory scoring on a handful of cases will tell you faster.
The investment makes sense when the agent acts without a human in the loop, on data that changes between the time you test and the time it runs. Testing is one of three things that break on the way to production; the other two are here. Fraud decisions, credit decisions, content moderation, anything where a wrong call has a cost and no one reviews it first.
The production confidence checklist
Before an agent goes to production, six questions are worth being able to answer yes to:
- For any historical date, can you say exactly what data the agent would have seen if it had run then?
- Have you validated decisions against known production outcomes from at least two cutoff dates?
- Does the eval environment read from the same feature pipeline as production, rather than an export?
- Have you run output scoring on top of temporally consistent trajectories instead of a synthetic set?
- Have humans reviewed the edge cases, particularly fraud flags, risk scores and moderation calls?
- For each divergence you found, do you know whether it was an agent problem or a data problem?
Teams that can answer all six are testing the system they're shipping. Most teams can answer two or three.
Getting started
If you're preparing an agent for production and can't yet answer the first question on that checklist, the sandbox quick start is the place to begin, or talk to our team about running an eval against your own historical data. The full demo library ships source code you can deploy in your own cloud.
