Run Evals

Chalk Evaluations

Close the improvement loop between development and production.

Teams running production AI on Chalk

End-to-end evals

Know your agents are getting better.

Replay your agent on real production decisions, with the data exactly as it was. Score every run, compare versions side by side, and keep scoring after you ship.

Explore Chalk Evaluations

Evaluate agents against real-world scenarios
Replay agent decisions with the exact data, context, and tools available at decision time. Build evaluations that reflect how your agents actually operate.
Run reliable, repeatable evaluations
Version datasets, tools, and evaluators to ensure consistent results across experiments. Compare runs, benchmark performance, and measure whether changes actually improve your agents.
Optimize and continuously improve agent performance
Trace every execution, inspect tool calls, queries, latency, and outputs. Identify failure points, understand agent behavior, and turn evaluation results into actionable improvements.

How it works

Build, test, and improve, all in one loop.

Collect representative inputs for the task, from your production historical datasets.
Datasets docs
Choose the criteria that determine success, such as answer quality, customer satisfaction, tool-call correctness, or cost.
Scorers docs
Give the evaluation a name, connect the dataset, task function, and scorers, then run the task across the dataset.
Running evaluations docs
Review aggregate scores, examples, and traces. Spot failures, extra tool calls, high costs, and regressions.
Reading results docs
Turn new production failures into eval cases and more specific scorers. Use for fine-tuning.
Scheduling docs

import chalkcompute

# Upload production history as a new dataset revision
revision = chalkcompute.DatasetClient().upload(
    "support-triage", ["jan.parquet", "feb.parquet"]
)

# Or point at the latest revision of an existing dataset
dataset = chalkcompute.DatasetRevisionRef(
    dataset_name="support-triage"
)

@chalkcompute.function
def triage(message: str) -> str:
    ...  # your agent

intent_match = chalkcompute.scorers.json_match(
    paths=["intent"], name="intent-match"
)

class IntentQuality(BaseModel):
    score: float = Field(ge=0, le=1)
    reason: str

resolved = chalkcompute.scorers.llm_judge(
    IntentQuality,
    model="openai/gpt-5",
    inputs=("message", "output", "expected"),
    instructions=(
        "Grade whether the output JSON correctly captures the "
        "customer's intent, using the message and expected JSON. "
        "Score 1 for correct, 0 for incorrect; explain the grade."
    ),
)

suite = chalkcompute.EvaluationSuite.create("Release")

evaluation = chalkcompute.Evaluation.create(
    "support-triage",
    dataset=dataset,
    task=triage,
    scorers=[intent_match, resolved],
    suite_id=suite.id,
)

run = evaluation.run().wait()
print(run.status)  # EvaluationRunStatus.SUCCEEDED

Time-correct context

Evaluate agents with the context they actually had.

Evaluate on production data. Turn real production decisions into reproducible evaluation cases.
Replay the entire trajectory. Apply one consistent time cutoff across every query and tool call.
Catch hindsight bias. Automatically flag answers that rely on evidence that wasn't available at decision time.

Try the time-correct demo

Inspect & compare

Every score opens into the reason behind it.

Compare experiments. Track scores, cost, latency, and tool usage across runs.
Benchmark against baselines. Measure improvements and regressions, including cost and performance.
Inspect individual results. Open any evaluation case to see the agent's output, scores, errors, and execution trace.

See how to compare results

Automatic evaluations

Keep scoring and improving after you ship.

Schedule with a cron string. Each window becomes its own dataset and evaluation.
Reuse suitable scoring logic as reward functions for post-training.
Run model-training jobs against those rewards.

Try the auto-evals demo

Everything around the eval

Evals run on the same platform as your agent.

The first product we ever deployed with Chalk paid for our team, probably more, in net revenue.

Eric SimonStaff Engineer, Medely

Go deeper

Run it yourself.

Demo
Time-correct evals
Score a dispute agent on what it knew: 81% vs. 100%.
Open demo
Demo
Automatic evals
Ship a bad prompt and watch the scheduled eval page you.
Open demo
Docs
Evaluations quick start
Upload a dataset, deploy a task, run your first eval.
Read docs
Webinar
Give agents context for evals and production
Watch the on-demand session.
Watch

Know which agent change is safe to ship.

We'll help you set up an evaluation that gives you the confidence to deploy your agent to production.

Talk to an engineer Read the docs

Chalk Evaluations

Evaluate agents against real-world scenarios

Run reliable, repeatable evaluations

Optimize and continuously improve agent performance

Time-correct evals

Automatic evals

Evaluations quick start

Give agents context for evals and production

Know which agent change is safe to ship.