Today we're announcing Chalk Notebooks, built for agentic ML.
Shipping a model is an exercise in confidence: that the model and features you trained on will behave the same way in the real world, and that the real world will keep behaving like the world you trained on.
Agents, even on frontier models, test that confidence. They rapidly analyze data and write code, but they can miss basic steps, fumble judgment calls, and hand back overconfident answers. Ask one if it's sure and it might fold or double down, with little in between. When the cost of being wrong is high (and in production ML it is), trusting agents on their word is professional negligence.
The chat log holds the agent's side of the story, but it can't answer the question a model change turns on: how will production behave with the new model? No test proves an analysis the way tests prove code, so validating a model change means following the reasoning and evidence behind it. Short of a randomized controlled experiment on live decisions, the best you can do is replay history using only what each decision knew at the time.
Whether you are in the loop, following the agent's steps, or on the loop, reviewing them afterward, the reasoning and evidence must outlive the chat session. That is reproducibility, and in ML it starts with the data.
Why we built Chalk Notebooks
Production ML is full of subtle sharp edges, and accidentally analyzing data that doesn't represent production is one of the sharpest. We built Chalk Notebooks to take that edge away.
Chalk assembles data from your production sources and serves it fresh to your models. A Chalk Notebook is a hosted notebook whose code runs inside your own cloud, next to your production deployment. It queries that deployment directly, so you and your agents record and reproduce analysis on the data your models actually see.
Chalk Notebooks have Python, SQL and markdown cells, a chalk notebook command-line interface (CLI) and a Model Context Protocol (MCP) server, so an agent can drive them. They also support three things a notebook off the platform cannot:
- One live federated query across your application databases, warehouses, streams and APIs, plus the online and offline features built on them
- Point-in-time correctness as a query parameter
- Production and a Chalk branch (a copy of your deployment serving no production traffic), queried from the same cell
Watch it work
In this demo, my agent will use a Chalk Notebook to run federated SQL queries across production databases to find the driver of a loan default spike, build point-in-time features, retrain the model on Chalk Model Training, test the candidate on a Chalk branch against production, and hand me a candidate model for promotion and the analysis that backs it up.
Say I run the loan approval model for a point-of-sale lending startup. Over Sunday coffee I see a streak of bad loans: first-payment defaults are up sharply over two months and my model never flagged it.
I open a connection to my coding agent running on a Chalk sandbox in my production environment. My sandbox has my model's repo mounted as a Chalk volume, the chalk CLI installed, and Chalk's MCP server connected. I ask it to investigate the issue.
The agent reads the repo and runs a few orientation queries via the Chalk MCP Server. After getting oriented to the problem, it creates a Chalk Notebook via the Chalk CLI.
I open the notebook it created in the Chalk dashboard and can see it start adding cells in real-time. But knowing the agent's investigation may take a while, I decide I don't want to be a human in the loop. I finish my cortado, shut my laptop and walk home.
Later, the agent notifies me that the notebook is complete, including the analysis and style peer reviews. I start my review.
(Feel free to explore the agentic ML notebook on your own here.)
Two things jump out from the summary. First, a Florida pawn shop fraud ring may be involved. Second, it got there using most of the platform, and the notebook shows its work.
What happened. The notebook's exploration starts with queries to understand the problem. Below, you can see how the agent used a federated SQL query to analyze weekly loan decision data and outcomes across the online lending_db and the offline analytic data warehouse (the join in blue). The accompanying charts show that the funded default rate had been at 2.0-2.4% from March to May, but jumped to 3.4-3.8% starting in June, while the model's mean prediction remained around 2.0%.
Who and where. The agent slices the default increase by merchant category, geography and merchant age, and one slice holds the key. Six Florida pawn shops onboarded this year hold 355 of the 435 pawn-shop defaults and together have a default rate of 90.8%, while every other merchant sits at 2.5%. Worse, my production model predicted a default rate of 1.91% for them, safer than the rest of the loan book. The signal appears to be related to the merchant, but my model was only looking at signals on the borrower.
The features. The agent identifies six point-in-time features about the merchant and its borrowers and builds a training frame of 72,837 loans in one ChalkSQL query, each computed as of that loan's own origination. The leak rules are written into the query predicates: a feature can use only what was knowable when the loan was approved. The resolvers that later compute and serve these features in real-time carry the same predicates, so training and serving agree by definition. The agent validates that 300 sampled loans on the Chalk branch all match the training data frame.
It shows me how these features differ between the six Florida pawn shops and all other merchants.
The models. Every model fit runs on Chalk Model Training. On a holdout of 12,242 labeled loans, set aside because they originated in the four weeks after the training window, and at a matched 95% approval rate, the model candidate cuts the default rate among approved loans from 3.4% to 2.7%. However, the agent sees that retraining on production's own six inputs with fresher training data gets down to 2.9% without new features, so most of the gain is from the fresher training data.
The agent realizes that, because loan outcomes aren't discovered for over a month after the loan is originated, the training window actually contains information about the six Florida pawn shops that the model couldn't have known when the defaults started.
So the agent runs a more honest walk-forward analysis that retrains the model every week on only the outcomes reported by that Monday. That candidate declines practically nothing from the six merchants for six weeks, then declines about two thirds of the six merchants' defaults. The agent finds an additional rule about the merchant on top of the model that catches all of the segment's defaults and fires on no other merchant's loans, bringing the default rate down to around 2%.
Checking against production. The agent then runs a live test between production and the Chalk branch. chalk apply --branch deploys the features, resolvers, and artifact beside production. On the last four weeks' worth of applications, production approved 189 of the six merchants' 190. The Chalk branch approves none, worth $364,390 of principal, and approves 96.7% of an 1,800-loan sample of everyone else against production's 95.8%.
Asked live, the way the approval service asks production, one borrower's $1,900 loan application at one of the six Florida pawn shops scores a loan default prediction of 3.1% on production (approved) and 73.1% on the Chalk branch (declined). The same borrower at an established merchant scores 3.0% on the Chalk branch (approved). The blue highlight below is the online query the agent used for the analysis.
The recommendation. The agent recommends promoting three things in this order of confidence: the merchant rule, the retrained model, and the six features. (I'm also left with the questions about how to handle the six Florida pawn shops, and how to treat new merchants with no reported outcomes.) Looks like I'll have a productive Monday.
Governed by default...
An agent with your production data is a different security posture than an engineer with it. If you skip permissions on a React frontend, you might lose an afternoon, but if you skip them on your production data, you might lose the data.
This one held a service token scoped to one environment, called its model through the Chalk Model Gateway on a budgeted, revocable key, and worked in a sandbox that mounted only the repo it needed. Write access is granted per environment: on in dev, off in prod. Row and column access policies apply to agent-generated SQL, and Chalk Notebook kernels are sandboxed compute with egress you control. More on governance and observability.
...with a human on the loop
Everything above happened in one notebook against the deployment that production runs on: federated queries found the segment, point-in-time features described it, Model Training fit the candidate, and a Chalk branch scored it against production. The agent was never able to change production.
This is as it should be. The agent investigates; the human ships.
Get started
Chalk Notebooks are available now. Open one, connect your coding agent to the Chalk MCP server, and get started with agentic ML on Chalk.
