TL;DR. Agentic machine learning is agents running the modeling loop itself: propose a hypothesis, implement it, evaluate it against a metric, keep it or discard it, repeat. A person still sets the objective. The agent optimizes toward it, completing dozens of cycles overnight instead of a handful a week. The loop is not the hard part to build. What decides whether a team can trust it is the data layer underneath: whether the agent sees what production saw, and whether the feature it proposes can be served without a rewrite.
A pricing model has been leaving margin on the table since Friday. Someone has to work out why on Monday morning.
Investigating it is manual and slow. An engineer has to reconstruct what the model saw at the moment it scored, then compare that against the training data. Finishing in a day is a good outcome. Now multiply that by every missed prediction worth chasing.
Teams don't need to devote an engineer to that work. An agent can take the first pass. It pulls the relevant decisions, replays the context each one ran on, and drafts a candidate fix. An engineer evaluates the work and ships the change, so a day-long effort becomes a few minutes of review.
That loop has a name now. Agentic machine learning is what happens when agents run the modeling loop. What follows is what the loop requires to be trusted, and that turns out to have less to do with the agent than with the data underneath it.
How does the agentic machine learning loop work?
Agentic machine learning hands the model development loop to an agent. The agent proposes a change, implements it, runs it, scores it against a metric, and keeps or discards it based on the result. A human sets the objective and the agent optimizes toward it.
The loop has six steps:
- Propose a hypothesis.
- Implement it in code.
- Execute it and produce a metric.
- Evaluate whether the result beat the baseline.
- Commit the change or roll it back.
- Repeat, building on what was kept.
It's the loop ML engineers have always run. Context engineering is a new name for feature engineering, which is a new name for data engineering. The name changes and the job does not: find the signals that drive decisions, get them to the model, and make sure they mean the same thing in training that they mean in production.
What changed is the rate. A person runs a handful of model iteration experiments a week. An agent runs dozens overnight.
What do teams gain from agentic machine learning?
Teams gain throughput on work they already know they should be doing and cannot reach.
The failures you currently let through. Every team triages: the loudest three misses get investigated and the rest go on a list nobody returns to. An uninvestigated miss still charges the business every time it fires. A team that can work the whole queue instead of the top of it stops paying for the same mistake twice.
Iteration speed. Model quality is a function of how many cycles a team completes, and most teams are capped on cycles by how long the hand-work takes rather than by ideas. Compressing the investigation means the same headcount runs more experiments against live results.
Headroom across the organization. ML teams are asked to support more use cases each year than they gain headcount for. Every new model brings its own investigation load and its own failures to chase. Teams stall when that load scales linearly with the number of models in production. Handing the repetitive layer to agents is one of the few ways to take on new use cases without growing the team.
How is agentic ML different from AutoML, coding agents, and recursive self-improvement?
AutoML searches a space you defined in advance, while an agent can redefine the space. A coding agent's output either compiles or it doesn't, while an ML agent's output is a number that looks the same whether it is right or wrong. Recursive self-improvement is the same loop taken further, where every accepted change becomes the baseline the next experiment has to beat.
Agentic ML vs AutoML
AutoML searches a space you defined in advance. You pick the model families and the ranges. The search explores inside those walls, and it has been good at that for years.
An agent changes the walls. It can decide the problem was framed wrong, redefine a label, or introduce a feature nobody had considered. AutoML optimizes within your hypothesis. An agent generates hypotheses.
Agentic ML vs coding agents
A coding agent ships code that either compiles or does not. Tests pass or they fail. The feedback is close to binary and it arrives fast.
An agent doing machine learning ships a number. The number looks the same whether it is right or wrong. That difference is why general-purpose coding infrastructure does not transfer cleanly to this work. The code compiling tells you almost nothing about whether the result is real.
Agentic ML vs recursive self-improvement
Recursive self-improvement sits at the further end of the same spectrum. Agentic ML describes agents running the loop. Recursive self-improvement describes a loop that compounds, where each accepted change becomes the baseline the next experiment has to beat.
Even Karpathy's autoresearch has an agent optimizing a different model rather than itself. The tool hands an agent a small language model training setup, runs a fixed five-minute experiment, keeps or discards the change on a single metric, and repeats. The README puts throughput at roughly 12 experiments an hour, or about a hundred while you sleep.
In the run he reported in March, an agent ran about 700 experiments over two days and found 20 optimizations. Applying those 20 changes to a larger model cut training time by 11 percent.
The agent tunes the training code and initial settings for a different, smaller model. It never touches its own weights. What is shipping today is a bounded loop with a human gate on promotion, moving toward fewer gates as evaluation gets more trustworthy.
Does agentic ML work on mature models, and how does it fail?
Yes, including on models teams had already spent years tuning. The failures are data problems.
Instacart published the most detailed account so far. Their delivery time model is a traffic-aware gradient-boosted tree with years of iteration behind it, and they were, in their words, genuinely skeptical that an autonomous research loop could improve it. Several engineers ran independent loops against the same data snapshot, the same splits, and the same validation metric. A 30-trial LightGBM search cut held-out mean absolute error by 3.6 percent against production. A tuned-MLP search cut it 4.8 percent against its own tree-based baseline. On catalog attribute extraction, a 25-round loop improved recall by 8.1 points with precision held above its floor, replacing a process that had taken about a week per attribute.
The gains stacked up out of small wins. Most of the agent's hypotheses failed.
Their write-up also lists what went wrong, which is helpful background because most teams only publish the wins.
- A point-in-time join error that merged the future onto the present.
- Feature leakage, recurrent, and described as particularly sinister because it is so hard to spot in offline analysis.
- An agent reading the held-out evaluation set while it selected hyperparameters.
- Features that added latency, or that needed data production does not have.
In every one of these the metric went up. A join that pulls the future into a training row produces a feature that predicts the label. So does a leaked one. An agent that has seen the held-out set scores well on it.
The loop keeps whatever improves the metric. That is the whole mechanism, and there is nothing in it to catch any of this.
At four experiments a week an engineer reads each one, and a feature correlating a little too well with the label gets noticed. At four an hour nobody is reading. The loop also prefers these features, because a leaked one wins. By morning it has twenty more experiments built on top of it.
How often should engineers review what an agent produces?
Often enough that no change reaches a production model without a human approving it, and rarely enough that review does not become the new bottleneck. Handing work to an agent mirrors a process your team already runs. The agent opens a pull request. It proposes a change and shows the work behind it. An engineer reviews, comments, and merges or rejects.
Engineers can review far more proposals than they can write. That moves a team's ceiling from how fast its people type to how fast they can judge. Changes that reach a production model still get human approval. Narrow, reversible changes with nothing downstream depending on them can run behind a shadow evaluation on their own, so review effort concentrates where the risk actually sits.
An agent may run unattended overnight and produce forty accepted changes by morning. Reviewing each one recreates the bottleneck. Reviewing none of them means trusting a metric that looks identical whether it is right or wrong.
How do you set up evaluations for agentic machine learning?
Start by deciding what you are evaluating, because there are two answers and they need different machinery. One is the model change the agent proposes, scored against a point-in-time correct backtest. The other is the agent's own behavior, meaning whether its plan and tool calls were reasonable and what they touched along the way. A loop that only measures the first will accept a good number produced by a bad process.
For the model change, the mechanics are the same as any prompt or model evaluation. Build a dataset that serves as ground truth, run candidates against it, and score them. Use Chalk functions to define the dataset, create a task and scorers, and make sure every run is stored.
For agent behavior you need the trajectory rather than the answer. Agent traces capture what the agent planned, which tools it called, and what those calls changed, so a run can be reconstructed and scored after the fact.
What data access does an agentic ML loop need?
An agentic ML loop needs access to the same data the production model uses, computed the same way, straight from the source. A loop confined to a single warehouse can only rediscover signal already in that warehouse. Agents that reach operational databases, streams, and APIs can find signals from the state of the world.
In practice, hunting for a signal nobody has used means pulling from Postgres and Snowflake and Kafka and S3, plus an internal service and whatever vendor API somebody wired up last quarter. Often on the same afternoon. What the loop needs is one layer for context that assembles data from the source itself.
What makes an agentic ML loop trustworthy?
Many teams acknowledge the need and benefits of adopting agentic ML. But what decides whether you can trust its output is the data layer under it, and two properties in particular: that the agent sees what production saw, and that anything it proposes can be served without a rewrite.
With Chalk's Context Engine teams define a feature once, in Python or SQL. That definition is what runs in training, in real-time serving, and in anything an agent reads. Values are computed from the source at the moment a decision needs them. Lineage and governance live in the same system as the context.
Point-in-time correct training sets mean an agent backtesting a feature sees what production would have seen at that timestamp, rather than data that arrived later. One definition across training and serving means a feature the agent invents is servable by construction, so the loop cannot propose something that dies on the way to production.
Computing from federated sources rather than a single warehouse is what lets the agent reach real-world signal without ETL jobs and data pipelines.
Agents multiply whatever a team already has, which is why the data layer decides the outcome. An ML Autoresearch Agent can complete this loop against features computed the same way production computes them with Chalk's Context Engine.
See what the loop looks like on your own data. Explore building an ML Autoresearch Agent with Chalk, or talk to an engineer about the model you would point it at first.
Frequently asked questions
What is agentic machine learning?
Agentic machine learning is agents running the model development loop: proposing a hypothesis, implementing it, evaluating it against a metric, and keeping or discarding it based on the result. A human sets the objective and the agent optimizes toward it. It is the loop ML engineers have always run, executed at a rate a person cannot sustain.
How is agentic machine learning different from AutoML?
AutoML searches a space you defined in advance, across model families and hyperparameter ranges you chose. An agent can change the space itself: redefine a label, resample the data, reformulate a feature, or reframe the problem. AutoML optimizes within your hypothesis. An agent generates hypotheses.
Does agentic machine learning replace ML engineers?
No. The human keeps the objective and the approval. What changes is where an engineer's time goes: from writing and running experiments to setting the target metric and judging proposals. Engineers can review far more proposals than they can write, which is why the ceiling moves.
What data access does an agentic machine learning loop need?
The same data the production model uses, computed the same way, plus the sources that have not been centralized yet. A loop confined to one warehouse can only rediscover signal already in that warehouse. A loop that reaches operational databases, streams, and APIs can find signal nobody has used.
Why do point-in-time correct training sets matter for agentic ML?
Because an agent evaluating a feature has to see what production would have seen at that timestamp. If later data reaches the backtest, the agent scores the feature on information the model would not have had, accepts it, and the result does not hold in production.
What should teams put in place before adopting agentic machine learning?
A review cadence, an evaluation setup, and data access. Decide what level of change needs human approval, decide how you score both the model change and the agent's own behavior, and make sure the agent computes features from the same sources and definitions production uses.
