Your evals are
the reward.
Point reinforcement learning at an agent evaluation you already run. A small open model works every task many times in Chalk sandboxes, your scorers grade each attempt, and a GRPO step teaches it to do more of what scored well.
Rollouts, scores, one step, repeat
The example is Larkspur, a fictional appliance company with 60 support tickets that sit on policy boundaries: refunds on day 14, a 48-hour damage window, recalls, billing errors. The student is Qwen3-4B-Instruct with a LoRA adapter. Every iteration runs four steps.
- 01
Run the evaluation
Every support ticket becomes a task in its own Chalk sandbox, with a simulated customer, a helpdesk backend, and a ledger the agent's actions are graded against.
- 02
Sample rollouts
The small model, served by vLLM with its current LoRA adapter, works each ticket several times. One iteration of 60 tickets is 120 conversations, done in 3½–8 minutes in parallel sandboxes.
- 03
Score them
The evaluation's own scorers grade every rollout: policy rubric, cost of service, LLM judges for accuracy and tone, and the customer's survey. Their weighted sum is the reward.
- 04
Take a GRPO step
A Chalk training run reinforces the rollouts that beat their ticket's average, saves the adapter, and hot-loads it into vLLM. On one A100 that took 8 min 20 s.
What one step bought
A real iteration on Chalk's staging GPUs: Qwen3-4B worked all 60 tickets twice, then one GRPO step trained its adapter, and the adapter worked them again. Five of six scorers improved, led by the accuracy of the agent's claims to the customer.
| Scorer | Base model | After 1 step | Change |
|---|---|---|---|
| Policy compliance | 0.742 | 0.747 | better +0.005 |
| Cost of service | 0.766 | 0.774 | better +0.008 |
| Claims accurate | 0.512 | 0.554 | better +0.042 |
| Customer satisfied | 0.750 | 0.765 | better +0.015 |
| CSAT survey | 0.673 | 0.681 | better +0.008 |
| Customer got irate | 0.025 | 0.025 | same |
| Cost per ticket | $178 | $184 | worse +$6 |
| Mean reward | 1.399 | 1.420 | better +0.020 |
One step is small on purpose: two samples per ticket and a learning rate of 1e-5 move the model gently, so the gains are modest and a single step is within noise on some scorers.
The honest detail is the cost per ticket: the adapter resolved tickets slightly more generously, so dollars per ticket rose while the bounded cost score improved. That is exactly the trade-off the reward weights exist to steer.
Rollouts took 3½–8 minutes and the training step 8 min 20 s on one A100 80GB, so a full run is a matter of hours.
The shape of a full run
IllustrativeTwelve iterations from the seeded demo run in the same environment, which replays real trajectories with synthetic scores. It is a sketch of what a longer run looks like, not a result. The purple points are the measured iteration above, and the dashed line is Claude Haiku 4.5 working the same tickets, scored with the same weights.
Show the numbers
| Iteration | Illustrative | Measured |
|---|---|---|
| 0 | 1.259 | 1.399 |
| 1 | 1.342 | 1.420 |
| 2 | 1.390 | — |
| 3 | 1.457 | — |
| 4 | 1.485 | — |
| 5 | 1.523 | — |
| 6 | 1.547 | — |
| 7 | 1.597 | — |
| 8 | 1.621 | — |
| 9 | 1.644 | — |
| 10 | 1.664 | — |
| 11 | 1.674 | — |
| 12 | 1.693 | — |
Why post-train on your evals
Prompting a frontier model gets an agent working. Reinforcement learning gets a model you own to do the job the way your business grades it.
Frontier quality, small-model cost
A 4B-parameter model you host can serve millions of tickets for a fraction of a frontier API's price, if it is trained on your job. Post-training is how it gets there.
No labeled data
The reward comes from the evaluation you already run. There is no dataset of ideal answers to write, and the model learns from its own best attempts.
Optimize what the business measures
Weights decide the trade-offs: compliance first, then cost, with judges and surveys shaping tone. An irate customer is a penalty, not just a lower score.
Your data stays in your cloud
Tickets, transcripts, trajectories and adapters live in your Chalk environment. The rollouts, the trainer and the policy server all run there.
One loop, many workloads
The same sandboxes, scorers, training runs and model registry run every iteration, so a post-training run is a configuration, not a new platform.
Every step is inspectable
Each iteration is an evaluation run with traces, and each GRPO step is a training run with metrics and a versioned adapter you can roll back to.
An evaluation in, a trained adapter out
Serve the small model, pick your reward weights, and start the loop. A Temporal workflow in Chalk drives each iteration: it runs the evaluation, starts the training run, and waits for the new adapter to be live before the next round of rollouts.
The trainer is plain PyTorch and PEFT in six small modules: rewards and Dr-GRPO advantages, chat reconstruction from the agent's trajectory, loss masking, the policy-gradient step, and the loop that saves and hot-loads each adapter.
chalk-ai/chalk-harborcd chalk-harbor/examples/appliance_support/post_training
# A vLLM server for the small model, with LoRA hot-loading, behind Chalk's AI router.
./serve_policy.py --gpu nvidia-a100-80gb:1
# Turn the evaluation's scorers into a reward and start the loop:
# each iteration runs the evaluation, then takes one GRPO step on a LoRA adapter.
./start_post_training.py \
--evaluation-id <larkspur evaluation> \
--trainer-image <uri from build_trainer_image.py> \
--policy-server-url <url from serve_policy.py>Train the model your evals describe
Bring an agent and the scorers you trust. Talk to an engineer about post-training it on GPUs in your cloud.