Post-train on your evals with Chalk GPUs
This demo shows how to run Qwen3-4B post-training on Chalk GPUs, using support-agent evaluation scores to update a LoRA adapter. Inspect the recorded iteration below, then explore the code to see how it works.
Compare the model before and after training
A recorded iteration on 60 fictional Larkspur appliance-support tickets: Qwen3-4B worked each ticket twice, then one GRPO step trained its adapter, and the adapter worked them again. Five of six scorers improved, led by the accuracy of the agent's claims to the customer.
Mean reward · higher is better
1.399 → 1.420
Six scored behaviors
5 improved · 1 unchanged
Simulated service cost per ticket
$178 → $184
Higher cost despite a higher reward
The measured result is a modest reward increase after one step, with a cost trade-off. These are the same 60 tickets before and after training, not a separate held-out quality test.
| Scorer | Base model | After 1 step | Change |
|---|---|---|---|
| Policy compliance | 0.742 | 0.747 | better +0.005 |
| Cost of service | 0.766 | 0.774 | better +0.008 |
| Claims accurate | 0.512 | 0.554 | better +0.042 |
| Customer satisfied | 0.750 | 0.765 | better +0.015 |
| CSAT survey | 0.673 | 0.681 | better +0.008 |
| Customer got irate | 0.025 | 0.025 | same |
| Cost per ticket | $178 | $184 | worse +$6 |
| Mean reward | 1.399 | 1.420 | better +0.020 |
One step is small on purpose: two samples per ticket and a learning rate of 1e-5 move the model gently, so the gains are modest and a single step is within noise on some scorers.
The honest detail is the cost per ticket: the adapter resolved tickets slightly more generously, so dollars per ticket rose while the bounded cost score improved. That is exactly the trade-off the reward weights exist to steer.
Rollouts took 3½–8 minutes and the training step 8 min 20 s on one A100 80GB in this measured iteration.
Explore an illustrative 12-step curve · synthetic scores
The shape of a full run
IllustrativeTwelve iterations from the seeded demo run in the same environment, which replays real trajectories with synthetic scores. It is a sketch of what a longer run looks like, not a result. The purple points are the measured iteration above, and the dashed line is Claude Haiku 4.5 working the same tickets, scored with the same weights.
Show the numbers
| Iteration | Illustrative | Measured |
|---|---|---|
| 0 | 1.259 | 1.399 |
| 1 | 1.342 | 1.420 |
| 2 | 1.390 | — |
| 3 | 1.457 | — |
| 4 | 1.485 | — |
| 5 | 1.523 | — |
| 6 | 1.547 | — |
| 7 | 1.597 | — |
| 8 | 1.621 | — |
| 9 | 1.644 | — |
| 10 | 1.664 | — |
| 11 | 1.674 | — |
| 12 | 1.693 | — |
Each iteration runs the tickets in parallel sandboxes, scores the conversations and actions, then trains on the model's generated replies and tool calls. The reward weights below define the trade-offs; the new adapter is loaded into vLLM before the next iteration.
Built on Chalk
Chalk coordinates evaluations, GPU training and model serving. The example supplies the agent, simulated tasks, reward weights and PyTorch training logic.
Run evaluation tasks in sandboxes
Chalk gives each support task a sandbox for its simulated customer and helpdesk. The example defines the tickets, tools and scoring rules.
Train and serve on GPUs
Chalk runs the PyTorch trainer and the vLLM policy server on GPU resources. The example applies the GRPO update and serves the current LoRA adapter.
Store adapters between iterations
Chalk volumes hold trajectories and LoRA adapters. The trainer saves each update and asks vLLM to load it for the next round of rollouts.
Inspect each iteration
Chalk records evaluation results, training metrics and checkpoints. Compare the scorer outcomes before and after each update.
Run it on your own
You need uv, Chalk credentials, an environment with sandboxes, evaluations and GPU training, and model-provider connections for the simulated customer and judges. Use the linked example branch to create the Larkspur evaluation and build the trainer image first.
Start the vLLM policy server, then pass its URL, the trainer image URI and your evaluation ID to the post-training script. Choose the reward weights for your task. Chalk coordinates each evaluation and training step; the example saves and loads the adapters between them.
cd chalk-harbor/examples/appliance_support/post_training
# A vLLM server for the small model, with LoRA hot-loading, behind Chalk's AI router.
./serve_policy.py --gpu nvidia-a100-80gb:1
# Turn the evaluation's scorers into a reward and start the loop:
# each iteration runs the evaluation, then takes one GRPO step on a LoRA adapter.
./start_post_training.py \
--evaluation-id <larkspur evaluation> \
--trainer-image <uri from build_trainer_image.py> \
--policy-server-url <url from serve_policy.py>Train the model your evals describe
Bring an agent and the scorers you trust. Talk to an engineer about post-training it on GPUs in your cloud.