Kill the worker.
The agent keeps going.
This demo shows a support-digest agent working through a week of fictional tickets for several minutes. Every model and tool call is a checkpointed workflow step, its memory lives in a Chalk Volume, and its waits are durable timers that survive restarts. Kill its worker mid-step and watch it pick up where it left off.
See a run recover after its worker exits
Start with recorded playback to understand the checkpoints before trying the longer live experiment.
Try this experiment · expected results
1Replay a completed run
Choose Replay a recorded run and follow completed steps, retries, and memory writes.
Expected: The timeline shows which work completed and which step was retried.
2Try recovery live
Start a live run, then choose Kill the worker while it is running. Allow about seven minutes for the full run.
Expected: A new worker retries the interrupted step; completed checkpoints and saved memory remain available.
Replay a recorded run first. Follow completed steps, worker loss, and recovery; Chalk Workflows coordinates the steps while agent memory is stored in a Chalk Volume.
Built on Chalk
Research, triage and monitoring agents do real work over long stretches. They should pay for the work, not for the waiting, and never start over.
Waits that cost nothing
asyncio.sleep in a workflow is a durable timer held by the orchestrator, not a thread on a worker. It survives restarts, and the worker scales to zero between runs.
Deploy the file as a worker
deploy_workflow_worker turns agent.py into a scale-to-zero worker group. It starts when a run is queued and stops shortly after the last one finishes.
Retries per step
Set retries, retry_delay and timeout on each @task. A rate-limited tool or a lost worker costs a retry of one step, not the whole run.
Models through the gateway
The agent calls its model through the Chalk Model Gateway, so keys, usage and traces stay in your environment.
An agent loop, made durable
Write the agent as an async @workflow and make each model or tool call a @task. Keep memory in a Volume so any step, on any worker, can read what earlier steps wrote.
Deployed workflow workers are in preview and ship in upcoming chalkcompute and chalkpy releases. Talk to us to try them in your environment today.
import chalkcompute
from chalk.client import ChalkClient
worker = chalkcompute.deploy_workflow_worker(
source="agent.py",
name="wf-durable-agent",
shutdown_delay_seconds=20,
env={"ROUTER_API_KEY": chalkcompute.Secret.from_chalk_env("CHALK_ROUTER_API_KEY")},
)
ChalkClient().trigger_workflow(
"support_digest",
{"run_key": "week-42", "ticket_ids": ["FH-1041", "FH-1042"]},
task_queue=worker.task_queue,
wait=False,
)Implementation details
How the example works
Every step is a checkpoint
Each model call and tool call is a workflow task. The orchestrator records its result the moment it finishes, so the run never redoes a finished step.
Kill it, it keeps going
When a worker dies mid-step, that attempt times out and runs again on a fresh worker. The run carries on from the last finished step, not from the start.
Memory in a Volume
Notes, a scratchpad, the report and a lessons file live in a Chalk Volume. The report is written from those files, and the next run starts by reading the lessons.
More capabilities
Memory you can inspect
Every Volume write is a new version. Read any file the agent wrote, or roll the memory back, from the SDK.
Plain async Python
The agent is an async function that calls tasks. Test it in-process, then deploy the same file as a worker.
Run your agents next to your data
Talk to an engineer about running durable agents on scale-to-zero workers in your cloud.