One image per step.
Zero workers between runs.
This demo shows how to run a fictional grocer's nightly reorder pipeline on Chalk Workflows. One step fans out to three parallel tasks on three different images, one of them rides out a rate-limited API with a retry, and every worker scales up for the run and back to zero after it.
Follow worker startup, parallel tasks, and a retry
A recorded run is the quickest way to read the timeline before submitting a live run.
Try this experiment · expected results
1Replay a recorded run
Choose a recording and select Replay. Follow the task lanes and final reorder plan.
Expected: Dashed stretches are worker startup waits, overlapping bars are parallel tasks, and coral marks a retried failure.
2Run from zero workers
Choose Run it live and compare startup, end-to-end time, and retry count.
Expected: The workflow finishes despite the example supplier rejecting an attempt; the output records the resulting plan.
Compare the cold and warm recordings. Follow worker startup and retries until the reorder plan appears; overlapping task bars show when work ran in parallel.
Built on Chalk
Batch jobs, data prep and inference pipelines run when they run. Nothing should be paid for in between.
An image per task
Give one step NumPy and another scikit-learn without one giant image. Equal images share a group, and steps without one use the main worker.
Scale to zero
Every group starts at zero, wakes when a run reaches it, stays up while a run is open, and returns to zero after its idle delay.
Retries and timeouts
Set retries, retry_delay and timeout on each @task. A rate-limited call or a flaky upstream costs a retry, not the run.
Parallel by default
Start independent tasks with submit() and gather the results. Each call keeps its task's image, retry policy and timeout.
Deploy once, submit by name
Define tasks with @task and the orchestration with @workflow, then run the script to deploy its worker. Any service with Chalk credentials can submit runs by name; it never imports your pipeline.
Deployed workers and per-task images are in preview and ship in upcoming chalkcompute and chalkpy releases. Talk to us to try them in your environment today.
# /// script
# dependencies = ["chalkcompute", "chalkpy[workflows]"]
# ///
import chalkcompute
from chalk.workflows import task, workflow
@task
def score(rows: list[int]) -> list[float]:
return [round(row * 0.1, 2) for row in rows]
@task
def summarize(batch_id: str, scores: list[float]) -> dict:
return {
"batch_id": batch_id,
"rows": len(scores),
"mean_score": sum(scores) / len(scores) if scores else None,
}
@workflow
async def score_batch(batch_id: str, rows: list[int]) -> dict:
scores = await score(rows)
return await summarize(batch_id, scores)
if __name__ == "__main__":
worker = chalkcompute.deploy_workflow_worker(name="wf-score-batch")
print("task queue:", worker.task_queue)Implementation details
How the example works
Deploy
deploy_workflow_worker packages the script into an image and creates a scaling group at zero replicas, plus one more group for each distinct task image.
Submit
A run is submitted by name to the worker's task queue. The open run wakes the group, and the orchestrator records every step as it happens.
Fan out
submit() starts independent tasks at once. Tasks with their own image wake their own group, and a failed attempt is retried by the task's policy.
More capabilities
Durable runs
The orchestrator records progress, so a run keeps going after the process that submitted it exits, and results stay retrievable.
Plain async Python
Workflows are async functions that call tasks. Call one directly to test it in-process, or with ChalkClient().run_workflow for durable local runs.
Run your pipelines next to your data
Talk to an engineer about running workflows on scale-to-zero workers in your cloud.