One image per step.
Zero workers between runs.
Run a fictional grocer's nightly reorder pipeline on Chalk Workflows. One step fans out to three parallel tasks on three different images, one of them rides out a rate-limited API with a retry, and every worker scales up for the run and back to zero after it.
Watch a run from cold
Press Run it live to submit a real run to workers that are sitting at zero replicas, or replay a recorded run. Each lane is a task; the dashed stretch is the wait for a worker to wake, and the red bar is the supplier API turning the first attempt away. Live runs are rate-limited.
How a run unfolds
A workflow is an async Python function that calls tasks. Chalk runs it on scaling groups that exist only as long as there is work for them.
Deploy
deploy_workflow_worker packages the script into an image and creates a scaling group at zero replicas, plus one more group for each distinct task image.
Submit
A run is submitted by name to the worker's task queue. The open run wakes the group, and the orchestrator records every step as it happens.
Fan out
submit() starts independent tasks at once. Tasks with their own image wake their own group, and a failed attempt is retried by the task's policy.
Deploy once, submit by name
Define tasks with @task and the orchestration with @workflow, then run the script to deploy its worker. Any service with Chalk credentials can submit runs by name; it never imports your pipeline.
Deployed workers and per-task images are in preview and ship in upcoming chalkcompute and chalkpy releases. Talk to us to try them in your environment today.
# /// script
# dependencies = ["chalkcompute", "chalkpy[workflows]"]
# ///
import chalkcompute
from chalk.workflows import task, workflow
@task
def score(rows: list[int]) -> list[float]:
return [round(row * 0.1, 2) for row in rows]
@task
def summarize(batch_id: str, scores: list[float]) -> dict:
return {
"batch_id": batch_id,
"rows": len(scores),
"mean_score": sum(scores) / len(scores) if scores else None,
}
@workflow
async def score_batch(batch_id: str, rows: list[int]) -> dict:
scores = await score(rows)
return await summarize(batch_id, scores)
if __name__ == "__main__":
worker = chalkcompute.deploy_workflow_worker(name="wf-score-batch")
print("task queue:", worker.task_queue)Pipelines without a standing cluster
Batch jobs, data prep and inference pipelines run when they run. Nothing should be paid for in between.
An image per task
Give one step NumPy and another scikit-learn without one giant image. Equal images share a group, and steps without one use the main worker.
Scale to zero
Every group starts at zero, wakes when a run reaches it, stays up while a run is open, and returns to zero after its idle delay.
Retries and timeouts
Set retries, retry_delay and timeout on each @task. A rate-limited call or a flaky upstream costs a retry, not the run.
Parallel by default
Start independent tasks with submit() and gather the results. Each call keeps its task's image, retry policy and timeout.
Durable runs
The orchestrator records progress, so a run keeps going after the process that submitted it exits, and results stay retrievable.
Plain async Python
Workflows are async functions that call tasks. Call one directly to test it in-process, or with ChalkClient().run_workflow for durable local runs.
Run your pipelines next to your data
Talk to an engineer about running workflows on scale-to-zero workers in your cloud.