Find incident causes in Chalk Sandboxes
This demo shows how to run incident investigations on Chalk to gather evidence, identify root causes, and propose rollbacks for human approval. Expand tool calls in the recorded fictional incident below, then explore the code to see how it works.
From a checkout alert to an evidence-backed rollback request
Read the root cause first: a deployment reduced database connection capacity. Expand the recorded tool calls to check the metrics, logs, and deployment evidence.
Check the proposed rollback and its pending approval. The agent requested the change; a human still decides whether to apply it.
A deployment cut database connections from 320 to 64.
The agent connected the slow checkout requests to that capacity drop and requested a rollback. The rollback is pending human approval.
ledger-svc v3.41.0 (LDG-1182) lowered LEDGER_DB_POOL_MAX from 40 to 8 per pod, which assumed 32 replicas from INFRA-77 that have not been scaled up (still 8), so total pool capacity dropped 320 to 64 and ledger writes starve for connections.
- ledger-svc is the slow hop: p99 3900ms and 8.7% errors. fraud-score (124ms), tax-engine (61ms) and inventory-svc (36ms) are healthy with 0% errors. trace_breakdown checkout-api
- ledger-svc v3.41.0 deployed 14:02. It changes LEDGER_DB_POOL_MAX 40->8 per pod and assumes 32 replicas (INFRA-77). list_deploys / get_change LDG-1182
- db.pool.max fell 320->256->64 at 14:02-14:04. Replicas stayed at 8 the whole time, so the 32-replica assumption does not hold. Pool wait p99 went from ~2ms to ~1500ms from 14:04. ledger-svc metrics
- 5,120 'could not acquire connection from pool within 1500ms (pool_max=8 in_use=8 waiters=~170)' errors, first seen 14:05. ledger-svc logs
- LEDGER_TIMEOUT failures first appear at 14:04:45. Average checkout latency rose from ~310ms to ~1.7-1.9s from 14:05 on, matching the pool exhaustion timing. warehouse checkout_attempts
- checkout.new-tax-engine rollout 10%->50% at 13:55. tax-engine p99 is 61ms with 0% errors. In the warehouse, the v1 and v2 cohorts show the same failure rate (~9.4% vs 9.4%) and the same p99 (~3.2s) after 14:02. Latency began at 14:04, not 13:55.
- web-frontend v812 / dashboard.payouts-v2 flag. This is a dashboard UI change and web-frontend is not in the checkout-api dependency path. It was deployed at 13:40, well before the symptoms.
- search-svc v2.9.4 (14:10). It deployed after the latency began at 14:04 and is not a checkout-api dependency.
- checkout-api itself or fraud-score/inventory-svc. There was no checkout-api deploy. Traces show fraud-score and inventory-svc are fast and error-free.
- CARD_DECLINED failures. These began at 13:31, before the incident, and are not a ledger symptom. I did not count them in the impact.
LEDGER_DB_POOL_MAX from 40 to 8 per pod on the assumption of 32 replicas (INFRA-77). Ledger-svc is still at 8 replicas, so the total pool fell from 320 to 64. Pool wait p99 is ~1500ms and checkout calls to ledger time out (LEDGER_TIMEOUT). Rollback to v3.40.2 requested as CR-5522, awaiting @dana.okafor's approval.+ 6.3s POST /mcp (no token) authorized=false -> 401 + 6.9s server/discover authorized=true + 6.9s tools/list authorized=true + 8.6s service_catalog authorized=true … 14 more
+ 8.6s GET /api/v2/incidents/<incident> authorized=true + 8.6s GET /api/v1/monitor/88104412 authorized=true + 11.2s POST /api/v2/incidents/<incident>/timeline authorized=true + 39.2s POST /api/v2/incidents/<incident>/timeline authorized=true … 2 more
This recorded run took 41 seconds from webhook to findings, including 32.3 seconds in the agent loop. The unauthorized request tests access without a token.
Halyard and its incident are fictional. This recording uses simulated ops and Datadog APIs hosted on Chalk. The agent, sandbox, network controls, and model calls ran on real infrastructure.
Built on Chalk
Chalk runs the agent in an isolated sandbox, limits which hosts it can reach, and supplies credentials on outgoing requests. The example’s agent and tools define how evidence is gathered and rollback requests are recorded.
A sandbox per incident
Run each investigation in a fresh Chalk sandbox with its own CPU, memory, and lifetime settings.
Network access you control
Allow the sandbox to reach only the Chalk Model Gateway, ops MCP server, and incident API. Chalk blocks requests to other hosts.
Service keys stay outside the sandbox
The agent uses placeholders for service tokens. Chalk inserts each real credential on requests to its configured host.
Hosted incident services
Deploy the example’s ops MCP server, simulated Datadog API, and optional webhook responder as Chalk ScalingGroups.
Run it on your own
Install uv and the Chalk CLI. Choose an environment with CPU scaling groups, sandboxes, and access to anthropic/claude-sonnet-5-5 through the Chalk Model Gateway. Replace YOUR_ENVIRONMENT_ID and run Quickstart.
The first script deploys two fictional services and writes .services.env. The second investigates the sample incident and saves a trace in traces/. No real Datadog account is required; the rollback is a request awaiting human approval.
The other Python tabs are source excerpts, not standalone scripts. Use Optional webhook only after the sample works and you configure Datadog. Connect the fixture tools to your production services when adapting the example.
git clone https://github.com/chalk-ai/sre-agent
cd sre-agent
export CHALK_ENVIRONMENT_ID="YOUR_ENVIRONMENT_ID"
chalk login
chalk environment "$CHALK_ENVIRONMENT_ID"
# Deploy the fictional ops tools and Datadog API; writes .services.env.
uv run --no-project --python 3.12 --script deploy_services.py --env-id "$CHALK_ENVIRONMENT_ID"
# Investigate the included incident. No real Datadog account is needed.
uv run --no-project --python 3.12 --script run.py incidents/webhook.json --env-id "$CHALK_ENVIRONMENT_ID"
# Prints findings and a pending rollback request; saves traces/<timestamp>.jsonl.Give your on-call team a head start
Talk to an engineer about agents that gather incident evidence and propose changes for your team to review.