An SRE agent that answers the page.
Root cause in 41 seconds.
A Datadog incident opens. The agent reads the runbook, pulls metrics, logs, traces and deploys through your internal MCP server, rules out the red herrings, and posts the root cause to the incident, with a rollback request waiting for a human to approve.
Incident #4127, investigated
A real run, replayed. At 14:02 a ledger deploy cut its database pool from 40 to 8 per pod, assuming a scale-up that never shipped; at 14:09 checkout latency paged. A tax-engine flag ramp and two other deploys look just as guilty on a timeline. Expand any call to see what came back.
ledger-svc v3.41.0 (LDG-1182) lowered LEDGER_DB_POOL_MAX from 40 to 8 per pod, which assumed 32 replicas from INFRA-77 that have not been scaled up (still 8), so total pool capacity dropped 320 to 64 and ledger writes starve for connections.
- ledger-svc is the slow hop: p99 3900ms and 8.7% errors. fraud-score (124ms), tax-engine (61ms) and inventory-svc (36ms) are healthy with 0% errors. trace_breakdown checkout-api
- ledger-svc v3.41.0 deployed 14:02. It changes LEDGER_DB_POOL_MAX 40->8 per pod and assumes 32 replicas (INFRA-77). list_deploys / get_change LDG-1182
- db.pool.max fell 320->256->64 at 14:02-14:04. Replicas stayed at 8 the whole time, so the 32-replica assumption does not hold. Pool wait p99 went from ~2ms to ~1500ms from 14:04. ledger-svc metrics
- 5,120 'could not acquire connection from pool within 1500ms (pool_max=8 in_use=8 waiters=~170)' errors, first seen 14:05. ledger-svc logs
- LEDGER_TIMEOUT failures first appear at 14:04:45. Average checkout latency rose from ~310ms to ~1.7-1.9s from 14:05 on, matching the pool exhaustion timing. warehouse checkout_attempts
- checkout.new-tax-engine rollout 10%->50% at 13:55. tax-engine p99 is 61ms with 0% errors. In the warehouse, the v1 and v2 cohorts show the same failure rate (~9.4% vs 9.4%) and the same p99 (~3.2s) after 14:02. Latency began at 14:04, not 13:55.
- web-frontend v812 / dashboard.payouts-v2 flag. This is a dashboard UI change and web-frontend is not in the checkout-api dependency path. It was deployed at 13:40, well before the symptoms.
- search-svc v2.9.4 (14:10). It deployed after the latency began at 14:04 and is not a checkout-api dependency.
- checkout-api itself or fraud-score/inventory-svc. There was no checkout-api deploy. Traces show fraud-score and inventory-svc are fast and error-free.
- CARD_DECLINED failures. These began at 13:31, before the incident, and are not a ledger symptom. I did not count them in the impact.
LEDGER_DB_POOL_MAX from 40 to 8 per pod on the assumption of 32 replicas (INFRA-77). Ledger-svc is still at 8 replicas, so the total pool fell from 320 to 64. Pool wait p99 is ~1500ms and checkout calls to ledger time out (LEDGER_TIMEOUT). Rollback to v3.40.2 requested as CR-5522, awaiting @dana.okafor's approval.+ 6.3s POST /mcp (no token) authorized=false -> 401 + 6.9s server/discover authorized=true + 6.9s tools/list authorized=true + 8.6s service_catalog authorized=true … 14 more
+ 8.6s GET /api/v2/incidents/<incident> authorized=true + 8.6s GET /api/v1/monitor/88104412 authorized=true + 11.2s POST /api/v2/incidents/<incident>/timeline authorized=true + 39.2s POST /api/v2/incidents/<incident>/timeline authorized=true … 2 more
41 s from webhook to findings, 32.3 s of it in the agent loop. The one unauthorized request is the policy probe, which sends no token.
Halyard, its services, and the incident are fictional. The internal ops MCP server and the Datadog Incidents API are stand-ins served from fixtures on Chalk ScalingGroups; the agent, the sandbox, its network policy, and the model calls are real.
How it works
The useful part of an SRE agent is not the model; it is reaching the systems that know what happened, safely, and leaving the decision with a person.
Paged
A Datadog incident fires a webhook. A small responder on Chalk checks its secret, ignores retries, and starts one fresh sandbox for that incident.
Connected
The agent asks the internal ops MCP server what it can do, and gets eleven tools: catalog, metrics, logs, traces, deploys, changes, flags, runbooks, warehouse, and rollback requests.
Investigated
It follows the runbook, lists what changed, and checks every suspect against at least two sources. Timing alone convicts nothing: the flag that ramped nine minutes before the symptoms is cleared by the data.
Reported, not deployed
Findings, evidence, impact in dollars, and ruled-out hypotheses land on the incident timeline. The rollback is a change request that waits for the owning team's on-call.
Put your own tools behind it
Clone the example and deploy two services, and the agent investigates the sample incident end to end. Point a Datadog webhook at the responder, and it does the same for every new incident.
The agent never hard-codes a tool. Replace the fixture-backed MCP server with one in front of your metrics store, log search, deploy system and warehouse, and the same agent investigates your incidents.
chalk-ai/sre-agentgit clone https://github.com/chalk-ai/sre-agent
cd sre-agent
chalk login
# Ops MCP server + Datadog API stand-in, as ScalingGroups
./deploy_services.py
# Investigate the sample incident in a fresh sandbox
./run.py incidents/webhook.json
# Optional: answer every new Datadog incident
./deploy_responder.pyWhat Chalk handles
An agent with production context is only useful if you can trust where it runs and what it can touch.
A sealed sandbox per incident
Each investigation starts in a gVisor sandbox in seconds and is torn down after. Its egress allowlist is empty except for three hosts.
Credentials it never holds
Tokens for the MCP server, Datadog and the model are placeholders inside the sandbox, swapped in at egress only for the host each belongs to.
Your internal tools, as MCP
The MCP server and the webhook responder are ScalingGroups deployed from Python, next to the data they front, with an HTTPS URL each.
Any model, one router
The agent calls the Chalk model router with an OpenAI-style request. Switching models is a one-string change.
An audit trail on both ends
The agent's trace records every call and result, and the MCP server logs every request it served, so you can check one against the other.
Minutes back on every page
In the recorded run: 19 tool calls and a root cause 41 seconds after the webhook, before most responders have opened the incident.
Give on-call a head start
Incident response, triage, or any agent that needs your internal data. Talk to an engineer about running it in sandboxes next to your systems.