Every issue to the model it needs.
Not your most expensive one.
A judge on the Chalk Model Gateway reads each GitHub issue a coding agent is handed and sends it to gpt-5-nano, gpt-5.4-mini or gpt-5.5. Paste any public issue to see which one it picks, the criteria it matched, and what that choice saves. Nothing gets solved.
Which model should take this?
Pick one of twelve real pandas issues or paste any public GitHub issue URL. The issue goes through a live route on the Model Gateway: the judge classifies it, and the request is served by the tier it picked with a 48-token budget, so you see the decision without paying for a fix.
The demo is rate limited per visitor. Issues are fetched from GitHub's public API, and only the decision is shown.
How it works
Most of an agent's queue is not frontier work. A route with a judge decides per request, so the caller never has to.
Send the task to a route
The caller asks for chalk/issue-triage instead of a model. Here the task is a GitHub issue, handed to a coding agent with the repo name, labels and body.
A judge reads it
Before anything is served, the Model Gateway asks a small judge model which level the request is. Each level has written criteria; the judge returns one label and nothing else.
The level picks the model
Level 0 goes to gpt-5-nano, level 1 to gpt-5.4-mini, level 2 to gpt-5.5. The chosen model comes back in the x-chalk-routed-model header.
A safe default
If the judge times out or answers with something unexpected, the route serves its default target, and every outcome can carry its own fallback chain.
Twelve pandas issues, one route
Docs fixes, bugs, refactors, API design questions and a performance regression, each routed once through the same route. Our own read of each issue sits next to the judge's pick; the judge never sees it.
Per-attempt cost prices a stated agent budget (150,000 tokens read, 15,000written) at each model's list rate; the judge's own cost is estimated from the request size. Recorded 2026-10-14 through the chalk/demo-issue-triage route, judged by gpt-4.1-mini. Borderline issues can land on either side of a boundary from one run to the next.
What the gateway handles
Routing is a policy on the gateway, not code in every agent.
Pay frontier prices for frontier work
On the recorded issues, routing came to about half the cost of sending everything to gpt-5.5, and the judge adds well under a tenth of a cent per decision.
Criteria you can read
Each level is a paragraph of plain-language criteria stored on the route. Change them, and the next request is judged by the new text.
Shadow before you switch
Run the judge in shadow mode first: the default model keeps serving while every decision the judge would have made is traced, so you can check it against your own labels.
Keys scoped to a route
This demo's key can call only its route, at 30 requests a minute with a daily token budget. Callers never hold provider credentials.
Every decision traced
Judge label, backend, latency, cache hit and the policy revision land on the request's trace, next to its tokens and cost.
Any provider behind the levels
Levels point at concrete connections and models: OpenAI, Anthropic, Gemini or a model you serve on Chalk. Callers keep asking for the same route.
Add a judge to a route
Create a route with a default target and a judge whose levels each name a model, then point your agent at the route. Use a small non-reasoning model as the judge: it only has to return a label.
Start the judge in shadow mode to compare its decisions with your own before it serves anything, then switch it to live.
Model Gateway docsimport httpx
# One route, three tiers. The judge returns a level; each level names a concrete model.
LEVELS = {
"0": ("gpt-5-nano", "Mechanical change with an obvious fix: typos, docs wording, renames..."),
"1": ("gpt-5.4-mini", "Localized bug or small feature confined to one function or module..."),
"2": ("gpt-5.5", "Cross-cutting refactors, public API deprecations, performance work, design questions..."),
}
admin = httpx.Client(
base_url="https://api.chalk.ai",
headers={"Authorization": f"Bearer {CHALK_TOKEN}", "X-Chalk-Env-Id": ENV_ID},
)
admin.post("/chalk.router.v1.ModelRouteService/CreateModelRoute", json={
"publicModelId": "chalk/issue-triage",
"connectionId": OPENAI_CONNECTION_ID,
"upstreamModelId": "gpt-5.4",
"published": True,
"routing": {
# Served when the judge is skipped or answers with an unknown level.
"defaultTarget": {"connectionId": OPENAI_CONNECTION_ID, "upstreamModelId": "gpt-5.4"},
"judge": {
"mode": "JUDGE_MODE_LIVE",
"classification": "CLASSIFICATION_DIFFICULTY",
# A small, non-reasoning model: it reads the request and returns one label.
"backend": {"completion": {"target": {
"connectionId": OPENAI_CONNECTION_ID, "upstreamModelId": "gpt-4.1-mini",
}, "timeoutMs": 8000}},
"outcomes": {
level: {"model": f"{OPENAI_CONNECTION_ID}/{model}", "description": criteria}
for level, (model, criteria) in LEVELS.items()
},
},
},
}).raise_for_status()Stop paying frontier prices for typo fixes
Routing, keys, budgets and traces for every model your agents call. Talk to an engineer about putting a judge in front of yours.