Find where your agent got confused.
In one SQL query.
ai_filter puts a language model in a ChalkSQL WHERE clause. Point it at a table of agent trajectories and ask, in plain English, which turns show a confused agent or an angry customer. Chalk plans the model calls like any other operator, so the question costs as few of them as the query allows.
400 conversations, one H100
The example generates 400 support conversations for an invented store, plants confused agents and angry customers in some of them, and keeps the labels. Each query below read every one and was scored against those labels. A recorded run, with Qwen3-4B on vLLM.
- 00CustomerHello, my bank shows two payments of $218.95 for order FH-86795. Can you look into it?
- 01AgentHi there, happy to help. One moment while I look that up.
- 02Agent callslookup_order({"order_id": "FH-68795"})
- 03Tool returned{"error": "order_not_found", "order_id": "FH-68795"}
- 04Agent● ConfusedI couldn't find an order with the number FH-68795. Could you double-check it for me?
- 05Customer● AngryI already gave you the order number. This is getting frustrating.
- 06Agent callslookup_order({"order_id": "FH-86795"})
- Rows
- 1,308
- Model calls
- 1,308
- Seconds
- 2.15
- Rows
- 2,725
- Model calls
- 2,725
- Seconds
- 4.61
- Rows
- 1,308
- Model calls
- 1,495
- Seconds
- 2.72
The confused-agent query only needs one TRUE per trajectory. With a stop key, Chalk's in-process LLM engine asks each trajectory's turns in order and stops at the first TRUE. Same model and prompts, one H100: it found 125 confused trajectories, against 126 without the key. An HTTP endpoint such as vLLM is still asked about every turn.
How it works
Keyword search finds the word "refund", not the agent that refunded an order nobody asked it to. A model can tell the difference, and SQL decides which rows it reads.
A table of turns
Each trajectory is rows: one per customer message, agent reply, tool call and tool result. Window functions and a self-join turn those rows into the context a question needs, from one message to the whole transcript so far.
A question per row
ai_filter(text, 'question') sends each row's text to any OpenAI-compatible model, here Qwen3-4B on vLLM, and reads back a single forced TRUE or FALSE token. The predicate then filters like any other.
A plan that saves calls
Cheap SQL conjuncts run before the model. Several ai_filter conjuncts run as a chain, cheapest expected prefill first, each asked only about rows the last one kept. A stop key stops asking about a trajectory once one turn has answered.
The SQL is the whole example
The repository generates the conversations, serves the model with vLLM, and runs four queries: angry turns, confused trajectories, where each agent first got confused, and customers who got mad right after the agent did.
Swap in your own trajectories table and your own questions. Nothing else changes: the verifier is a url and a model name.
chalk-ai/agent-trajectory-searchgit clone https://github.com/chalk-ai/agent-trajectory-search
cd agent-trajectory-search
# 400 synthetic support chats, with planted episodes and labels.
./generate.py
# Any OpenAI-compatible endpoint works; here, vLLM on one GPU.
./serve.sh Qwen/Qwen3-4B-FP8
# Runs the four queries in sql/ and scores them.
python run.py --url http://localhost:8000/v1What Chalk handles
You write the question. Chalk decides which rows it reaches, in what order, and how the calls get to the model.
Predicate ordering
The planner orders LLM predicates by expected prefill tokens, from selectivity and text-length hints. On this data that is 1,495 calls instead of 1,923, for the same answer.
Stop keys
stop_key => trajectory_id pushes a DISTINCT into the filter: once any turn of a trajectory answers TRUE, the rest of it can go unasked. Chalk's in-process LLM engine skipped 25% of the prompts here.
Prefix-cache locality
Prompts put the document first, so several questions about the same text share its cached KV. cache_tokens sizes each batch to stay inside the server's prefix cache.
Throughput controls
max_concurrency bounds requests in flight per endpoint, and prompts_per_request packs several rows into one HTTP request.
Any OpenAI-compatible model
vLLM on your own GPUs, a gateway, or a hosted API. extra_body forces a one-token TRUE/FALSE answer, so a verdict costs one decode step.
Just SQL around it
Joins, window functions, GROUP BY and DISTINCT compose with ai_filter. The question is a string literal; everything around it is the query you would write anyway.
Ask your own trajectories
Agent traces, support transcripts, or any text you already keep in tables. Talk to an engineer about putting a model in your queries.