Find and filter data by calling AI models.
In one SQL query.
This demo shows how ai_filter, a ChalkSQL function, puts a language model inside a WHERE clause. Point it at a table of agent trajectories, ask in plain English, which turns show an angry customer. Chalk plans the model calls like any other operator, so each question costs as few calls as the query allows.
Filter 400 conversations
This is a recorded search over 400 synthetic support conversations, scored against their planted labels.
Try this experiment · expected results
1Read the highlighted turns
Follow the agent’s failed lookup and the customer’s response in the sample transcript.
Expected: The confusion and anger labels point to specific turns, rather than a verdict about the whole chat.
2Compare search quality and work
Read precision and recall for each question, then compare prompts with and without the stop key.
Expected: Fewer model calls are a work reduction; precision and recall separately show which labeled episodes were found.
The agent transposes the order number, then the customer gets frustrated. Chalk’s queries find these turns by meaning and sequence; the highlighted messages are model predictions.
- 00CustomerHello, my bank shows two payments of $218.95 for order FH-86795. Can you look into it?
- 01AgentHi there, happy to help. One moment while I look that up.
- 02Agent callslookup_order({"order_id": "FH-68795"})
- 03Tool returned{"error": "order_not_found", "order_id": "FH-68795"}
- 04Agent● ConfusedI couldn't find an order with the number FH-68795. Could you double-check it for me?
- 05Customer● AngryI already gave you the order number. This is getting frustrating.
- 06Agent callslookup_order({"order_id": "FH-86795"})
Precision measures how often a flagged episode matches its label. Recall measures how many labeled episodes the query finds.
- Rows evaluated
- 1,308
- Model calls
- 1,308
- Seconds
- 2.15
- Rows evaluated
- 2,725
- Model calls
- 2,725
- Seconds
- 4.61
- Rows evaluated
- 1,308
- Model calls
- 1,495
- Seconds
- 2.72
Inspect how stopping early reduces model work
The confused-agent query only needs one TRUE per trajectory. With a stop key, Chalk's in-process LLM engine asks each trajectory's turns in order and stops at the first TRUE. Same model and prompts, one H100: it found 125 confused trajectories, against 126 without the key. An HTTP endpoint such as vLLM is still asked about every turn.
Built on Chalk
You write the question. Chalk decides which rows it reaches, in what order, and how the calls get to the model.
Predicate ordering
The planner orders LLM predicates by expected prefill tokens, from selectivity and text-length hints. On this data that is 1,495 calls instead of 1,923, for the same answer.
Stop keys
stop_key => trajectory_id pushes a DISTINCT into the filter: once any turn of a trajectory answers TRUE, the rest of it can go unasked. Chalk's in-process LLM engine skipped 25% of the prompts here.
Prefix-cache locality
Prompts put the document first, so several questions about the same text share its cached KV. cache_tokens sizes each batch to stay inside the server's prefix cache.
Throughput controls
max_concurrency bounds requests in flight per endpoint, and prompts_per_request packs several rows into one HTTP request.
The SQL is the whole example
The repository generates the conversations, serves the model with vLLM, and runs four queries: angry turns, confused trajectories, where each agent first got confused, and customers who got mad right after the agent did.
Swap in your own trajectories table and your own questions. Nothing else changes: the verifier is a url and a model name.
git clone https://github.com/chalk-ai/agent-trajectory-search
cd agent-trajectory-search
# 400 synthetic support chats, with planted episodes and labels.
./generate.py
# Any OpenAI-compatible endpoint works; here, vLLM on one GPU.
./serve.sh Qwen/Qwen3-4B-FP8
# Runs the four queries in sql/ and scores them.
python run.py --url http://localhost:8000/v1Implementation details
How the example works
Use AI model calls in SQL to filter and understand your data, finding matches that keyword search misses. SQL determines which rows the model reads.
A table of turns
Each trajectory is rows: one per customer message, agent reply, tool call and tool result. Window functions and a self-join turn those rows into the context a question needs, from one message to the whole transcript so far.
A question per row
ai_filter(text, 'question') sends each row's text to any OpenAI-compatible model, here Qwen3-4B on vLLM, and reads back a single forced TRUE or FALSE token. The predicate then filters like any other.
A plan that saves calls
Cheap SQL conjuncts run before the model. Several ai_filter conjuncts run as a chain, cheapest expected prefill first, each asked only about rows the last one kept. A stop key stops asking about a trajectory once one turn has answered.
More capabilities
Any OpenAI-compatible model
vLLM on your own GPUs, a gateway, or a hosted API. extra_body forces a one-token TRUE/FALSE answer, so a verdict costs one decode step.
Just SQL around it
Joins, window functions, GROUP BY and DISTINCT compose with ai_filter. The question is a string literal; everything around it is the query you would write anyway.
Ask your own trajectories
Agent traces, support transcripts, or any text you already keep in tables. Talk to an engineer about putting a model in your queries.