How Chalk’s query engine runs AI inference from SQL

David Percy - SWE
by David Percy

From agent traces to evals in one SQL query: use Chalk SQL to analyze agent behavior, classify potential violations, and prioritize sessions for review.

November 19, 2026

Agents generate a lot of traces. Every session records what the user asked, what the agent did, and which tools it called along the way.

When you're running thousands of sessions a week, that's a lot of behavior to review. You can sample traces manually, but you'll inevitably miss things. You can write rules to catch obvious problems, but rules aren't great at understanding intent. An agent deleting a file might be exactly what the user asked it to do. It might also be a serious mistake.

I'd like to know when an agent does something the user didn't authorize. That could mean accessing data outside the task, making an unexpected change, or calling a tool that has nothing to do with the request.

The traces already contain the information I need. The problem is getting through them.

One approach is to build an application around model calls. Query the database, send each record to a model, parse the response, write the results back, and query them again. It works, but there's a surprising amount of plumbing around what is essentially a few questions about your data.

With Chalk SQL, I can do the whole thing in one query. AI functions like ai_filter, ai_classify, and ai_score let me ask questions about rows, get structured answers, and use those answers in the rest of a SQL query. Here's what that looks like.

Reviewing agent traces

I'll start with a table called agent_sessions. There's one row per session, with the user's request, the agent's transcript, and a few columns describing when the session ran and how many tools it called.

I want to look at the last seven days of completed sessions, find potential authorization violations, classify what happened, and rank the results by severity. Here's the query:

The full query, run on synthetic agent sessions.

There are a few things happening here. I'm filtering sessions before asking a model to review them, using one model call to identify potential violations, another to classify what happened, and a third to score severity. The result is a ranked list of 20 sessions worth reviewing. Let's walk through it.

Give the model the right context

The first thing I need is a useful representation of each session.

context:
  'User authorization: ' || user_request || chr(10) ||
  'Recorded activity: ' || transcript

I'm combining two fields into one string: what the user asked for and what the agent actually did.

The labels are intentional. I want the model to understand that these are two different things and compare the agent's behavior against the user's instructions.

This is just a SQL expression. I can concatenate fields, add labels, insert newlines, or use other SQL functions to shape the input. That also means I don't need to build a separate Python function to construct prompts for every record. The model's input is defined alongside the rest of the query.

Filter before calling the model

Before reviewing anything, I can eliminate sessions that aren't relevant.

WHERE status = 'completed'
  AND completed_at >= current_timestamp()::TIMESTAMP - INTERVAL '7 days'
  AND tool_call_count > 0

I'm only interested in completed sessions from the last week that involved at least one tool call. The last condition is particularly useful for this example. If an agent never called a tool, there's no tool action to review for unauthorized activity.

These are inexpensive SQL predicates. There's no reason to send every session to a model when I can narrow the dataset first.

Ask a yes-or-no question with ai_filter

Now I can ask whether the agent did anything outside the user's authorization.

AND ai_filter(
  context,
  'Did any action exceed the user''s authorization?',
  model => 'gpt-4.1-mini'
)

ai_filter takes a text input and a question, then returns a boolean. It works like any other predicate in a WHERE clause. If the model returns true, the row moves forward. If it returns false, the row is filtered out.

I don't need to write application code to loop through sessions, make individual API calls, or interpret different ways a model might express yes or no. I can ask the question directly in the query. This is also a useful way to narrow the work for subsequent model calls. Instead of classifying every session, I only need to classify the ones that might contain a violation.

Classify what happened with ai_classify

Finding a potential violation is a start, but I also want to know what kind of behavior I'm looking at.

ai_classify(
  context,
  'What unauthorized activity occurred?',
  ['data_exfiltration', 'unauthorized_change',
   'unrelated_tool_use', 'consistent', 'unclear'],
  model => 'posttrain/Qwen/Qwen3-4B-Instruct-2507'
)

ai_classify takes the input text, a question, and a list of allowed labels. It returns one label for each row.

I've defined five categories:

  • data_exfiltration: the agent accessed or exposed data outside its authorization.
  • unauthorized_change: the agent modified something the user didn't authorize.
  • unrelated_tool_use: the agent called a tool unrelated to the task.
  • consistent: the behavior was consistent with the user's request.
  • unclear: there isn't enough information to make a determination.

The useful part is that the output is structured. I get one of the labels I specified, rather than free-text responses that I need to normalize and parse.

I can use finding like any other column. For example, I could group by it to count how many sessions fell into each category over the last week.

Rank the results with ai_score

Once I've identified and classified potential violations, I still need to decide which ones deserve attention first.

ai_score(
  context,
  'Is the violation severe?',
  model => 'gpt-4.1-mini'
)

ai_score returns a score, based on the model's probabilities that the answer to the given question would be yes. That gives me a value I can use to rank results.

I can then sort the sessions. Rather than handing someone thousands of traces, I can give them a smaller review queue. The score is a useful ranking signal, not a guarantee that the model has correctly assessed the severity of every incident. I'd still want a human to review the results before treating them as confirmed violations.

Use different models for different jobs

You might have noticed that I'm using two different models in the query. gpt-4.1-mini handles filtering and scoring. A smaller, post-trained Qwen3-4B model handles classification.

There's no reason every operation needs the same model. A straightforward yes-or-no question may not need the same capabilities as a more nuanced classification task. I can choose a model for each function without changing the structure of the query.

Chalk's Model Gateway gives me a single endpoint to route requests to different models and track usage across them. That means I can experiment with different models for different operations without building a separate integration for each provider.

What happens when the query runs?

One thing worth understanding is that the user sends Chalk a whole SQL query at once. So Chalk, instead of the user, manages how the API requests are sent to the AI provider.

In a traditional application, I'd have to orchestrate the entire sequence myself. I'd query the database, decide which records to send to a model, make the calls, process the responses, and then join the results back to the original data.

With Chalk SQL, the query planner sees the whole operation. It can reason about the SQL filters and AI functions together, and make holistic decisions about how to order the work efficiently. For example, it can push supported filters down to the data source so that Chalk only requests the relevant subset of data.

The execution looks roughly like this:

  1. Chalk queries the data source using filters that can be pushed down.
  2. The data source returns the requested results.
  3. Chalk applies any remaining SQL filters.
  4. AI functions run against the rows that survive those filters.
  5. The engine processes the results through joins, aggregations, sorting, or limits.

Filter pushdown is a planning decision that happens before execution. When the query runs, Chalk requests the relevant subset of data from the source rather than retrieving everything and filtering it afterward.

The planner can also coordinate model calls with the other operations in the query. This matters because model calls are expensive compared with ordinary SQL predicates. If a session can be eliminated by a cheap filter, there's no reason to spend money evaluating it with a model.

Chalk can batch the API requests it makes to AI providers, and keep multiple API requests in flight at once. The exact execution plan depends on the query and the backend, but the important point is that Chalk can reason about the whole workflow.

Keep the cost of model calls under control

When you're applying AI functions across thousands or millions of records, the cost of each operation adds up. There are a few things I can do to keep that under control.

Filter the rows first

This is the most obvious one. In the example above, I remove incomplete sessions, old sessions, and sessions without tool calls before asking the model to review anything.

Keep the context small

The model doesn't necessarily need the entire transcript. If I'm looking for unauthorized tool use, I might only need the user request and the recorded tool calls. I can change the context expression to include just those fields, or truncate long transcripts. Because the context is a SQL expression, I can make those changes directly in the query.

Use smaller models where they work

A classification task with a fixed set of labels may work well with a smaller model. A more complicated evaluation might need something larger. I can choose a different model for each AI function and compare the tradeoffs.

Think about filter ordering

If I'm applying multiple AI filters, their selectivity matters. For example, say I have 10,000 sessions. Filter A keeps 5% of rows, while filter B keeps 50%.

If A runs first, B only needs to evaluate the 500 rows that survive. If B runs first, A needs to evaluate 5,000 rows. That's 10,500 versus 15,000 model evaluations.

The example is illustrative, but it shows why the order of model calls can have a meaningful effect on cost. The exact ordering behavior depends on the query planner, so it's worth checking the execution plan for your workload.

The same pattern works beyond agent traces

Agent audits are one example. The same functions are useful anywhere I want to ask questions about a dataset and do something with the answers.

  • For customer support, I could find tickets where someone wants to cancel, classify the reason, and rank them by urgency.
  • For evaluations, I could score model outputs against a rubric across an entire evaluation dataset.
  • For trust and safety, I could classify flagged content and prioritize the highest-risk cases for human review.
  • For sales, I could identify buying signals in call transcripts and rank conversations by purchase intent.

In each case, the workflow is similar: filter the data, ask a model a question, get a structured result, and use that result in the rest of the query.

From traces to a review queue

The interesting thing about this example isn't that I can call a model from SQL. It's that I can combine model calls with the rest of my data workflow without building a separate application to manage them.

I can define what the model sees, filter the rows it evaluates, choose different models for different tasks, and use the results in subsequent SQL operations. For agent traces, that means turning a week of sessions into a prioritized list of things worth investigating.

Try it yourself: run the interactive demo of ai_filter, ai_classify, and ai_score, see ai_filter search agent trajectories, or read the Chalk SQL documentation.

Want to stay up-to-date with Chalk?

Subscribe for updates on what we’re building (and shipping!) at Chalk