Last time I wrote about building a working memory for every account we have ever touched. Defined, queryable fields a model can read instead of reconstructing thousands of histories. Some of those fields are arithmetic. Others are an LLM’s reading of a conversation or a company’s website.
This post is about the second kind, and the obvious problem with those fields: an LLM’s interpretation can be wrong. We use Chalk Agent Evals to test the features we save and reuse. From there, we apply the same discipline to the briefs and summaries LLMs produce for our team.
A wrong label doesn’t stay in one place
We do the extraction once and reuse the result, so a question can read a saved fit assessment instead of repeating the research. It also means the same assessment might inform the priority score, the account briefing and a leadership analysis at the same time.
A prospect who agrees to a meeting, a prospect who is curious, and a prospect who asks us to come back next quarter are in three different states. Collapse them into one “engaged” label and every workflow downstream treats them the same way.
So I treat the prompts that produce these fields the way I would treat code that several systems depend on.
An eval gives a prompt a set of examples and checks its outputs against expectations we wrote down in advance. Chalk Agent Evals keeps the inputs, task versions and scores together.
Turning business judgment into examples
I start with real conversations and write down the assessment I expect from each one. A request to reconnect after a hiring milestone should keep that condition. A polite reply should not become meeting intent. A prospect talking about agents may be expressing a different product focus, not a lack of fit.
This is also what makes the definitions from the last post tangible. A definition describes what engagement or timing or product focus means. An eval example shows how that definition lands on language a person actually used. When a case is ambiguous, we review it and write down what we expect, so a disputed label doesn’t stand as though it were settled.
How do you check whether an LLM output is any good?
Can the workflow use it? Structure checks confirm the required fields are there and the categories are valid. If a label is supposed to be one of a known set and the model returns a sentence, nothing downstream can filter on it.
Did it make the right call? Deterministic checks compare the output against examples a person already reviewed. Does it hold Timing - Not Now when the prospect is open but names a future condition? These make specific disagreements visible instead of averaging them into a score.
Does the assessment follow from the evidence? For this I give another model a grading rubric and let it judge whether the label and its reasoning are supported by what was actually said.The judge is evidence to weigh, not a source of truth.
Each check catches something the others miss. Valid JSON can still carry the wrong label, and a fluent explanation can invent buying intent that was never in the transcript.
If you want the product mechanics rather than the practice, Sai wrote up how prompt evaluation works in Chalk, and we covered evals in a recent session.
Comparing two versions of a prompt
When I change a prompt, I run the current and proposed versions against the same examples in Chalk and read the difference. What improved, what regressed, and where an explanation stopped matching the evidence. We used the same approach to compare three models across 208 sales conversations, checking which preserved the distinctions our workflows depend on.
I keep a set of examples I did not look at while tuning. Checking against those is how I tell whether a change generalized or just learned the cases in front of it.
AI Fit uses an LLM to research companies and score how well their agent use cases fit Chalk’s infrastructure. In testing, one version gave too much credit to products marketed as agents that only answered questions. We tightened the rubric to distinguish those from agents that take actions, such as processing a refund, then tested it again. We also had to preserve the other side of that distinction: a conversational interface can still be an agent if it actually completes the task. That gave us a clearer scoring rule for a feature used across our account workflows.
Evaluating what the team reads
The same discipline applies to what our team reads. For our “why now” briefs, we check whether the claims follow from the account evidence and whether they give the rep useful direction. We apply it to our Fathom call summaries too.
The out-of-the-box summary did not consistently surface what our team needs from a call: the use case, why Chalk might fit, the technical constraints, what could hold up a purchase, and who committed to the next step. So we give the model Chalk’s product context and a rubric for what matters in a sales conversation, and send the request through Model Gateway.
The eval asks two things. Did the summary capture those items when they were in the transcript, and did it avoid inventing ones that were not.
Here the output is a set of bullets, a read on the deal, and next steps with owners. One review caught a summary turning someone leaving the company in a month into someone being away on vacation for a month.
What evals don’t do
Evals do not make compression lossless, and they do not make every assessment correct. They tell you whether the parts you are about to reuse have earned it.
The cost of a wrong output scales with how many places read it. That is why we check, and why we keep the source. When a task needs the nuance a label cannot hold, the original conversation is still right there.
This is the third post in a series on what people at Chalk build on Chalk. The first was MealBot, and the second was a working memory for our GTM.
