What Is a Context Engine?

An agent that can't see current state doesn't fail loudly. It fails confidently and plausibly. Three examples that might sound familiar:

• A support agent processes a refund for a customer with three opened disputes. It should have blocked it and triggered de-escalation.
• A financial advisory agent quotes a portfolio balance that was correct an hour ago. Retrieval worked as designed. The index was just old.
• A retention agent offers a win-back discount to a customer the risk system just froze. The agent scored on lifetime spend, while risk saw net chargebacks.

Each of these agents was asked to make a call about a specific customer at a specific moment, and each was handed a pile of documents or a table. In every case the business already had what it needed to get the answer right. The building blocks of the right decision existed somewhere, but didn’t get to the model in time.

All three are missing the same thing: a system that computes values and data relevant to customer state at the instant the request arrives, and assembles it in the prompt at inference time. That's a context engine.

Great context makes prompts smaller and AI less wrong. Here’s how:

Everything the agent knows about you is in the prompt

Agents are powered by large language models. Ask any LLM about current events, and its frozen worldview quickly shows. Real-time accurate answers require a streaming view of the world, meaning something external must provide that information. An agent's current view of the world is strictly limited to its prompt at call time. Everything specific to this business, this account, this moment either arrives in the prompt or doesn't arrive at all. All action the agent takes is inference over a snapshot, the context in the prompt.

This means most agent failures are assembly failures. Model quality is one variable. The other is how you build the world the agent sees on every reasoning loop: how close it is to what's true right now, and whether you can replay what the agent saw.

Most of what gets loaded into context windows today are documents, because documents are what the current tooling makes easy to fetch. The values that describe what's happening right now are the harder half, and in durable systems, these decide the agent’s path forward.

Why agents miss

Three pathways put data in front of most models powering an agent. A vector index over documents. A memory store over history. And tool calls, which work differently from the first two: the agent reads its prompt, decides it needs something, calls an API, and the result gets appended to the context for the next turn. Tool calls are how an agent fetches context it wasn't given, one round trip at a time. None of the three paths were built for live data.

Most retrieval infrastructure was designed to make written knowledge findable and it does that extremely well. It was never designed to compute a value from real-time data at the instant of a decision, so it doesn't. That gap shows up in three ways for fast moving companies:

Business and world state. The agent sees what documents, databases, and semantic layers say, which is not the same as what's true at the moment it runs. A nightly load can't answer "what is this account's status this second." It also can’t know "is the payment corridor mentioned in this doc down right now" before it fails.

Situational judgment. The agent gets raw records and is asked to infer. It never receives the derived values the business already uses to judge this kind of case, so it re-derives them badly or skips the step. Your risk team spent two years on a chargeback model. The agent is doing arithmetic on a transaction list.

Workflow. Nothing tells the model which internal path to take. Escalate or resolve. Refund or withhold. Disclose or don't. With no deterministic signal, it improvises, and there's no record of how it decided.

Dumping in more context is worse

The instinct when an agent gets something wrong is to give it everything. Longer context windows make this feel free. It isn't, from both a cost and wasted outcomes lens:

• Chroma tested 18 production models including GPT-4.1, Claude 4, Gemini 2.5, and Qwen3, and found accuracy falls as input grows. On LongMemEval every model did better on a focused ~300-token history than on the full ~113k-token log, and a single distractor was enough to pull it down. (source)
• "Lost in the Middle" (Liu et al) found the shape of it. Models use what's at the start and end of a long context far better than what's buried in the middle, including models built for long context. (source)
• Building legal agents, Harvey AI found that capping how many tokens a tool could return produced large gains across open and closed models, before any training at all. Their base model, with no way to know which documents mattered, bulk-read everything: 104 tool calls, 461k tokens, a 0.061 rubric score. Trained to be selective, 42 calls, 250k tokens, 0.803. (source)

The reasoning models of agents aren’t short on context. They’re short on a few real-time values that would have told them what mattered most.

Teams mostly know this. What blocks them is plumbing. An automation company described their invoice extraction to us as a set of silos: a vision model pulls line items with no access to that customer's purchase history, and a separate statistical layer catches implausible results afterward. When something looks wrong the invoice routes to a human, and the customer ends up touching the same document three times. What they wanted was to feed history into the extraction step so the model corrects itself. The real-time value was calculable in their systems. The model was never seeing that context.

What engineered context looks like

Take the support agent from the top. Before the prompt is assembled, the system resolves a dozen values for this customer: tenure, plan tier, open disputes in 30 days, chargeback risk, whether the jurisdiction is regulated, and failed payments in the last hour.

That last one is specifically interesting, because it's the kind of value nothing else in the stack can produce. It isn't stored anywhere. It's an aggregation computed at inference time over a stream of payment events, some of which landed forty seconds ago. No document contains it. No nightly table has it. The warehouse won't know about it until tonight. The agent needs it now, because a customer whose card failed three times in the last hour is in a completely different situation than one whose card failed three times last spring.

Those values don't inform the agent. They direct it.

• Chargeback risk above threshold with open disputes: the refund tool is withheld and the case escalates.
• Failed payments in the last hour with low tenure: pull the last three transaction attempts.
• Regulated jurisdiction: suppress two tools and inject the disclosure.

What reaches the model is a dozen values, several of them computed on the spot, plus two targeted retrievals: transcripts of how this exact situation resolved well before. The full account record and six months of tickets stay where they are.

The principle underneath: the feature value is the control plane for context assembly. It's how the agent learns what kind of situation it's in before it starts reasoning about it.

Where this sits next to your stack

A fair question at this point is whether you already have this. A few things are adjacent.

• Vector databases do semantic recall over documents. No computation over live systems.
• Memory layers hold what was said. Conversation history goes stale the moment things change.
• Knowledge graphs hold structure and relationships, as current as their last load.
• Storage-first feature stores serve consistent precomputed values, with freshness set by whenever the pipeline last ran.
• Semantic layers govern what a metric means. They map warehouse tables to business concepts so that revenue means one thing everywhere, and resolve queries against that definition.

Vendors often considered for the context layer do a few different jobs. Glean runs enterprise search across workplace systems. LangChain gives you orchestration to wire retrieval and tools together. Neo4j holds the relationship graph. Mem0 and Zep hold conversation history. All of it is real, and most teams need some of it.

What none of them do is compute a value from a live operational system at the moment of the decision. They can tell an agent everything that has been written about a customer and nothing about what that customer did four minutes ago. Retrieval still finds the policy document. The context engine decides whether this particular customer should be shown it.

Semantic layers are a newer layer worth being specific about. When an analytics team works with AtScale, Dremio, or Cube: they’re building a governed definition layer over the warehouse, built to make reporting consistent. That answers "what does this term mean." A context engine answers "what is true and relevant right now." Different jobs for different challenges.

Wire the two together and the semantic layer gets better. A semantic layer holds the definition of churn risk and the right way to reason about it, and today that logic runs against warehouse tables loaded last night. With a context engine, it reasons correctly and resolves on demand against live data instead. The governance doesn't change. The definition doesn't change. The number stops being twelve hours old.

The primitives haven't changed

None of the components for a context engine are new. It's the feature engine and model serving architecture that ML teams have run in production for years, pointed at a different consumer.

• Agent and context are defined together in Python, so your AI and your models don't disagree about what a word means.
• On-demand resolution from live sources. A value is current because it was computed when you asked.
• Windowed aggregations, because "failed payments in the last hour" decides agent behavior and no document contains it.
• Point-in-time correctness, which made training data honest and now makes evals replayable against the context production was served.
• Millisecond latency, so the reasoning loop doesn't crawl.
• Model serving and data lineage, so there's a record of what the agent saw and did.
• Deployment inside your VPC, so sensitive data never crosses the open internet.

Which values to build for agents

Having the primitives doesn't tell you what to compute. Your user base has thousands of possible attributes. Which do you build out first? A useful filter: for each candidate value, name what agent path it decides. If a value doesn't change what the agent does, retrieves, or refuses, it's just decoration.

Four classes are worth building, roughly in this order. Two are lookups. Two are calculations, and the calculations are where the work is.

Entity state. The slow facts the agent keeps getting wrong. Tenure, plan tier, jurisdiction, account status. Cheap to define, and they resolve against the live source.

@features
class User:
    id: str
    tenure_days: int
    plan_tier: str

Windowed counts. A windowed aggregation runs over a stream of events at request time, bounded by a window that moves with the clock. A one-hour window at 3:04pm and the one-hour window at 3:05pm are different computations over different sets of events. Nothing stores the answer, which is why nothing can look it up.

@features
class User:
    failed_payments: Windowed[int] = windowed(
        "1h", "24h", "30d",
        expression=_.payments[_.status == "failed"].count(),
        materialization={"bucket_duration": "10m"},
    )

Derived judgments. The scores and tiers your business already computes, usually inside a model or a SQL view no agent can reach. Define them once so the agent and the risk system can't disagree.

@online
def get_chargeback_risk(
    failed_1h: User.failed_payments["1h"],
    disputes_30d: User.open_disputes["30d"],
    tenure: User.tenure_days,
) -> User.chargeback_risk:
    return risk_model.predict(failed_1h, disputes_30d, tenure)

Routing flags. The booleans that gate tools. This is where the control plane stops being a metaphor.

@online
def get_refund_eligible(
    risk: User.chargeback_risk,
    disputes: User.open_disputes_30d,
) -> User.refund_eligible:
    return risk < 0.4 and disputes == 0

The agent doesn't need to reason about whether to offer a refund here. It reads a boolean computed the same way the refund system computes it.

These last two classes also decide what to retrieve

Gating tools is the obvious use. The less obvious one is that the same values should be choosing which documents the agent sees.

Say refund_eligible resolves false and the case is heading toward a denial. What helps is the ten past denials in this product line that customers accepted without escalating, plus the two policy sections that apply to this jurisdiction and plan tier. The full policy library does nothing.

Most stacks retrieve first and reason later, which means the model gets a semantically similar pile and has to work out which half applies. Resolve the values first and retrieval runs over a shelf instead of a warehouse. Same index, far better inputs, and the agent stops spending its reasoning budget figuring out which situation it's in.

Tying it all together

Let's revisit the three agents from the beginning.

The support agent never saw the fraud model. It didn't need to. The fraud model's output is a derived judgment, chargeback_risk, and the agent reads it through a routing flag: refund_eligible resolves false, the refund tool never enters the tool list, and the case escalates. Two of the four classes, chained. The agent didn't get better at judgment. It stopped being asked to have any.

The advisor quoted an hour-old balance because the balance was entity state served from an index. Resolve that same entity state against the system that owns it and it gets computed when the question is asked. There's no window during which the answer is quietly wrong.

The agent that called a blocked customer high value needed one definition instead of two. value_tier is a derived judgment computed once and read by both the agent and the risk system, so the two can't contradict each other in front of the customer.

None of these are model problems. Swap in a better model and it gets all three wrong the same way, because none of them were errors of reasoning. The information needed never made it into the prompt.

This is already running

These systems aren't theoretical. Mercury, Whatnot, Socure and more run risk decisions, recommendations, embeddings, and agents in sandboxes on context engines today. Typical time to compute a value is 5 milliseconds. The question isn’t if your AI stack has the time, it’s whether you can build reliably without it.

So, do you need a Context Engine?

Not always. If your agent works over static documents and a nightly refresh is just as accurate, you don't need this. Check below. You know you need a context engine if any of these describe your agent:

• It acts rather than answers. Issuing a refund, approving a limit, releasing a shipment, routing a case. An action taken on stale information is a decision the business has to unwind.
• The answer changes within a session. A balance, an inventory count, a fraud score, a rate. If a value can move between the user's first message and their third, an index can't carry it.
• Recency changes the meaning. Three failed payments this hour and three failed payments last spring are the same number describing two different customers.
• Another system already has an opinion. A risk model, an eligibility service, a pricing engine. If the agent computes its own version, the two will contradict each other in front of a customer eventually.
• Someone will ask what it saw. Regulated decisions, disputes, audits. Reconstructing why an agent did something requires knowing what was in the prompt at the time.

None of these apply to an agent that summarizes analysis. Most of them apply to anything touching a customer account.

AI is in a paradigm shift. We're handing agents real decisions while keeping them ignorant of the situation they're deciding about. The systems that fix this already exist. They were built for fraud and risk teams who couldn't afford a stale answer, and they deliver context consistently and point-in-time correct under load. Pointing them at agents is the shortest path to an agent that knows what's going on.