What is LLM inference?

TL;DR

LLM inference is the process of running a trained language model against new input to produce output, without changing the model's weights. Training sets the parameters. Inference applies those frozen parameters to whatever arrives at the endpoint, one request at a time, under a latency budget. Most of that budget is spent before the model generates a single token, assembling the context the prompt is made of.

Teams tuning inference reach for tokens per second first, and the reason is that the number exists. Serving engines publish it, benchmarks compare it, and a dashboard will show it to you.

The time spent assembling the prompt has no leaderboard. It is spread across a retrieval service, a feature lookup and two API calls, each owned by a different team, and none of them appear on the inference dashboard. A response that generates in 200ms behind a 900ms context fetch is a 1.1 second response. Quantizing the model attacks the smaller number.

That arithmetic is illustrative, not measured, and the ratio varies by workload. The asymmetry it points at does not: the half of the request that gets optimized is the half that gets measured.

LLM inference is a specific case of model inference, the general discipline of a trained model answering a live request, whether that model is an XGBoost risk score or a language model. The language model case inherits all of it and adds a much larger memory footprint, a cache to manage, and output produced one token at a time instead of returned in a single pass. It is also distinct from LLM serving, which is the infrastructure around the computation: the endpoint, batching, autoscaling, failover and access control.

This guide covers what happens inside a request, why prompt length lands on your GPU bill, and where the data layer decides whether the output is correct at all.

How does LLM inference work?

An inference request runs through tokenization, then prefill, then decode, with a KV cache that keeps decode affordable.

Tokenization converts the input text into the integer token IDs the model operates on. Token count, not word count, drives cost and context limits. What inflates that count in production is rarely the user's question. It is everything retrieval decided to put in front of it.

Prefill processes the entire prompt at once. Every input token is known up front, so the model computes attention across all of them in parallel. The stage is compute-bound, and it produces your time-to-first-token number.

Decode generates the response one token at a time. Each new token depends on the one before it, so the work cannot be parallelized the way prefill can. This stage is bound by memory bandwidth rather than compute, and it sets tokens per second.

The KV cache keeps decode from becoming quadratic. It stores the key and value tensors already computed for previous tokens, so generating token 500 does not require recomputing attention over the preceding 499. It is also one of the largest consumers of GPU memory during serving, and at long context lengths it becomes the dominant one.

Why does prompt length show up on your GPU bill?

Prompt length gets charged twice.

Prefill does the obvious version of this. Twice the prompt is roughly twice the attention computation before the first token comes back.

The KV cache does the less obvious version. Its size grows with sequence length, and it competes for the same GPU memory as the model weights. Longer prompts mean fewer sequences resident at once, which means fewer concurrent requests per card, which means more cards.

So prompt length sets time to first token and serving density at the same time. And prompt length is decided by retrieval precision.

A team that can compute the three aggregates a decision needs, say transactions in the last 24 hours, declines in the last 7 days, and average ticket size over 90 days, sends three numbers. A team that cannot compute them retrieves the transaction history and asks the model to find them. The second approach ships faster and reads the same in application code. It arrives here, as prefill latency and cache pressure.

What makes this hard to catch is where the decision lives. It is made in a retrieval service by someone solving a correctness problem, and it is paid for on a GPU utilization graph that a different team is watching. This is an argument about the architecture, not a measured benchmark, and it is the reason so much inference optimization work turns out to be memory management work.

Why do stale inputs break a model that tests fine?

Everything above is still only cost. The inputs can also be wrong.

Most writing about LLM inference assumes the prompt is sitting there ready when the request lands. In production it rarely is. Something has to fetch the account balance, the last five transactions, the open support tickets and the current price, then assemble them into context before the model sees anything.

If those values come from a batch job that runs hourly or nightly, the model is reasoning about a version of the world that has already moved. That ceiling is a property of the architecture instead of a tuning problem, which is the argument in why feature stores have a freshness ceiling. A support agent tells a customer their refund has not been processed, because the refund landed forty minutes ago and the feature table refreshes at midnight. A fraud model clears a transaction on a session state that stopped being true before the request arrived. Both models performed exactly as trained.

The property that closes this is point-in-time correctness: the values fed to the model reflect the state of the world at the moment the request arrived, and when you examine that decision later, you can reconstruct the exact inputs it saw.

The common workaround is to compute the inputs inline, in the application code handling the request. It works for the first model. It stops working predictably once a prediction waits on several database reads, a third-party API and an embedding call, because each one adds variance to a budget you have already spent most of on generation. It also puts the feature logic in application code owned by a backend team while the training pipeline keeps its own copy, and the two drift. That is training-serving skew arriving through the side door.

Context, in other words, has a timestamp. Inference infrastructure that ignores it produces fast wrong answers.

How does Chalk approach LLM inference?

Chalk's position is that the data path and the model path spend the same latency budget, so they belong in one system instead of two products someone has to integrate.

Real-time serving computes the inputs at request time, directly from the source systems, using resolvers written as ordinary Python or SQL. One definition serves the training job and the production request, which removes the skew rather than monitoring for it. Cached and materialized values return in under 5ms. The resolver overview is the reference, and the feature graph demo runs a loan decision assembled from BigQuery SQL, Python and a registered scikit-learn model as a single planned query.

Model Serving runs the model on autoscaling GPUs inside your own cloud account, next to the data it needs. You hand a trained model object to register_model_version and write a class with a predict method, and Chalk builds the serving image. There is no Dockerfile, no separate model registry and no deployment YAML, and for scikit-learn, PyTorch, XGBoost and CatBoost models there is no schema to write either. Weights mount from a versioned volume, so retraining a model does not rebuild a container. The same path handles classical model inference, so an XGBoost risk score and a fine-tuned language model are served by one system. The same deployment answers a live request and scores a backfill, and both weights and data stay in your account. The range runs further than tabular models: streaming speech-to-text and text to video both run as live demos in the browser, and model deployments documents the calling path. Open-weight models follow a second path, deployed as a scaling group running vLLM's OpenAI-compatible server, which serving open and custom models covers and this guide walks through end to end.

That co-location is the argument. When the serving layer lives at another vendor, every feature-dependent prediction pays for a round trip out and back, and that round trip comes out of the same budget as generation.

For calls to frontier models, Chalk Model Gateway gives you one OpenAI-compatible endpoint in your cloud. It holds the provider credentials, enforces a model allow-list and a token budget per key, fails over automatically when a provider is unavailable, and tracks token usage and upstream model cost against the key that spent it. There is a model routing demo for the routing behavior and the gateway docs for the reference. The alternative most teams start with is every service calling providers directly with its own key, which ships the first feature quickly and then turns key rotation, spend attribution and outage handling into a separate job inside every codebase.

What are the main LLM inference optimization techniques?

Once the data path is sound, the remaining work is hardware-level, and most of the available gains come from how the serving engine manages GPU memory.

Quantization reduces the numerical precision of model weights, commonly from FP16 to INT8 or INT4. Lower precision means a smaller memory footprint, so more of the model and more of the KV cache fit in GPU memory. The tradeoff is accuracy, and how much you lose depends heavily on the model and the method.

Continuous batching groups incoming requests dynamically instead of waiting for a fixed batch to fill. Because decode is bound by memory bandwidth, a GPU serving one request at a time is mostly idle. Batching raises utilization and throughput, at some cost to the latency of any individual request.

KV cache management addresses the memory fragmentation that naive allocation creates. Techniques like PagedAttention, which borrows the idea of virtual memory paging, let the serving engine pack more concurrent sequences onto the same GPU.

None of it is required at small scale. Heavily quantized models run acceptably on CPU for low-volume or latency-tolerant work, and a GPU becomes necessary once you need real concurrency or a low time to first token. These techniques belong to your serving engine, not your application, and full implementations sit outside the scope of this guide. NVIDIA's guide to inference optimization is the most thorough neutral treatment of the systems work involved.

When do you not need inference infrastructure?

If your application calls a single provider, tolerates a provider outage without anyone filing a ticket, and builds its prompts from values that arrive in the request itself, direct API calls are the correct answer and a platform would be overhead.

The same applies to the data layer. A model whose inputs are genuinely static, or that runs as a nightly batch where an hour of staleness costs nothing, does not need point-in-time correctness. Plenty of useful models are in exactly that position.

The picture changes when predictions become customer-facing, when several teams start shipping against the same providers, or when someone in a review asks which agent is allowed to call which model and nobody can answer. Agents raise the stakes again, because each step shapes what the next one retrieves. Testing an agent against production data is the step most teams skip, and an evaluation that scored the agent's behavior on data production would never have served will pass regardless.

Conclusion

LLM inference performance is the whole path, from the moment the request lands through context assembly to the last generated token. The slowest part of that path is usually the part nobody profiled, and the most expensive part is usually a retrieval decision made three services away from the GPU it bills.

The Chalk demo library has live examples of model serving, routing and federated queries, each with source code you can deploy in your own cloud. To run one against your own data, create a Chalk account.