Long-context LLM serving: the real tradeoffs in memory, latency, cost, and accuracy

Context is everything you send to a model in one request: the prompt, documents, code, conversation history. It’s measured in tokens, which are word pieces, roughly 750 words per 1,000 tokens. A 128K-token window (about a novel’s worth of text) is the common standard today. Llama 3.1 and GPT-OSS-120B both ship with 128K windows. Frontier closed models go further. Claude Sonnet 4.6, 4.5, and 4 accept up to 1M input tokens, about ten novels, per the model listings on DigitalOcean’s pricing page.

The number on the spec sheet tells you what the model can accept. It tells you nothing about what happens when many users actually send inputs that big to the same infrastructure at the same time. Support is a model capability. Performance is a serving question, and the two come apart in four specific places. This article walks through them, with the numbers.

You need two pieces of background first.

Prefill and decode. Inference has two phases. In prefill, the model reads your whole input in one parallel pass before it writes anything. Prefill is compute-bound, limited by how fast the GPU can do math. In decode, the model generates its answer one token at a time. Decode is memory-bound, limited by how fast data moves out of GPU memory, not by arithmetic. Long inputs mostly stress prefill. Long outputs mostly stress decode.

The KV cache. For every token the model reads, it computes and stores two vectors, a key and a value, so it doesn’t have to reread the whole input for each new token it generates. That store is the KV cache. It sits in GPU memory for the entire life of the request, and it grows in direct proportion to how much you sent.

The KV cache footprint follows a simple formula:

2 × layers × KV heads × head dimension × sequence length × bytes per element

The 2 counts the two vectors stored per token, one key and one value. This is the standard cache accounting from the serving literature; the PagedAttention paper (Kwon et al., SOSP 2023), the work behind vLLM, derives the same per-token calculation. The formula here uses KV heads rather than total attention heads because models like Llama 3 use grouped-query attention, where many query heads share a few KV heads, which shrinks the cache considerably.

Here it is worked out, following the DigitalOcean tutorial, for Llama 3 70B in BF16 precision (80 layers, 8 KV heads, head dimension 128) with a 128K-token input:

2 × 80 × 8 × 128 × 131,072 × 2 bytes ≈ 43 GB

Sit with that number for a second. 43 GB of GPU memory for one user’s context, roughly what a heavily compressed copy of the model itself would take. An NVIDIA A100 or H100 has 80 GB total, and the model weights already claim most of it. Push to a 1M-token context and the cache outgrows the model.

Run the same formula across the Llama 3 family (all three use 8 KV heads and a head dimension of 128; the layer counts are 32, 80, and 126 per their published configurations) and you can see how the cache scales with model size and context length together. Cache sizes only; weights come on top:

Read the 128K column against an 80 GB card. One request takes 21% of the card’s memory on the 8B model, 54% on the 70B, 85% on the 405B, and that’s before the weights. Now look at the 1M column. Every number in it is bigger than 80 GB, which means a single million-token request cannot fit on one GPU no matter what. The cache has to be split across several cards, and during generation those cards constantly pass attention data back and forth over the links between them. Those links are many times slower than a GPU reading its own memory, so serving gets slower and more complicated at exactly the moment it’s already most expensive. (One honest note about the table: Llama 3.1 itself only accepts 128K tokens, so the 1M column is not something you can actually run on these models. It applies the same formula at 1M to show what the math looks like for the frontier models that do advertise windows that size.)

Where this actually hurts is batching. GPUs stay affordable because many requests get processed together and share the fixed costs. Long requests break that in two ways. A single huge cache leaves no headroom for other requests to join the batch, and a mix of small and large requests fragments GPU memory so that even the free space can’t be packed well.

PagedAttention, the technique vLLM uses, allocates cache memory in pages instead of one continuous block and recovers some of the waste. But a 128K request still needs 128K worth of memory. So as average context length creeps up across a fleet, batch sizes shrink, utilization falls, and cost per token climbs, all without anything being wrong with the hardware.

Latency is waiting time: how long between sending a request and getting something back. For a language model it has two parts that users feel differently. The first is the pause before any output appears at all, called time to first token (TTFT). The second is how quickly the rest of the answer streams in once it has started. People tolerate a slow stream far better than a long silent pause, which makes TTFT the number that decides whether an app feels responsive. And TTFT is exactly what long context inflates.

Here’s why. Attention, the operation at the core of every transformer, compares every token in the input against every other token. Compute grows with the square of input length. Double the context, quadruple the work. From 32K to 128K is 4x the length and 16x the attention compute.

All of that happens in prefill, before the first word of the answer appears. Which is why TTFT goes from milliseconds at short contexts to whole seconds at long ones. Optimizations like FlashAttention do help here. They cut down the time the GPU wastes moving data back and forth between its fast and slow memory, and the speedup is noticeable. What they can’t do is change the underlying math. The work still grows with the square of the input; it just grows from a lower starting point. As long as standard attention is quadratic, each token you add makes the next one more expensive.

The second latency cost is easier to miss: what your long request does to everyone else. A 128K-token prefill holds the GPU for seconds, and every request queued behind it waits. The people hurt most aren’t the ones who sent long requests, since they expect to wait. They’re the ones who sent short requests that had the bad luck to land behind a long one. On a dashboard, this looks like healthy average latency while p95 and p99 quietly rot, and that’s the pattern that does the most damage to interactive products like agents and copilots.

Even with unlimited compute, decode hits a wall that has nothing to do with arithmetic. To generate each output token, the GPU has to read the entire KV cache out of high-bandwidth memory (HBM). Flagship GPUs move data at roughly 3 to 4 TB/s, and that number is fixed. Once the cache is tens of gigabytes, reading it once per generated token becomes the constraint that matters.

The relationship is nearly linear. Double the context, double the cache, double the data read per decode step, and your tokens-per-second roughly halves. A cluster can have compute sitting idle and still serve long contexts slowly, because the bottleneck has moved off the compute units and onto the memory bus. If you buy and measure GPU capacity in FLOPS, you’re measuring the wrong constraint for this workload.

You can put an upper bound on this yourself with one division. For a single request, each decode step reads the model weights plus the KV cache, so the best possible speed is bandwidth divided by bytes read. For Llama 3 70B in BF16 (141 GB of weights) on an H100 SXM (3.35 TB/s):

Two caveats. This is a theoretical ceiling for one unbatched request; nothing exceeds it and most systems sit well below it. And for a single request, the weights dominate the read until the cache grows to a comparable size, which is why the ceiling falls gently at first and then steeply. In production the effect is worse than the table suggests, because batching shares one weight read across many requests while every request brings its own cache traffic. Once cache reads dominate the bus, you get the double-the-context, halve-the-throughput behavior. The point of the arithmetic isn’t precision. It’s that you can sanity-check any provider’s throughput claim in thirty seconds.

Cost follows. You might assume a 128K-token request costs about 128 times a 1K request, in proportion to tokens. It usually costs more, once quadratic prefill, shrunken batches, saturated bandwidth, and the blocked queue behind it are counted. Some pricing already admits this. Anthropic’s Claude Sonnet models cost $3.00 per million input tokens up to 200K tokens of input and $6.00 per million past it, with output rising from $15.00 to $22.50, as listed on DigitalOcean’s model pricing page (checked July 2026). Prompt caching exists mostly to soften this: you pay once to process a long, stable context ($3.75 per million tokens to write the cache, with a 5-minute lifetime, for Claude Sonnet), then $0.30 per million each time you reuse it.

The cleanest way to see what this means is to price a session instead of a token. Take a 10-turn agent conversation carrying a stable 128K-token context, say a codebase or a document set, where each turn adds a 500-token question and gets a 500-token answer, at the Claude Sonnet rates above:

Cumulative cost of a 10-turn agent session with a 128K-token context, compared across three strategies: resending the full context each turn reaches $3.93, caching it reaches $0.95, and retrieving an 8K slice reaches $0.33

The plot makes the shape of each strategy visible. Resending grows steeply and never stops. Caching starts higher than retrieval because of the one-time cache write on turn one (you can see it cross the resend line between turns one and two), then flattens to almost nothing per turn. Retrieval stays low and flat throughout.

Same conversation, same model. Caching cuts the bill 4x and retrieval cuts it 12x. At thousands of sessions a day, the architecture choice matters more than the per-token rate. Retrieval isn’t free either; it needs its own infrastructure, which brings us to the next section.

The first three tradeoffs are infrastructure problems. The fourth starts as a model problem and ends up as an infrastructure problem.

Models answer questions about long inputs less reliably than short ones. This is well documented. RULER showed that the effective context length of many models, the length at which they still perform well, falls far short of the advertised maximum. Lost in the Middle showed accuracy dropping sharply when the relevant information sits in the middle of a long input instead of at the start or end.

A 2026 measurement study from IBM Research and TU Delft, Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving (Yoshimura, van de Beek, and Chiba, EuroMLSys '26), adds two findings worth knowing if you run long-context workloads in production.

The first is that accuracy degrades in ways that break intuition. The authors tested five models on key-value lookup tasks at context lengths from 4K to 64K tokens, in English, Japanese, and Chinese. Model size didn’t predict long-context accuracy. Phi3-mini, the smallest model tested, was often the most accurate, and beat the larger Phi3-medium. A 2B model outperformed its 8B sibling at 32K and 64K tokens. One model held up fine to 16K and then collapsed at 32K. Language mattered too: the best model in English wasn’t necessarily the best in Japanese or Chinese. Latency rankings across models stayed stable. Accuracy rankings didn’t.

The second finding is the paper’s core argument: accuracy failures turn into latency. When a model answers wrong, the user or the system retries, and the clock keeps running. The paper gives this a name, Time-to-Correct-Answer (TTCA): the total wall-clock time from the first attempt until the first correct answer. A fast model that’s wrong half the time can feel worse than a slower model that’s right the first time, because the retries add up. Under long-context serving, as the authors put it, accuracy becomes speed.

They also show you can act on this. Their routing design, LAAR (Lightweight Accuracy-Aware Routing), scores each candidate model by expected latency divided by expected success probability, using nothing fancier than input length and language as features, and it won’t resend a retry to a model that already failed. Across five models on A100 GPUs, this cut mean TTCA by up to 31% against load-aware routing and up to 49% against session-affinity routing. The specific numbers matter less than the principle. If you serve long contexts across more than one model, context length should help decide which model gets the request, because the best choice changes with it.

Fair question. If models accept a million tokens, why keep a vector database, a chunking pipeline, and a retrieval step around? Why not paste the whole corpus into the prompt?

Because everything above is the price of that paste. Stuffing the window means paying the quadratic prefill cost, the multi-second time to first token, and the tiered per-token pricing on every request, even when the answer needed three paragraphs of the input. It also means working in the region where models are least reliable. RULER found effective context routinely falling short of the advertised window, and Lost in the Middle found models weakest exactly when the relevant detail is buried mid-context, which is the normal case when a whole corpus goes in. Retrieval sidesteps all four tradeoffs the same way: it keeps requests short. The model reads only the slice that matters, so prefill is cheap, the cache is small, the answer starts fast, and the model works at lengths where it’s dependable.

So long context didn’t make retrieval obsolete. It changed when each approach wins, and the caution runs both ways: work on combining the two, such as Long-Context LLMs Meet RAG, found that retrieving more passages into a big window doesn’t reliably improve answers and can degrade them as irrelevant text piles up. A bigger window is not a license to retrieve carelessly.

The costs of skipping retrieval

The four tradeoffs are the visible price. Teams that drop retrieval and go all-in on the window also hit a set of costs that don’t show up until the system is in production. Five come up again and again.

The bill scales with the window, not with the question. With retrieval, you pay for the few thousand tokens that were actually relevant. Without it, you pay for the whole window on every request, whether the model needed it or not. At the Claude Sonnet rate of $3.00 per million input tokens, one 128K-token request costs about $0.38 in input alone. At 10,000 requests a month, that’s $3,840 a month of input spend before a single output token is billed. The same traffic through an 8K retrieved context is about $240. Nothing is wrong, nothing is broken, and the bill is 16x higher.

Caches expire, and idle time bills you again. Prompt caching looks like the fix for the cost above, and it often is, but the cheap cache tier on Claude Sonnet has a 5-minute lifetime. A user who reads something for six minutes and then asks a follow-up triggers a full rewrite of the cache, another $0.48 for a 128K context at the $3.75 per million write rate. Systems with bursty, human-paced traffic re-pay this constantly, and it rarely appears in the original cost model.

Quality drops silently, in the middle. Lost in the Middle measured double-digit percentage-point accuracy drops when the relevant information sits mid-context rather than at the start or end. Nothing errors and nothing logs. The answers just get worse depending on where in your corpus the truth happened to live, which is a failure mode you can’t see without evaluation sets that vary answer position.

Requests can fail inside the advertised limit. The window on the spec sheet assumes the memory to serve it is free at that moment. Under concurrent load, the KV cache and activations for a very long request can exceed what’s available, and the request fails even though it was within the model’s stated limit. Retrieval-shaped traffic, thousands of small requests instead of a few enormous ones, is simply easier for a serving system to keep promises about.

Interactive latency budgets don’t survive the window. A well-built retrieval step adds tens to a couple hundred milliseconds. Prefill on a very large context costs seconds, and no amount of output streaming hides a slow start. If your product’s responsiveness target is a couple of seconds, the architecture decision is effectively made for you.

None of this says retrieval is free. It says the window’s costs are back-loaded: cheap to demo, expensive to operate. Retrieval’s costs are front-loaded in infrastructure and go down per query. That asymmetry is why the demo and the production bill so often disagree.

A practical way to choose

Long context plus caching wins when the working set is stable and reused. Many requests against the same document set, and the set fits in the window? Cache it once and skip retrieval infrastructure entirely. A day-long agent session that keeps accumulating history is the same shape: the context is the state, and there’s nothing to retrieve from.

RAG wins when the corpus is large, changing, or mostly irrelevant per query. A knowledge base bigger than any window, documents that update daily, or questions that each touch a tiny fraction of the data. Resending the unneeded majority on every request buys nothing but cost and latency.

Precision lookups favor RAG even when the corpus would fit. If the job is finding one specific fact, retrieval keeps the model at short lengths where the accuracy measurements above say it holds up.

Most production systems end up using both: retrieval to decide what enters the context, a long window to hold a generous amount of it, and caching where it repeats.

Every known mitigation buys something and gives something up:

None of these gets you long-context capability at short-context cost. What they add up to is a short playbook:

Don’t send what you don’t need. Retrieval, summarizing old turns, and pruning dead context beat every GPU-side fix, because the cheapest token is the one you never send.

Cache what repeats. A long, stable prefix shared across requests (a system prompt, a document, a codebase) is exactly what prompt caching is for. Pay the prefill once, reuse it cheaply.

Match the model to the length. The recent research results say the most accurate model at 8K may not be the most accurate at 64K, and bigger isn’t automatically better. If you control routing, feed context length into it. If you don’t, test your model at your lengths rather than trusting the advertised window. RULER exists because the two routinely differ.

Watch tail latency and context mix, not averages. The damage hides in p95 and p99, and in autoscaling signals that count requests instead of tokens. Ten 128K requests and a thousand 1K requests can look identical to a request-rate dashboard right up until latency collapses.

Before committing to a provider for long-context work, these are the four questions worth asking. How is long-context throughput measured, alone or under concurrent load? What happens to other tenants’ latency when a long request arrives? Does pricing scale linearly with tokens, or does it reflect the real non-linear cost? And can they show you the prefill versus decode latency split? That last one is a decent maturity test. A provider that has instrumented that split knows where its time goes.

If you’re evaluating how your own stack holds up as context grows, DigitalOcean’s Inference Engine serves long-context models (including Claude Sonnet’s 1M-token input window) with per-token pricing published per model, prompt caching rates listed alongside, and Dedicated Inference for workloads that need isolation from noisy neighbors. Test with your real context lengths and real concurrency, not a single request in a demo. That’s the fastest way to learn whether a context window is something your product can rely on or just something it technically supports.

Sources

• Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving, Yoshimura, van de Beek, and Chiba, EuroMLSys '26 (accuracy variability measurements, TTCA, LAAR results)
• Long-Context Inference at Scale: The Hidden Infrastructure Cost, DigitalOcean, May 2026 (KV cache formula and worked example, prefill/decode analysis, bandwidth ceiling, provider questions)
• Efficient Memory Management for Large Language Model Serving with PagedAttention, Kwon et al., SOSP 2023 (KV cache accounting, paged cache allocation)
• GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints, Ainslie et al., 2023 (grouped-query attention, the KV-heads term in the cache formula)
• RULER: What’s the Real Context Size of Your Long-Context Language Models? (effective vs. advertised context length)
• Lost in the Middle: How Language Models Use Long Contexts (positional accuracy loss)
• Long-Context LLMs Meet RAG: Overcoming Challenges for Long Inputs in RAG (retrieval quality in long windows)
• RAG vs Large Context Window: The Real Trade-offs for AI Apps, Redis, February 2026 (operational cost categories referenced in the hidden-costs subsection)
• DigitalOcean Gradient AI Platform Pricing (tiered long-context and prompt caching rates, checked July 2026)