Why is p99 Time to First Token (TTFT) High When Everything Else Looks Normal? Debugging Tail Latency in LLM Inference

• Time to First Token (TTFT): the time between when a client sends a request and when the first token comes back. At the tail end, this delay usually isn’t just about how long the model takes to compute that first token. A request can also lose time sitting in a queue waiting for a free GPU slot, or waiting to get routed to the right server. Those delays add to TTFT just as much as the actual compute time does.
• Time per Output Token (TPOT): the average time it takes to generate each token after the first one.
• Inter-Token Latency (ITL): the time gap between one token and the next during generation.
• 99th percentile (p99): the value below which 99% of requests fall. If p99 TTFT is 2 seconds, that means 99 out of 100 requests start responding in under 2 seconds, and the slowest 1 request takes longer. It’s a way to measure the worst-case experience instead of the typical one, which is why it can look bad even when the average looks fine.
• KV-cache (key-value cache): the GPU memory that holds the intermediate attention data (the “key” and “value” tensors) for every token in every active request. The model reuses this cached data instead of recomputing it on each step, which is what makes generation fast. But it takes up GPU memory, and that memory is limited, so it fills up as prompts get longer or more requests run at once.
• GPU utilization: the percentage of time the GPU is actively doing compute work. It’s a useful health signal, but it only tells you how busy the GPU is, not whether requests are stuck waiting to be admitted into a batch. A GPU can show high utilization while new requests still sit in a queue.

One thing to know: different tools define these terms slightly differently. Some treat TPOT and ITL as the same thing. Others, like NVIDIA’s genai-perf, treat TPOT as an average over the whole generation and ITL as a per-token measurement. Check which definition your tooling uses before comparing numbers across systems.

A production inference service can look completely healthy while some users are already having a bad experience:

• Average TTFT barely moves.
• Median TTFT still hits its target.
• GPU utilization looks normal.
• Tokens per second show no drop.

But a small, and often commercially important, group of requests can take several seconds just to start responding. That’s the danger of a bad p99 TTFT: the problem is invisible in the averages.

The cause is usually not “the GPU is slow.” It’s more often something happening between request queueing, continuous batching, prompt length, KV-cache allocation, model restarts, routing, and general infrastructure contention. Queueing delay and compute time depend on when a request shows up relative to everything else in the system. Two identical requests, sent to the same model on the same GPU, can have very different TTFTs depending on what else is running at the time.

To debug this well, you need more than a single latency chart. You need to trace the request all the way from the load balancer to the first streamed token, and measure queueing time and compute time separately.

This article walks through a practical, incident-based way to debug p99 TTFT problems in production LLM inference. It covers how to read TTFT alongside TPOT and ITL, five common failure patterns, how to recreate hidden problems by replaying real traffic patterns, and how to pick fixes based on actual data. The goal isn’t just to notice that inference is slow. It’s to find exactly where the delay is coming from and what change removes it.

A simple way to break down TTFT:

TTFT ≈ network time + router queue time + scheduler queue time + prefill (compute) time

Say 99 requests finish in about 400ms, but one request with a long prompt takes six seconds to even start. The average latency barely moves, but the p99 blows past your target. If your only dashboard shows the average or the median, you might not even know there’s a problem until users start complaining that the app “freezes” sometimes.

Tail latency is also sensitive to shifts you won’t see in overall throughput. A service could be handling the same number of requests per second, but now with longer prompts, more requests arriving at once, or a different mix of customers. The GPU can keep producing tokens at a normal rate while new requests sit and wait behind ones that are already running.

A good rule of thumb: never look at p99 TTFT by itself. Always check it against queue time, prefill time, prompt length, batch state, KV-cache usage, routing delay, and which instance handled the request.

TTFT, TPOT, and ITL each describe a different part of the request. Comparing them tells you where to look:

• TTFT goes up, TPOT and ITL stay normal: the GPU decodes tokens at a normal speed once generation starts. So the problem is upstream: network delay, router queueing, scheduler delay, a cold model, KV-cache allocation, or overlapping prefill work from other requests.
• Both TTFT and TPOT get worse: there’s probably a shared compute bottleneck. Think GPU saturation, thermal or power throttling, memory bandwidth contention, tensor-parallel communication overhead, a bad instance, or noisy neighbors.
• TTFT stays normal but ITL becomes jumpy: the request starts on time, but something interrupts decoding along the way. This can be scheduler preemption, unstable batch sizes, delays in collective communication, or long prefills getting mixed in with decode work.

Queue time and prefill time are what really tell the two apart. If queue time is the main driver of a slowdown, tuning your model’s kernels won’t help. If queue time looks fine but prefill time grows with prompt length, the problem is in how the model executes.

Continuous batching improves throughput by letting requests join and leave a running batch instead of waiting for a fixed batch to finish. But it doesn’t remove contention. At high concurrency, the scheduler still has to decide between new prefills, running decodes, and what to preempt.

Batching contention usually looks like this:

• p99 TTFT jumps sharply once concurrency crosses a certain point.
• Queue time grows faster than prefill compute time.
• TPOT for requests already running stays close to normal, or only gets a bit worse.
• The number of running sequences stays close to its configured max.
• Requests pile up waiting even though GPU utilization looks fine.

To confirm this, replay the same mix of prompts at increasing concurrency. Plot p50, p95, and p99 queue time against concurrency. You should see a clear bend in the curve where new requests are arriving faster than the scheduler can admit them within your target latency.

Next, break requests down by prompt length. A single overall percentile can hide the fact that long prompts are slowing down short, interactive ones. Compare prompts under 512 tokens against buckets like 512 to 2048, 2048 to 8192, and anything above 8192 tokens.

Fixing high queue time is usually about admission control, not just making the queue bigger:

• Cap concurrency before queue time gets out of hand, and send back a retryable error or route extra traffic elsewhere.
• Add more replicas for traffic that’s consistently high.
• Route requests based on queue depth as well as GPU utilization, not utilization alone.

If you’re running self-hosted on GPU Droplets, look into chunked prefill, max batch size, max batched-token limits, and keeping long-context traffic in a separate pool from interactive traffic. For Dedicated Inference clusters, use saturation and per-endpoint latency numbers, not just overall GPU utilization, to decide on capacity and routing. DigitalOcean’s Inference platform supports both serverless and dedicated deployments, and the Inference Router (currently in public preview) can send serverless requests to different models based on cost or latency rules you set. Routing can help soak up traffic spikes in some cases, but it shouldn’t be used to hide a saturated endpoint behind an average.

KV-cache memory gets used up by every active sequence. It fills up faster with longer prompts, bigger batches, more concurrency, and longer generations. When free cache blocks get low, the system may start rejecting requests, kicking out active sequences, swapping state to slower memory, or recomputing state it dropped.

This can look like a scheduler problem at first, since new requests spend more time waiting. The way to tell them apart is KV-cache occupancy. If queue time rises as the cache fills up and free blocks disappear, memory pressure, not the scheduler, is the real cause.

Watch for:

• KV-cache usage getting close to its limit.
• Free cache blocks dropping.
• Preemption, eviction, swap, or recompute counts going up.
• TTFT rising the most for long-context requests.
• Uneven throughput even when GPU compute isn’t maxed out.
• Latency improving right away once you cut context length or concurrency.

Confirm this with two matched tests, one with short prompts and one with long prompts, at the same concurrency. If longer prompts alone cause the cache to fill up and start preempting requests, memory pressure, not the total number of requests, is the bottleneck.

The real fix is planning capacity around token count, not request count. A queue with ten 16K-token prompts is a very different load than one with ten 500-token prompts.

Other options: lower the max context length, cap concurrent sequences, add more cache, switch to a smaller or compressed model, add GPU capacity, or send long-context requests to a separate pool.

Prefix caching can help when prompts share a common prefix, but it’s not a cure-all since unique prompts still use up cache space. A high cache hit rate just means good reuse, it doesn’t mean the cache is under control. Rising occupancy, allocation failures, eviction, and preemption together are what actually signal pressure.

Cold starts shouldn’t be mixed in with normal steady-state latency. A brand-new GPU Droplet, a restarted container, a newly scaled-up replica, or a reloaded model all need time to download weights, allocate GPU memory, set up kernels, build CUDA graphs, or warm up code paths. Requests that land on that replica belong to a different, separate group of latencies.

Signs of this include:

• Very high TTFT tied to specific instance IDs.
• A spike right after a deployment, restart, scaling event, or failed health check.
• Normal queue time, but the model isn’t actually ready yet, or requests get accepted before it is.
• Latency returning to normal after the first few requests.
• More container restarts or longer model-load times.

Confirm it by matching slow requests to instance_id, container_start_time, model_loaded_at, model version, and deployment version. If slow requests cluster around recently started replicas for a limited window, it’s a startup issue, not a general lack of capacity.

Make sure your readiness checks actually check that inference is ready to go. A process that’s just listening on a port isn’t necessarily ready to handle a real request. Keep a replica out of the pool until its weights are loaded, memory is allocated, and a test warm-up request succeeds.

During deployments, roll out changes gradually and keep a minimum number of warm replicas available to take traffic. GPU Droplets give you GPU-backed virtual machines for self-managed inference, in single-GPU or 8-GPU setups. On these, you’re responsible for your own model loading and warm-up process.

A request can spend most of its TTFT before it even reaches the inference server. Without timestamps at the edge and application layer, it’s easy to blame the GPU by default, since TTFT feels like an inference metric.

Network-side queueing usually shows up as:

• High TTFT on the client side, but normal TTFT on the inference side.
• A gap between when the load balancer got the request and when it reached the backend.
• Steady GPU queue depth and KV-cache usage during the incident.
• Latency concentrated in one region, route, tenant, or connection pool.
• More connection retries, slow TLS setup, or proxy buffering.

Use a correlation ID that follows the request through the client, load balancer, router, gateway, and inference runtime. Timestamp everything with both wall-clock time and monotonic time. Your traces should clearly show where the request sat waiting.

Streaming setup deserves a closer look too. A proxy might buffer the generated output instead of forwarding the first token right away. In that case, the server will report a healthy first-token time while the client still sees a slow TTFT.

An inference router matters any time requests get sent dynamically to different models or providers. Track which route was picked, how long the routing decision took, how many retries happened, how many fallbacks were tried, and the latency to wherever the request ended up. A fast model can’t make up for a slow router.

GPU workloads can compete for GPU memory and compute, CPU time, host memory, storage bandwidth, or network bandwidth. Even when GPUs are properly isolated, there’s still CPU and network overhead from tokenizing, parsing requests, logging, and streaming output. A noisy-neighbor problem usually looks like:

• Tail latency concentrated on one host or replica.
• CPU steal time, memory pressure, disk latency, or network retransmissions rising at the same time.
• TTFT and TPOT both changing with no clear connection to prompt length.
• Overall service metrics looking fine, since the other replicas are healthy.
• Latency improving once the workload moves or the instance gets replaced.

Always keep per-instance data around. A cluster-wide p99 with no host information just blends one bad replica together with nine healthy ones.

For self-managed setups, compare GPU metrics against host-level ones: utilization, memory use, power state, clock speed, PCIe throughput, CPU load, run-queue depth, memory faults, storage latency, and network errors. Draining a suspected replica and watching if tail latency improves is a quick way to confirm it’s a bad host.

Using dedicated capacity removes some kinds of contention, but it doesn’t remove contention between different workloads sharing the same model endpoint. Keep tenant, route, and workload labels around even on dedicated hardware.

1. Lock in the regression window and compare it to a known-good baseline. Plot p50, p95, and p99 TTFT, not just the worst case. Check whether the problem is limited to one model, region, endpoint, tenant, prompt-length range, or instance.
2. Break TTFT down into network time, routing time, scheduler queue time, and prefill time. This tells you which branch of the problem to chase.
3. Check the decode path. If TPOT and ITL look fine, the decode path probably isn’t the bottleneck, and broad GPU compute or memory issues are less likely. If generation is also slow, look at compute saturation, memory bandwidth, communication overhead, and preemption.
4. Compare scheduler state to KV-cache pressure. The number of waiting requests plus available cache blocks tells you more than GPU utilization alone.
5. Line up the timing with recent changes: deployments, restarts, autoscaling events, config changes, traffic shifts, and model updates.

If you don’t have a clean production incident to work from, recreate the problem on a DigitalOcean GPU Droplet instead. This works well because it gives you real, checkable numbers without touching customer data. Set up a fixed model and inference runtime on a specific GPU Droplet configuration, and write down the GPU type, GPU count, model version, numeric precision, tensor-parallel size, inference runtime version, cache settings, and scheduler limits.

Use a load-testing tool like vllm bench serve, genai-perf, or your own async client. Run at least three prompt setups:

• Short prompts for interactive use: 256 input tokens, 128 output tokens.
• Realistic mixed traffic across a few prompt-length ranges.
• Long-context load: 8,192 or 16,384 input tokens.

Increase concurrency step by step (1, 4, 8, 16, 32, 64, 128) without changing anything else. Let each step run long enough to reach a steady state, and repeat each one at least three times. Collect client-side TTFT, server queue time, prefill time, TPOT, ITL, request throughput, waiting and running sequence counts, KV-cache usage, preemption count, GPU utilization, GPU memory, CPU load, and network latency.

Report the actual numbers you measure, including how much they vary, instead of guessing at expected values. For example:

Under the short-prompt workload, p99 TTFT stayed below [measured value] up to concurrency [value]. With [input length] prompts, KV-cache occupancy hit [value]%, preemptions rose from [value] to [value], and p99 TTFT went from [value] ms to [value] ms. TPOT only changed by [value]%, which points to the regression happening before decoding. Reducing [setting] or adding [capacity change] brought p99 TTFT down to [value] ms under the same load.

Note: the bracketed placeholders above are just an example structure, not real results. Before publishing, replace every placeholder with numbers from an actual benchmark run. Include the full benchmark setup used to get them. This section is the reason this piece is currently blocked. Until CPTO benchmarking capacity is scoped and scheduled for this piece, it should stay in the same holding pattern as the Long-Context Serving Tradeoffs and Inference Observability pieces, not go out with placeholder numbers.

A bad p99 TTFT almost never comes down to one simple “slow inference” number. It’s a symptom that shows up from waiting, prefill cost, cache allocation, model readiness, routing, and general infrastructure contention. The fastest way to find the cause is to break TTFT down at the request level. High queue time points to admission and scheduling. High prefill time points to context length and computation. Cache saturation and preemption point to memory pressure. A mismatch between client and server numbers points to routing or buffering. Slowdowns tied to one replica point to cold starts or noisy neighbors.

DigitalOcean GPU Droplets give you the fine-grained control needed to test and reproduce these problems. Dedicated Inference and the Inference Router offer managed ways to deploy and route traffic, but you should still watch them at the request level. Infrastructure metrics and overall GPU utilization are not a substitute for actually breaking latency down.

• DigitalOcean Inference documentation
• NVIDIA GenAI-Perf documentation
• vLLM production metrics
• USENIX: Orca paper and presentation
• vLLM KV-cache and preemption guidance