Why AI responses are slow: tracing retrieval, tools and model generation

A user reporting “the AI is slow” might mean a blank screen before an answer, long pauses while text arrives, or a long wait for a finished result. Start by selecting that milestone and its clock boundaries.

The first two are application measurements proposed here. They are not standard GenAI metric names. The model client may run in your backend; “client” in a provider metric does not necessarily mean the browser.

OpenTelemetry's Development conventions define gen_ai.client.operation.time_to_first_chunk for streaming calls. They separately describe server gen_ai.server.time_to_first_token as covering the wait through queueing and prefill. Different start points and observation locations make these different measurements. GenAI metric definitions

Record the first protocol chunk and the first answer-bearing chunk separately if the protocol can send metadata or tool arguments before answer text. Neither receipt proves that the UI has displayed anything. Likewise, an HTTP first-byte timer may stop at headers or a status event.

Streaming can make partial output visible before generation finishes. It does not, by itself, establish less total generation work or a faster completed answer; Hugging Face's TGI documentation illustrates earlier visibility with unchanged completion time. Measure both milestones in your application. TGI streaming

Create a trace for the application operation, with child spans for retrieval, each tool execution and each model call. OpenTelemetry spans represent work with timestamps, parentage and attributes; events can mark individual instants, and propagated context joins work across services. These primitives support a request timeline, but missing instrumentation still leaves missing evidence. OpenTelemetry traces

Use the conceptual timeline below to decide which boundaries you can observe. Prefill means processing the model input before producing the initial output; subsequent generation is commonly called decode. A provider's internal queue and computation need provider-side measurements. An application timer around its SDK cannot separate them from connection setup, transit or other handling.

The depicted route performs retrieval and tools before its final generation. A model-driven tool loop also has a model call that chooses the tool, followed by execution and another model call. Include every round; do not hide those calls inside a single “tools” label.

Measure the critical path: the sequence of dependencies that determines when the chosen milestone can happen. If two required lookups overlap, their elapsed contribution is their combined time coverage up to the join, not the sum of both durations. A parent span already includes its children. Adding the parent and children counts the same time twice.

For each large interval, ask whether the next step actually waits for it. Work continuing after the final result, such as an asynchronous export, should not be blamed for that result's latency unless it contends for resources in a way you can measure. Gaps in a trace are unresolved time; add instrumentation before calling them queueing or model computation.

For application queues, capture enqueue and work-start boundaries. For retrieval, split query preparation or embedding, search, reranking and context assembly when the aggregate span is large enough to investigate. For each tool, distinguish client wait from downstream execution if both sides are observable. These are proposed instrumentation boundaries, not fields every tracing integration supplies automatically.

Compare requests from the same route and workload class before changing infrastructure. “High” in this table means worse than your relevant baseline or product objective, not a universal threshold. The table is a diagnostic procedure derived from the measurement boundaries above.

A concrete server-side example is vLLM's V1 metrics design, which documents queue, prefill and decode intervals as well as frontend timing. Its TTFT starts at frontend input processing, whereas some intervals come from engine events. Read the installed engine's definitions before combining them. The same design distinguishes a gap between output events from a per-request time-per-output-token calculation, because one event can contain multiple tokens. vLLM metrics design, pinned revision

Use that distinction with any backend: a chunk-gap measurement is not automatically a token-generation measurement. Without server telemetry, report what the application observed rather than deriving a GPU explanation from a streaming gap.

Hypothetical trace; all timings are assumed. No application, benchmark or load test was run. Imagine a support assistant that retrieves a policy and then runs two read-only lookups: order status and delivery estimate. Application logic already knows both lookup inputs after retrieval; no model call chooses these tools in this example. It waits for both results before sending one final generation request.

Assume one successful attempt, no cache hits, no output validation delay, and a first protocol chunk that already contains answer text. The unified time axis below is an explanatory assumption; real measurements across machines require clock care and correlation.

Three conclusions follow from these assumptions:

1. First visible answer takes 3.70 s; completion takes 6.10 s. The model client's first-chunk time is only 0.80 s. Calling all three values “TTFT” would hide most of the user's wait.
2. The model request starts after 2.78 s. Its 0.80 s first-chunk interval cannot explain that earlier delay. The application has 1.80 s of serial tool work worth investigating. The 0.80 s provider-call interval remains unsplit; this trace contains no provider queue or prefill measurements.
3. The model-call duration is 3.20 s, from 2.78 to 5.98. The 0.12 s first-display interval overlaps the 2.40 s remaining stream interval, so adding every table row would overcount. A non-overlapping completion calculation is 2.78 + 0.80 + 2.40 + 0.12 = 6.10 s.

Now suppose inspection confirms the tools are independent, both inputs are ready at 0.88 s, and running together changes neither result nor duration. Their joined duration would be max(0.70, 1.10) = 1.10 s instead of 1.80 s. If every later duration also stayed unchanged, the first visible answer would move to 3.00 s and completion to 5.40 s: a conditional 0.70 s saving.

That is arithmetic under stated assumptions, not a promised improvement. Shared rate limits or resource contention can invalidate it. If delivery estimate actually needs the order-status result, the dependency prevents this overlap.

For comparison, halving the assumed retrieval interval saves only 0.30 s under the same unchanged-downstream assumptions. This comparison helps choose a focused experiment; it does not justify removing evidence the answer needs. Enabling streaming is not the next fix here because the assumed request already streams.

Use this checklist to validate an existing trace before adding a dashboard or optimization.

• Define two user milestones. Record submission, first answer display and usable completion in the client. Document how the display timestamp is captured; a DOM update callback is a proxy unless actual rendering is measured. Track status-only updates separately.
• Correlate boundaries. Preserve a request/trace relationship across the application, retrieval service and tools; retain provider request IDs when available. If the provider does not participate in your trace, mark that visibility limit.
• Time local intervals locally. Use a monotonic clock for elapsed durations. Do not subtract unrelated process or browser clock origins. Treat cross-host absolute timestamps as subject to clock skew; use locally measured spans and explicit correlation when reconciling the timeline. vLLM interval calculations
• Cover waiting and every round. Record ingress, enqueue/start, connection acquisition where observable, retrieval stages, each model/tool attempt, retry backoff and final processing. Identify missing intervals rather than assigning them a guessed cause.
• Keep the stream inside the timer. Record first protocol chunk, first answer-bearing chunk, subsequent receipt events or a suitable gap distribution, end of consumption and terminal status. Check whether SDK instrumentation ends when the iterator is returned or when it is exhausted. For non-streaming calls, first-chunk telemetry is inapplicable; measure completion rather than inventing a zero. GenAI metric definitions
• Keep delivery observable. Record when the application forwards content and when the browser receives and displays it. Start with coarse boundary events; collecting every chunk can add overhead and volume. Verify the instrumentation under the workload it is meant to diagnose.
• Record comparison context. Include route, model identity/version where available, prompt/configuration revision, streaming mode, input/output token counts and their accounting source, tool-round count, cache outcome, concurrency and outcome. Keep request IDs on traces rather than making each one a metric label.
• Keep failures visible. Distinguish success, timeout, cancellation, retry and partial-stream failure. A response that emitted one chunk and then failed is not a completed answer. Track the count of requests that never reached each milestone, alongside durations for those that did.
• Use comparable distributions. Report count, time window and p50/p95 for first-visible and completion separately; add higher percentiles when the sample supports them. Separate streaming modes, workloads and outcomes before interpreting a change. Do not use only deliberately selected slow traces to estimate fleet latency.
• Aggregate before computing quantiles. Merge compatible histogram observations/buckets across instances and then estimate the percentile. Averaging instance p95s is not the overall p95; histogram resolution also limits precision. Prometheus histograms and summaries

Likewise, a request's completion p95 cannot be reconstructed by summing the p95 of each stage: the slowest observations may belong to different requests, and stages may overlap. Measure completion directly, then inspect representative traces from that distribution to explain it. This is a consequence of the boundaries and dependencies, not an additional telemetry metric.

Turn the diagnosis into a small decision record: target milestone → observed delaying stage → proposed change → expected signal → correctness check. Keep an explicit “unknown” when the trace cannot yet attribute a delay.

Treat these as investigations to run, not results reported by this guide. Hold the task, model configuration, cache conditions, output requirements and load comparable where possible; retain differences you cannot control. Repeat enough cases to distinguish a useful change from generation and traffic variability. Compare the user milestone, completion/error counts and answer usefulness, not just the one span you optimized.

Documentation context, checked 26 September 2026: the OpenTelemetry GenAI definitions cited here are pinned to document revision 8ffdf568e1b4 (22 September 2026) and marked Development. The illustrative vLLM V1 design is pinned to b78fbab4be55 (17 September 2026); it includes historical and proposed material, so this guide uses only its stated measurement boundaries and interval distinctions, not a promise that every release exports every metric. TGI's streaming page, OpenTelemetry's trace concepts and Prometheus guidance were fetched on the same date. Record your installed instrumentation and server versions before adopting metric names or comparing results. No SDK execution, measured latency result or deployment recommendation is claimed.