Measure AI gateway latency and investigate slow calls

What AI gateway latency measures

A request's recorded duration is not the same as gateway overhead. Separate provider processing, response streaming and the client network before deciding which part of an AI call is slow.

What the latency numbers measure

• Gateway request duration: recorded server-side time for the routed request, including provider work and response transfer. For streamed calls, final metering happens after the stream finishes or stops.
• External event duration: the duration_ms your backend or supported completed GenAI span reports. Its measurement boundary is owned by the sender.
• First response byte: request detail metadata can include first_byte_ms when the gateway observes a non-empty response body. That byte may be stream metadata, not the first generated token.
• Gateway overhead: extra time attributable to routing through the gateway. Use a controlled direct-versus-gateway benchmark to estimate it; the average duration chart measures the whole request.

Investigate the same traffic across screens

1. 1
   Choose a representative period

   Start in Overview and select the provider, model or traffic source you want to investigate. Check the latency sample count as well as the average. A broad average can mix short JSON calls with long generations.

2. 2
   Compare duration, failures and model usage

   Use Latency and Usage with the same period, then narrow Requests by the model or status involved. These are measurements of recorded traffic, not provider-wide benchmarks. Periods and chart buckets use UTC.

3. 3
   Inspect individual requests

   Open the corresponding request to check status, duration and routing or fallback outcome. A fast rejected request should not be compared with a successful long stream as evidence of generation speed.

Unknown is different from zero

A dash or No latency samples means the selected traffic has no recorded duration. It does not mean the gateway added zero milliseconds. A measured 0 ms remains a valid recorded value at millisecond precision. Missing values are excluded from averages, and empty chart intervals are gaps, not zero-duration requests.

How to benchmark gateway overhead

From the same client and region, compare direct-provider and gateway calls using the same provider account, model ID, payload, output limit and streaming mode. Run warm-up calls, alternate the two paths and collect enough samples to compare their distributions. Keep errors and retries separate. Provider calls in both paths incur provider usage.

Measure time to first response byte and total completion time separately at the client. For streaming, also measure time to the first generated token in your application. A gateway-minus-direct difference is an estimate under those test conditions; provider load, network variation and connection reuse can change it. Calculate any p50 or p95 from those samples; the Latency page displays averages, not percentiles.

For an existing backend, send completed-call metrics or use OpenTelemetry ingestion. Keep measurement boundaries consistent before combining sources.

Verify it works

Open Latency and Requests for the same period. Recorded durations appear as measured values, including a real 0 ms. Traffic without recorded duration shows a dash and no latency samples. The duration average is not a measurement of gateway overhead.

Troubleshooting

Requests exist but latency is missing

For external events, send an actual duration_ms or a supported completed span with valid start/end timestamps. Do not insert zero as a placeholder. Gateway receipts without a usable timing remain unknown.

A streamed request looks slower than a JSON request

Compare like-for-like streaming mode and output length. Total stream duration and time to the first generated token answer different questions.

The average improved while failures increased

Inspect statuses and routed attempts. Short rejected requests can lower an overall average without improving successful generation speed.

Full error-code reference

Next steps

1,000 requests per month shared across projects. No card required. Provider charges are separate.