A four-state LLM serving-latency scatter: time to first token against time per output token for every request, nearest-rank p50/p95 crosshairs, an inclusive budget crosshair that cuts the plane into four filterable zones, and queue/prefill/decode phase bars for the median and p95 requests.
Preview in your theme
Loading preview…
This component is part of Pro
The install command, prompt and concept map are included with a Pro license — one purchase unlocks every Pro component, forever.
status="ready" · 40 sampled requests from one afternoon, judged against a 300 ms first-token budget and a 20 ms/token streaming budget. The solid crosshair is the budget and the grey L is everything outside it; the dashed and dotted crosshairs are the p50 and p95 of the whole sample, so the p95 first-token line lands outside the budget while the median sits well inside it. Press a zone chip to take that zone out of the plot — the axes and the percentile lines hold still, and the shares re-apportion by largest remainder so every filter state still reads exactly 100%.
Serving latency · chat endpoint
40 requests · Tue 14:00–18:00 UTC · p95 497 ms to first token · p95 25.9 ms/token · 26 within budget
65%
of requests met both budgets
26 of 40 requests metered
budget 300 ms · 20 ms/token
21.9 s
p95 end to end
p50 3.28 s · 40 requests
TTFT + TPOT × (tokens − 1)
x: time to first token (ms) · y: time per output token (ms/token) · dot area: output tokens
p50 236 ms · 13.2 ms/tokenp95 497 ms · 25.9 ms/tokenbudget 300 ms · 20 ms/token · 50 tok/s
Where the wall clock goes · queue, prefill, decode
Median request, end to end3.28 s
4819 · orbit-3 · slow first token
Queue 18 ms · <1%Prefill 316 ms · 10%Decode 2.94 s · 90%
p95 request, end to end21.9 s
d1a8 · orbit-3 · slow streaming
Queue 68 ms · <1%Prefill 221 ms · 1%Decode 21.6 s · 99%
End-to-end time is derived, never supplied: TTFT plus TPOT times one fewer token than the response carried, because the first token is already inside TTFT. The p50 and p95 lines are nearest-rank over all 40 plotted requests and hold still while the filter moves the dots — and they are marginal, so the point where the two p95 lines cross is not a request anyone made. These 40 were sampled from 128,400 served, so p95 here is the 3rd slowest of 40 — quote your metrics store for the window as a whole. orbit-3 misses a budget on 13 of the 27 requests it served (48%), against 1 of 13 (8%) for orbit-3-mini.
Serving latency · chat endpoint: every plotted request with its first-token time, per-token time, response length, derived end-to-end time and budget zone, unfiltered. 40 requests, 26 within both budgets.
Request
Model
Time to first token (ms)
Time per output token (ms)
Output tokens
End to end (ms)
Zone
a17c
orbit-3-mini
96
7.8
128
1,087
Within budget
b204
orbit-3-mini
108
8.3
96
897
Within budget
c33f
orbit-3-mini
117
8.9
212
1,995
Within budget
d0a8
orbit-3-mini
126
8.6
164
1,528
Within budget
e51b
orbit-3-mini
134
9.4
268
2,644
Within budget
f27d
orbit-3-mini
143
9.1
112
1,153
Within budget
0a93
orbit-3-mini
151
9.8
305
3,130
Within budget
8c04
orbit-3-mini
162
10.3
88
1,058
Within budget
9d72
orbit-3-mini
173
9.6
244
2,506
Within budget
ae15
orbit-3-mini
184
10.7
132
1,586
Within budget
04d5
orbit-3-mini
119
8.1
176
1,537
Within budget
15e9
orbit-3-mini
195
11.2
198
2,401
Within budget
1b46
orbit-3
203
12.4
226
2,993
Within budget
2c7e
orbit-3
212
11.9
168
2,199
Within budget
3d18
orbit-3
224
13.1
342
4,691
Within budget
4e60
orbit-3
231
12.7
198
2,733
Within budget
5f2a
orbit-3
240
13.6
276
3,980
Within budget
6a81
orbit-3
248
14.2
384
5,687
Within budget
7b39
orbit-3
256
13.8
152
2,340
Within budget
bf88
orbit-3
219
12.1
296
3,789
Within budget
c096
orbit-3
263
14.6
210
3,314
Within budget
d1a3
orbit-3
271
15.3
358
5,733
Within budget
e2b7
orbit-3
278
14.9
174
2,856
Within budget
f3c1
orbit-3
285
16.1
402
6,741
Within budget
26f2
orbit-3
236
13.4
268
3,814
Within budget
3708
orbit-3
290
17.2
330
5,949
Within budget
4819
orbit-3
334
13.2
224
3,278
Slow first token
5920
orbit-3
372
12.6
168
2,476
Slow first token
6a31
orbit-3
418
14.1
296
4,578
Slow first token
7b42
orbit-3-mini
455
10.6
208
2,649
Slow first token
8c53
orbit-3
497
15.2
342
5,680
Slow first token
9d64
orbit-3
560
16.5
262
4,867
Slow first token
ae75
orbit-3
226
21.5
512
11,213
Slow streaming
bf86
orbit-3
258
23.4
688
16,334
Slow streaming
c097
orbit-3
187
22.1
434
9,756
Slow streaming
d1a8
orbit-3
289
26.8
806
21,863
Slow streaming
e2b9
orbit-3
244
24.3
596
14,703
Slow streaming
f3ca
orbit-3
383
22.6
742
17,130
Slow both ways
04db
orbit-3
461
25.9
918
24,211
Slow both ways
15ec
orbit-3
545
29.5
1,164
34,854
Slow both ways
p50 across all 40, each column ranked separately
—
236
13.2
—
3,278
—
p95 across all 40, each column ranked separately
—
497
25.9
—
21,863
—
status="empty" · a valid contract carrying budgets but no traced requests, worded so it cannot be mistaken for a failed fetch — a budget is a service target, known long before anything has been measured against it.
Serving latency · chat endpoint
First-token latency against per-token latency, one dot per request
No requests in this window yet
A dot needs a time to first token, a time per output token and a response length; budgets alone draw the crosshair but not the cloud.
status="loading" · a skeleton holding the two stat tiles, the plot box with its per-token gutter and the four zone chips, aria-hidden, honouring prefers-reduced-motion, with no timer involved.
Serving latency · chat endpoint
First-token latency against per-token latency, one dot per request
Loading serving latency
status="error" · the tracing source failed. Try again only appears when onRetry is passed, and clicking it flips the card to ready.
Serving latency · chat endpoint
First-token latency against per-token latency, one dot per request
Zyeon UI