A four-state cluster-utilisation card: a device-by-time SM occupancy strip, one memory line per device group against the HBM ceiling, and idle stalls drawn as bubbles sized by their duration — with the idle share, the longest stall and a wall-clock budget that totals exactly 100% derived in-component.
Preview in your theme
Loading preview…
This component is part of Pro
The install command, prompt and concept map are included with a Pro license — one purchase unlocks every Pro component, forever.
node-a · pp stage 0peak 171.2 GB · 95%node-b · pp stage 1peak 149.8 GB · 83%highest device in each group · 8.8 GB headroom at the peak
SM occupancy is the share of each 10 s sample with a warp resident — busy, not efficient: a memory-bound kernel holds a row bright at 100%. Idle share charges the union of the stall intervals, so overlapping stalls are never billed twice; each stretch goes to whichever stall opened first. The memory axis starts at 130 GB rather than zero — what a lane is read for is its distance to the 180 GB ceiling. Click a bubble, or focus the lane and use the arrow keys, to read what occupancy and memory did inside it.
Cluster utilisation · pretrain-8b. SM occupancy for 8 devices across 5 min, sampled every 10 s, with resident memory read against a 180 GB ceiling. 3 stalls cost 90 s of the 5 min window, 30.0% of it. All-reduce wait is the biggest single drain: 38 s across 1 stall. Outside the stalls occupancy averages 90%, against an overall mean of 70%. Peak memory 171.2 GB on gpu 3 at 4:00, 95% of capacity. Exact values follow in the data tables.
Every drawn device: how many samples it reported, its mean SM occupancy and its peak resident memory.
Device
Group
Samples
Mean SM occupancy
Peak memory
gpu 0
node-a · pp stage 0
30
68%
169.2 GB at 4:00
gpu 1
node-a · pp stage 0
30
67%
169.2 GB at 4:00
gpu 2
node-a · pp stage 0
30
69%
167.7 GB at 4:10
gpu 3
node-a · pp stage 0
30
66%
171.2 GB at 4:00
gpu 4
node-b · pp stage 1
30
69%
147.0 GB at 4:00
gpu 5
node-b · pp stage 1
30
68%
149.5 GB at 4:10
gpu 6
node-b · pp stage 1
30
76%
145.9 GB at 4:10
gpu 7
node-b · pp stage 1
30
70%
149.8 GB at 4:00
The wall-clock budget of this 5 min window, apportioned to whole percents that total exactly 100.
Bucket
Share
Time
Compute
70%
3m 30s
All-reduce wait
13%
38 s
Data-loader starvation
10%
30 s
Checkpoint write
7%
22 s
3 stalls, in time order, with what occupancy and memory did inside each.
Opened
Cause
Duration
Inside
1:06
All-reduce wait
38 s
SM occupancy 16% inside against 90% elsewhere, busiest gpu 6 at 84%, peak memory 164.2 GB. Rank 6 finished its backward 3.9 s late; every other rank waited in the bucket.
2:26
Data-loader starvation
30 s
SM occupancy 18% inside against 90% elsewhere, busiest gpu 7 at 20%, peak memory 164.9 GB. Input queue drained — 14 of 16 loader workers blocked on shard fetch.
3:56
Checkpoint write
22 s
SM occupancy 37% inside against 90% elsewhere, busiest gpu 7 at 40%, peak memory 171.2 GB. Sharded optimiser state flushed to object storage, 1.4 TB.
empty
Cluster utilisation · pretrain-8b
step 41,200 · 1.94 nats · 8xB200
No samples in this window
One row per device fills in as the profiler reports SM occupancy and resident HBM. Stall intervals are charged against the seconds those samples span, so the card needs at least one sampled device before it can call anything idle.
Zyeon UI