A small-multiples grid of causal attention maps — one miniature per (layer, head), sortable by row entropy or previous-token mass, with the picked head enlarged beside it under token labels on both axes and its mass split into four shares that add to exactly 100.
Preview in your theme
Loading preview…
This component is part of Pro
The install command, prompt and concept map are included with a Pro license — one purchase unlocks every Pro component, forever.
Aurora-3 34B · 36 of 1,920 heads · 6 of 48 layers · 11-token prompt · 262K context
checkpoint 1.42M steps · val loss 1.87 nats · attentions from one greedy pass
Order heads by
L0L2L6L14L27L41
H0
H3
H7
H11
H19
H26
H0
H3
H7
H11
H19
H26
H0
H3
H7
H11
H19
H26
H0
H3
H7
H11
H19
H26
H0
H3
H7
H11
H19
H26
H0
H3
H7
H11
H19
H26
In the checkpoint's own head order, one row per layer. 5 of 36 heads send more than half of their mass to the token immediately before.
< 2%5%20%50%100%attention weight
Layer 14 · head 0
induction-like
0.44 nats
row entropy
uniform rows: 1.75
1.6 tokens
effective span
uniform reach: 5.8
18%
previous token
mass landing on q − 1
self (q)0%
previous (q − 1)18%
first token57%
other earlier25%
Induction score 81% — mean mass on the token that followed an earlier copy of the query, over the 3 rows that repeat one.
Every row is one query token's post-softmax distribution over the keys it can see, so it sums to 1 and the map is triangular. Row 0 is left out of every mean — a first token can only attend to itself. The four shares are disjoint and tested in order (self at q, then previous at q − 1, then the first token, then anything earlier), so a row-1 cell that is both the first token and the previous one is counted once, as previous; largest remainder makes them add to exactly 100. The induction score is measured separately and may overlap them. A head is tagged by the first of induction, previous, first token or self to reach 45%, and diffuse when its effective span reaches 85% of a uniform row's.
Attention maps for 36 heads of Aurora-3 34B, 6 layers sampled from 48, over a prompt of 11 tokens. In the checkpoint's own head order, one row per layer. Arrow keys move between heads and select as they go: left and right walk one layer, up and down change layer. Selected: layer 14, head 0.
Every plotted head with the numbers the miniatures are made of. Entropy is in nats over query rows 1 and up; the four shares are disjoint and add to 100.
Layer
Head
Pattern
Row entropy (nats)
Effective span (tokens)
self, the query's own key
previous, the key one position back
the first token
any other earlier key
0
0
previous-token
0.53
1.7
3%
85%
11%
1%
0
3
previous-token
0.54
1.7
3%
84%
12%
1%
0
7
mixed
1.38
4.0
10%
36%
19%
35%
0
11
self-attending
0.41
1.5
89%
1%
9%
1%
0
19
diffuse
1.74
5.7
22%
20%
18%
40%
0
26
attention sink
0.41
1.5
5%
13%
81%
1%
2
0
previous-token
0.51
1.7
3%
85%
11%
1%
2
3
mixed
1.39
4.0
10%
35%
19%
36%
2
7
attention sink
0.38
1.5
5%
13%
82%
0%
2
11
self-attending
0.37
1.5
89%
1%
9%
1%
2
19
diffuse
1.74
5.7
22%
21%
17%
40%
2
26
mixed
1.39
4.0
10%
35%
19%
36%
6
0
attention sink
0.40
1.5
5%
13%
81%
1%
6
3
previous-token
0.52
1.7
3%
85%
11%
1%
6
7
mixed
1.38
4.0
10%
35%
19%
36%
6
11
diffuse
1.74
5.7
22%
20%
18%
40%
6
19
induction-like
0.46
1.6
0%
18%
57%
25%
6
26
self-attending
0.39
1.5
89%
1%
9%
1%
14
0
induction-like
0.44
1.6
0%
18%
57%
25%
14
3
attention sink
0.40
1.5
5%
13%
81%
1%
14
7
diffuse
1.74
5.7
22%
20%
17%
41%
14
11
previous-token
0.51
1.7
3%
85%
11%
1%
14
19
mixed
1.38
4.0
10%
35%
19%
36%
14
26
induction-like
0.44
1.6
0%
18%
57%
25%
27
0
attention sink
0.38
1.5
5%
13%
82%
0%
27
3
diffuse
1.74
5.7
22%
20%
18%
40%
27
7
induction-like
0.46
1.6
0%
18%
57%
25%
27
11
attention sink
0.41
1.5
5%
13%
81%
1%
27
19
mixed
1.38
4.0
10%
35%
19%
36%
27
26
diffuse
1.74
5.7
21%
21%
17%
41%
41
0
attention sink
0.37
1.5
5%
13%
82%
0%
41
3
attention sink
0.40
1.5
5%
13%
81%
1%
41
7
diffuse
1.74
5.7
22%
20%
18%
40%
41
11
mixed
1.38
4.0
10%
35%
19%
36%
41
19
diffuse
1.74
5.7
22%
21%
17%
40%
41
26
self-attending
0.39
1.5
89%
1%
9%
1%
Layer 14, head 0: every weight of the enlarged map, query rows down, key columns across. A blank cell is a key the row cannot read.
checkpoint 1.42M steps · val loss 1.87 nats · attentions from one greedy pass
No attention maps captured yet
One miniature per (layer, head) appears here once a forward pass is captured with attention returned. Each map is a token-by-token softmax over the prompt, so the card needs the token strings too — they label both axes.
Zyeon UI