Scaling Law | Zyeon UI

status="ready" · 21 runs across four model shapes. The mock generates the losses from laws the card never sees — Dense was drawn with α = 0.145 and the fit recovers 0.149. MoE starts worst and ends best: the steepest exponent (0.160) and the lowest floor (1.605), which is why it projects lowest at 6e25 FLOPs. Hybrid SSM is the trap — cheapest at the bottom of the ladder, flattest curve, 433× compute to halve what is left. Deep-narrow has three runs, one short of a three-parameter fit, so its points are plotted with no line. Toggle a family: the shares, the frame and the verdicts recompute, and the shares are apportioned by largest remainder, so they total 100 in every one of the fifteen subsets.

Pretraining ladder · loss vs. training compute

One mixture, one held-out slice · compute is 6 · params · tokens

21 runs · 4 families · 3 fitted · 4.0 decades of compute

Families

Denseα 0.149
floor 1.692 · R² 0.9997 · 6 runs · 35% of compute
→ 1.846 at 6e25 FLOPs
MoE (sparse)α 0.160
floor 1.605 · R² 0.99994 · 6 runs · 45% of compute
→ 1.791 at 6e25 FLOPs
Hybrid SSMα 0.114
floor 1.674 · R² 0.99946 · 6 runs · 17% of compute
→ 1.884 at 6e25 FLOPs
Deep-narrownot fitted
3 runs · 4 needed for a three-parameter fit · 3% of compute

Every family is fitted on its own runs — least squares on log(L − E) against log C, with the floor E searched in one dimension. Pooling shapes would fit the mixture, not a law. MoE (sparse) has the steepest exponent, α 0.160: every 10× of compute takes 30.9% off its reducible loss, against 23.1% for Hybrid SSM. Halving the gap to the floor costs 75× compute for MoE (sparse) and 433× for Hybrid SSM. The lowest floor belongs to MoE (sparse) at 1.605 — the loss it cannot train away. Past 3e24 FLOPs nothing has been run: the hatched band is arithmetic. At 6e25 FLOPs — 20× the largest run here — MoE (sparse) projects lowest at 1.791, with the fits 0.093 apart. Deep-narrow carries fewer than 4 runs, so its points are plotted unfitted.

Pretraining ladder · loss vs. training compute: Held-out loss (nats/token) against Training compute (FLOPs), both logarithmic, over 21 runs from 3.1e20 to 3e24 FLOPs. Each family is fitted separately to L = E + A · C^(−alpha) by least squares on log(L − E), the floor E found by a one-dimensional search. Dense: alpha 0.149, irreducible loss 1.692, R squared 0.9997, from 6 runs, projecting 1.846 at 6e25 FLOPs. MoE (sparse): alpha 0.160, irreducible loss 1.605, R squared 0.99994, from 6 runs, projecting 1.791 at 6e25 FLOPs. Hybrid SSM: alpha 0.114, irreducible loss 1.674, R squared 0.99946, from 6 runs, projecting 1.884 at 6e25 FLOPs. Deep-narrow: 3 runs, fewer than the 4 a three-parameter law needs, so it is plotted unfitted. Every run follows in the data table. Up to 32 runs, evenly spaced by rank.
Run	Family	Compute (FLOPs)	Held-out loss (nats/token)	Fitted	Residual	Params	Tokens
hyb-1.6b	Hybrid SSM	3.1e20	2.5188	2.5169	+0.0019	1.6B	32B
dense-1.8b	Dense	3.9e20	2.6057	2.6051	+0.0006	1.8B	36B
moe-1.3b	MoE (sparse)	6.1e20	2.7887	2.7861	+0.0026	1.3B	78B
hyb-3.7b	Hybrid SSM	1.6e21	2.372	2.3701	+0.0019	3.7B	74B
dense-4.5b	Dense	2.4e21	2.3842	2.3869	−0.0027	4.5B	90B
moe-3.2b	MoE (sparse)	3.6e21	2.4905	2.4911	−0.0006	3.2B	190B
dn-6.5b	Deep-narrow	5.1e21	2.3948	not fitted	not fitted	6.5B	130B
hyb-8.7b	Hybrid SSM	8.9e21	2.245	2.2482	−0.0032	8.7B	170B
dense-11b	Dense	1.5e22	2.2274	2.2244	+0.003	11B	220B
moe-7.5b	MoE (sparse)	2e22	2.2749	2.2781	−0.0032	7.5B	450B
dn-16b	Deep-narrow	3.1e22	2.2416	not fitted	not fitted	16B	320B
hyb-20b	Hybrid SSM	4.8e22	2.1434	2.1476	−0.0042	20B	400B
dense-26b	Dense	8.1e22	2.1062	2.104	+0.0022	26B	520B
moe-17b	MoE (sparse)	1e23	2.1243	2.1243	0.000	17B	1T
dn-39b	Deep-narrow	1.8e23	2.1195	not fitted	not fitted	39B	780B
hyb-47b	Hybrid SSM	2.7e23	2.0692	2.0638	+0.0054	47B	940B
dense-61b	Dense	4.4e23	2.0075	2.0123	−0.0048	61B	1.2T
moe-41b	MoE (sparse)	6.1e23	1.9966	1.9942	+0.0024	41B	2.5T
hyb-96b	Hybrid SSM	1.1e24	2.0042	2.0056	−0.0014	96B	1.9T
dense-140b	Dense	2.4e24	1.9432	1.9414	+0.0018	140B	2.8T
moe-91b	MoE (sparse)	3e24	1.9058	1.9067	−0.0009	91B	5.5T

One ladder with no family on any run and no params or tokens either. The filter disappears (a filter with one option is furniture), the single series is still fitted — α 0.154 over a floor of 1.107, because code is more predictable than prose — and the data table says "not reported" rather than inventing a model size.

Code mixture ablation · loss vs. training compute

One ablation ladder, unlabelled runs · loss on the held-out code slice

6 runs · 1 fitted · 3.8 decades of compute

All runsα 0.154
floor 1.107 · R² 0.99994 · 6 runs · 100% of compute
→ 1.286 at 5e25 FLOPs

Every family is fitted on its own runs — least squares on log(L − E) against log C, with the floor E searched in one dimension. Pooling shapes would fit the mixture, not a law. The fit lands at α 0.154: every 10× of compute takes 29.8% off the reducible loss, and halving the remaining gap to the floor of 1.107 costs 90× compute. Past 4.5e24 FLOPs nothing has been run: the hatched band is arithmetic. At 5e25 FLOPs — 11× the largest run here — the fit projects 1.286.

Code mixture ablation · loss vs. training compute: Code loss (nats/token) against Training compute (FLOPs), both logarithmic, over 6 runs from 8e20 to 4.5e24 FLOPs. Each family is fitted separately to L = E + A · C^(−alpha) by least squares on log(L − E), the floor E found by a one-dimensional search. All runs: alpha 0.154, irreducible loss 1.107, R squared 0.99994, from 6 runs, projecting 1.286 at 5e25 FLOPs. Every run follows in the data table. Up to 32 runs, evenly spaced by rank.
Run	Family	Compute (FLOPs)	Code loss (nats/token)	Fitted	Residual	Params	Tokens
8e20	All runs	8e20	2.0863	2.0833	+0.003	not reported	not reported
5e21	All runs	5e21	1.8405	1.8434	−0.0029	not reported	not reported
3e22	All runs	3e22	1.6638	1.6659	−0.0021	not reported	not reported
1.6e23	All runs	1.6e23	1.5415	1.539	+0.0025	not reported	not reported
9e23	All runs	9e23	1.4381	1.4382	−0.0001	not reported	not reported
4.5e24	All runs	4.5e24	1.3653	1.3655	−0.0002	not reported	not reported

status="empty" · a valid contract with no runs at all, worded so it cannot be mistaken for a failed fetch: it names the four runs a three-parameter law needs before it can be fitted at all.

Pretraining ladder · loss vs. training compute

One mixture, one held-out slice · compute is 6 · params · tokens

Held-out loss (nats/token) against Training compute (FLOPs), both on log axes

No runs to fit

A scaling law is fitted from finished runs, not declared: send at least 4 of them — a compute budget and the loss it reached — and this card derives the exponent, the irreducible term and the projection itself. Three runs can only be interpolated by a three-parameter law.

status="loading" · a skeleton holding the filter row, the plot box and the legend tiles, with a falling ladder over its own floor — the silhouette of every scaling law. aria-hidden, honouring prefers-reduced-motion, no timer involved.

Pretraining ladder · loss vs. training compute

One mixture, one held-out slice · compute is 6 · params · tokens

Held-out loss (nats/token) against Training compute (FLOPs), both on log axes

status="error" · the run store failed. Try again only appears when onRetry is passed, and clicking it flips the card to ready.

Pretraining ladder · loss vs. training compute

One mixture, one held-out slice · compute is 6 · params · tokens

Held-out loss (nats/token) against Training compute (FLOPs), both on log axes

Couldn't load this scaling law

The run store didn't respond.