Scaling Law

Preview in your theme

status="ready" · 21 runs across four model shapes. The mock generates the losses from laws the card never sees — Dense was drawn with α = 0.145 and the fit recovers 0.149. MoE starts worst and ends best: the steepest exponent (0.160) and the lowest floor (1.605), which is why it projects lowest at 6e25 FLOPs. Hybrid SSM is the trap — cheapest at the bottom of the ladder, flattest curve, 433× compute to halve what is left. Deep-narrow has three runs, one short of a three-parameter fit, so its points are plotted with no line. Toggle a family: the shares, the frame and the verdicts recompute, and the shares are apportioned by largest remainder, so they total 100 in every one of the fifteen subsets.

Pretraining ladder · loss vs. training compute

One mixture, one held-out slice · compute is 6 · params · tokens

21 runs · 4 families · 3 fitted · 4.0 decades of compute

Families

Denseα 0.149
floor 1.692 · R² 0.9997 · 6 runs · 35% of compute
→ 1.846 at 6e25 FLOPs
MoE (sparse)α 0.160
floor 1.605 · R² 0.99994 · 6 runs · 45% of compute
→ 1.791 at 6e25 FLOPs
Hybrid SSMα 0.114
floor 1.674 · R² 0.99946 · 6 runs · 17% of compute
→ 1.884 at 6e25 FLOPs
Deep-narrownot fitted
3 runs · 4 needed for a three-parameter fit · 3% of compute

Every family is fitted on its own runs — least squares on log(L − E) against log C, with the floor E searched in one dimension. Pooling shapes would fit the mixture, not a law. MoE (sparse) has the steepest exponent, α 0.160: every 10× of compute takes 30.9% off its reducible loss, against 23.1% for Hybrid SSM. Halving the gap to the floor costs 75× compute for MoE (sparse) and 433× for Hybrid SSM. The lowest floor belongs to MoE (sparse) at 1.605 — the loss it cannot train away. Past 3e24 FLOPs nothing has been run: the hatched band is arithmetic. At 6e25 FLOPs — 20× the largest run here — MoE (sparse) projects lowest at 1.791, with the fits 0.093 apart. Deep-narrow carries fewer than 4 runs, so its points are plotted unfitted.

Pretraining ladder · loss vs. training compute: Held-out loss (nats/token) against Training compute (FLOPs), both logarithmic, over 21 runs from 3.1e20 to 3e24 FLOPs. Each family is fitted separately to L = E + A · C^(−alpha) by least squares on log(L − E), the floor E found by a one-dimensional search. Dense: alpha 0.149, irreducible loss 1.692, R squared 0.9997, from 6 runs, projecting 1.846 at 6e25 FLOPs. MoE (sparse): alpha 0.160, irreducible loss 1.605, R squared 0.99994, from 6 runs, projecting 1.791 at 6e25 FLOPs. Hybrid SSM: alpha 0.114, irreducible loss 1.674, R squared 0.99946, from 6 runs, projecting 1.884 at 6e25 FLOPs. Deep-narrow: 3 runs, fewer than the 4 a three-parameter law needs, so it is plotted unfitted. Every run follows in the data table. Up to 32 runs, evenly spaced by rank.
Run	Family	Compute (FLOPs)	Held-out loss (nats/token)	Fitted	Residual	Params	Tokens
hyb-1.6b	Hybrid SSM	3.1e20	2.5188	2.5169	+0.0019	1.6B	32B
dense-1.8b	Dense	3.9e20	2.6057	2.6051	+0.0006	1.8B	36B
moe-1.3b	MoE (sparse)	6.1e20	2.7887	2.7861	+0.0026	1.3B	78B
hyb-3.7b	Hybrid SSM	1.6e21	2.372	2.3701	+0.0019	3.7B	74B
dense-4.5b	Dense	2.4e21	2.3842	2.3869	−0.0027	4.5B	90B
moe-3.2b	MoE (sparse)	3.6e21	2.4905	2.4911	−0.0006	3.2B	190B
dn-6.5b	Deep-narrow	5.1e21	2.3948	not fitted	not fitted	6.5B	130B
hyb-8.7b	Hybrid SSM	8.9e21	2.245	2.2482	−0.0032	8.7B	170B
dense-11b	Dense	1.5e22	2.2274	2.2244	+0.003	11B	220B
moe-7.5b	MoE (sparse)	2e22	2.2749	2.2781	−0.0032	7.5B	450B
dn-16b	Deep-narrow	3.1e22	2.2416	not fitted	not fitted	16B	320B
hyb-20b	Hybrid SSM	4.8e22	2.1434	2.1476	−0.0042	20B	400B
dense-26b	Dense	8.1e22	2.1062	2.104	+0.0022	26B	520B
moe-17b	MoE (sparse)	1e23	2.1243	2.1243	0.000	17B	1T
dn-39b	Deep-narrow	1.8e23	2.1195	not fitted	not fitted	39B	780B
hyb-47b	Hybrid SSM	2.7e23	2.0692	2.0638	+0.0054	47B	940B
dense-61b	Dense	4.4e23	2.0075	2.0123	−0.0048	61B	1.2T
moe-41b	MoE (sparse)	6.1e23	1.9966	1.9942	+0.0024	41B	2.5T
hyb-96b	Hybrid SSM	1.1e24	2.0042	2.0056	−0.0014	96B	1.9T
dense-140b	Dense	2.4e24	1.9432	1.9414	+0.0018	140B	2.8T
moe-91b	MoE (sparse)	3e24	1.9058	1.9067	−0.0009	91B	5.5T

One ladder with no family on any run and no params or tokens either. The filter disappears (a filter with one option is furniture), the single series is still fitted — α 0.154 over a floor of 1.107, because code is more predictable than prose — and the data table says "not reported" rather than inventing a model size.

Code mixture ablation · loss vs. training compute

One ablation ladder, unlabelled runs · loss on the held-out code slice

6 runs · 1 fitted · 3.8 decades of compute

All runsα 0.154
floor 1.107 · R² 0.99994 · 6 runs · 100% of compute
→ 1.286 at 5e25 FLOPs

Every family is fitted on its own runs — least squares on log(L − E) against log C, with the floor E searched in one dimension. Pooling shapes would fit the mixture, not a law. The fit lands at α 0.154: every 10× of compute takes 29.8% off the reducible loss, and halving the remaining gap to the floor of 1.107 costs 90× compute. Past 4.5e24 FLOPs nothing has been run: the hatched band is arithmetic. At 5e25 FLOPs — 11× the largest run here — the fit projects 1.286.

Code mixture ablation · loss vs. training compute: Code loss (nats/token) against Training compute (FLOPs), both logarithmic, over 6 runs from 8e20 to 4.5e24 FLOPs. Each family is fitted separately to L = E + A · C^(−alpha) by least squares on log(L − E), the floor E found by a one-dimensional search. All runs: alpha 0.154, irreducible loss 1.107, R squared 0.99994, from 6 runs, projecting 1.286 at 5e25 FLOPs. Every run follows in the data table. Up to 32 runs, evenly spaced by rank.
Run	Family	Compute (FLOPs)	Code loss (nats/token)	Fitted	Residual	Params	Tokens
8e20	All runs	8e20	2.0863	2.0833	+0.003	not reported	not reported
5e21	All runs	5e21	1.8405	1.8434	−0.0029	not reported	not reported
3e22	All runs	3e22	1.6638	1.6659	−0.0021	not reported	not reported
1.6e23	All runs	1.6e23	1.5415	1.539	+0.0025	not reported	not reported
9e23	All runs	9e23	1.4381	1.4382	−0.0001	not reported	not reported
4.5e24	All runs	4.5e24	1.3653	1.3655	−0.0002	not reported	not reported

status="empty" · a valid contract with no runs at all, worded so it cannot be mistaken for a failed fetch: it names the four runs a three-parameter law needs before it can be fitted at all.

Pretraining ladder · loss vs. training compute

One mixture, one held-out slice · compute is 6 · params · tokens

Held-out loss (nats/token) against Training compute (FLOPs), both on log axes

No runs to fit

A scaling law is fitted from finished runs, not declared: send at least 4 of them — a compute budget and the loss it reached — and this card derives the exponent, the irreducible term and the projection itself. Three runs can only be interpolated by a three-parameter law.

status="loading" · a skeleton holding the filter row, the plot box and the legend tiles, with a falling ladder over its own floor — the silhouette of every scaling law. aria-hidden, honouring prefers-reduced-motion, no timer involved.

Pretraining ladder · loss vs. training compute

One mixture, one held-out slice · compute is 6 · params · tokens

Held-out loss (nats/token) against Training compute (FLOPs), both on log axes

status="error" · the run store failed. Try again only appears when onRetry is passed, and clicking it flips the card to ready.

Pretraining ladder · loss vs. training compute

One mixture, one held-out slice · compute is 6 · params · tokens

Held-out loss (nats/token) against Training compute (FLOPs), both on log axes

Couldn't load this scaling law

The run store didn't respond.

Loading preview…

"use client"

import * as React from "react"

import { cn } from "@/lib/utils"
import {
  CHART_SCALING_LAW_MIN_RUNS,
  type ChartScalingLawData,
  type ChartScalingLawRun,
} from "./chart-scaling-law.contract"

export interface ChartScalingLawProps
  extends Omit<React.HTMLAttributes<HTMLDivElement>, "title">,
    ChartScalingLawData {

Installation

npx shadcn@latest add https://ui.zyeon.ai/r/chart-scaling-law.json

Prompt

Build a React + TypeScript + Tailwind "ChartScalingLaw" research card in plain
SVG with zod. Not recharts: both axes are logarithmic, the curve on them is
fitted by the component rather than supplied, and the mark that matters most —
a hatched region past the last real run — is a statement about evidence, not a
series.

Contract
- One zod schema is the source of truth:
  { status: "loading" | "empty" | "error" | "ready"; title: string;
    caption?: string; computeLabel: string; lossLabel: string;
    runs: { compute: number > 0; loss: number > 0; family?: string;
            params?: number; tokens?: number; label?: string }[];
    extrapolateTo?: number > 0 }.
- Component props = z.infer of that schema plus height?: number,
  showFamilyFilter?: boolean, onRetry?: () => void and className. No
  hand-written parallel interface.
- Refinements, because these are contract errors and not render-time surprises:
  a ready payload needs at least four runs (L = E + A·C^(−α) has three free
  parameters, so three points interpolate it exactly), and extrapolateTo must
  lie beyond the largest run — the hatched band is an extrapolation, not a
  redraw of the fitted range.
- compute is total training FLOPs (6 · params · tokens, counting active
  parameters for a sparse model); loss is whatever lossLabel names. params and
  tokens are reported in the tooltip and the data table, never fitted: this is
  loss against compute, not the two-variable L(N, D) surface.

Behavior
- Fit, do not accept a fit. For each family: at a candidate floor E the law
  linearises to ln(L − E) = ln A − α·ln C, so A and α come from closed-form
  least squares; only E is searched — a grid over [0, 0.98 · min(loss)) and
  then golden section inside the bracketing cell. The 0.98 ceiling is real:
  as E approaches the best run's loss its residual collapses and the objective
  starts describing the parameterisation instead of the data. Least squares
  runs in log space, which weights a 1% miss at the bottom of the ladder like a
  1% miss at the top. Report R² of that log-space fit, and NaN → "—" when the
  residual is constant.
- Fit each family separately and never pool them: a mixture of model shapes has
  no single exponent, and the whole reason to draw several on one pair of axes
  is to compare their α. A family shorter than the four-run minimum is plotted
  as points with no line and says why.
- Runs with no family collect into one implicit series; when NO run declares a
  family, the series is "All runs" and the filter disappears — a filter with
  one option is furniture.
- Extrapolation is drawn twice over, because two different things are being
  admitted: the hatch starts at the largest run ON THE CARD, while each
  family's line goes dashed at ITS OWN last run. A projection marker sits at
  extrapolateTo as a hollow ring — a filled dot is a run somebody paid for.
  Refuse to project a fit whose exponent came out ≤ 0 and say so: loss rising
  with compute is a broken ladder, not a scaling law.
- Filter: one aria-pressed toggle per family in a role="group", all pressed to
  start. Hiding the last visible family is refused rather than obeyed (that
  button carries aria-disabled and an sr-only hint). Everything downstream
  recomputes from the visible set — the frame, the hatch's left edge, the
  compute shares and every printed verdict.
- Compute shares are apportioned by largest remainder over the visible
  families, so they total exactly 100 in every subset; rounding each share on
  its own is how three families print 97 under one filter state and 101 under
  another.
- Derived readouts, all computed and none hard-coded: percentage of reducible
  loss bought per 10× of compute (1 − 10^(−α)), the compute multiplier that
  halves the remaining gap to the floor (2^(1/α)), the projected loss at the
  target, the spread between projections, and a caveat for any family fitted
  over less than one decade of compute.
- Four first-class branches: loading (skeleton with a falling ladder over its
  own floor, aria-hidden, animate-pulse + motion-reduce:animate-none), empty
  (names the four-run minimum instead of implying a failed fetch, and reports
  runs dropped for a non-positive compute or loss), error (role="alert" plus a
  Try again button only when onRetry exists), ready.
- A ResizeObserver measures the plot box and is disconnected on unmount and on
  node change; it is the only subscription in the component.

Rendering & styling
- Axes: decade gridlines on x labelled "1e22", thinned by a stride so labels
  never collide; 2× and 5× minor rules only while a decade is wide enough. On y
  the ticks come from a mantissa ladder — the finest ladder whose ticks all
  clear a minimum pixel gap wins, because a loss axis spans a fraction of a
  decade and plain decade ticks would print two labels.
- The frame opens far enough to show a fitted floor sitting just under the runs
  (that asymptote is the point of the chart) but not one half a decade below,
  which would squash every run into a stripe.
- Colours come only from tokens: family i takes var(--chart-{(i % 5) + 1}) for
  its points, its law, its floor rule, its projection ring and its legend
  swatch — one formula, five consumers — plus a dash pattern per family so the
  card survives greyscale. Panel is rounded-xl border bg-card; the hatch is
  fill-muted with stroke-muted-foreground lines, since a low-alpha wash alone
  disappears on a near-black card.
- role="img" on the plot with an aria-label and aria-describedby; the only
  controls are the family pills above it, and every number is repeated in an
  sr-only table of runs (rank-sampled to a cap) with the fitted value and the
  residual beside each measured loss.
- cn() merges className, numbers are tabular-nums, and any expression followed
  by a unit is separated with an explicit {" "} — JSX deletes a newline next to
  an expression rather than folding it to a space.

Customization levers
- Plot density: height (200–520) and the y-axis mantissa ladders; drop the
  minor 2×/5× rules for a cleaner frame, or add 3× and 7× for log-paper feel.
- Which blocks to keep: the family filter (showFamilyFilter), the legend's
  projection row, the floor rules, the verdict paragraph — each is independent
  of the plot.
- The fit: FLOOR_STEPS / GOLDEN_STEPS trade search cost for precision; pin the
  floor to a known entropy estimate by skipping the search entirely; or swap
  the objective for Huber in log space, which is what the Chinchilla paper
  does, when a ladder carries a diverged run.
- Minimum ladder length: CHART_SCALING_LAW_MIN_RUNS is exported — raise it to
  six if your reviewers refuse fits on four points.
- The projection: extrapolateTo drives the hatch, the dashed extensions and the
  rings; omit it and the card draws only the measured range. Swap the "halve
  the gap" multiplier for "compute needed to reach loss X" by inverting the law
  — C = ((L − E) / A)^(−1/α).
- Palette: the colour formula cycles five chart tokens; map a fixed token per
  family (brand colours for shipped architectures) by replacing inkFor.

Concepts

The fit is the deliverable — the card is handed (compute, loss) pairs and nothing else, so the exponent it prints can be checked against the runs sitting next to it. A pre-fitted line passed in as props is a claim the reader has to take on faith; here the line, the exponent and the residual column all come out of the same four lines of algebra.
Three parameters, one of them awkward — A and α are linear in log space once the floor E is fixed, so only E needs searching. That asymmetry is why the component grids E first and refines by golden section: the objective has a wall at E → min(loss) where the best run's residual collapses, and any plain descent walks straight into it.
The irreducible term is the honest half — E is the entropy the data has and the model cannot remove. Without it a power law promises zero loss at infinite compute, and every conversation about "just scale it" quietly assumes that. Drawing E as a rule the runs approach and never cross turns the argument into a number.
Per-family fits, never pooled — different model shapes have different exponents; averaging them produces a law describing a mixture nobody trained. Each family gets its own colour, its own dash, its own α, and the comparison the axes were drawn for.
Extrapolation is admitted twice — the hatched band says "past anything on this card", the dashed line says "past this family's own last run". They are different admissions, they start at different places, and a chart that draws only one of them is over-claiming for the family that stopped early.
Every subset totals 100 — compute shares are apportioned by largest remainder over the visible families, so hiding one re-apportions the rest instead of leaving 97 or 101 on screen. Rounding per slice is a defect that survives review because it only shows up in the filter states nobody screenshots.

Pretraining ladder · loss vs. training compute

Code mixture ablation · loss vs. training compute

Pretraining ladder · loss vs. training compute

Pretraining ladder · loss vs. training compute

Pretraining ladder · loss vs. training compute

Installation

Prompt

Concepts

On This Page

Pretraining ladder · loss vs. training compute

Code mixture ablation · loss vs. training compute

Pretraining ladder · loss vs. training compute

Pretraining ladder · loss vs. training compute

Pretraining ladder · loss vs. training compute