Prompt Arena

npx shadcn@latest add https://ui.zyeon.ai/r/prompt-arena.json

Build a React + TypeScript + Tailwind "PromptArena" component with zod and
lucide-react. It reports a PAIRWISE prompt test: prompt A and prompt B answered
the same inputs, a judge said which answer it preferred, and this card turns the
pile of verdicts into a decision.

Contract
- A zod schema (`promptArenaRunSchema` in a sibling contract file) is the single
  source of truth for one run: { id, promptA, promptB, matchups[], totals,
  dataset?, judge?, completedAt?, note? }.
- A side is { label, text?, model? }. A matchup is { id, input, winner?: "a" |
  "b" | "tie", votes?: { a, b, tie }, outputA?, outputB?, note?, tag?, error? }.
  Totals are { a, b, tie, runs? } — counts, never fractions.
- TOTALS ARE THE TRUTH, MATCHUPS ARE A SAMPLE. `a + b + tie` is the number of
  judgements behind the scoreboard; `matchups` is the handful of examples the
  report happened to carry. Never recompute the scoreboard from the visible rows,
  and always say how many rows are being drawn. Eight anecdotes are not a result.
- `runs` (attempted) may exceed `a + b + tie` (judged). The difference is
  rendered as "N without a verdict" rather than being folded into either side,
  and it is clamped so a bad report can never produce a negative count.
- `winner` is OPTIONAL: a matchup still running, or one whose judge crashed, is
  "not judged" — never a tie, never a zero. When `winner` is absent but `votes`
  are present, derive it from a STRICT majority; an even split between the two
  sides is a tie, and screen readers are told the verdict was derived.
- A separate envelope status — "loading" | "empty" | "error" | "ready" — is the
  card's own render state, independent of the verdict. `status="error"` means the
  report failed to load; a matchup failing is `matchup.error`, and the two read
  differently on screen.
- Props: run: Run | null; status; order? ("given" | "regressions" | "contested");
  maxMatchups? (8); excerptChars? (320); confidence? (0.9 | 0.95 | 0.99, default
  0.95); minDecisiveRuns? (30); defaultOpenMatchupId?; defaultShowPrompts?;
  onOpenMatchupChange?(id | null); onRerun?(runId); actions?: ReactNode;
  onRetry?; density? ("comfortable" | "compact"); formatPercent?; formatDate?;
  emptyState?; errorMessage?; label? ("Prompt arena"); className plus the rest of
  the element props, ref forwarded to the <section>.
- Every numeric prop is clamped (maxMatchups >= 1, excerptChars >= 40,
  minDecisiveRuns >= 1) and an unsupported confidence level falls back to 0.95,
  so a 0 or a NaN can never empty the list or void the verdict.

Behavior
- A LEAD IS NOT A WIN. Compute the head-to-head rate as B / (A + B) — ties are
  excluded from the denominator, because folding "both fine" into either side
  inflates whichever prompt is more agreeable — then put a Wilson score interval
  around it at the chosen confidence level. The verdict is a position, not a
  comparison of two numbers: interval entirely above 50% names B, entirely below
  names A, and an interval that still contains 50% reads "Too close to call".
  Use Wilson, not the naive p ± z·sqrt(p(1-p)/n): the naive interval returns
  bounds outside [0,1] and collapses to zero width on a clean sweep, so a 5–0
  run would read "100% ± 0" — exactly the run you must not ship on.
- Below `minDecisiveRuns` decisive judgements the card refuses to name anybody at
  all, however lopsided the split is, and states the shortfall ("only 7 decisive
  judgements of the 30 needed"). Zero decisive judgements with ties on the board
  says so in the same sentence: every verdict so far was a tie.
- The confidence sentence always names its denominator. "58%" over 12 judgements
  and over 1,200 judgements are different claims, and this card is the only place
  the reader can see which one they are looking at. The interval is computed once,
  on B; A's reading is its mirror image (1 − high, 1 − low), so the two sides can
  never disagree with each other.
- Row order is a lever: "given" keeps the report's order, "regressions" puts the
  inputs the CANDIDATE LOST first (then ties, then wins, then unjudged — the only
  list anyone reads before reverting), "contested" floats the closest vote splits
  up and sinks rows with a single judgement, which have no contest to measure.
- Rows never truncate silently: `maxMatchups` is applied after ordering and the
  footer says "Showing 3 of 8 sampled matchups" next to the run's own judgement
  count. A row with no outputs, no note and no error is not a button — an
  expander that expands nothing is a keyboard stop that wastes a Tab.
- ONE PAIR OPEN AT A TIME. Opening a matchup closes the previous one: comparing
  two long answers is the whole job, and a stack of open pairs turns the card into
  a wall of prose. The open panel carries the full input, the two outputs side by
  side (an auto-fit grid that drops to one column as soon as a column would fall
  under 18rem, so neither answer is ever squeezed into a ribbon), the judge's note
  and the matchup error; the winning side's pane is marked with the word
  "preferred", not with a glow.
- Long outputs are excerpted with an EXPLICIT count and a reveal control
  ("Showing 320 of 1,504 characters" + "Show all"), and Copy always writes the
  full text, never the visible slice. A silently clipped answer is how you
  conclude that the new prompt "stops early". The matchup error is the one thing
  never excerpted.
- RERUN IS ONE-SHOT. `onRerun` fires at most once per run id no matter how fast
  the button is clicked, because the lock lives in a ref read and written
  synchronously inside the click handler: several clicks dispatched in one task
  all read the same stale state, so a state-only guard would re-judge a paid
  batch twice. The lock expires by itself — a generation counter ticks whenever a
  new run id arrives, so the next run is rerunnable while a duplicate click is
  not. The button uses aria-disabled, never the native `disabled` attribute,
  which would blur it mid-action and drop focus to <body>.
- Identity changes reset the card (adjust state during render, no effect): a new
  run id restores both disclosures to their defaults — `defaultOpenMatchupId` and
  `defaultShowPrompts` — and re-arms the rerun lock, so a recycled card can never
  show the previous run's pair or refuse a legitimate rerun.
- A single judgement never draws a distribution bar. `votes` totalling 1 is
  reported as "one judgement" and the row's verdict chip carries the reading; a
  100%-wide bar off one sample is a lie with a nice gradient.
- The card owns no clock, no fetch and no arithmetic on your behalf beyond the
  interval: `completedAt` is formatted in a fixed locale and UTC so the server and
  the browser render the same string.

Rendering & styling
- Semantic tokens only: bg-card / border / text-muted-foreground for chrome,
  var(--chart-1) for side A, var(--chart-3) for side B (a blue/amber pair that
  survives colour blindness), bg-muted-foreground/40 for the tie band, and
  text-destructive with bg-destructive/5 for failures. No hard-coded colours.
- THE SIDE COLOURS PAINT FILLS AND BORDERS, NEVER TEXT: the bar bands, the side
  dots, the winning pane's tint, and the outline of the verdict chip / matchup
  mark / "preferred" badge. Every label in those elements stays on
  text-foreground. Chart tokens are calibrated for a 3:1 fill contrast against
  the card, which leaves --chart-1 at 3.63:1 on a light card — an 11px label
  painted in it misses the 4.5:1 AA text threshold. The letter, the word and the
  shape carry the verdict anyway; the hue is the second channel, not the one
  anybody has to read.
- The split bar is A | tie | B with a parity tick drawn at 50% of the WHOLE bar:
  a side whose band alone crosses the tick holds an outright majority of every
  judgement. Position carries the reading, so it survives greyscale and a
  screenshot. A run with nothing judged draws a dashed empty track — geometry
  that says "no measurement", where three zero-width bands would read as a bug.
- Every verdict is a letter plus a word plus a shape (A / B / equals sign /
  dashed circle), each with an sr-only phrase, so nothing depends on colour
  alone. Bars grow from scaleX(0) with a `from`-only keyframe, which means
  motion-reduce lands on the finished geometry instead of on an empty track;
  every transition is motion-reduce-guarded.
- The root is a <section aria-label="Prompt arena: A-label versus B-label">
  holding ONE persistent sr-only role="status" line with the headline and the
  confidence sentence, so a verdict CHANGE after a rerun is announced from an
  element the screen reader is already watching. Disclosures are real buttons
  with aria-expanded, their panels are UNMOUNTED when closed (a zero-height
  collapse leaves every copy button reachable by Tab), and aria-controls is only
  set while open.
- Long inputs use line-clamp-2 in the row and are shown WHOLE in the panel; every
  free-text node pairs `min-w-0` with `wrap-anywhere`, not `break-words`, since
  only overflow-wrap: anywhere lowers the min-content width that would otherwise
  push the card wider than its container.
- cn() merges the consumer's className into the root, which also spreads the
  remaining element props and forwards its ref.

Customization levers
- Statistics: `confidence` picks the band (0.9 / 0.95 / 0.99) and
  `minDecisiveRuns` is your ship gate — raise it for a decision that costs money,
  lower it for an exploratory sweep. Both change the WORDING and the verdict, not
  the geometry, so the bar stays comparable across cards.
- Reading order: `order="regressions"` for a revert review, `"contested"` for
  picking inputs to send to a human, `"given"` when your harness already ranked
  them.
- Budget: `maxMatchups` and `excerptChars` decide how much prose the card carries
  before the explicit reveal controls. Raise them in a full report page, lower
  them in a PR comment — but never make either silent; the count is part of the
  reading.
- Density and chrome: `density="compact"` for a row inside a bigger report,
  `defaultShowPrompts` to lead with the two prompt texts, `defaultOpenMatchupId`
  to deep-link a specific pair, `label` to rename the region.
- Actions: `onRerun` renders the one-shot "Run again" button, and `actions`
  is a free slot for your own header buttons ("Promote B", "Export CSV", "Open in
  harness"). Both are yours to wire — the card never mutates the run.
- Wording: `formatPercent` swaps percentages for raw fractions or odds,
  `formatDate` for local time; `emptyState`, `errorMessage` + `onRetry` own the
  envelope branches, and `run.note` is a free line from your harness under the
  verdict (judge version, position-swap policy, dataset caveat).