Tokenizer Preview

npx shadcn@latest add https://ui.zyeon.ai/r/tokenizer-preview.json

Build a React + TypeScript + Tailwind "TokenizerPreview" component: a panel that
shows how a piece of text splits into tokens, what that costs, and where the
expensive parts are. Ship a deterministic built-in splitter (no vocabulary
download) and let the caller inject a real tokenizer.

Contract
- forwardRef<HTMLDivElement>, extends React.HTMLAttributes<HTMLDivElement> minus
  children / defaultValue / onChange / onRateChange; the rest spreads on the
  root. The last two collide with DOM members of the same name — `onRateChange`
  is a media event on every element — so they have to be dropped for ours to win.
- value?: string, defaultValue?: string, onValueChange?(value): controlled or
  uncontrolled text.
- mode?: "edit" | "read" (default "edit"). "edit" puts a textarea above the
  split; "read" renders a given string with no field.
- tokenize?: (source: string) => TokenChunk[], default the built-in splitter.
  TokenChunk = { text: string; start: number; end: number; id?: number } —
  OFFSETS are the contract, `text` is informational.
- rate?: number / defaultRate?: number (default 3) — price per MILLION tokens;
  onRateChange?(rate) fires whenever the box parses; currency?: string ("USD").
- showCost?: boolean (default true), showIds?: boolean (default false).
- longTokenChars?: number (default 8) — the "unusually long" flag.
- maxRenderedTokens?: number (default 800) — a draw ceiling, never a count cap.
- label / hint / placeholder / rows, plus textareaRef for focus management.

Behavior — the built-in splitter (deterministic, vocabulary-free)
- Stage 1, pre-tokens: one regex whose alternatives cover every character, so
  the pass is total — contractions ('s 't 're 've 'm 'll 'd), then " ?\p{L}+",
  " ?\p{N}+", " ?[^\s\p{L}\p{N}]+", "\s+(?!\S)", "\s+". A leading space is
  glued to the run AFTER it: " tokenizer" bills as " token" + "izer", never as
  " " + "token" + "izer". Letters, digits, symbols and whitespace never mix.
- Stage 2, merge-ish: a run of <= 8 characters is assumed to be in a real
  vocabulary and stays whole. Longer runs split at the seams a merge table
  would have learned: camelCase boundaries, then the longest matching suffix
  (ization / ative / ment / izer / ing / ed / ly …), then the longest matching
  prefix (counter / inter / under / re / un …), each only if >= 3 characters
  remain. What survives all of that is sliced into 5-char pieces, and a 1–2
  char tail is glued back onto its neighbour.
- Digits go in groups of three, left to right. Code points outside the scripts
  BPE merges into words (CJK, Hangul, kana, astral emoji) cost one token per
  code point — iterate with for…of so a surrogate pair is never cut in half.
  Whitespace splits per repeated character and caps at four.
- Build a fresh scanner per call: a module-level /g regex keeps `lastIndex`
  between calls and would skip matches on the second render. Guard against a
  zero-width match advancing nothing.
- Say out loud in the UI that it is an approximation ("approximate split —
  counts are estimates") and flip that line to "custom tokenizer" when
  `tokenize` is not the default.

Behavior — the lossless cover (the invariant that makes injection safe)
- Never render the strings a tokenizer hands back. Sort the chunks by offset,
  clip anything that overlaps its predecessor, drop zero-width chunks, and
  re-slice every run out of the SOURCE.
- Gaps become visible "unclaimed" runs (destructive tint, wavy underline), are
  excluded from the token count, and add one muted line: "N characters are not
  covered by any token". A third-party tokenizer may be wrong; the text on
  screen must still be the text the user typed.

Behavior — reading the split
- Tint tokens by index modulo 5 over --chart-1..5 at ~16%, plus a 1px inset
  hairline of the same tone: hue is a convenience, the BOUNDARY is carried by
  geometry so the split survives both themes and colour vision deficiency.
- Whitespace-only tokens print stand-in glyphs (· for space, → for tab, ⏎ for
  newline, and the real newline after it so the flow still breaks) and carry an
  aria-label like "4 spaces" — they are billed and otherwise invisible.
- Tokens of longTokenChars or more get a dashed underline: one token swallowing
  many characters is the density outlier worth noticing.
- In edit mode the LAST token is provisional whenever the text does not end on
  whitespace — the next keystroke can merge into it and renumber the rest. Mark
  it with a dashed bottom border whose colour blinks between currentColor and
  transparent, and turn the blink off under prefers-reduced-motion (the dashed
  border stays, so the meaning survives without the motion).
- Counts: tokens, characters as CODE POINTS (Array.from(text).length, not
  .length), UTF-8 bytes, words (one per CJK glyph, one per Latin run), and
  density = chars / token, which is the number that actually explains a bill.
- Cost = tokens * rate / 1e6. Fraction digits follow the magnitude (2 above $1,
  4 above $0.01, 6 below) so a real cost is never rounded down to "$0.00", and
  an unknown currency code falls back to "12.3456 XYZ" instead of throwing.

Behavior — the rate box and the inspector
- The rate input holds TEXT, not a number: "" and "3." are legal mid-typing
  states. Parse the WHOLE trimmed string on every change (a decimal with an
  optional exponent — never parseFloat, which reads "3.5abc" as 3.5), call
  onRateChange only when the parse is a finite number >= 0, show "—" for the
  cost while it is not, and set aria-invalid only when the box is non-empty AND
  unparseable.
- A controlled `rate` rewrites the box only when the prop disagrees with what
  the box already parses to — otherwise a parent echoing onRateChange back eats
  the decimal point mid-keystroke.
- The split is a role="listbox" of role="option" spans with a roving tabindex:
  exactly one option is tabbable (the selected one, or the first), so the whole
  split costs one Tab. Arrow keys / Home / End move focus, focus moves the
  selection (one source of truth), Escape clears it, and hover previews without
  stealing it. Memoise the rendered option spans on the selection, not on the
  hover position, so a pointer sweep re-renders only the inspector line.
- One line below shows index, the token as JSON.stringify (so whitespace and
  quotes are visible), char count, byte count, the vocabulary id when there is
  one, and the whitespace / long / provisional flags.
- Clamp the selection DURING RENDER against the current token list — never in
  an effect — so an edit that shortens the split cannot leave a frame pointing
  at a token that is gone.
- Empty text renders a dashed placeholder box, zero counts and a zero cost. No
  invented numbers.
- Beyond maxRenderedTokens, draw the first N runs and add "showing the first N;
  the counts above cover all M". Counts always describe the whole text.

Rendering & styling
- Semantic tokens only: bg-card / text-card-foreground for the panel, border,
  bg-muted/30 for the split surface, text-muted-foreground for glyphs and
  legends, text-destructive + bg-destructive/10 for unclaimed runs, ring for
  focus, --chart-1..5 for the tints. No hex, no rgb(), no oklch().
- Tints are inline color-mix(in oklab, var(--chart-N) 16%, transparent) so the
  same markup works on a card, a muted panel or a dialog.
- The split surface is font-mono, whitespace-pre-wrap, wrap-anywhere; token
  spans use box-decoration-clone so a token that wraps keeps its full tint on
  both lines.
- The textarea is a real <textarea> (never contenteditable): IME composition,
  spellcheck, undo and every caret shortcut keep working. field-sizing-content
  with min/max height grows it with zero JS where supported, `rows` is the
  fallback everywhere else.
- Merge every className through cn(); the panel is the only element that takes
  the consumer's className.

Customization levers
- Splitter constants: KEEP_WHOLE (8) is how long a run may be before it is
  suspected of being out of vocabulary — raise it to model a bigger vocabulary,
  lower it for a character-level model. FALLBACK_PIECE (5) is the slice width
  for hashes and base64. The affix lists are plain arrays: add your domain's
  morphology (chemistry, legal, an inflected language) and it improves.
- longTokenChars: 8 suits the built-in approximation; raise it to 12 when you
  inject a real BPE, whose common tokens are longer.
- Palette: swap the 5-tone cycle for 2 tones if you only want an A/B stripe, or
  drive the tint from the token id instead of its index to make identical
  tokens share a colour across the text.
- Sub-blocks are independent: drop the cost line (showCost={false}) for a pure
  counter, drop the legend, or hide the inspector line if the surrounding page
  already explains the markers. Adding a context-window bar means comparing
  tokenCount against a limit — a different component's job.
- Density knobs: text-[13px] / leading-7 on the split surface and the panel's
  gap-3 are the only spacing levers; keep leading generous or wrapped tints
  collide.
- Performance: the whole text is re-tokenized on every keystroke (O(n), fine to
  tens of thousands of characters). Above that, debounce the value before it
  reaches the component, or lower maxRenderedTokens — the counts stay exact
  either way.