Zyeon UI
AI

Tokenizer Preview

A tokenizer playground for input text — alternating-tinted token chunks from a deterministic built-in splitter or your own, token/char/word counts, a live cost estimate at an editable per-Mtok rate, and markers for whitespace and unusually long tokens.

Preview in your theme

Loading preview…

"use client"

import * as React from "react"
import { TriangleAlert } from "lucide-react"
import { cn } from "@/lib/utils"
import { Input } from "@/components/ui/input"
import { Label } from "@/components/ui/label"

/* -------------------------------------------------------------------------- *
 * Token model
 *
 * A token is an OFFSET RANGE over the source, never a detached string. Offsets
 * are what makes the render lossless: the component slices every run out of the
 * source itself, so a tokenizer that lies about `text` — or leaves a hole, or

Installation

npx shadcn@latest add https://ui.zyeon.ai/r/tokenizer-preview.json

Prompt

The prompt behind this component — paste it into your AI assistant to recreate or adapt it.

Build a React + TypeScript + Tailwind "TokenizerPreview" component: a panel that
shows how a piece of text splits into tokens, what that costs, and where the
expensive parts are. Ship a deterministic built-in splitter (no vocabulary
download) and let the caller inject a real tokenizer.

Contract
- forwardRef<HTMLDivElement>, extends React.HTMLAttributes<HTMLDivElement> minus
  children / defaultValue / onChange / onRateChange; the rest spreads on the
  root. The last two collide with DOM members of the same name — `onRateChange`
  is a media event on every element — so they have to be dropped for ours to win.
- value?: string, defaultValue?: string, onValueChange?(value): controlled or
  uncontrolled text.
- mode?: "edit" | "read" (default "edit"). "edit" puts a textarea above the
  split; "read" renders a given string with no field.
- tokenize?: (source: string) => TokenChunk[], default the built-in splitter.
  TokenChunk = { text: string; start: number; end: number; id?: number } —
  OFFSETS are the contract, `text` is informational.
- rate?: number / defaultRate?: number (default 3) — price per MILLION tokens;
  onRateChange?(rate) fires whenever the box parses; currency?: string ("USD").
- showCost?: boolean (default true), showIds?: boolean (default false).
- longTokenChars?: number (default 8) — the "unusually long" flag.
- maxRenderedTokens?: number (default 800) — a draw ceiling, never a count cap.
- label / hint / placeholder / rows, plus textareaRef for focus management.

Behavior — the built-in splitter (deterministic, vocabulary-free)
- Stage 1, pre-tokens: one regex whose alternatives cover every character, so
  the pass is total — contractions ('s 't 're 've 'm 'll 'd), then " ?\p{L}+",
  " ?\p{N}+", " ?[^\s\p{L}\p{N}]+", "\s+(?!\S)", "\s+". A leading space is
  glued to the run AFTER it: " tokenizer" bills as " token" + "izer", never as
  " " + "token" + "izer". Letters, digits, symbols and whitespace never mix.
- Stage 2, merge-ish: a run of <= 8 characters is assumed to be in a real
  vocabulary and stays whole. Longer runs split at the seams a merge table
  would have learned: camelCase boundaries, then the longest matching suffix
  (ization / ative / ment / izer / ing / ed / ly …), then the longest matching
  prefix (counter / inter / under / re / un …), each only if >= 3 characters
  remain. What survives all of that is sliced into 5-char pieces, and a 1–2
  char tail is glued back onto its neighbour.
- Digits go in groups of three, left to right. Code points outside the scripts
  BPE merges into words (CJK, Hangul, kana, astral emoji) cost one token per
  code point — iterate with for…of so a surrogate pair is never cut in half.
  Whitespace splits per repeated character and caps at four.
- Build a fresh scanner per call: a module-level /g regex keeps `lastIndex`
  between calls and would skip matches on the second render. Guard against a
  zero-width match advancing nothing.
- Say out loud in the UI that it is an approximation ("approximate split —
  counts are estimates") and flip that line to "custom tokenizer" when
  `tokenize` is not the default.

Behavior — the lossless cover (the invariant that makes injection safe)
- Never render the strings a tokenizer hands back. Sort the chunks by offset,
  clip anything that overlaps its predecessor, drop zero-width chunks, and
  re-slice every run out of the SOURCE.
- Gaps become visible "unclaimed" runs (destructive tint, wavy underline), are
  excluded from the token count, and add one muted line: "N characters are not
  covered by any token". A third-party tokenizer may be wrong; the text on
  screen must still be the text the user typed.

Behavior — reading the split
- Tint tokens by index modulo 5 over --chart-1..5 at ~16%, plus a 1px inset
  hairline of the same tone: hue is a convenience, the BOUNDARY is carried by
  geometry so the split survives both themes and colour vision deficiency.
- Whitespace-only tokens print stand-in glyphs (· for space, → for tab, ⏎ for
  newline, and the real newline after it so the flow still breaks) and carry an
  aria-label like "4 spaces" — they are billed and otherwise invisible.
- Tokens of longTokenChars or more get a dashed underline: one token swallowing
  many characters is the density outlier worth noticing.
- In edit mode the LAST token is provisional whenever the text does not end on
  whitespace — the next keystroke can merge into it and renumber the rest. Mark
  it with a dashed bottom border whose colour blinks between currentColor and
  transparent, and turn the blink off under prefers-reduced-motion (the dashed
  border stays, so the meaning survives without the motion).
- Counts: tokens, characters as CODE POINTS (Array.from(text).length, not
  .length), UTF-8 bytes, words (one per CJK glyph, one per Latin run), and
  density = chars / token, which is the number that actually explains a bill.
- Cost = tokens * rate / 1e6. Fraction digits follow the magnitude (2 above $1,
  4 above $0.01, 6 below) so a real cost is never rounded down to "$0.00", and
  an unknown currency code falls back to "12.3456 XYZ" instead of throwing.

Behavior — the rate box and the inspector
- The rate input holds TEXT, not a number: "" and "3." are legal mid-typing
  states. Parse on every change, call onRateChange only when the parse is a
  finite number >= 0, show "—" for the cost while it is not, and set
  aria-invalid only when the box is non-empty AND unparseable.
- A controlled `rate` rewrites the box only when the prop disagrees with what
  the box already parses to — otherwise a parent echoing onRateChange back eats
  the decimal point mid-keystroke.
- The split is a role="listbox" of role="option" spans with a roving tabindex:
  exactly one option is tabbable (the selected one, or the first), so the whole
  split costs one Tab. Arrow keys / Home / End move focus, focus moves the
  selection (one source of truth), Escape clears it, and hover previews without
  stealing it.
- One line below shows index, the token as JSON.stringify (so whitespace and
  quotes are visible), char count, byte count, the vocabulary id when there is
  one, and the whitespace / long / provisional flags.
- Clamp the selection DURING RENDER against the current token list — never in
  an effect — so an edit that shortens the split cannot leave a frame pointing
  at a token that is gone.
- Empty text renders a dashed placeholder box, zero counts and a zero cost. No
  invented numbers.
- Beyond maxRenderedTokens, draw the first N runs and add "showing the first N;
  the counts above cover all M". Counts always describe the whole text.

Rendering & styling
- Semantic tokens only: bg-card / text-card-foreground for the panel, border,
  bg-muted/30 for the split surface, text-muted-foreground for glyphs and
  legends, text-destructive + bg-destructive/10 for unclaimed runs, ring for
  focus, --chart-1..5 for the tints. No hex, no rgb(), no oklch().
- Tints are inline color-mix(in oklab, var(--chart-N) 16%, transparent) so the
  same markup works on a card, a muted panel or a dialog.
- The split surface is font-mono, whitespace-pre-wrap, wrap-anywhere; token
  spans use box-decoration-clone so a token that wraps keeps its full tint on
  both lines.
- The textarea is a real <textarea> (never contenteditable): IME composition,
  spellcheck, undo and every caret shortcut keep working. field-sizing-content
  with min/max height grows it with zero JS where supported, `rows` is the
  fallback everywhere else.
- Merge every className through cn(); the panel is the only element that takes
  the consumer's className.

Customization levers
- Splitter constants: KEEP_WHOLE (8) is how long a run may be before it is
  suspected of being out of vocabulary — raise it to model a bigger vocabulary,
  lower it for a character-level model. FALLBACK_PIECE (5) is the slice width
  for hashes and base64. The affix lists are plain arrays: add your domain's
  morphology (chemistry, legal, an inflected language) and it improves.
- longTokenChars: 8 suits the built-in approximation; raise it to 12 when you
  inject a real BPE, whose common tokens are longer.
- Palette: swap the 5-tone cycle for 2 tones if you only want an A/B stripe, or
  drive the tint from the token id instead of its index to make identical
  tokens share a colour across the text.
- Sub-blocks are independent: drop the cost line (showCost={false}) for a pure
  counter, drop the legend, or hide the inspector line if the surrounding page
  already explains the markers. Adding a context-window bar means comparing
  tokenCount against a limit — a different component's job.
- Density knobs: text-[13px] / leading-7 on the split surface and the panel's
  gap-3 are the only spacing levers; keep leading generous or wrapped tints
  collide.
- Performance: the whole text is re-tokenized on every keystroke (O(n), fine to
  tens of thousands of characters). Above that, debounce the value before it
  reaches the component, or lower maxRenderedTokens — the counts stay exact
  either way.

Concepts

  • Pre-token seam — before any merging, text is cut where a tokenizer can never merge across: a leading space joins the word after it, and letters, digits, symbols and whitespace never share a token. This one rule explains most of the surprise in a token count.
  • Lossless cover — the view treats the tokenizer's output as offsets, not strings: overlaps are clipped, holes are shown as unclaimed characters, and every run is re-sliced from the source, so a buggy injected tokenizer can distort the count but never the text on screen.
  • Provisional tail — while the caret is still inside a word the last token has no right-hand neighbour; the next keystroke can absorb it and renumber everything after it, which is why a streaming token count wobbles. It is drawn with a blinking dashed edge that goes static under reduced motion.
  • Whitespace is billed — indentation, blank lines and trailing spaces are real tokens with no glyph, so they print as ·, → and ⏎ and get spoken names ("4 spaces") rather than disappearing into the layout.
  • Density over count — characters per token is the portable read: English prose lands near 4, JSON and hashes fall to ~2, CJK and emoji sit at or below 1, and the same sentence in two languages differs in price by that ratio.
  • Estimate honesty — the built-in splitter ships no vocabulary, so the panel labels itself an approximation and flips to "custom tokenizer" the moment a real one is injected; the cost line is arithmetic over whichever count is on screen.

Cache Indicator

A prompt-cache chip for one request — hit / partial / miss / write derived from the token counts, a tooltip breakdown of cached vs fresh tokens and money saved, and a self-correcting TTL ring that flips the chip to expired and fires onExpire exactly once.

Few Shot Editor

A few-shot example manager — paired input/output cards you add, duplicate, drag-reorder and switch off, with per-example and total token estimates against a budget, and gentle hints for blank outputs and near-duplicate inputs.

On This Page

InstallationPromptConcepts