Detection Overlay
Object-detection results drawn over the picture they came from — per-class tinted boxes with label and confidence chips, a live confidence gate, a class legend, and a rail whose hover, focus and selection stay in sync with the boxes.
Loading preview…
Installation
npx shadcn@latest add https://ui.zyeon.ai/r/detection-overlay.jsonPrompt
Build a React + TypeScript + Tailwind "DetectionOverlay" component with zod and
lucide-react. No other runtime dependency: the confidence slider is hand-rolled
so it can sit flush in a dense control bar.
Contract
- A zod schema (`detectionOverlaySchema` in a sibling contract file) is the single
source of truth: { status, items, image, model?, inferenceMs? }.
- One detection = { id, label, confidence, bbox: { x, y, w, h }, cls }.
- bbox is NORMALISED: x / y / w / h are fractions (0 → 1) of the image's own
width and height, origin top-left. Never pixels. Pixel boxes are only true for
the exact resolution the model saw — serve a thumbnail, put the picture in a
narrower column or export at 2x and every box slides off its object.
- `label` and `cls` are two fields on purpose. `cls` is the machine class key
(`traffic_light`): it drives the colour slot, the legend and the per-class
filter, so it must be stable across runs. `label` is what a human reads on the
chip and may carry an instance ("Person · rider") without splintering the
palette into a new colour every frame.
- `image` is { src, alt, width?, height? } or null. Sending width/height reserves
the frame's aspect ratio before the bytes arrive, so the layout never jumps.
- Envelope status is "loading" | "empty" | "error" | "ready", and it is about the
RESULT, not about the picture: `ready` with a 404 image is a real situation and
renders as a broken frame beside a perfectly working list.
- Props: the contract fields, plus view ("split" | "image" | "list", default
"split"); threshold? / defaultThreshold (0.5) / onThresholdChange?; selectedId?
/ onSelectedChange?; order ("confidence" | "given", default "confidence");
classColors?; showLegend (true); showThreshold (true); formatConfidence?;
aspectRatio?; errorMessage?; onRetry?; emptyState?; label ("Detections");
className plus the rest of the element props, ref forwarded to the <article>.
Behavior
- NORMALISE BEFORE RENDER, never trust the payload. Duplicate ids are dropped (a
duplicate key makes hover and selection address two boxes at once); x / y are
clamped into the image; w / h are clamped against the box's OWN origin so an
over-long side is trimmed at the edge instead of hanging outside the picture;
every side is floored at ~0.8% of the image, because a zero-width detection is
a pointer target nobody can hit but still a detection you must be told about;
confidence is clamped to 0 → 1 and NaN degrades to 0 rather than throwing.
- Classes take colour slots BY POSITION — first appearance gets slot 1, and the
palette wraps after five. Never hash the class name: a hash reshuffles every
box the day someone renames `traffic_light`, and two hashed neighbours can
collide on the same hue with nothing to stop them.
- The confidence gate filters BOTH surfaces at once, live. The legend filters by
class the same way. The counter ("9 / 12") is always on screen, so a filter can
never hide something silently.
- FILTERED-TO-NOTHING IS NOT EMPTY. When the model returned detections but the
filters hid all of them, say so, name the cause and render a "Clear filters"
button. "No objects were detected" would be a lie about the model.
- A pin that the filters hide simply STOPS APPLYING; it is never cleared. Drop
the gate again and the same box lights up. That is what makes the slider feel
like a lens instead of a delete button.
- Three separate signals, deliberately: hover (pointer, transient), focus
(keyboard, transient) and selection (the sticky pin). The highlight is
`hover ?? focus ?? selection`; when anything is highlighted, every other box
drops to ~35% opacity. Only the PIN calls `onSelectedChange`, so arrowing
through the whole list costs the consumer zero change events — collapsing the
three into one "active id" is exactly what produces a change event per row.
- Boxes are painted big-first so a bus can never swallow the pointer target of
the cyclist in front of it; the rail is ordered by confidence. Paint order and
reading order are different problems and must not share a sort.
- EXACTLY ONE SURFACE IS IN THE ACCESSIBILITY TREE. When the rail is on screen it
is the listbox (role=listbox / role=option / aria-selected) and the box layer
is aria-hidden decoration that still answers the pointer. In view="image" the
boxes become the listbox instead. Two listboxes over one collection would make
a screen reader read every detection twice.
- The list surface uses a ROVING TAB INDEX: one tab stop for the whole list,
landing on the row you last used (focused → pinned → first). Arrow keys move
real DOM focus (the focused row's own onFocus lights the box), Home / End jump,
Enter / Space toggles the pin, Escape unpins. Selection arriving from a box
click scrolls the matching row into view with block:"nearest".
- Boxes are placed inside the PAINTED rectangle of a contain-fitted image, not
inside the frame: measure the frame with a ResizeObserver, compare its ratio to
the image's, and letterbox accordingly. Without this, a caller who pins a
height or forces `aspectRatio` gets boxes floating off their objects.
- The label chip sits above its box, flips INSIDE when the box is near the top
edge and anchors right when the box's centre is past ~58% of the width — a chip
clipped by the frame is a chip that made the reader guess.
- The picture has its own load state, independent of the envelope: a shimmer
while loading, and on failure a corner note plus a Reload that REMOUNTS the
<img> via a key (reassigning the same src reuses the cached failure). The boxes
stay drawn on the failed frame — the geometry is still real data.
- A cached or server-rendered image can finish loading before hydration attaches
onLoad; re-read `img.complete` once per mount / src / reload inside a rAF, or
the shimmer never lifts. `complete` is true for failures too, and a zero
naturalWidth is exactly how a failure reports itself.
- Every observer and animation frame is cancelled on unmount; the slider uses
pointer capture on its own track instead of window listeners, so a drag that
leaves the track keeps working and there is nothing left to clean up.
- `status="ready"` with zero items renders the EMPTY branch. A data layer that
forgets to flip the flag should not produce a card with an empty rail and a
live slider.
Rendering & styling
- Semantic tokens only. Chrome: border / bg-card / bg-muted / text-muted-
foreground / ring; classes: var(--chart-1..5) applied through inline
borderColor + color-mix(in oklab, <token> N%, transparent) fills, so the same
hue reads at three intensities (idle 8%, highlighted 18%, pinned 30%). The
error envelope uses text-destructive. No hard-coded colours anywhere.
- Chips are bg-card with the class token as text and border colour, never the
token as a background: an opaque card chip stays legible over ANY photograph
and in both themes, while coloured-fill text does not.
- Layout is container-driven (@container + @2xl), not viewport-driven: the
overlay is dropped into side panels and full pages alike, and the split only
happens when the component itself is wide enough.
- Every transition is motion-reduce-guarded and the skeleton's pulse stops under
prefers-reduced-motion; filtering, highlighting and pinning keep working.
- The root is an <article aria-label> holding one persistent sr-only role=status
line that reports the visible / total count — the gate is the only control
whose effect has no other announcement, since rows announce themselves as
options.
- cn() merges the consumer's className into the root, which also spreads the
remaining element props and forwards its ref.
Customization levers
- Surfaces: `view` picks split / boxes-only / list-only, and the keyboard surface
follows automatically. `showLegend` and `showThreshold` strip the control bar
when the filters live elsewhere in your app.
- Palette: `classColors` overrides any class ("person" → var(--chart-2)) while
unlisted classes keep their positional slot; widen or shrink the base palette
by editing the token array — five slots is a default, not a limit.
- Density: the rail row is one line (swatch · label · class · bar · percent) —
drop the class tag or the bar for a tighter rail, or add a thumbnail crop per
row for a review tool.
- Framing: `aspectRatio` forces the frame's shape (a fixed grid track, a square
card) and the painted-box maths keeps the boxes honest; omit it to follow the
image's own ratio.
- Wording: `formatConfidence` swaps percentages for logits, a 0–1 decimal or a
bucket word ("high"); `label` names the region; `emptyState` and
`errorMessage` + `onRetry` own the two failure sentences.
- Control: pass `threshold` + `onThresholdChange` and `selectedId` +
`onSelectedChange` to lift the gate and the pin into your own store — one gate
shared by a whole gallery of frames, or a pin that also scrolls a video
timeline. Omit them and the component owns both.
- Ordering: `order="given"` keeps the model's own ranking when your backend has
already sorted by NMS score or by track id.Concepts
- Normalised coordinates are the contract — a box is stored as fractions of the picture, never as pixels. That single decision is what lets the same payload render on a thumbnail, in a narrow side panel, on a retina export and inside a forced square frame without a box ever sliding off the object it belongs to.
- Painted rectangle, not frame rectangle — a contain-fitted photo letterboxes inside its frame, so "40% of the frame" and "40% of the image" stop being the same point the moment the two ratios disagree. Boxes are placed inside the measured painted rect, which is why forcing an aspect ratio changes the picture's shape and not the detections' truth.
- Three signals, one highlight, one event — hover is a pointer look, focus is a keyboard look, the pin is a decision. The highlight resolves
hover ?? focus ?? pin, but only the pin reachesonSelectedChange; fusing them is what turns arrowing through a list into a storm of change events. - One accessibility tree over two surfaces — the rail and the boxes show the same collection, so exactly one of them is ever the listbox: the rail when it is on screen, the boxes when it is not. The other is decoration that still answers the pointer, and a screen reader never hears the same detection twice.
- The gate is a lens, not a delete — raising the threshold hides a pinned detection without unpinning it, so lowering the threshold brings the same box back exactly as it was. A filter that destroys state teaches people not to touch it.
- Filtered-to-nothing is a different sentence from empty — "the model found nothing" and "your filters hid all twelve" are opposite diagnoses, and only one of them comes with an undo. Naming the cause is the difference between a user adjusting a slider and a user filing a bug against the model.
Labeling Queue
A one-item-at-a-time human labeling queue — number-key label buttons, skip and flag escapes, a resumable "34 / 200" progress bar, undo for the last call, an all-done screen, and a one-shot lock per item so a double press can never double-label.
OCR Viewer
A scanned page beside the text read off it, kept in sync both ways — hover or focus a line to light up its source box, click a box to focus its line, with low-confidence words underlined in place.
