AI

Detection Overlay

Object-detection results drawn over the picture they came from — per-class tinted boxes with label and confidence chips, a live confidence gate, a class legend, and a rail whose hover, focus and selection stay in sync with the boxes.

Preview in your theme

Loading preview…

"use client"

import * as React from "react"
import { ImageOff, RefreshCcw, ScanSearch, SquareDashed, TriangleAlert } from "lucide-react"

import { cn } from "@/lib/utils"
import type { DetectionOverlayData } from "./detection-overlay.contract"

/**
 * The class palette: five theme slots, handed out in order of first appearance
 * and cycled after that. Classes get a colour by POSITION, never by hashing the
 * string — a hash would silently reshuffle every box the day someone renames
 * `traffic_light` to `traffic-light`, and two hashed neighbours can collide on
 * the same hue with nothing to stop them.

Installation

npx shadcn@latest add https://ui.zyeon.ai/r/detection-overlay.json

Prompt

Build a React + TypeScript + Tailwind "DetectionOverlay" component with zod and
lucide-react. No other runtime dependency: the confidence slider is hand-rolled
so it can sit flush in a dense control bar.

Contract
- A zod schema (`detectionOverlaySchema` in a sibling contract file) is the single
  source of truth: { status, items, image, model?, inferenceMs? }.
- One detection = { id, label, confidence, bbox: { x, y, w, h }, cls }.
- bbox is NORMALISED: x / y / w / h are fractions (0 → 1) of the image's own
  width and height, origin top-left. Never pixels. Pixel boxes are only true for
  the exact resolution the model saw — serve a thumbnail, put the picture in a
  narrower column or export at 2x and every box slides off its object.
- `label` and `cls` are two fields on purpose. `cls` is the machine class key
  (`traffic_light`): it drives the colour slot, the legend and the per-class
  filter, so it must be stable across runs. `label` is what a human reads on the
  chip and may carry an instance ("Person · rider") without splintering the
  palette into a new colour every frame.
- `image` is { src, alt, width?, height? } or null. Sending width/height reserves
  the frame's aspect ratio before the bytes arrive, so the layout never jumps.
- Envelope status is "loading" | "empty" | "error" | "ready", and it is about the
  RESULT, not about the picture: `ready` with a 404 image is a real situation and
  renders as a broken frame beside a perfectly working list.
- Props: the contract fields, plus view ("split" | "image" | "list", default
  "split"); threshold? / defaultThreshold (0.5) / onThresholdChange?; selectedId?
  / onSelectedChange?; order ("confidence" | "given", default "confidence");
  classColors?; showLegend (true); showThreshold (true); formatConfidence?;
  aspectRatio?; errorMessage?; onRetry?; emptyState?; label ("Detections");
  className plus the rest of the element props, ref forwarded to the <article>.

Behavior
- NORMALISE BEFORE RENDER, never trust the payload. Duplicate ids are dropped (a
  duplicate key makes hover and selection address two boxes at once); x / y are
  clamped into the image; w / h are clamped against the box's OWN origin so an
  over-long side is trimmed at the edge instead of hanging outside the picture;
  every side is floored at ~0.8% of the image, because a zero-width detection is
  a pointer target nobody can hit but still a detection you must be told about;
  confidence is clamped to 0 → 1 and NaN degrades to 0 rather than throwing.
- Classes take colour slots BY POSITION — first appearance gets slot 1, and the
  palette wraps after five. Never hash the class name: a hash reshuffles every
  box the day someone renames `traffic_light`, and two hashed neighbours can
  collide on the same hue with nothing to stop them.
- The confidence gate filters BOTH surfaces at once, live. The legend filters by
  class the same way. The counter ("9 / 12") is always on screen, so a filter can
  never hide something silently.
- FILTERED-TO-NOTHING IS NOT EMPTY. When the model returned detections but the
  filters hid all of them, say so, name the cause and render a "Clear filters"
  button. "No objects were detected" would be a lie about the model.
- A pin that the filters hide simply STOPS APPLYING; it is never cleared. Drop
  the gate again and the same box lights up. That is what makes the slider feel
  like a lens instead of a delete button.
- Three separate signals, deliberately: hover (pointer, transient), focus
  (keyboard, transient) and selection (the sticky pin). The highlight is
  `hover ?? focus ?? selection`; when anything is highlighted, every other box
  drops to ~35% opacity. Only the PIN calls `onSelectedChange`, so arrowing
  through the whole list costs the consumer zero change events — collapsing the
  three into one "active id" is exactly what produces a change event per row.
- Boxes are painted big-first so a bus can never swallow the pointer target of
  the cyclist in front of it; the rail is ordered by confidence. Paint order and
  reading order are different problems and must not share a sort.
- EXACTLY ONE SURFACE IS IN THE ACCESSIBILITY TREE. When the rail is on screen it
  is the listbox (role=listbox / role=option / aria-selected) and the box layer
  is aria-hidden decoration that still answers the pointer. In view="image" the
  boxes become the listbox instead. Two listboxes over one collection would make
  a screen reader read every detection twice.
- The list surface uses a ROVING TAB INDEX: one tab stop for the whole list,
  landing on the row you last used (focused → pinned → first). Arrow keys move
  real DOM focus (the focused row's own onFocus lights the box), Home / End jump,
  Enter / Space toggles the pin, Escape unpins. Selection arriving from a box
  click scrolls the matching row into view with block:"nearest".
- Boxes are placed inside the PAINTED rectangle of a contain-fitted image, not
  inside the frame: measure the frame with a ResizeObserver, compare its ratio to
  the image's, and letterbox accordingly. Without this, a caller who pins a
  height or forces `aspectRatio` gets boxes floating off their objects.
- The label chip sits above its box, flips INSIDE when the box is near the top
  edge and anchors right when the box's centre is past ~58% of the width — a chip
  clipped by the frame is a chip that made the reader guess.
- The picture has its own load state, independent of the envelope: a shimmer
  while loading, and on failure a corner note plus a Reload that REMOUNTS the
  <img> via a key (reassigning the same src reuses the cached failure). The boxes
  stay drawn on the failed frame — the geometry is still real data.
- A cached or server-rendered image can finish loading before hydration attaches
  onLoad; re-read `img.complete` once per mount / src / reload inside a rAF, or
  the shimmer never lifts. `complete` is true for failures too, and a zero
  naturalWidth is exactly how a failure reports itself.
- Every observer and animation frame is cancelled on unmount; the slider uses
  pointer capture on its own track instead of window listeners, so a drag that
  leaves the track keeps working and there is nothing left to clean up.
- `status="ready"` with zero items renders the EMPTY branch. A data layer that
  forgets to flip the flag should not produce a card with an empty rail and a
  live slider.

Rendering & styling
- Semantic tokens only. Chrome: border / bg-card / bg-muted / text-muted-
  foreground / ring; classes: var(--chart-1..5) applied through inline
  borderColor + color-mix(in oklab, <token> N%, transparent) fills, so the same
  hue reads at three intensities (idle 8%, highlighted 18%, pinned 30%). The
  error envelope uses text-destructive. No hard-coded colours anywhere.
- Chips are bg-card with the class token as text and border colour, never the
  token as a background: an opaque card chip stays legible over ANY photograph
  and in both themes, while coloured-fill text does not.
- Layout is container-driven (@container + @2xl), not viewport-driven: the
  overlay is dropped into side panels and full pages alike, and the split only
  happens when the component itself is wide enough.
- Every transition is motion-reduce-guarded and the skeleton's pulse stops under
  prefers-reduced-motion; filtering, highlighting and pinning keep working.
- The root is an <article aria-label> holding one persistent sr-only role=status
  line that reports the visible / total count — the gate is the only control
  whose effect has no other announcement, since rows announce themselves as
  options.
- cn() merges the consumer's className into the root, which also spreads the
  remaining element props and forwards its ref.

Customization levers
- Surfaces: `view` picks split / boxes-only / list-only, and the keyboard surface
  follows automatically. `showLegend` and `showThreshold` strip the control bar
  when the filters live elsewhere in your app.
- Palette: `classColors` overrides any class ("person" → var(--chart-2)) while
  unlisted classes keep their positional slot; widen or shrink the base palette
  by editing the token array — five slots is a default, not a limit.
- Density: the rail row is one line (swatch · label · class · bar · percent) —
  drop the class tag or the bar for a tighter rail, or add a thumbnail crop per
  row for a review tool.
- Framing: `aspectRatio` forces the frame's shape (a fixed grid track, a square
  card) and the painted-box maths keeps the boxes honest; omit it to follow the
  image's own ratio.
- Wording: `formatConfidence` swaps percentages for logits, a 0–1 decimal or a
  bucket word ("high"); `label` names the region; `emptyState` and
  `errorMessage` + `onRetry` own the two failure sentences.
- Control: pass `threshold` + `onThresholdChange` and `selectedId` +
  `onSelectedChange` to lift the gate and the pin into your own store — one gate
  shared by a whole gallery of frames, or a pin that also scrolls a video
  timeline. Omit them and the component owns both.
- Ordering: `order="given"` keeps the model's own ranking when your backend has
  already sorted by NMS score or by track id.

Concepts

  • Normalised coordinates are the contract — a box is stored as fractions of the picture, never as pixels. That single decision is what lets the same payload render on a thumbnail, in a narrow side panel, on a retina export and inside a forced square frame without a box ever sliding off the object it belongs to.
  • Painted rectangle, not frame rectangle — a contain-fitted photo letterboxes inside its frame, so "40% of the frame" and "40% of the image" stop being the same point the moment the two ratios disagree. Boxes are placed inside the measured painted rect, which is why forcing an aspect ratio changes the picture's shape and not the detections' truth.
  • Three signals, one highlight, one event — hover is a pointer look, focus is a keyboard look, the pin is a decision. The highlight resolves hover ?? focus ?? pin, but only the pin reaches onSelectedChange; fusing them is what turns arrowing through a list into a storm of change events.
  • One accessibility tree over two surfaces — the rail and the boxes show the same collection, so exactly one of them is ever the listbox: the rail when it is on screen, the boxes when it is not. The other is decoration that still answers the pointer, and a screen reader never hears the same detection twice.
  • The gate is a lens, not a delete — raising the threshold hides a pinned detection without unpinning it, so lowering the threshold brings the same box back exactly as it was. A filter that destroys state teaches people not to touch it.
  • Filtered-to-nothing is a different sentence from empty — "the model found nothing" and "your filters hid all twelve" are opposite diagnoses, and only one of them comes with an undo. Naming the cause is the difference between a user adjusting a slider and a user filing a bug against the model.

On This Page