Eval Scorecard | Zyeon UI

Preview in your theme

status='ready' — 89.1% against an 85% gate, so the run ships. Every bar carries a tick at its own gate: Context precision reads 78.1% against 80% and is called a regression because it was passing at 82.4% on main@a1b2c3. Answer relevance moved 0.001 and says 'unchanged' instead of drawing an arrow; Tone has no baseline and says 'new'. Open a failing-cases chip to read the worst examples.

89.1%overall

gpt-4o-mini-2024-07-18

Gate passed

5 of 6 criteria passed · 1 regression

+4.1 pts vs the 85% gate+1.7 ptsvs main@a1b2c3

Faithfulness91.2%, gate 85%, passed, up from the baseline (+2.9 pts), 41 failing cases
Every claim in the answer is supported by a retrieved passage.
gate 85%480 cases judged
Answer relevance87.4%, gate 85%, passed, unchanged from the baseline, 62 failing cases
The answer addresses what the ticket actually asked.
gate 85%480 cases judged62 failing cases · none sampled
Context precision78.1%, gate 80%, failed, a regression from the baseline, down from the baseline (-4.3 pts), 105 failing cases
Share of retrieved passages that were actually used.
gate 80%480 cases judgedregression — was passing on the baseline
Refusal calibration94.3%, gate 90%, passed, up from the baseline (+4.1 pts), 12 failing cases
Refuses when the knowledge base has no answer — and only then.
gate 90%480 cases judged12 failing cases · none sampled
JSON validity99.8%, gate 99%, passed, unchanged from the baseline, 1 failing case
The reply parses against the ticket-reply schema.
gate 99%480 cases judged
Tone83.6%, gate 80%, passed, no baseline to compare against, 79 failing cases
Reads as the support voice guide, not as a model.
gate 80%480 cases judged

Dataset: support-tickets-v3
Cases: 480 cases
Finished: Mar 4, 2026, 9:41 AM UTC
Run: run_7Y2kQ

A run that should stop a deploy: 74.2% against the same 85% gate, four regressions, ordered worst-first with order='failing-first'. Conciseness dropped 37.2 points and its worst cases are hundreds of words long — every field is excerpted to 240 characters with an explicit reveal, and Copy always writes the whole case.

74.2%overall

internal-support-7b-lora-r64

Gate failed

1 of 5 criteria passed · 4 regressions

-10.8 pts vs the 85% gate-13.2 ptsvs main@a1b2c3

Faithfulness69.4%, gate 85%, failed, a regression from the baseline, down from the baseline (-18.9 pts), 147 failing cases
Every claim in the answer is supported by a retrieved passage.
gate 85%480 cases judgedregression — was passing on the baseline
Answer relevance80.2%, gate 85%, failed, a regression from the baseline, down from the baseline (-7.1 pts), 95 failing cases
The answer addresses what the ticket actually asked.
gate 85%480 cases judgedregression — was passing on the baseline95 failing cases · none sampled
Context precision77.1%, gate 80%, failed, a regression from the baseline, down from the baseline (-5.3 pts), 110 failing cases
Share of retrieved passages that were actually used.
gate 80%480 cases judgedregression — was passing on the baseline110 failing cases · none sampled
Conciseness41.8%, gate 70%, failed, a regression from the baseline, down from the baseline (-37.2 pts), 288 failing cases
Answers the ticket without padding; under 120 words unless quoting policy.
gate 70%480 cases judgedregression — was passing on the baseline
JSON validity99.6%, gate 99%, passed, unchanged from the baseline, 2 failing cases
The reply parses against the ticket-reply schema.
gate 99%480 cases judged2 failing cases · none sampled

Dataset: support-tickets-v3
Cases: 480 cases
Finished: Mar 5, 2026, 2:18 AM UTC
Run: run_9F1pD

no case opened yet — expand a row and press “Open case”

defaultOpenFailures — the messy run: the toxicity judge timed out so that criterion is 'not scored' with a dashed track (never a zero), Faithfulness counted 7 failures but sampled none, and two cases point at a criterion that was removed from the rubric — they are grouped as unmatched instead of vanishing.

81.2%overall

gpt-4o-mini-2024-07-18

Gate passed

1 of 2 criteria passed · 1 not scored · 1 regression

+1.2 pts vs the 80% gateunchangedvs run_7Y2kQ

Faithfulness86.9%, gate 80%, passed, down from the baseline (-1.4 pts), 7 failing cases
Every claim in the answer is supported by a retrieved passage.
gate 80%96 cases judged7 failing cases · none sampled
Toxicity screennot scored, gate 80%, the judge returned no score for this criterion, no baseline to compare against
The judge for this criterion timed out on the whole batch.
gate 80%0 cases judged
Answer relevance75.5%, gate 80%, failed, a regression from the baseline, down from the baseline (-11.8 pts), 3 failing cases
The answer addresses what the ticket actually asked.
gate 80%96 cases judgedregression — was passing on the baseline
ticket-51002scored 20%
Input
```
Do you have a Terraform provider?
```
Expected
```
Yes — link the registry page and the minimum provider version.
```
Got
```
You can use our REST API to automate workspace creation. Here is how authentication works…
```
Judge note
```
Answers a different question; never mentions Terraform.
```
1 of 3 failing cases were sampled into this report

Unmatched failing cases

These cases point at a criterion that is not in this rubric. They are listed here instead of being dropped from the report.

ticket-51244scored 12%

Input

What is the rate limit on the search endpoint?

Expected

600 requests per minute per workspace, burst 1200.

Got

The search endpoint allows 10,000 requests per second per API key.

Judge note

Criterion `hallucination-rate` was removed from the rubric between runs.

ticket-51301scored 34%

Input

Which regions do you store data in?

Expected

eu-central-1 and us-east-1, selectable per workspace at creation.

Got

We store data in twelve regions worldwide including ap-southeast-2 and sa-east-1.

Judge note

Region list is invented; the docs list two.

Dataset: support-tickets-v3-subset
Cases: 96 cases
Finished: Mar 6, 2026, 8:02 AM UTC
Run: run_3Kx8b

The first run of a new suite: no baseline exists anywhere, so every row reads 'new' rather than a fabricated +0.0. One open-ended case has no gold answer and says so instead of printing an empty Expected box.

82.3%overall

claude-haiku-4.5

Gate passed

2 of 3 criteria passed

+2.3 pts vs the 80% gatenew

Instruction following88%, gate 80%, passed, no baseline to compare against, 14 failing cases
Honours the brief: length, audience, call to action.
gate 80%120 cases judged14 failing cases · none sampled
Factuality84.7%, gate 80%, passed, no baseline to compare against, 18 failing cases
Feature claims match the product docs bundled with the prompt.
gate 80%120 cases judged18 failing cases · none sampled
Tone74.4%, gate 80%, failed, no baseline to compare against, 31 failing cases
Warm, direct, no superlatives.
gate 80%120 cases judged

Dataset: onboarding-emails-v1
Cases: 120 cases
Finished: Feb 19, 2026, 2:05 PM UTC
Run: run_1A0zz

density='compact' with showDial={false} — the same card as a row inside a bigger run header. The rubric is empty here, so the card states that instead of drawing an empty list, and the footer still carries dataset, cases, finish time and run id.

mistral-large-2411

Gate failed

0 of 0 criteria passed

-2.9 pts vs the 80% gateunchangedvs yesterday

The run reported an overall score but no individual criteria.

Dataset: smoke-suite
Cases: 40 cases
Finished: Mar 6, 2026, 11:59 PM UTC
Run: run_0Q4mn

The same run under three levers: order='delta' puts the biggest drop first, formatScore switches the wording to the raw 0–1 the harness emitted (the geometry never changes), and a tighter case budget — maxFailuresPerCriterion={1}, excerptChars={80} — still says out loud how much it withheld.

0.891overall

gpt-4o-mini-2024-07-18

Gate passed

5 of 6 criteria passed · 1 regression

+4.1 pts vs the 0.850 gate+1.7 ptsvs main@a1b2c3

Context precision0.781, gate 0.800, failed, a regression from the baseline, down from the baseline (-4.3 pts), 105 failing cases
Share of retrieved passages that were actually used.
gate 0.800480 cases judgedregression — was passing on the baseline
ticket-40912scored 0.240
Input
```
Our team plan says 20 seats but billing charged us for 25. Can you explain the d
```
Expected
```
Seat count is billed on the peak seats used in the period, so a temporary invite
```
Got
```
Your plan includes 20 seats. If you need more seats you can upgrade at any time 
```
Judge note
```
Retrieved the plan-limits page (0.31) and ignored the peak-seat billing passage 
```
Excerpted to 80 characters
2 of 105 failing cases were sampled into this report
Answer relevance0.874, gate 0.850, passed, unchanged from the baseline, 62 failing cases
The answer addresses what the ticket actually asked.
gate 0.850480 cases judged62 failing cases · none sampled

JSON validity

The reply parses against the ticket-reply schema.

gate 0.990480 cases judged

ticket-40233scored 0.000

Input

Refund request for invoice INV-2291, duplicate charge on 2 March.

Expected

{"action":"refund","invoice":"INV-2291","amount_cents":4900,"reason":"duplicate_

Got

{"action":"refund","invoice":"INV-2291","amount_cents":49.00,"reason":"duplicate

Judge note

Trailing comma and a float where the schema requires an integer number of cents.

Excerpted to 80 characters

Faithfulness0.912, gate 0.850, passed, up from the baseline (+2.9 pts), 41 failing cases
Every claim in the answer is supported by a retrieved passage.
gate 0.850480 cases judged
ticket-40771scored 0.180
Input
```
Is SSO included on the Team plan?
```
Expected
```
SSO is an Enterprise feature. Team plans can use Google and GitHub sign-in, but 
```
Got
```
Yes — SAML SSO is available on the Team plan, you can enable it under Settings →
```
Judge note
```
Unsupported claim: no retrieved passage mentions SAML on Team. The linked settin
```
Excerpted to 80 characters
2 of 41 failing cases were sampled into this report
Refusal calibration0.943, gate 0.900, passed, up from the baseline (+4.1 pts), 12 failing cases
Refuses when the knowledge base has no answer — and only then.
gate 0.900480 cases judged12 failing cases · none sampled
Tone0.836, gate 0.800, passed, no baseline to compare against, 79 failing cases
Reads as the support voice guide, not as a model.
gate 0.800480 cases judged
ticket-41655scored 0.380
Input
```
This is the third time I am writing about the same broken invoice.
```
Expected
```
Acknowledge the repeat contact, apologise once, state what is being done now and
```
Got
```
Thanks so much for reaching out again!! I totally understand your frustration!! 
```
Judge note
```
Three exclamation marks, no acknowledgement of the previous two tickets, no comm
```
Excerpted to 80 characters
1 of 79 failing cases were sampled into this report

Dataset: support-tickets-v3
Cases: 480 cases
Finished: Mar 4, 2026, 9:41 AM UTC
Run: run_7Y2kQ

status='loading' — the run record itself hasn't arrived.

status='empty' — this dataset has never been evaluated, which is a different thing from scoring zero.

No evaluation yet

Nothing has been scored against this dataset. Run the suite to get a baseline to compare against.

status='error' — the report failed to load. A different failure from the run failing, so it reads differently and offers a retry.

Couldn't load this evaluation

The eval service returned 503 while fetching run_7Y2kQ. request-id: 9ad1-7f2c

Loading preview…

This component is part of Pro

The install command, prompt and concept map are included with a Pro license — one purchase unlocks every Pro component, forever.

Unlock with Pro