slp.blue Model Bench
Model Bench · scoring

How an answer gets its score

Nobody — no person and no model — hands out a number. Code checks what the chatbot actually did, a judge lists the defects it can quote, the quotes are verified, and the score is computed from what is left.

score = 0 if a hard check fails, else min over dimensions ( min level of that dimension's findings, 5 if none )

Each finding type carries a fixed level. A dimension takes the lowest level among its findings; the case takes the lowest dimension.

From run to score

Five steps, in this order

  1. Run the case

    The harness plays the user: the prompt, any follow-ups, and the answers a tester would give if the chatbot asks. It records every tool call and what was written.

    → transcript, tool calls, tenant state

  2. Hard checks

    Code checks the outcome that can be checked exactly: a draft exists, a rename is complete, a sentence sits in the right section, in red.

    → any failure scores 0

  3. List findings

    The judge reads the evidence and lists every defect it can quote, each typed from a fixed list of 27.

    → typed, quoted findings

  4. Verify quotes

    Every quote is looked up in the reply, the written record or the documents the model read. What cannot be found is dropped.

    → findings that survive

  5. Compute

    Code maps each finding type to its level and takes the minimum. The judge never picks the number.

    → 0–5
The scale

What each score means

The worst defect decides. One invented action in a record meant to be signed outweighs an otherwise perfect answer, and a single English word caps an answer at 4 no matter how good the rest is.

The finding types

27 defects the judge can report

Grouped by the dimension they lower. The square is the level a single finding of that type sets.

Before the judge

Hard checks

What they check

The outcome of the task, straight from the tool calls and the tenant's state after the run — not from what the reply says happened.

  • a new draft of the right document exists
  • a rename reached every occurrence in the text
  • the requested sentence is in section 5, and it is red
  • a safety warning survived a rewrite, still highlighted
  • a nonconformity or corrective action was actually created

How they count

  • Any failed check scores the case 0. The judge's findings are still listed, so the board shows what else went wrong.
  • Some checks are diagnostics: they flag something worth a look — a name shared with the template document, a follow-up that was never delivered — and never fail a case.
  • A model that keeps calling tools until its context is full, or until the app's 60-round tool cap, has not finished the task: that is a 0 too.
Keeping the judge honest

What the judge is held to

One fixed judge

Every answer is judged by the same model, Gemini 3.8 Flash, with a larger thinking budget when a case's evidence runs over 100,000 characters — long SOP rewrites, where a copied paragraph is easy to miss.

Inventories first

For a written record the judge first lists every action, check, cause and number the record asserts, and where each one comes from — the user, a document it read, or nowhere. Where the rule is mechanical, code turns those lists into findings: a critical fact that is absent, a section that only says “see the manual”.

Quotes must exist

Every finding quotes the reply, the record or a source document, and the quote is looked up. A quote that is not there is sent back once; if it is still not there, that finding does not count — and a case whose penalty rests on an unquoted finding is left unscored rather than guessed.

Fact tables

Each Czech case has a table of the values a correct answer contains, checked word for word against the documents the models read, with the forms that count as the same value (unit conversions, number words), what the answer must include, and what it must not state. Values and omissions are judged against the table, not the judge's memory.

What the app can do

Questions answered without tools are judged against a maintained page of what slp.blue can and cannot do, plus an index of ISO 15189 and ISO 22367 clauses. An answer that offers a feature the app does not have is an invented feature, whatever the judge believes.

One fingerprint per board

Every result stores a fingerprint of the prompts, rules, fact tables and code that graded it. The board only shows results that share one fingerprint: .

A real answer, scored

Worked example

CB14

New work instruction from a manufacturer's manual

Gemini 3.8 Flash · low, one replicate. The task: write a work instruction for the Erytra Eflexis analyser from its manual, using an existing instruction for a different analyser (Wadiana) only as a layout template.

slp.blue Model Bench · the scoring code and the full judge prompt live with the benchmark harness.