Which model, and which cases tell them apart
Six chat models on 21 tasks from a Czech laboratory — 8 handbook questions, 4 document edits, 5 synthesis tasks, 4 incident records — three replicates of every case per model, one for DeepSeek. Each answer is scored 0–5 from the defects a judge can quote — see how we score. The models barely separate on most cases; the board shows where they do, and how noisy each case is.
Six arms at a glance
Handbook questions are answered without tools; document edits, synthesis tasks and incident records read and write the tenant's documents and records. Click a column header to sort any table; click a row to spotlight that model on the case board.
Fabrications: findings that fail a case by themselves — a record statement the user’s account rules out, an action written into a non-draft record as done when nobody reported it, a value or product name with no source, an app feature that does not exist — per full pass of the 21 cases. Czech: the language level the findings leave, 1–5.
What goes wrong, per model
Of every ten judged answers, how many have at least one finding of each kind. A finding is a quoted defect the judge found and the audit confirmed in the answer, the written record or the documents the model read. The largest figure in each column is bold.
What a task consumes
Tokens are the part that carries over to any setup: on your own hardware, time per task is the tokens below divided by how fast the machine reads and writes them. Measured from the gateways' per-call logs, as the median task of each kind per model; a tool-using task sums every model call it made.
On your own hardware
Set how fast a local server processes the prompt and generates text. The estimate counts every input and output token of the median task, one call after another.
A server with prefix caching reprocesses little of an edit, synthesis or incident task's repeated context, so treat those figures as upper bounds.
Why some models use more input tokens
The handbook. A handbook question sends the whole slp.blue handbook (build 3.18, about 130,000 tokens) with every question. On this tier the input is the handbook; the model adds only its tokenizer.
The tokenizer. The same Czech text is a different number of tokens for each model. Qwen and DeepSeek both read the 3.18 build: Qwen counts it as about 135,000 tokens, DeepSeek as about 147,000 — DeepSeek splits Czech into roughly 8% more tokens. Per byte of handbook, Gemini counts about the same as Qwen.
Tool rounds. In the edit, synthesis and incident tasks every model call re-sends the whole conversation so far — instructions, the user's messages, every document read and every tool result. Input therefore grows with the number of calls and with how much document text the model pulls in, which is why document tasks run to several hundred thousand input tokens.
Caching. Gemini reports repeated context as cached tokens, billed at a tenth of the input price; the other vendors in these runs report no cache hits, so all their input is counted and billed as fresh. For a local server, cached and fresh input are the same work unless it keeps a prefix cache.
Output includes reasoning. Thinking tokens are counted as output. Gemini 3.8 at medium thinking and Qwen with reasoning on write several times more output than their low or reasoning-off settings — and output is the slow part on local hardware.
Where the models separate
Each dot is one replicate of one case, on the 0–5 score; the short bar is the mean. Rows are sorted by how far apart the models' means sit; click a model's header to sort by its score instead (worst first, again for best first). Click a row to read what the judge said about every replicate.
What extra tokens buy
Method and caveats
Every answer gets a 0–5 score computed from the defects a judge can quote and an audit confirms. The full rules — hard checks, the 27 finding types and how they become a score — are on How we score.
Runs and replicates
Every pass runs against a freshly seeded copy of the same Czech laboratory tenant. The Gemini and both Qwen arms have three replicates of every case; DeepSeek has one. The same model on the same case varies by 1–2 points between replicates — that spread is on the board, and it is why a single-replicate difference is not a ranking.
Every handbook question on the board was answered from the same handbook build (3.18), on the same test instance, so handbook answers compare case by case across models. The edit, synthesis and incident tasks run on assistants that do not load the handbook.
Lost and unfinished runs
A run that fails on a vendor connection error is repeated. A replicate that still cannot be completed is shown on the board as lost and left out of the means.
A model that keeps calling tools until its context window is full, or until it reaches the app's 60-round tool cap, has not finished the task, and scores 0.
Cost and latency
Tokens and prices come from the gateways' per-call logs, priced at list price. Latency is the time from the question to the final answer as the harness measured it, the vendor's queue included.