Which LLM should classify your email? I ran six.

2026-08-26 · one run per model · every number re-runs with pnpm eval:judge

Every AI email tool picks a model for you and never shows its work. Klorn's eval set is committed to the repo and gated in CI, so instead of an opinion, here is a measurement: six current models, one classification job, identical prompt, identical decision rule.

The job is narrow on purpose. The model does not pick the folder or the priority. It scores four features per email — confidence, sender trust, reversibility, urgency — and a deterministic ~200-line rule maps those numbers to exactly one of five lanes. That split is what makes this benchmark possible at all: the model's task is small enough to measure.

Committed 56-email gate set, 2026-08-26, temperature-0, --context=fixture, JUDGE_INCLUDE_BODY=true — the same environment CI runs. Gate floors: overall ≥80%, urgent recall ≥90%, silenced precision ≥90%.

Three things in this table matter more than the ranking.

1. Price does not order the table

The $5.00/M model places fifth. The spread among frontier models — 19.6 points — is more than five times the gap between the best of them and a pin that costs eight times less. If you chose your model because it tops a general benchmark, you chose on a number that has nothing to do with this job.

2. The interesting failure has a mechanism

claude-sonnet-5 missed eight urgent emails. Seven of the eight failed on confidence, not urgency: it scored urgency 0.80–1.00 — correct — and then reported confidence 0.55–0.60, under the 0.70 bar the rule requires before interrupting you. It read the mail right and declined to say it was sure.

I can tell you that because the threshold is a number in a file. In a “model picks the label” design, the same result reads as “this model is worse at email,” which is both wrong and unfixable.

3. The floor held for every model

Across a 20-point accuracy spread, including two models that failed the gate outright, the number of urgent emails classified as hide-forever was zero — every miss degraded to the visible queue. That invariant is a unit test, not a hope. It is also the only line in this post that will still be true when the next model generation ships.

JUDGE_MODEL=<any-openrouter-id> pnpm eval:judge

The eval set, the rule, the floors and the receipts pipeline are AGPL-3.0: github.com/k08200/klorn. Or skip the setup and point Klorn at your own inbox: