Measured, not promised.
The figures Klorn publishes, where each one comes from, and what each one cannot tell you. Every set and script named here is in the public repository.
Three figures, and their sources
- ≥90%Urgent recall, CI floor
A change to the classifier that drops urgent recall below 90% on the synthetic gate set fails the build.
Source: eval-floors.ts - 95.5%Silent precision
Measured on the hand-labelled real-mail set: of 22 messages Klorn filed as Silent, 21 were labelled silent. Three runs on 2026-08-26, identical each time.
Source: eval/README.md - 0Unapproved actions
Sends, permanent deletes and forwards to outside addresses wait for approval and log a receipt pinned to a hash of what went out. An auto-reply rule you set up yourself counts as that approval.
Enforced in code: attention-floor.ts
No headline accuracy. The real-mail set predates the Meeting and Info lanes: its labels use three lanes, so it cannot score the other two, and it needs re-labelling against all five. Until it is re-labelled we quote no headline accuracy from it. The silent-precision figure above comes from the same set and carries the same limits.
How evaluation runs
The committed sets
Two sets live in packages/api/eval/. The real-mail set (real-eval-set.json) holds
53 real emails, hand-labelled and scrubbed of personal data. The synthetic gate set
(judge-eval-set.json) holds 56 made-up emails with no personal data.
The runner
From packages/api, pnpm eval:real scores the classifier against the real-mail set;
pnpm eval:judge does the same for the synthetic set.
The gate in CI
Every pull request that changes the classifier runs the synthetic gate set through the model, with three floors enforced: 80% overall, 90% urgent recall and 90% silent precision. A run below any floor fails. If the job has no model key, as on a pull request from a fork, it skips with a notice instead of passing.
A second check needs no model and runs on every CI test run: the rule-based fast path and keyword fallback must reach 70% on the same set, a missed urgent message must fall to Queue rather than Silent, and a marketing message must never be marked urgent.
The real-mail set is scored on the same pull requests as a report only. Its urgent lane has 4 labelled messages, too few for a 90% floor to mean anything.
Read the sets, the runner and the measurement history on GitHub
Checked 2026-10-14 against the repository. Something here wrong? Tell us.