How we run honest evals
Every week we grade ourselves on the same test and publish the result — including the numbers that make us wince. Here's how, and why.
Benchmark numbers in AI have a credibility problem, and everyone in the field knows it. A company picks the suites it wins, runs them under favorable settings, and publishes a chart where every bar points up and to the right. It's not usually lying, exactly. It's selection — the quiet art of only measuring what makes you look good.
We wanted our evals page to be the opposite of that. Not a marketing chart, but a report card: the same test, every week, published no matter the grade. This post is how the machine behind that page works, and the reasoning for a few choices that might look strange.
What the harness actually does
Every Monday at 06:00 Pacific, a cron job runs a fixed suite of graded tasks against each Wren tier — Seed, Field, and Grove. The suite is versioned in a repository; we don't quietly swap questions between runs, because week-over-week comparison only means something if the test held still. When we do change the suite — add a category, retire a stale one — we bump the version and say so in the run notes, so a drop or jump can be read correctly.
The current suite covers five things we think matter for a thinking-and-building tool:
- Faithful summary — given a source document, does the summary stay grounded in it, or does it invent? Scored as a grounded percentage.
- Instruction follow — multi-constraint prompts ("three bullets, no adjectives, under 40 words"). Pass or fail, no partial credit.
- Frames render — asked to build something, does it emit valid, running HTML? Measured by whether the artifact loads without error.
- Multi-step reasoning — math and logic chains where one wrong step sinks the answer.
- Refusal calibration — a mixed set of benign and genuinely harmful requests. Correct means answering the safe ones and declining the unsafe ones. This is the one we score worst on.
Who grades the grader
Some tasks grade themselves — valid HTML either parses or it doesn't; a math answer is right or wrong. Those are exact checks, no judgment involved. The softer ones (is this summary faithful? is this refusal appropriate?) use a rubric and an LLM judge, and here we have to be careful, because a model grading a model is a hall of mirrors if you're not disciplined about it.
So we do two things. First, the judge runs against a written rubric with concrete pass/fail criteria, not vibes. Second, every week we hand-audit a random sample of the judge's calls — a human reads the transcript and checks whether the automated grade was right. When the human and the judge disagree more than we're comfortable with, we fix the rubric, not the score. The audit rate and disagreement numbers are part of what we publish.
A number you'd be embarrassed to publish is exactly the number worth publishing.
Why we publish the bad numbers
Our refusal calibration sits in the low-to-mid seventies. That's not a great number and we print it in the same size font as the good ones. People occasionally ask why we'd surface our weakest result so plainly. Three reasons.
First, it's the honest thing, and honesty is the entire pitch of this company — it would be incoherent to hide a number on the page whose whole purpose is not hiding numbers. Second, a weakness you've published is a weakness you're accountable for; the number going up over time is a promise we've made in public. Third, and least obviously: an all-green report card is a signal that someone chose the test to win. The wince-worthy row is what makes the green ones believable.
How to read the evals page
When you open the page, a few habits will serve you:
- Check the date and suite version first. A score only means something relative to when it was run and which test produced it. Both are stamped on every run.
- Read across tiers, not just down. The interesting story is usually the gap — where Grove pulls ahead of Field, and where the extra thinking buys nothing. (We wrote about one such gap when we launched Grove.)
- Watch the trend, not the point. Any single week has noise; the questions are the same but model serving drifts. A number that climbs three weeks running is real. A one-week blip usually isn't.
- Mind what we're not measuring. This is a self-eval on a small, fixed suite. It is not a third-party benchmark and it doesn't cover everything you might care about. We say this on the page, and we'll say it here too.
The limits of the whole exercise. These are our own evals — we wrote the suite, we run the harness, we publish the results. That's transparent, but it isn't independent, and we won't dress it up as if it were. A small fixed suite can also be gamed by accident: optimize toward it long enough and you improve the test scores without improving the product. We guard against that by rotating in fresh tasks and keeping the suite deliberately broad, but we can't claim to have solved it. Treat these numbers as an honest window into how we're doing — not as a benchmark authority we don't have.
None of this is novel science. We're a product company standing on hosted frontier models, not a research lab, and evaluating models well is genuinely hard work that people far deeper than us have written about. What we can offer is discipline and daylight: the same test, every week, published whole. If you want to see this week's — good rows and bad — it's one click away.
See the receipts.
Every dated run, good rows and bad, on one page. Nothing curated to flatter.