Evals Are Broken, Use Them Anyway

In plain English

An “eval” is a test that scores how well an AI does a job. Ara says two groups get these wrong. One trusts the published scores too much, though models with near-identical numbers behave very differently in practice. The other ignores tests and goes on gut feel.

His middle path is to treat lab-published numbers as rough, avoid being first to switch to each new model, and prefer new, realistic tests over old, standardised ones. Then run your own tests, study the failures and fix them in order of impact.

Three ideas worth keeping

• Vendor scores are approximations, and some well-known tests no longer measure real work.
• When an AI agent fails a task, read what it actually did, group the causes and fix the cheapest ones first.
• Chasing a higher score by tuning to the test itself is cheating, so keep checking that the product feels good to use as well.

What it means for your business

Do not choose an AI tool because it topped a leaderboard. Ask the supplier to show results on tasks like yours, and test it yourself on a handful of real cases. Wait a few weeks after a new model launches before switching.

What it means for software teams

Run a realistic test set in isolated environments so many tasks run in parallel. Use an agent to read the failure traces. A low score can mean the model, the surrounding software or a badly set task. Cline found that a model working well on one family of models did poorly on others until they tuned for each.

What we found, and what we did

We hold our own AI work to the same discipline: sealed test sets the system never trained on, scoring that gives the same answer every time, and a judge that never did the work and can’t see whose answer is whose.

When we chose an AI code reviewer, we tested 17 setups on 29 real past changes that held 11 known defects, all judged blind. The best setup caught only 4 of the 11, so high-risk changes still get a deeper review by other means.

We also wrote down how we would measure our agents’ memory feature before running the test, so we couldn’t pick a flattering result afterwards. Our guide How to Build AI Agents That Run in Production and our note Stop Measuring AI by Test-Pass Rate explain the approach.

Worth watching

• 05:23: why older, standardised tests stop being useful.
• 12:55: the routine of running every task, reading the failures and fixing them in order of impact.
• 16:03: the three zones of improvement, ending with the temptation to overfit.