How We Took an AI Agent From 41% to 97% Accurate

In plain English

Milind’s company builds AI agents (software that carries out tasks on its own) for large businesses. He tells the story of one for a big pizza chain. The job was to check competitors’ prices on delivery apps and websites and fill in a spreadsheet, work an analyst used to do by hand.

The first version got 41% of the prices right. The eleventh got 97%. He argues the gain came from a repeatable routine, not a smarter model: agree what “good” means, write down every term, measure honestly, start tiny, then fix one thing at a time.

Three ideas worth keeping

• If you do not write down what a term means, such as “the price the customer actually pays”, the agent will guess, and often guess wrong.
• Version one is there to be measured, not to be good, so start with one item in one city and widen it later.
• Each real mistake becomes a written rule and a new test case, so the agent does not repeat it.

What it means for your business

Before you ask anyone to automate a back-office task, write down how your best person does it and how you would check their work. Expect the first version to be poor, and judge the supplier on how fast it improves. Plan who reviews the agent’s flagged answers and who approves each new version once it is live.

What it means for software teams

Build a test set from past human answers before building the agent, and mix cheap automatic checks with a second AI that reviews the work. Read the agent’s step-by-step traces, because some failures, like an agent looping on image-heavy pages, only show up there. Change one thing at a time and fix the instructions before swapping the model.

What we found, and what we did

We saw the same pattern when we made a small AI model, one that runs on a laptop, reliable at one narrow task. On a sealed test set (questions kept apart so the model never trained on them), it went from 17% to 97.8%.

Rewriting the instructions did most of the work, taking it to about 87%, and a round of extra training added the rest. Most of the misses left were unclear definitions, not model errors: on inputs with one clear answer it scored about 99.6%, so our next step is clearer categories, not a bigger model.

The full story is in our field note From 17% to 97.8%: Making a Laptop-Sized AI Actually Reliable.

Worth watching

• 01:23: the headline result, 41% at version one and 97% at version eleven.
• 03:58: why every term needs a written definition, with the pricing examples.
• 10:22: the agent that kept zooming into images, and the simple rule that cut cost by more than half.