Harness Engineering: How to Build a Software Factory

In plain English

Dru leads product and design at Tessl, a company that sells tools for teams working with AI coding agents (software that writes and changes code on its own). In a software factory, agents produce everything that ships and engineers improve the factory instead.

He tracks progress on three measures: autonomy, how often a person has to correct the agent; automation, how far you trust its work without checking it; and quality, how good the end product is. The work of getting there he calls harness engineering, which means building the checks and feedback loops around the agents.

He argues the real reward is better quality, not just speed, because the backlog of fixes finally gets done.

Three ideas worth keeping

• Use three loops: quick, cheap checks while the agent works, heavier checks once a change is submitted, and a slower loop that studies past mistakes and stops them repeating.
• Move agent work onto shared systems, such as an issue tracker and code review, so every correction is recorded and can be learnt from.
• Watch for fewer human takeovers and review comments, and more changes started without a person.

What it means for your business

If a supplier promises an AI-built product, ask how they check the agents’ work and how mistakes feed back into the process. Dru is candid that this improvement work is unplanned and competes with shipping features, so budget time for it rather than expecting it to happen on the side.

Expect steady, small steps, not a jump straight to full automation.

What it means for software teams

Start with one place where work begins, one place where it is reviewed, and a shared library of written procedures for the agents. Expect more access and setup problems than you think before agents can work alone. Then spend most of your effort on the loops that catch errors and feed fixes back.

What we found, and what we did

Our own platform, Mibble, is built this way, as described in Why GitHub Wasn’t Built for AI Agents: agents write most of the changes, automated checks are the first reviewer, low-risk changes that pass every check merge on their own, and anything hard to undo waits for one person’s approval.

Over the last 30 days that came to 1,406 merged changes, about 45 a day, with 91% merging without a person approving them; in September it was about 20 a day, and people approved about 70%.

A typical change is about 740 lines of code, many times the size of a typical pull request, and together they add up to over 1.5 million lines a month, directed by one operator.

Our slower loop is partly by hand: after a bad week we study the agents’ session records and turn failure patterns into fixes and written rules.

We’re more cautious than Dru about AI checks standing in for human review. In our blind test the best AI review setup caught only 4 of 11 known defects, so ours advises but cannot block. More in Our Journey to Automated Code Review.

Worth watching

• 02:30: the three measures, and the difference between autonomy and automation.
• 06:16: harness engineering and the inner, outer and meta loops.
• 09:16: why the work is hard: fast-changing advice, unplanned effort and hidden information.