No, That's Not a Software Factory

In plain English

A software factory is a setup where AI agents (software that carries out tasks on its own) write most of the code, and the team builds the pipeline around them. Ryan, an engineer at WorkOS, says most companies judge theirs by output, such as how many code changes the AI produced.

That number can rise while nothing useful ships. WorkOS’s first version, an agent in a sealed-off workspace turning prompts into code changes, was no better than engineers using AI on their own laptops. Things improved only when they built their wider work habits into the factory: planning documents, project tickets and bug reports, not just the coding.

Three ideas worth keeping

• Count outcomes, such as features delivered, defects and time to recover from problems, not the volume of code an AI writes.
• An agent that drafts the plan and the task list solves the blank-page problem, even when people have to cut back what it proposes.
• One shared connection to all your internal tools, with notes on how each is organised, ended up useful well beyond coding.

What it means for your business

If a supplier or your own team reports how much code AI is writing, ask what has shipped to customers and whether things break more often. The bigger gains came from automating the work around coding: planning, breaking work into tasks and sorting incoming bug reports.

What it means for software teams

Feed the agent the same signals your people react to, like finished tickets and new bug reports, so it can pick up the next piece of work itself. Invest early in giving agents a clear map of your internal systems.

Owning the infrastructure lets you study past agent sessions and fix the gaps they reveal, such as instructions that have gone stale.

What we found, and what we did

We learnt the output lesson the hard way. Our pipeline looked busy, but when we measured three days of it, about 60% of the automated test runs were wasted re-runs and a person was quietly merging by hand to keep it moving, as we wrote in We Were the Bottleneck.

Today our own platform, Mibble, merges about 45 changes a day, up from about 20 in September, and people approve under 10% of them instead of about 70%.

A typical change is about 740 lines of code, many times the size of a typical pull request, and together they add up to over 1.5 million lines a month, directed by one operator.

Ryan would rightly point out that those are output numbers. For outcomes, we lean on one question: would a senior engineer merge the change? (Stop Measuring AI by Test-Pass Rate). An automatic brake also pauses merging when too many recent changes have had to be undone.

Like WorkOS, we go back through the agents’ own session records after a bad week and turn what we find into fixes.

Worth watching

• 01:53: why counting AI-written code changes can hide whether the factory is working.
• 05:15: the agent picking up the next ticket on its own when the one before it is done.
• 16:05: the outcome measures WorkOS actually wants: shipping, customer impact, defects and recovery time.