How We Run Our Software Factory (and What Broke)

Our own platform, Mibble, gets built every day by a software factory.

Last month it shipped over 1.5 million lines of code across about 1,300 changes. One operator directs all of it.

AI agents write most of the changes. Automated checks, not people, are the first line of review. A person steps in for the changes that matter most.

It’s been running since mid-June. Here’s how it works, what we measured and what went wrong.

What a Software Factory Is

Strip away the jargon and it’s simple.

Every change, whoever wrote it, goes through the same automated gate before it reaches the main code. Low-risk changes that pass every check merge on their own. Changes that touch things you can’t easily undo, such as payments, customer data or database structure, wait for one person’s approval.

That’s it. A pipeline, a gate and a short list of things a human still decides.

We judge it by one question: would a senior engineer merge this? Not by how many tests went green.

The Rules We Run By

Checks first, people second. Tests, type checks, linting and the build run on every change and have to pass at every risk level. A failing test can’t hallucinate.

Approval by risk, not by habit. Changes are sorted by how much they could break, how hard they are to undo and how new they are. Only the irreversible level needs a person. Everything else merges on evidence.

One human approval, never two. In a small team, a two-person sign-off is the same person clicking twice. It deadlocks.

Nothing clears its own work. The thing that wrote a change can never be the thing that clears it. That includes a “review panel” of helper agents the author started itself. More on this in how we automate code review.

Big changes wait for a person. Above a certain size, a change can’t merge on its own. Work stacks up in steps on trunk, with a single queue landing them in order. We wrote up why we don’t branch and the tooling that makes stacking work.

Guardrails, not instructions. If a rule matters, the tooling enforces it. Writing it in the docs isn’t enough.

Gate what’s reliable, advise what isn’t. An AI reviewer can comment on a change. It can’t block one until it has proved reliable.

Measure before building. Changes to the agents’ own tools get benchmarked on speed, cost, accuracy and mergeability. The benchmark includes tasks where the plain approach should win, so a rigged result shows up.

The Numbers

Lines of code aren’t how we judge the work. Each change is judged by whether a senior engineer would merge it. Lines are how we show capacity: how much one operator can direct through the factory. Counting changes alone undersells it, because our changes are big.

A typical one touches about 740 lines of code across a dozen files. The median pull request Graphite measured is 47 lines. So we count lines of code too, leaving out generated files, lockfiles, data and docs.

Month	Code lines changed	Merged changes	Merged with no person approving
March to May (average)	about 150,000	(before the pipeline)	none
June (pipeline live on the 12th)	471,000	484	47%
July	702,000	1,087	35%
August	291,000	538	32%
September	465,000	621	31%
October	2,030,000	1,298	88%

The jump in October didn’t come from adding more agents. It came from letting low-risk changes merge on evidence instead of waiting for a person. Over the 30 days to 6 November, 91% merged with no person approving them. In the most recent week, every one of the 307 changes was cleared by the pipeline alone.

For context, the median developer merges about two pull requests a week. One operator now directs about 1,300 changes a month, each typically 16 times that size. Every one goes through the same checks.

We also tested AI code reviewers properly before trusting one. Seventeen review setups, run on the same 29 real changes that had already merged. Nine of those changes held 11 defects we knew about because they were fixed later. Every finding was judged blind.

The best setup caught 4 of the 11. None caught the deep, cross-file bugs. Noise decided it: false alarms ranged from 0.8 down to 0.1 per clean change. Giving the chosen reviewer written guidelines lifted its catch from 2 to 6 at the same precision (with the caveat that some guidelines came from incidents in the test set).

The bake-off also turned up real defects already on main.

So the AI reviewer advises. It doesn’t decide. High-risk changes still get a deeper review.

What Went Wrong

This is the part that matters.

A person was quietly covering for a failure. Auto-merge was built and switched on, but constant churn kept knocking it over, so someone kept merging by hand. It looked like a safeguard. It was an alarm. We wrote that one up in We Were the Bottleneck.

An unreliable reviewer jammed the pipeline. We promoted an AI reviewer to a hard gate. Under load it hung or returned unreadable output, and blocked good changes. We did this twice before turning it back to advisory.

Agents claimed audits they never ran. Changes got marked “audited” with no audit behind them. We replaced the honour system with a step that actually runs the audit and records it.

A written rule was broken by the agent that wrote it. One agent wrote the rule “never work in the shared copy”. Then it broke that rule itself for convenience, and another agent’s work collided with it. Only a hard check in the tooling stopped it.

A change passed every check and still broke older devices. A database change cleared all 61 required checks, then crashed older devices on launch. No check used a real, older database. The checks proved the pipes were connected, not that anything flowed through them.

A shared, busy machine turned a 5-minute gate into 53. Many agents, the desktop app and CI all on one machine. We’d designed for ideal resources, not real ones.

“All green” changes that never merged. A change re-pushed faster than CI could finish sat at the back of the queue forever, with every visible check green on an old version.

Two required checks contradicted each other. Running the formatter to pass one broke the other. Before a check becomes mandatory, a change has to be able to pass every mandatory check at once.

A two-person approval rule crept in and deadlocked a solo operation. We took it out.

After one bad week, we noticed every failure had broken a rule nobody had written down. So we wrote them down. Every pipeline change now gets checked against them.

What This Means for Your Business

If you’re weighing up something like this, the lessons travel.

1. Start with deterministic checks. They’re cheap, they don’t guess, and they’re the floor everything else stands on.
2. Spend human attention by risk. Put people on the irreversible changes and let evidence clear the rest.
3. Never let the work clear itself. Not directly, not through helpers it started.
4. Test AI reviewers blind before you trust them. Ours caught at best 4 of 11 known defects, so advisory is the right default.
5. Enforce, don’t instruct. If an agent can break a rule for convenience, it will.
6. Test real conditions. Old data, busy machines, changes arriving faster than checks finish.
7. Measure before you build. The pipeline you imagine isn’t the one you have.

We went from about 150,000 lines of code a month to over 2 million, and from a person approving 70% of changes to under 10%, with one operator directing it.

None of that came from a clever model. It came from rules, checks and being honest about what broke. If you’re running AI agents on tools built for people, that’s where we’d start.

---

Thinking about a software factory of your own? We’ve made the mistakes already. Let’s talk.