The dark software factory: levels of autonomy, autocomplete to lights-out

“Dark factory” comes from manufacturing: a plant that runs with the lights off because nobody needs to see. In software it means a pipeline where specifications go in and working, shipped code comes out, and no person writes the code or reads it on the way.

It sounds like a slogan, and for most teams it still is. But the idea has become specific enough to be useful. There is a vocabulary of levels, at least one public team claiming to run at the top one, and enough data on agent capability to argue about when the rest of us get there. This article lays out the levels, what changes at each step, and what has to be true before you take the next one.

What “dark factory” means in software development

The clearest framing I know is Dan Shapiro’s essay “The Five Levels: From spicy autocomplete to the dark factory”, published on 23 January 2026. He borrows the self-driving car scale. At the bottom you drive and the car helps. At the top there is no steering wheel. Applied to code, the question at each level is simple: who writes the code, and who checks it?

A dark factory answers both with “not a person”. People still exist in it. They write the specification, decide what the product should do, and judge whether the result satisfies users. They just do not sit inside the loop that turns a ticket into a merged change.

The six levels, from autocomplete to lights-out

Shapiro’s levels, in my paraphrase:

Levels of AI coding autonomy

After Dan Shapiro's five levels (plus level 0)

• Person
• Agent
• Automated check

Steps as text

Step	What happens
1. 0. Manual	Person — AI as search; you write everything
2. 1. Discrete tasks	Person — you hand off "write a test for this"
3. 2. Pairing	Agent — flow state, but you are not done
4. 3. Human reviews	Person — you read every diff the AI writes
5. 4. Spec and leave	Agent — you plan, it runs, you check later
6. 5. Dark factory	Automated check — specs in; software out; no human loop

Level 2 is where most “AI-native” developers sit today, and Shapiro’s warning about it is the most useful line in the essay: it feels like you are done, and you are not. Level 3 is where many teams land next, and it is often worse than level 2 for the people involved. The AI writes most of the code, and the engineer becomes a full-time reader of diffs. That is the review bottleneck everyone is complaining about.

Level 4 is the robotaxi: you write the spec, the agent runs for a long stretch, you look at the result later. Level 5 removes the “you look at the result” step. The checking still happens. It is just done by something other than a person reading code.

Here is the same scale as a table, with the gate I would want passed before moving up.

Level	Who writes	Who checks	Gate to the next level
0 to 1	Person	Person	You can describe a task well enough to hand it off
1 to 2	Person and AI together	Person	Your tests run in one command
2 to 3	AI	Person reads every diff	Tickets have acceptance criteria a script can check
3 to 4	AI	Person checks results, not diffs	The pipeline collects evidence itself; an independent model reviews
4 to 5	AI	Independent model plus scenarios	Deploys are reversible, risky areas are fenced, escaped defects stay low

Why level 3 is the trap

Level 3 is where AI-heavy teams get stuck, and it is easy to see why. Everyone already uses an agent to write code, so the code arrives faster. Nobody has replaced the human check, so every change waits in a review queue. The team is busier and the delivery date does not move.

Getting out of level 3 does not mean reviewing faster. It means changing what the review is made of. Instead of a person reading a diff and forming an opinion, the pipeline runs the build, the tests and the checks the agent registered, records the exit codes, and hands all of it to a second model that did not write the code. I go through that switch step by step in can you take people out of code review, and the evidence half of it in “all tests pass” is a claim.

Are agents capable enough for level 4 and 5?

The capability question has a reasonable public answer. METR measures the length of task, in human working time, that an agent completes half the time. Their 2025 paper found that this horizon doubled roughly every seven months over six years. Their later data (version 1.1, last updated September 2026) puts the doubling time since 2023 at about 129 days.

How long a task AI agents can complete

50% time horizon, minutes of human work, frontier model at each date

Data as a table

	50% time horizon
2019	0.1 min
2023	4 min
Feb 2025	60 min
Aug 2025	203 min
Nov 2025	293 min
Feb 2026	719 min

From a few seconds in 2019 to about twelve hours of human work for the best model in February 2026. METR notes that its current task set cannot measure reliably above 16 hours, so the most recent points are less certain than the early ones. The trend is still clear: a single agent run can now cover a day of a person’s work, which is what level 4 needs.

Two caveats keep me honest. A 50% success rate is not good enough to ship without a check, which is why the reviewer matters more than the writer. And benchmark tasks are cleaner than your codebase, so run your own evaluation before you trust a number.

A public example of level 5: StrongDM’s software factory

StrongDM published its approach at factory.strongdm.ai. The team was formed on 14 July 2025, and Simon Willison describes it as three people. Their two rules are blunt: code must not be written by humans, and code must not be reviewed by humans.

What replaced review is the interesting part:

• Scenarios: end-to-end user stories, often kept outside the codebase, so the coding agent cannot simply write code that passes its own tests.
• Satisfaction: a probabilistic measure of the fraction of runs through all scenarios that likely satisfy the user, often judged by an LLM rather than a strict pass or fail.
• A “digital twin universe”: behavioural clones of the third-party services the software depends on, such as Okta, Jira and Slack, so agents can test against realistic systems without touching real ones.

Willison’s reservations are worth repeating. StrongDM suggests that a factory spending less than $1,000 a day on tokens per engineer has room to improve, which he puts at roughly $20,000 a month per engineer. And he asks the question everyone should ask: how is no human review sensible when LLMs make inhuman mistakes? Their answer, as far as I can tell, is that the checking moved from reading code to proving behaviour. That is the right direction. Whether it holds for your product is something you have to measure.

What the public data says about review placement

My own public benchmark (published at bulava.app/benchmark.json) is one task, so treat it as a hint. The same game-rebuild task ran three ways: Claude Code alone, Claude with a review at the end, and two models writing independent positions before any code.

Where the second opinion sits changes the run

Hours for the same task, three setups

Data as a table

	Value
Claude Code alone	3.7 h
Claude plus one end review	4.2 h
Two positions before code	1.5 h

The end review halved the bugs (about 13 to about 7) but made the run longer. Moving disagreement to the start, before code existed, cut the run to 1.5 hours and found no broken mechanics. For a dark factory this matters: an independent check is not one gate at the end. It belongs at the plan as well as at the diff.

How to move up one level without breaking things

A bad approach jumps from level 2 to level 5 because a slide said so. It removes human review before anything replaces it, and the first escaped bug ends the experiment. Another bad one stays at level 3 forever and calls it “responsible AI”, while engineers spend their days reading generated diffs.

A good approach moves one product, one level at a time, and lets evidence decide:

1. Measure where you are. For two weeks, log who writes each change, who checks it, and how long it waits.
2. Make tickets checkable. If a script cannot tell whether a ticket was done, no model can either.
3. Make the pipeline collect evidence: build, test and scenario output attached to every change, run by the pipeline rather than reported by the agent.
4. Add an independent reviewer with the power to reject. Track how often it rejects. If it never does, it is decoration.
5. Fence risky areas. Payments, auth, migrations and deletions keep a human signature until escaped defects there are near zero.
6. Make deploys reversible, then start merging without human review on low-risk changes. Widen the scope as the escaped-defect rate allows.

At level 4 and 5 the team changes shape, which I model in from nine developers to one. The short version: people stop being the loop and start owning the intent.

Questions

What is a dark factory in software development?

A pipeline that turns specifications into shipped software with no person writing or reviewing the code. People still write the intent and judge the outcome, but the build, test and review loop runs without them.

What are the levels of AI coding autonomy?

Dan Shapiro’s scale runs from level 0, where AI is a search tool, through pairing at level 2 and human review of AI code at level 3, to level 4 (write a spec and leave) and level 5, the dark factory.

Is any company running a dark software factory?

StrongDM publicly describes a software factory with two rules: no human-written code and no human-reviewed code. It is a public claim by the company, not an independent audit.

Is it safe to ship code no human has reviewed?

Only if something better than a quick human read replaces it: evidence collected by the pipeline, an independent reviewer model, end-to-end scenarios and reversible deploys. Keep a human signature on payments, auth and data migrations until the numbers say otherwise.