250 subagents, 300 uncommitted files, and a 92% wrong answer
Handing work to more agents without a limit looked like speed. It was 250 agents editing the same shared files and grading the wrong source of truth, together, at once.
The job I handed over
- Use a large number of helper agents working at the same time to review a big set of candidate items and remove unsuitable ones quickly.
What happened
- The task was spread across roughly 250 helper agents working at the same time with no limit.
- They collided on shared files, leaving around 300 files in an uncommitted, half-edited state.
- The audit itself was graded against the wrong source of truth, and the resulting recommendations were wrong about 92% of the time — roughly 151 valid items were confidently marked for removal that shouldn't have been.
How I checked it
- Recovery required manually reverting all ~300 uncommitted files by hand before any of the audit's real conclusions could even be evaluated.
- Traced the failure to two problems that made each other worse: no limit on agents editing shared files at the same time, and no check that the review was using the correct reference information before it started grading.
What it took from me
- A full afternoon of manual file-by-file recovery to get back to a clean state.
- Set a hard cap of 16 helper agents working at the same time on any job that touches shared files, going forward.
What I took from it
- More parallel agents isn't more throughput once they start stepping on the same files — it's just a faster way to produce a bigger mess, especially if the thing they're all agreeing on is wrong.
Try this
- Before handing a job to many workers at once, check that they aren't going to read or edit the same shared files at the same time.
- Set a firm limit on how many workers can use shared files or data at once — don't rely on 'it'll probably be fine.'
- Check the reference information the job is grading against before trusting the volume of output it produces.
Applies to any large parallel job — bulk data cleanup, mass content review, batch agent tasks — where workers share files, a database, or any other shared data.
Source
Fan-out incident lesson, internal ops log