Brendan Cody-Kenny, October 2026
I ran AI coding agents unattended on my own codebase, and over 51 days they merged 229 changes. The agents merged most reliably on tasks with a clear written spec and a test a machine could run. About four in ten tasks merged on the first try. Retrying a task without changing its spec rarely helped, and most tasks that merged late did so after I had changed or re-queued them. Which AI model wrote the code mattered less than I expected. I would not leave anything unattended that no test can check. The main open problem is that a run of small changes, each passing the tests, gets stuck at a design that no single small change improves.
I keep a list of tasks. Each one is a short written spec plus one command that passes when the task is done. Overnight, and whenever the list has work, an AI agent takes a task and writes code on its own branch. The change then runs through the test suite, first where the agent worked and again on a clean machine, and it merges only if both runs pass. If they fail, the task is retried, and after a couple of failures it is set aside for me. Every attempt goes into an attempt log, a file with one line per attempt saying which task, which model, what happened and how long it took. The harness is everything around the agent, meaning the task list, the test suite, the retries and the log.
1. The agent is only as good as the test suite. Ten changes the agent marked as done failed on a clean machine, because it had skipped the unit tests it was told to run. Across the whole log, that clean rerun caught 18 of 247 changes that the first run had passed.
2. An agent given its test in the prompt can pass it without fixing the problem. Mine were given theirs. Some other systems write the tests before the agent starts and store them where it cannot open them, and that is my next experiment.
3. Only give it work a test can check. I kept anything visual for myself from the start.
4. Rewrite a failing task instead of retrying it. Retries of an unchanged task merged about one time in five, against four in ten for first tries. Most late successes came after I had changed or re-queued the task.
5. When the harness breaks, it looks as if the task failed, or worse, succeeded. A provider outage made sixteen tasks fail in 34 minutes. A merge script once dropped the real change and recorded success. A stuck lock stopped everything for 34 hours.
6. Small tasks are not cheap. Every task runs the full test suite twice, and a typical successful attempt took about 23 minutes.
7. Per-task tests do not measure the design. Each change passes the tests it touches, so a change that makes the structure worse passes too. Big structural changes still needed me, working outside the task list.
8. Nobody looks at what merges unless something notifies them. For a month, 36 set-aside tasks went unreviewed because nothing notified me. A summary now goes out after each run.
9. Log every attempt. One line per attempt answered "has it run, and when?" in seconds, where it had taken an hour, and every number here comes from that log.
10. Check numbers with a script. Three confident claims in my own notes turned out wrong when a script checked them, including the total hours.
11. Comparing AI models fairly needs random assignment. Which model got which task depended on the time of day, so the log cannot show which model is best. The log does show that two of the five were worse than the other three.
I want as much of my development as possible to go through the harness unattended. Each problem below limits how much can.
Getting stuck in a local optimum. Each task is one small change, and it merges only if every test passes. A better design that needs several changes in a row, some of which break things along the way, cannot be reached one passing change at a time. The agents patch around the old structure, and the codebase stays at the best design that small passing changes can reach. Lesson 7 is one result of this. I end up pausing the harness and asking for large structural changes that I expect will break a lot of things, and then I run the harness to clean up.
Writing tasks takes my time. Most tasks that failed and later merged did so after I rewrote them. A failed attempt shows where the spec was ambiguous, where the test checked the wrong thing, or where the task was too big. I find that out only after an attempt has failed, and the rewrite is by hand. Nothing checks a spec for ambiguity before the first attempt.
The same logic gets reimplemented and scattered. An agent will write its own version of something the codebase already has, such as a second function that opens the same data file, in place of using the shared one. The tests pass either way. Over many changes one job ends up done in several places, each slightly different, and a later fix that misses one of them leaves the bug in place.
Pass or fail is the only signal. An attempt that got most of the way scores the same as one that did nothing. The next attempt gets the failure output and no measure of how close the last one came.
One candidate per attempt. Each attempt produces one change, which merges or fails. Some systems generate several changes for the same task and keep the best, which costs several times as much per task. I have not tried it.
The attempt log is not fed back. I analyse the log by hand. No part of the harness uses it to change the prompts, the task list or the harness itself.
Understanding falls behind. The agents merged 229 changes I did not write, and I have no measure of how much of that code I understand.