Working notes — 15 October 2026
Coding-agent harnesses often implement a search loop: propose a change, evaluate it with tests or other feedback, retain or reject it, and try again. That is recognisably related to search-based software engineering (SBSE). Not every agent workflow is an explicit optimiser, and the analogy does not imply that the problems are solved.
These notes separate established search ideas, practical hypotheses for coding agents, and questions that still need evidence.
| Observed harness problem | SBSE lens | Intervention worth testing | What remains uncertain |
| Repeated small patches without structural progress | Local optima; neighbourhood design | Detect stagnation; restart or perturb; change edit granularity | Which stagnation signals predict genuine architectural blockage? |
| All tests pass but the code gets harder to change | Fitness specification; multi-objective search | Track change cost and structural indicators alongside correctness | Which proxies predict future maintenance cost rather than just rewarding neat-looking code? |
| Multiple agents reach similar solutions | Premature convergence; diversity | Compare structurally distinct plans, not just independently sampled responses | How should architectural diversity be measured? |
| Success on known tests but failure on new requirements | Overfitting; generalisation | Hidden tests, mutation testing, holdout tasks, follow-on change tasks | What evaluation predicts performance on future work? |
| A needed refactor cannot be made in one passing step | Representation and search operators | Allow isolated branches with temporarily failing intermediate states, then validate the final state | How should the harness budget and safely manage these excursions? |
| Tests and model calls dominate runtime | Expensive fitness evaluation | Cheap checks first; selective full-suite runs; budget-aware search | When does screening discard the best candidates? |
My harness makes small changes and merges them only when every test passes. It works well for incremental work, but when a feature needs a structural change, the agent can spend a long time patching around the existing architecture.
My current intervention is to stop, make or request a larger structural change, tolerate the temporary breakage in an isolated working branch, then restart the harness to repair failures with small passing tasks. Rewriting a repeatedly failing task also works better than simply retrying it.
This resembles perturbation followed by local search. It is an observation, not yet a controlled result.
Control for model, budget, repository, task difficulty and evaluation harness. Report negative results. Avoid treating a proxy metric as the objective itself.
Anthropic's Automating eval design and hillclimbing with Claude describes iteratively improving prompts, skills, tool descriptions and harness code against evaluations. See also the eval-hillclimb implementation notes.
This is a concrete example of the SBSE connection: candidate generation, evaluation, selection and repeated local improvement. Held-out evaluations help check generalisation, but they do not by themselves solve architectural quality or the problem of reaching improvements that require temporarily worse intermediate states. It would be worth comparing its search operators and stopping rules with established SBSE approaches.
These are starting points, not claims that each technique has been validated for LLM coding agents.
Related Substrate notes: Harness engineering glossary and Learnings from harness engineering.
The point is not to rename harness engineering as SBSE. It is to find out which decades-old lessons save us from repeating the same mistakes.