Brendan Cody-Kenny, October 2026
Many ideas in harness engineering already have names in search-based software engineering (SBSE). Harness engineering is about two years old and its terms are still changing. Each row below gives one idea and its name in SBSE, in software engineering, in harness engineering and in agent factories, with a reference for each term.
The agent factory column uses the 2026 sense of the term, from Addy Osmani. "Software factory" also names Cusumano's Japanese factories and Greenfield and Short's model-driven factories (2004), and this table covers neither.
Strength says how closely the SBSE term matches the harness term. Equivalent means both name the same role and the words could be swapped. Analogy means the roles match and the mechanism differs. Weak means the terms do related but different jobs. No match means one field has no counterpart.
| Concept | Strength | SBSE / GP | Software engineering | Harness engineering | Agent factory |
|---|---|---|---|---|---|
| Pass or fail check on a change | equivalent | oracle | acceptance tests, test assertions | verifier, grader; the verifiable reward in RLVR | review gate |
| A score for how close a change is | equivalent | fitness function | mutation score, coverage, benchmark time, static-analysis score | reward, score, graded eval; a model-based grader or LLM-as-judge | no term |
| The overall loop | analogy | the metaheuristic | the iterative development process, such as Scrum | the harness, the scaffold; owns the loop, context and stop conditions | the loop: gather context, take an action, check the result, go again until some condition is met |
| The thing being changed | analogy | the representation (genotype), AST, representation | the diff | candidate patch | no term |
| How a new change is made | weak | mutation, crossover | hand edit, refactoring | an LLM (the policy) producing a rollout; one rollout contains many edits and test runs | the action step of the loop |
| Trial and error inside one attempt | analogy | memetic algorithms, local search, hybrid metaheuristics | debugging | the agent's own edit and test loop | no term |
| Changes made using what the code is for | no match | none | a developer | an LLM given the repository's code, tests and docs | no term |
| One attempt and its log | weak | individual, variant (GP) | commit, pull request | rollout, trajectory | no term |
| Several attempts at one task at the same time | analogy | population | set-based development (Lean), competing prototypes | K independent attempts, best-of-N | many harnessed loops running at once |
| Accepting a change | analogy | selection, elitism (always keep the best so far) | code review, protected main | verifier-guided selection, rejection sampling | merged after a review gate |
| A failed attempt | equivalent | failed evaluation | failed CI run | failed rollout | no term |
| When to stop | equivalent | termination criterion, evaluation budget | timebox | step limit, max turns, budget | no named stop rule |
| Picking which model to use | analogy | algorithm portfolio, hyper-heuristics | assigning a ticket to the developer best suited to it | model routing, cascade (a cheap model first, a stronger one on failure) | no term |
| Trying differently after a failure | analogy | restart, perturbation, simulated annealing (accept a worse candidate for a while) | rewrite it | resample; retry with feedback | no term |
| Changing one thing at a time and keeping it only if the score improves | equivalent | hill climbing, local search | no term | hillclimbing a prompt, skill or model setting against an eval | no term |
| Stuck where no small change helps | no match | local optimum, premature convergence | no settled term | no settled term | no term |
| Passing the check while still wrong | equivalent | overfitting to the test suite; plausible patch (APR) | teaching to the test | reward hacking, specification gaming, verifier gaming | no term |
| What an attempt starts from | analogy | seeding (start from known programs) | boilerplate | initial repository state, system prompt, context | a queue of work |
| The task description | analogy | specification; examples (PBE) | requirements, acceptance criteria | task spec, instruction | a queue item |
| Improving an existing program | analogy | genetic improvement | refactoring, optimisation | automated program repair; SWE-bench-style repair | no term |
| No test can check it | equivalent | the oracle problem | "you have to look at it" | unverifiable or hard-to-verify domains | light factory, where people check what the loop cannot |
| One task | equivalent | problem instance | ticket | task, environment instance | a queue item: an issue (OpenAI Symphony), a scenario (StrongDM) |
| Where changes are tested | equivalent | evaluation environment | dev box | sandbox, environment | the sandbox the loop runs in, then the review gate |
| Chance of success within k tries | equivalent | success rate over k independent runs | no term | pass@k, BEST@K; pass^k, the chance that all k tries succeed | no term |
| Cost of testing a change | equivalent | evaluation cost | CI minutes | rollout cost | verification cost |
| Using past runs to improve the setup | weak | fitness landscape analysis | retrospectives | Retrospective Harness Optimization | no term |
| Rerunning a fixed set of tasks to measure the harness or model | equivalent | benchmark suite of test problems | benchmark | eval suite, as in SWE-bench | no term |
| Evals of new abilities versus evals that guard old ones | analogy | no term | new tests versus the regression suite | capability evals and regression evals | no term |
| An eval too easy to show a difference | no match | no term | no term | eval saturation | no term |
| Splitting eval cases into ones you tune on and ones you check on | equivalent | training and test sets | no term | train and test split | no term |
| A big change made in small passing steps | weak | no settled term; genetic improvement edits are small and the program keeps passing its tests between them | parallel change (expand, migrate, contract), strangler fig, long-term refactoring | long-running agent work with a progress file and incremental commits; long-horizon whole-repository migration | no term |
| Grow, then clean up | analogy | diversification and intensification in tabu search | expansion and consolidation phases (Foote and Opdyke; Big Ball of Mud) | no settled term | no term |
| A count that may only go down | analogy | elitism, and never accepting a worse result | ratchet, as in Notion's eslint ratcheting system | no settled term | no term |
| A test for architecture rules | equivalent | fitness function | architectural fitness function, a term borrowed from evolutionary computing | verifier, rarely applied to structure | no term |
| Many agents running at once | analogy | island model (separate populations that swap candidates), parallel evolutionary algorithms | continuous integration pipeline | orchestrator, as in OpenAI Symphony | the factory: many loops taking work from one queue, with every change passing a review gate |
| Where people review | analogy | interactive evolutionary computation | code review | human in the loop | light factory (humans in the loop) and dark factory (nobody looks at the code); Shapiro's scale ends at the dark factory |
| How much to leave unchecked | weak | no settled term | no settled term | no settled term | back pressure: a loop gets only as much autonomy as can be checked cheaply and reliably |
| Code nobody understands | analogy | bloat, introns (code with no effect on the output); evolved programs are often unreadable | technical debt, dead code | no settled term | comprehension debt |
| Tests the agent cannot see | equivalent | held-out test set, as in training, validation and test sets | independent acceptance testing | held-out eval | "the writer never grades the work" (addyosmani/factory); sealed tests the builder never sees (DUBSOpenHub/dark-factory); scenarios held out like a model-training holdout set (StrongDM) |
A harness has two nested loops and an evaluator. The outer loop chooses a task, a model and a budget, and then retries, merges or sets the task aside. The inner loop is the agent inspecting, editing and testing. The test suite checks each candidate and returns the result. In a Scrum team, people run the outer loop by hand, choosing the next ticket, assigning it and deciding whether it is done. A harness automates that part of the development process. The inner loop is already a search, so the outer loop picks between searches. The closest SBSE work for the outer loop is algorithm selection and hyper-heuristics.
Genetic programming uses a cheap random mutation and runs thousands of evaluations. An agent writes a patch from the repository's code and tests, at far higher cost per candidate, and a harness can afford a handful of evaluations per task. With so few evaluations, the result depends on the agent and the tests.
Links were checked on 2026-10-15. The harness and agent factory terms are recent, so date them wherever you reuse them.