We know that the harness influences an AI coding agent’s performance. But is it possible to measure how good a harness is on its own? The answer is more complicated, and more interesting, than it first appears.
Imagine two AI coding agents using exactly the same language model.
Both receive the same bug report, work on the same repository, and have access to the same tests. Yet one manages to fix the issue while the other gets stuck repeatedly inspecting files, running commands, and eventually exhausting its context window.
The difference may have nothing to do with the model’s intelligence.
It could be the harness.
The harness is the software surrounding the language model that manages its interactions with the development environment. It provides tools, constructs context, executes commands, handles failures, and determines how the agent continues working.
As coding agents become more sophisticated, this surrounding infrastructure is attracting increasing attention.
But something about the way we benchmark it bothers me.
If the harness contributes to an agent’s performance, shouldn’t we be able to benchmark the harness separately from the model?
And if we cannot, what does it actually mean to call one harness better than another?
What does it mean to benchmark a harness?
There are two fundamentally different questions hiding behind the idea of a harness benchmark.
The first is whether a harness works correctly.
Does it execute tool calls reliably? Does it preserve the necessary context? Does it handle errors, manage permissions, and recover from interruptions?
These are software engineering questions. We can test many of them without relying on a capable language model.
For example, we could replace the LLM with a scripted mock model that produces predetermined tool calls and responses. The harness would then be evaluated on whether it executes the expected behavior correctly.
Interestingly, a recent research project called Doc2Harness uses precisely this principle. It asks language models to construct agent harnesses, then evaluates those implementations using scripted mock-model interactions and deterministic tests.
Strictly speaking, Doc2Harness benchmarks the ability of models to build harnesses, rather than ranking existing production harnesses. Nevertheless, it demonstrates that harness implementation quality can be tested without depending on an LLM’s coding-task performance.
But the second question is harder.
Does a particular harness actually make an AI coding agent better at solving problems?
To answer that, we need a model.
After all, the harness does not reason about a bug or decide which algorithm to implement independently. It creates the environment in which a model can perform those tasks.
We can hold the model constant and vary the harness. That isolates the harness as the experimental variable.
But what we measure is not the harness’s absolute quality. We measure its contribution when paired with that particular model.
That distinction turns out to matter enormously.
The evidence is surprisingly contradictory
Consider two recent studies.
In Same Model, Different Harness, published in August 2026, the researcher compared two configurations of the same coding harness while keeping the model, tasks, and execution conditions fixed.
The improved configuration managed context more selectively and responded to repeated or stalled actions.
Under a constrained context window, the number of completely resolved SWE-bench Verified tasks increased from 43 to 72 out of 169.
That is a substantial improvement without changing the underlying model.
Now consider a different study.
In October 2026, researchers published What Does a Harness Buy? Tokens, Mostly.
They compared coding-agent harnesses including Claude Code, mini-SWE-agent, and OpenCode, holding the model fixed and repeating runs to account for variability.
On a 447-task evaluation involving two models, Claude Code and mini-SWE-agent performed within five percentage points of each other.
On a smaller, harder subset, changing the harness altered the outcome of about as many tasks as simply rerunning the same configuration.
Yet the cost per task differed by as much as threefold across harnesses.
How do we reconcile these findings?
One study demonstrates a substantial improvement in task resolution. Another finds that changing harnesses often makes little reliable difference to success rates, even while significantly affecting cost.
I think this apparent contradiction is the most interesting finding.
Perhaps we are asking the wrong question when we ask which harness is best.
A harness is only as useful as the constraint it removes
Think about why context management might help a coding agent.
If the agent repeatedly fills its context window with file contents, terminal outputs, and test logs, an effective compaction strategy could allow it to continue working instead of stopping prematurely.
In that environment, context management is extremely valuable.
But what happens if the model already has enough context to complete the task comfortably?
The same machinery might provide little improvement. It might even introduce additional processing, latency, or information loss.
This pattern appears in An Empirical Study of Harness Design for Coding Agents, published in September 2026.
Across four models and 176 matched experimental settings, the researchers investigated planning, tool interfaces, and context management.
They found that the usefulness of these components depended on the model and the available resources.
Planning helped weaker models complete tasks but served mainly as an efficiency improvement for stronger models. Predefined tools supported models with weaker command-line capabilities, whereas stronger bash-capable models could operate effectively with simpler interfaces.
These findings suggest something important:
Harness effectiveness is conditional, not universal.
The same component can be essential, redundant, or costly depending on the model, task, and operating conditions.
This reminds me of control systems in scientific instrumentation.
A feedback controller cannot be judged meaningfully without understanding the system it controls, the disturbances it must handle, and the performance requirements.
The controller that works best in one configuration may not be optimal in another.
Why should we assume that coding-agent harnesses behave differently?
Perhaps a harness leaderboard is the wrong abstraction
Suppose we build a leaderboard that ranks coding-agent harnesses using a single language model.
Harness A comes first. Harness B comes second.
What have we established?
Only that Harness A performed better with that model, on those tasks, under those evaluation conditions.
Now replace the underlying model.
What if Harness B moves into first place?
Nothing necessarily went wrong with the benchmark. The experiment may have revealed an interaction between model capability and harness design.
In fact, this is exactly why a fixed-model comparison, while useful, is not sufficient to establish universal harness superiority.
I would find a different type of benchmark much more informative.
Test several models across several harnesses, under controlled conditions, and repeat each combination enough times to distinguish meaningful improvements from random variation.
But instead of publishing another single ranking, report the marginal contribution of each harness.
How much does it improve task resolution compared with a minimal baseline? How much additional cost does it introduce? Does the improvement persist across different models? And under which conditions does it disappear?
This last question is particularly important.
We typically celebrate the addition of more sophisticated agent capabilities: planning, memory management, reflection loops, recovery mechanisms.
But perhaps we should also celebrate discovering that some of these mechanisms are no longer necessary.
A sufficiently capable model might perform better with a simpler harness that gives it direct access to tools without excessive orchestration.
In that case, removing complexity could be an improvement rather than a regression.
So, can we benchmark only the harness?
My answer is yes for implementation correctness, but not independently for task-solving effectiveness.
We can test whether a harness reliably performs its intended functions without involving a capable coding model.
We can also measure how changing a harness affects agent performance while keeping the model fixed.
But neither produces a universal measure of harness quality.
For that, we would need to understand how its performance transfers across models, tasks, and operating conditions. Even then, the result would remain conditional on the distribution we chose to evaluate.
This changes how I think about coding-agent benchmarks.
The model and harness are not simply two components whose independent scores can be added together.
They form an interacting system.
And the most valuable harness may not be the one with the most sophisticated architecture or the highest isolated benchmark result.
It may be the simplest harness that allows a particular model to perform reliably, without adding more overhead than the capability it unlocks.
Perhaps that is what we should be benchmarking.
Not which harness is the best.
But what does this harness actually enable the model to do that it otherwise could not?