Skip to content
Chat with AI Agent
Foundations

Harness Engineering: Same Model, Better Results, and the Scaffolding You Can Delete

Orange ITS — AI engineering team 10 min read

A model upgrade lands, your agent behaves differently, and half the rules in your prompt now look like superstition. Somebody added each one after a bad week. Nobody remembers which ones still matter.

That is the practical problem behind harness engineering, the craft of improving an agent by changing everything around the model while the model stays put. The results can be large, and the vendors who sell the tooling have started saying, in public, that much of it is temporary.

If you want a team that builds this around your own systems and reviews it as models change, that is the core of our forward-deployed AI engineering work.

The honest summary is that harness work can move results a long way, that the best-known number is weaker evidence than it looks, and that you should build most of it so you can delete it later. This article covers what the evidence shows, what a gain might be worth in money, and how to decide which scaffolding stays.

Scope note: the six parts of a harness are covered in agent harness architecture. This piece is about what changing them is worth and when to remove them.


What Changed When Only the Harness Changed

The best-known case comes from LangChain. In a post dated 17 February 2026, Vivek Trivedy describes taking a coding agent from 52.8% to 66.5% on Terminal-Bench 2.0, a gain of 13.7 points, with gpt-5.2-codex held fixed for every run. The post reports that the agent moved from just outside the top 30 of the leaderboard into the top 5.

What did they change? According to the post, the list includes:

  • A build-and-verify loop in the system prompt, aimed at an agent that re-read its own code, decided it looked fine and stopped.
  • Middleware that intercepts the agent before it exits and prompts one more verification pass against the task.
  • A map of the working directory and available tools, supplied at the start.
  • A warning about time budgets.
  • Loop detection that counts edits to the same file and suggests rethinking after repeated attempts.
  • A schedule of reasoning effort that is high at planning, lower while building and high again at verification.

Most of that is not exotic. These are the checks a careful colleague would do on their own, written into the system so the model cannot skip them. The vocabulary is also worth noting. Elsewhere in this article we call these pieces scaffolding, because they hold something up while it is being built.


How Far to Trust That Number

Say it plainly: this is one vendor’s run on one benchmark. LangChain sells agent tooling. As far as the post describes, the harness was tuned on the same benchmark it was scored on. LangChain gives no trial count per configuration, and it gives no breakdown of how many points each change contributed. The only isolated comparison it reports concerns reasoning effort.

The benchmark has 89 tasks, so 52.8% is about 47 tasks and 66.5% is about 59. That is roughly 12 more tasks passed. Our own rough arithmetic, which assumes one run per configuration and ignores correlation between tasks, puts one standard error on that gap at around 7 points, which makes a 95% range of roughly 0 to 28 points. A gap of 13.7 is then suggestive and short of conclusive. Treat that estimate as ours rather than LangChain’s.

The most useful part of the post for a buyer is what its authors say about their own work. They write that “changes that overfit to a task are bad for generalization and can lead to regressions in other Tasks.” They also write that “these guardrails will almost surely dissolve over time.”

So the finding is modest and still valuable. Harness changes can matter a lot, a vendor with an incentive to say so has shown one case, and the same vendor expects its own additions to become unnecessary. Scores from different Terminal-Bench versions are also not comparable, so do not set this figure beside numbers from later releases.


What a Harness-Only Gain Might Be Worth to You

Benchmark points are not business outcomes. There is no published test of whether a gain on a coding benchmark carries over to invoice matching, ticket triage or document handling. What follows is arithmetic, and it is illustrative.

Take a workflow of 200 cases a month.

Unattended successCases handled without reviewCases needing a person
Before52.8%about 106about 94
After66.5%about 133about 67

The difference is about 27 cases a month. If each escalated case costs 20 minutes of staff time at CHF 60 an hour, that is CHF 20 per case, so about CHF 540 a month, or about CHF 6,500 a year.

Three things sit around that figure. It ignores build cost, token cost and maintenance. It assumes the gain transfers, which nobody has shown. And it depends heavily on case volume: at 2,000 cases a month the same assumptions give ten times the saving, while at 40 cases a month the harness work probably never pays for itself.

Consistency matters as much as the average. Anthropic’s evals guidance notes that a 75% per-trial success rate falls to about 42% if the same task must succeed three times in a row. A process that has to be right every time needs a different bar than one where a person reviews each output.

Cost belongs on the same page. An ETH Zurich study of instruction files, arXiv 2602.11988, reports that context files did not generally improve task success and raised inference cost by over 20% on average. Added scaffolding is not free, and sometimes it buys nothing.


Scaffolding With a Shelf Life

Three sources describe the same pattern from inside the companies that make the models.

Anthropic. Lance Martin, writing for the Claude Platform team on 2 April 2026, says harnesses encode assumptions about what the model cannot do on its own, and that these assumptions grow stale as models improve. His example: Sonnet 4.5 tended to wrap up early when it sensed its context limit, so Anthropic added context resets. With Opus 4.5 the behavior was gone, and the resets had become dead weight. His advice is to strip the harness down and ask what you can stop doing.

The Claude Code team. In a fireside chat transcribed by Simon Willison, Thariq Shihipar said that removing examples from the prompt was extremely helpful, and that the earlier Opus 4-class models wanted many examples while the team now aims for fewer hard constraints, more context and fewer instructions overall. Cat Wu said the team now keeps a different system prompt per model. Thariq also said they are trending towards fewer tools. Cat described softening an absolute rule after finding that a statement true 90% of the time was wrong in the other 10%.

Vercel. In a post on its internal text-to-SQL agent, Andrew Qu describes replacing 17 specialised tools with one, a bash tool (alongside a SQL executor) for exploring a documented semantic layer. On five test queries, success went from 4 of 5 to 5 of 5. That is a tiny sample, and the author states the approach worked because the semantic layer was already well documented. A poor data layer, the post warns, would likely produce faster but still wrong queries.

The pattern across all three is consistent: rules, examples and tools added to cover a weakness tend to outlive that weakness. The cost of keeping them is token spend and constraints the newer model no longer needs. The Claude Code staff also said that the behavioral evals used to check these decisions are still being built up and do not cover everything, which is where the next section picks up.


Expiring and Durable Scaffolding

Not everything in a harness has a shelf life. Some parts exist because of what the model cannot do, and those shrink as models improve. Others exist because of what your business must be able to show, and those stay.

ScaffoldingLikely to expire?Reason
Context resets for early wrap-upExpiresBuilt for one model generation, dead weight on the next (Anthropic)
Long rule lists and worked examplesExpiresNewer models were said to need fewer (Claude Code team)
Absolute rules such as “always verify”Expires or needs rewordingExceptions turn up as models change (Claude Code team)
Pre-completion checks and loop detectionProbably expiresLangChain expects its guardrails to dissolve, though it gives no per-change effect
Long tool lists and tool-selection rulesShrinksClaude Code is moving to fewer tools, and Vercel reported a better result with fewer
Permission boundaries and sandboxingDurableA prompt is not a control, and access limits decide what an action can reach
Narrow credentials, audit logs, approval for irreversible actionsDurableAccountability does not depend on model quality
Your eval set of real casesDurableIt is the test that tells you whether deleting something is safe
Written definition of a correct resultDurableAny model needs a target

The sort order in that table is our reading of the sources. None of them publishes such a list as a finding, so adapt it to your own workflow.


A Skeptic’s View

Not everyone agrees the harness matters much. Fabio Akita, in an August 2026 blog post, argues that for a frontier model the harness is noise, and that large swings between setups usually reveal plumbing bugs such as a transport fault rather than a real harness effect. His benchmark is self-run and has not been independently reproduced, and he promotes his own tools, so we treat his numbers as unverified.

His warning is still useful. Before you credit a harness change with a gain, check the logs for something boring: a failed API call, a truncated response, a tool that errored silently. A big jump or a big drop is a reason to look for a bug first.

On the other side, the summary of an ICML 2026 workshop paper, “How good is your harness?”, by Jiwoo Han and Yuekai Sun says the harness matters as much as the model, that the gain from choosing the best harness is comparable to the gain from choosing the best model, and that the effects vary by model.

The disagreement does not need resolving to act on it. If the effect is large, you want to capture it. If it is small, you want to avoid paying to maintain scaffolding that does little. Both views point toward measuring on your own cases.


Harness Engineering Versus Prompt Engineering

Prompt engineering is a subset. A prompt shapes what the model is told, whereas a harness also decides which tools exist, what is checked before the agent may finish, how loops are interrupted and what the agent is permitted to touch. If you are weighing a bought copilot against your own harness, custom AI harness vs SaaS copilot covers that choice.

That difference changes where you put effort. A sentence in a prompt asking the agent to verify its work is a request, and a middleware step that blocks completion until a verification pass has run is a mechanism. The second kind can be tested and logged. It is also easier to remove cleanly, because it sits in one place instead of being spread through paragraphs of instructions.

For a technical lead, the useful habit is to ask of any rule whether it lives in the prompt or in code, who wrote it and what failure it was meant to prevent. A rule with no recorded failure behind it is a candidate for deletion.


Remove One Rule, Rerun the Evals

Here is the test we recommend. This is Orange ITS’s recommendation, and no source describes this exact procedure. It follows the spirit of Lance Martin’s advice that harness assumptions should be tested, and of the Claude Code team’s use of behavioral evals.

  1. Keep a set of real cases with agreed pass marks. Twenty to fifty drawn from past failures is a workable start, and our guide to testing and evals for AI agents shows how to build it.
  2. Pick one rule, example, tool or check and remove it.
  3. Rerun the set on the new model, several times if results vary between runs.
  4. Compare the pass rate. If it dropped, restore the rule and write down why it matters. If it held, leave it out.
  5. Repeat for the next candidate, one change at a time, and again at every model change.

Two cautions apply. A set that is too small cannot detect a small drop, so a rule that looks harmless may be guarding a rare case. And the eval set itself must be trustworthy, which means checking outcomes in the real system rather than reading the agent’s own claim that it succeeded.

Build the scaffolding with this in mind. Put each rule in one named place, record the failure it answers and add a review date. Deletion becomes a routine edit instead of an archaeology project.

The scaffolding that survives each pass is the part worth owning: the eval set, the permissions, the logs and the written standard of a correct result. The rest was only ever holding the model up until the model could stand on its own.

Frequently asked questions

What is harness engineering?

Harness engineering is the work of designing everything around a language model that decides how an agent behaves: its instructions, tools, checks, retries and limits. The model stays the same while the surrounding code and configuration change. LangChain uses the term for a run in which only these parts changed and the benchmark score rose from 52.8% to 66.5%.

How is harness engineering different from prompt engineering?

Prompt engineering tunes the wording of instructions, while harness engineering covers the whole system around the model, including tools, verification steps, loop detection and permissions. A prompt is one component of a harness. The practical difference is that harness changes can be enforced in code and tested against outcomes, which a sentence in a prompt cannot guarantee.

Will a better harness raise my agent''s success rate by 13 points?

Not by any rule we can cite. The 13.7-point gain comes from LangChain on a coding benchmark, tuned on that benchmark, by a company that sells agent tooling. No source tests whether benchmark points carry over to invoice matching, ticket triage or other business workflows, so measure the effect on your own cases before planning around it.

Which parts of a harness will become obsolete as models improve?

The parts that compensate for a model weakness are the likeliest to expire, such as context resets, long rule lists, worked examples and verification nudges. Anthropic and LangChain both say this about their own harnesses. Permissions, audit logs, narrow credentials and your eval set tend to stay, because they exist for accountability rather than model limits.

How do I know which rule to delete?

Remove one rule, rerun your eval set on the new model and compare the pass rate. If it does not drop, the rule was dead weight and can go. This is Orange ITS's recommendation rather than a published finding, and it only works if you already have a set of real cases with agreed pass marks. Anthropic says harness assumptions should be tested in this spirit.

Insights

Put these ideas to work

A 30-minute call is enough to find out whether an AI agent fits your workflow — and what it would return.