Skip to content
Chat with AI Agent
Custom vs platform

Custom AI Harness vs SaaS Copilot: Where a Defensible AI Advantage Actually Sits

Orange ITS — AI engineering team 8 min read

Your team already has a copilot in the mail client, and a vendor has just offered you an agent platform on top. Somebody on the board asks the sensible question: if everyone can license the same models and the same assistants, where does an advantage come from? The loudest answer in 2026 is “the harness”, meaning everything between the model and the business outcome. It is a good answer to half the question.

A custom harness only pays off where it holds assets a competitor cannot license, and most of the generic parts of it are being sold as products. This article separates the two, shows what the evidence supports, and gives you a test to run before you pick a side.

If you want the vocabulary first, AI agent vs AI assistant vs copilot explains the autonomy levels behind each label.

If your workflows run through your own systems and rules, our custom AI development service is where we build that layer.


What a harness is, and what a copilot already gives you

A harness is the system around the model. It decides which context the model receives, which tools it can call, what it remembers between sessions, what checks run on its output, how it is tested and how the steps are orchestrated. The model supplies reasoning. The harness decides what that reasoning has to work with and whether the result is allowed through.

A SaaS copilot has one too, built by the vendor for the average customer. It brings a context pipeline into your mailbox and documents, a fixed set of tools, vendor guardrails and a vendor-run evaluation process. For generic work such as drafting, summarising and searching your own files, that is often enough, and it can be switched on quickly.

The real choice is whose layer runs your work, who controls its parts, and which parts you can carry out the door.


What changes to the layer around the model are worth

The best published evidence comes from LangChain’s February 2026 write-up. With the model fixed, the team changed only the harness: a build-and-verify loop in the system prompt, a verification step before the agent could finish, loop detection and a schedule for reasoning effort. Its coding agent moved from 52.8% to 66.5% on Terminal-Bench 2.0. That is a large jump with no model change. The anatomy of an agent harness goes through the six parts one by one.

Three caveats come from the same post:

  • It is one benchmark run by a company that sells agent tooling, and the post does not separate how much each change contributed.
  • The authors warn that changes tuned to one task can cause regressions on others.
  • They write that the guardrails will likely become unnecessary as models improve.

The result shows that the layer around the model matters. It does not show that a custom layer beats a good generic one. We found no independent study that compares a custom build with a generic copilot on the same tasks, at the same cost and in the same time. That gap is the honest centre of this debate.

Anthropic’s guidance points the same way from a different direction. Its Building effective agents post recommends finding “the simplest solution possible” and adding complexity only when needed. It also cautions that frameworks can obscure the underlying prompts and tells readers to understand the code they adopt. For an SME, that argues for a thin harness that you understand over a large one that you merely own.


Why vendors selling the plumbing weakens the “harness as moat” case

Look at what model vendors now offer. Anthropic’s Agent SDK gives developers the agent loop, tools and context management that power its own coding product. Its Managed Agents beta goes further and runs the loop, tool execution and runtime for you. OpenAI updated its Agents SDK in April 2026 along similar lines, as Help Net Security reported. Shared standards are forming as well: the Linux Foundation formed the Agentic AI Foundation in December 2025 around MCP, goose and AGENTS.md.

When the loop, the tool protocol and the runtime are packaged and standardised, building them yourself gives you cost and control, but it does not give you a lead. A competitor can license the same parts next quarter. Data handling also still matters: Anthropic’s own documentation says Managed Agents is not currently eligible for zero data retention and that transcripts persist until you delete them. That is a constraint to check before you route client data through any managed harness.


Where a defensible advantage sits

Two commentators who agree this layer is important stress different risks. Johnson Lee puts it as an engineering fact (it matters) set against a business judgment (whether it resists substitution). He expects the machinery of verification to commoditise, and he locates the durable part in a company’s own ground-truth process: production outcomes, expert judgment, business constraints and a feedback loop that turns failures into regression tests.

Kai Waehner argues that lock-in has moved up a level, into the harness layer, and advises choosing the model for capability and the harness for what it lets you keep. Waehner works for a workflow-orchestration vendor, and neither piece offers outcome data, so treat both as informed argument rather than measurement.

Read together, they suggest a practical list of assets worth owning:

  • Your definition of a correct output, written down well enough that a stranger could grade against it.
  • A test set built from real tickets, orders or cases from your own work. To build such a test set from your own support tickets, see AI evals from real tickets.
  • The corrections your staff make, stored as reusable rules rather than chat history.
  • An audit trail of what the agent saw, did and decided.
  • Agent definitions and prompts in a format that runs on more than one vendor.

The first four are specific to you by construction. The fifth is what keeps the others usable if you change supplier. We cover that exposure in AI agent platform lock-in.

This list is our synthesis of the two arguments. It is a hypothesis about where advantage sits, and nobody has measured it.


What a copilot costs, and what it can be extended to do

A fair comparison starts with a fair picture of the copilot. Microsoft’s Swiss Copilot pricing page, read in November 2026, lists the Microsoft 365 Copilot Business add-on at CHF 17.00 per user per month, with a promotional price of CHF 14.57 on annual billing, both excluding VAT. The promotional price covers the first year on orders until 31 December 2026 and applies to up to 300 users. The add-on requires an eligible Microsoft 365 Business plan. As a rough illustration, 20 users come to about CHF 3,500 in the promotional year and about CHF 4,100 a year at the CHF 17.00 list price, before the base licence.

The belief that a generic copilot cannot take custom tools is wrong. Microsoft’s documentation lists declarative agents, custom API actions, Copilot Studio and the Microsoft 365 Agents SDK, and Copilot Studio has an agent evaluation feature with test sets that you can rerun to check for regressions. The gap is control. Copilot-native agents run on Microsoft’s orchestrator, and the more advanced build options trade convenience for more say over orchestration, model integration and deployment.

Two things deserve a check before a Swiss contract. Microsoft announced in November 2025 that in-country processing of Copilot interactions for Switzerland is planned by the end of 2026. That is a target rather than a delivered state, so confirm the current position. And if you build your evaluation set inside a vendor’s tool, ask whether you can export the cases, the graders and the history. Our custom vs platform TCO model shows how the cost curves cross as volume grows.


A decision test to run before you commit

Run these questions in order. Stop at the first one that settles it.

  1. Is the work generic? Drafting, summarising and searching your own files usually are. Buy a copilot and spend the effort on training your staff to use it.
  2. Does the work depend on your own systems, rules or quality bar? If a generic assistant gets it wrong in ways your staff must correct daily, you have a candidate for a custom build.
  3. Can you write the pass mark? If you cannot define a correct output and score ten real cases against it, a custom build will have nothing to optimise.
  4. Can you take your assets with you? Ask any vendor, including us, whether test cases, correction rules, logs and agent definitions export in a documented format.
  5. Does the volume justify it? As a rule of thumb from our own TCO work, a platform usually wins below a few hundred tasks a month, and a custom build tends to catch up above a few thousand.

Most firms will land on a mix. A copilot covers generic work, and a small custom layer covers the work that is theirs. That mix is also a sensible hedge. If the layer commoditises, the custom layer shrinks to prompts and tool wrappers, and the assets in the list above remain.


The debate stays open

The pro-build camp points at gains from changing only the layer around the model and at the way lock-in shifts upward. The pro-buy camp points at guardrails that models absorb and at vendors packaging the loop. MIT NANDA’s widely quoted finding, as reported by Fortune, that bought tools succeed about two-thirds of the time while internal builds succeed a third as often is itself contested: the sample sizes reported in coverage differ and the definition of success is unclear.

So treat any confident claim, ours included, with care. The conclusion you can defend today is narrow. Own what is specific to your business and keep it portable. Rent what everyone can rent. Revisit the split each time a model upgrade makes a piece of your scaffolding unnecessary.

Frequently asked questions

What is a custom AI harness?

A custom AI harness is the system you build around a language model: the context it receives, the tools it may call, its memory, the guardrails and checks on its output, the evaluation set that scores it, and the orchestration that runs the steps. The model supplies reasoning. The harness decides what it sees, what it can do and how its work is verified.

Is a custom AI harness a real competitive advantage?

Partly, and the evidence is still thin. A harness-only change lifted one vendor's coding agent from 52.8% to 66.5% on a benchmark, but that vendor expects some of its guardrails to become unnecessary as models improve. The more durable assets are your ground truth, test cases, correction data and audit trail, because a competitor cannot buy those.

Can Microsoft 365 Copilot be customised with our own tools?

Yes. Microsoft documents declarative agents, custom API actions, Copilot Studio and the Microsoft 365 Agents SDK for extending Copilot. The trade-off is control. Copilot-native agents run on Microsoft's orchestrator, and custom agents give you more say over orchestration, model choice and deployment at the price of building and operating more yourself.

How much does Microsoft 365 Copilot cost for a Swiss SME?

As of November 2026, Microsoft's Swiss page lists Copilot Business at CHF 17.00 per user per month at list price, and CHF 14.57 in the first year on orders until 31 December 2026 (annual billing, excluding VAT, up to 300 users). It needs an eligible Business plan underneath, so 20 users cost roughly CHF 3,500 in the promotional year and CHF 4,100 at list, before the base licence.

When should we buy a copilot instead of building a harness?

Buy when the work is generic, the volume is modest, and you cannot name an output standard that is specific to your firm. Build, or build in part, when workflows depend on your own systems, rules and quality bar, and when you can keep your evaluation cases and correction data in a format that you can move to another vendor.

Insights

Put these ideas to work

A 30-minute call is enough to find out whether an AI agent fits your workflow — and what it would return.