You sit through a vendor demo. The assistant answers fluently, handles two follow-up questions and then the salesperson says the word: agent. The price list says the same. Three weeks later the pilot produces transcripts, and your staff still copy the results into the ERP by hand.
This pattern has a name, agent washing, and it is common enough that analysts now track it. Spotting it before signing takes a few specific checks that cost an hour of demo time. If you already know the product category you need and want it built to your process, our AI development work starts from the same logic: define what the system must write, when it must stop and how you will verify it.
A real agent can be proven in the room, because it changes something in a system you can open. The five tests below are built to be run live, by you, without the vendor’s help.
Scope note: the definition of an agent versus a chatbot is covered in AI agents vs chatbots. This article assumes that definition and focuses on what to do in the demo.
What Agent Washing Means, and What Gartner Actually Said
Gartner uses the term for vendors rebranding existing products as agentic. According to Reuters’ report on the Gartner release, the rebranded products include AI assistants and chatbots, and Gartner estimates that only about 130 of the thousands of agentic AI vendors are real. Reuters’ report does not describe the criteria or sample behind that 130, so read it as an analyst judgement. It signals direction, and you cannot plan around it as a percentage.
Gartner’s press release carries a second number that is often misquoted. Gartner forecasts that more than 40 percent of agentic AI projects will be cancelled by the end of 2027. That is a forecast without a measured cancellation rate behind it, and it says nothing about 2026. Reuters reports the stated reasons as escalating costs and unclear business value.
For a buyer, the practical reading is simple. Costs and value are the two things you can test before you commit money, and the tests below are aimed at exactly those.
A regulator has already looked at this kind of claim
In January 2025 the SEC issued an order against Presto Automation. The SEC said Presto had falsely claimed that its AI product eliminated the need for human order-taking at drive-thrus, while the vast majority of orders placed through that version of the product required human intervention. Presto consented to a cease-and-desist order without a civil penalty.
The case is useful for one reason. A demo can look autonomous while a person is doing part of the work off screen, and only evidence from inside the system will show it. Tests 1 and 5 exist for that situation.
Test 1: Make It Write to a System You Can Open
Pick one record in a system you own: a CRM contact, a purchase order, a ticket. Ask the vendor to have the agent change it, live, using your credentials or a sandbox copy of your system. Then open the system yourself, on your own screen, and look at the before and after state.
A transcript that says “done” proves nothing. A changed field with a timestamp and a user ID proves the action happened.
What to watch for:
- The write path. Which API or connector did the agent call? A real agent has tool definitions it can name, with documented inputs and outputs. OpenAI’s practical guide to building agents lists action tools such as updating a CRM record as one of the core tool types, and rates tool risk by whether it is read-only or write, and whether it can be reversed.
- Fixed scripts in disguise. If the write happens the same way every time, with no variation when you change the input, you may be looking at RPA with a language model on the front. That can be a fine product. It is not an agent.
- Reversibility. Ask what happens if the write is wrong. A vendor who has thought about tool risk will tell you which actions are reversible and which need approval.
Anthropic’s Building effective agents makes the same point from the builder side: effort should go into the interface between agent and tools, with thorough documentation and testing. If the vendor cannot show you that documentation, the tool layer may not exist.
Test 2: Change the Request Halfway Through
Single-turn demos are the easiest kind to rehearse. Real work does not arrive that way. A customer corrects an address, a manager changes the approval limit, an invoice turns out to belong to a different entity.
During the demo, start a task, then change one detail after the second turn. Remove a required input and see whether the agent asks for it or invents it. Contradict an earlier instruction and watch which one wins.
The research suggests why this matters. Salesforce AI Research’s CRMArena-Pro benchmark reported around 58 percent single-turn success for leading agents, falling to about 35 percent in multi-turn settings. Two caveats apply. The benchmark was built by a vendor that sells agents, and the models tested date from 2025, so current systems may do better. Use the result as a reason to run the test, and do not treat it as a league table.
A good sign is a vendor who shows a failure here and explains the recovery behaviour. A worse sign is a demo script that never leaves the first turn.
Test 3: Ask What Makes It Stop
An agent that cannot halt is a liability. Ask the vendor to show what happens when something goes wrong, and ask for it live.
Three things to request:
- A failure threshold. The agent retries a tool call that fails, then stops after a set number of attempts and returns control with its state saved.
- A high-risk trigger. OpenAI’s guide names examples such as cancelling orders, authorising large refunds and making payments as actions that should route to a person.
- A stopping condition. Anthropic describes agents that pause for human feedback at checkpoints or when they hit blockers, and that use stopping conditions such as a maximum number of iterations.
Then ask the awkward question: who gets notified, through which channel, and how fast? A halt that sends an email to a mailbox nobody reads is the same as no halt.
A rebadged chatbot has a different answer. It either fails silently or returns a polite generic fallback (“I could not help with that”), because nothing in the product was built to hand a task over with context.
Test 4: Ask for Repeated Runs on Your Own Tasks
One good run proves little. Language-model systems vary between runs, and the variation is where production trouble comes from.
The tau-bench paper from Sierra gives the clearest framing. It scores an agent by comparing the final database state with the goal state, and it reports pass^k: the probability that the agent solves the same task correctly on all of k repeated attempts. The authors reported that function-calling agents succeeded on fewer than 50 percent of tasks and were quite inconsistent, with pass^8 below 25 percent in the retail domain for the best model tested, gpt-4o. That was a 2024 model, so the numbers show the method rather than today’s performance.
Turn that into a request:
- Ask for the vendor’s evaluation set, how many tasks it contains and how results are scored.
- Give them five to ten tasks from your own operation and ask for results across repeated runs, with the pass rate for every attempt.
- Ask what changed between the two most recent model versions and whether the evals were rerun.
OpenAI’s guide advises setting up evals to establish a performance baseline before scaling. A vendor with no eval set is asking you to be the test suite. That is possible to accept, but then the price and the contract should reflect it.
Test 5: Ask for the Trace of Every Run
A conversation log shows what was said. A trace shows what was done. For each run you want the tool called, the input it received, the output it returned, the timestamp, the reasoning step that led to the call and, for any high-risk action, who approved it.
Ask to see a trace from a run that went wrong. Vendors who have operated agents in production have these on hand, because debugging without them is guesswork.
A note on regulation, kept narrow on purpose. The EU AI Act requires automatic event logging for high-risk AI systems under Article 12. That duty does not extend to every agent, so do not let a vendor present it as a universal legal obligation. Your reason to demand traces is operational: without them nobody can tell you why a record changed, and the SEC’s Presto order shows what can sit behind a confident product claim.
The Demo-Day Table
Print this and take it to the meeting. The right-hand column describes the pattern, and a single product may mix several.
| Test | A real agent shows | A rebadged chatbot, RPA tool or assistant shows |
|---|---|---|
| 1. Writes | A changed record in your system, with before and after state, a named tool call and a reversal path | A chat message that says “done”, or a fixed script that writes the same way every time |
| 2. Multi-turn | Asks for missing inputs, keeps state, adapts when the request changes | Handles turn one, then drifts, repeats itself or invents a value |
| 3. Halts | Defined thresholds and risk triggers, a handoff with state and a named recipient | A generic fallback message, or silent failure |
| 4. Repeated runs | An eval set, pass rates over several attempts and results on your tasks | One polished showcase run and a headline accuracy figure |
| 5. Traces | Per-run log of tool, input, output, timestamp and approver | A transcript of the conversation |
These five tests are a synthesis built from vendor-published guidance and benchmark methods. No standards body has issued them as a checklist, so treat them as a practical screen and add your own questions.
When a Deterministic Workflow Is the Right Buy
Some demos should end with the vendor saying: you do not need an agent for this.
Anthropic distinguishes workflows, which orchestrate models and tools through predefined code paths, from agents, and recommends finding the simplest solution possible and adding complexity only when needed. OpenAI’s guide says that applications using a model without letting it control workflow execution, such as simple chatbots and sentiment classifiers, are not agents. It also advises validating your use case against its criteria before committing, because a deterministic solution may suffice.
Both organisations sell agents. Their advice to buy less when less will do carries weight for that reason.
Take a process with a stable sequence: invoice arrives, fields are extracted, amounts are checked against the purchase order, an exception goes to a person. If the branches are known in advance, a workflow with one model call for extraction is more predictable and cheaper to run. Calling it an agent adds nothing.
If your process has open-ended steps, varied inputs and decisions that depend on context, an agent starts to earn its cost. For the build-or-buy question that follows, see build vs buy for AI agents.
Putting the Tests Into a Swiss Pilot Contract
Running the tests once in a demo is a start. Writing them into the pilot agreement is what protects budget.
- Acceptance criteria. Convert each test into a measurable criterion. For example: the agent writes to a named system of record in a sandbox with a verifiable before and after state, and it passes an agreed share of your own tasks over repeated runs. Set the threshold yourself, based on what the manual process delivers today.
- Logging deliverables. Specify the trace fields (tool, input, output, timestamp, approver) and the retention period. State who owns the logs.
- Halt behaviour. List the actions that must route to a person, and name the recipient role.
- Exit right. Agree that you can end the pilot at a defined date without further fees if the criteria are not met. Any amounts should be stated in CHF, with a cap on pilot spend.
- Model changes. Require notice before the vendor swaps the underlying model, plus a rerun of the evals afterwards.
Given Gartner’s forecast about cancellations, the exit clause is the mechanism that turns unclear business value into a decision you can take on evidence.
If the pilot passes, you have a product that earned its label. If it fails, you have paid for an hour of demo time and a short test period rather than a year-long licence.
A Last Check Before You Sign
Review your notes from the demo. Did the vendor let you open the target system? Did they show a failure honestly? Did they offer to run your tasks, and did they ever tell you that a simpler solution might do?
Vendors who answer yes to all four have probably built something. Those who steer around the questions are selling a label.
Frequently asked questions
What is agent washing?
Agent washing is the practice of relabelling existing products, such as chatbots, AI assistants and robotic process automation, as agentic AI without substantial agentic capability. Gartner uses the term. A real agent uses tools, writes to your systems and decides its next step, while a relabelled product only converses or follows a fixed script behind a new name.
How can I tell if a vendor's AI agent is real in a demo?
Ask the vendor to run a task live and then open the target system yourself. A real agent leaves a changed record, a sent message or a filed ticket that you can see outside the chat window. Then change the request mid-conversation, force a halt and ask for the per-run log. A rebadged chatbot usually has only a transcript to show.
Is the Gartner figure of 130 real agentic vendors reliable?
Treat it as an analyst estimate. According to Reuters, Gartner said only about 130 of the thousands of agentic AI vendors are real, but the coverage does not describe the criteria or sample behind the count. The number is a useful warning about market hype, and it is not an audited census, so test each product on its own behaviour.
Does a simple workflow count as a worse purchase than an agent?
No. OpenAI and Anthropic both say a deterministic workflow can be the better answer when the task is well defined. Anthropic notes that workflows offer predictability and consistency for well-defined tasks, while agents trade latency and cost for flexibility. A vendor who tells you that your use case needs a workflow rather than an agent is giving you honest advice.
What should a Swiss company put in an AI agent pilot contract?
Write the demo tests into the contract as acceptance criteria, with pass thresholds measured on your own tasks, and agree what logs the vendor must deliver. Keep a right to exit at the end of the pilot without penalty and without further fees. That protects your budget if the product turns out to be a relabelled chatbot.