Your support agent has been live for six weeks. Someone suggests a cheaper model, or a rewritten system prompt, or a new refund rule. Before you change anything, you want to know whether the agent gets worse. You have no way to tell, because the only test you ever ran was the demo.
You do have the material for a real test, though. It sits in your ticket system: last month’s actual customer and employee requests, already answered by people. This article walks through the recipe for turning those tickets into a regression set that you run before every change. If you want a team to build the harness around it, that is part of our custom AI development work.
Build the test set from real tickets before you touch the agent, and run it before every change to the prompt, the model or the rules. What follows is the sequence we recommend, with the contested numbers attributed and the data-protection traps named.
Scope note: Testing AI Agents covers what a buyer should demand at sign-off. This piece is the build recipe for the set itself. Corrections your staff make today become tomorrow’s test cases through the routine in the AI feedback loop.
Why Last Month’s Tickets Beat Invented Test Cases
Invented cases are tidy. Real tickets contain a customer who writes three questions in one message, a pasted screenshot description, a half-finished sentence and a refund request buried in the fourth paragraph. Your agent will meet exactly that.
OpenAI’s evaluation guide warns about this directly, listing “biased design” among its common mistakes: eval datasets that do not reproduce production traffic. Anthropic’s engineering team gives similar advice in its guide to agent evals. It recommends starting from the manual checks you already run before a release and from your bug tracker and support queue.
Anthropic also writes that converting user-reported failures into test cases ensures the suite reflects actual usage. Tickets are the cheapest source of such cases you have, because someone already named the problem and someone already answered it.
Step 1: Sample the Month, Do Not Cherry-Pick It
Pull a month of closed tickets and sample from all of them. The temptation is to pick the interesting ones, which produces a set full of hard cases and blind to the dull middle where most volume sits.
We recommend stratifying the sample. The scheme is our own and no source prescribes it. Group the month by ticket category and by channel (email, chat, web form, phone notes), then draw from each group in rough proportion to volume. Add a deliberate extra slice from the categories where the agent will have the least room for error, such as refunds or access to accounts.
For an IT helpdesk, that might mean password resets, VPN problems, hardware requests and onboarding questions each represented, plus every ticket that was escalated. Our helpdesk article describes those categories in more detail.
Keep the multi-turn tickets. A thread with six exchanges and a changed requirement is harder for an agent than a one-shot question. Salesforce AI Research’s CRMArena-Pro benchmark reports around 58% single-turn success and approximately 35% in multi-turn settings, across 19 expert-validated business tasks. It is a vendor benchmark, it tests 2025-era models in a simulated CRM environment, and the figures say nothing about your agent. It does support one design choice: do not filter your set down to single messages.
Step 2: Redact Before Anyone Else Sees the Data
A ticket is free text written by people who put names, addresses, contract numbers and sometimes health details into it. Moving that text into a test set is a new use of personal data. Do the protective work first.
What the Swiss FADP means in practice
We are not lawyers and this is no legal advice. A legal check is advisable before you act on any of it. With that said, the points below are what we would put in front of the lawyer.
- Identifiability is relative. Husi-Stämpfli and Gubser, in a 2026 article in the Swiss journal AJP/PJA, read the FADP as asking whether a person is identifiable given the means and perspective of the actor holding the data. Pseudonymised data stay personal data for whoever holds the key. The statute itself only speaks of a person being identifiable, so the relative test comes from doctrine and case law. Their reading is scholarly commentary, so have counsel confirm it.
- Purpose and risk. Tickets were collected to serve the customer, and reusing them as test data is a new purpose that your privacy notice and records should cover. Large-scale use of sensitive content may also call for a data protection impact assessment.
- Anonymise before use. If you can strip the identifying content before the tickets reach the test set, the exposure shrinks. The authors add that pseudonymisation reduces risk only gradually.
- Redaction is best-effort. The Microsoft Presidio project states that there is no guarantee it will find all sensitive information, and that additional systems and protections should be employed. Treat any automated pass as a first filter.
- Restrict access. Keep the set in a location with named users and a log, separate from general project folders.
- An external LLM judge changes the picture. If a hosted model reads the tickets to grade answers, you are sending data to a service provider. The same commentary describes a processing contract as a requirement for that arrangement and notes that disclosure abroad depends on the recipient country offering adequate protection or, failing that, on other safeguards such as contract clauses. Whether your specific provider and region meet that test is a question for your legal adviser and the provider’s current terms.
A redaction routine that holds up
Run a recogniser such as Presidio over every ticket, replace names, emails, phone numbers and identifiers with typed placeholders, and then read a sample of the output by hand. Fifty redacted tickets is a reasonable spot-check. Look for what the tool misses: a nickname, a company name that identifies a person, a rare incident that only one customer could have described.
Where a ticket cannot be cleaned, drop it from the set. A smaller clean set serves you better than a larger risky one.
Step 3: Read and Label Before You Write Any Rubric
Resist the urge to define metrics up front. Hamel Husain’s field guide for improving AI products argues for reading traces first and building the failure taxonomy from what you see, in contrast to starting with generic scores such as “hallucination” or “helpfulness” that often miss domain-specific issues.
So read. Open the redacted tickets next to what the agent produces for them, and write short notes on every undesired behaviour. After a few dozen, patterns appear: the agent quotes an outdated return window, ignores the second question in a message, promises a callback it cannot schedule.
Turn each pattern into a named failure mode, then label every ticket pass or fail against each mode that applies. Husain’s LLM-as-judge guide prefers binary labels to numeric scales, and says that evaluations built on 1 to 5 scores are a mistake. That view is contested: Shopify’s production loop uses four scored criteria with defined anchors for each score. A scored rubric can work once a team has calibrated it. For a first set we would take pass/fail, because two people argue less over a binary call.
Step 4: Add Reference Answers and Negative Cases
Each case needs something to compare against. Anthropic’s guide recommends a reference solution for every task, meaning a known working output that passes all graders. For a ticket, that is the answer a careful expert would give, written down once.
Then add the cases where the agent should do nothing, or refuse, or hand over to a person. Anthropic puts it plainly: test both the cases where a behaviour should occur and where it should not, because one-sided evals produce one-sided optimisation. If every ticket in your set rewards action, you will tune an agent that takes action on everything.
The human agent may have been wrong
The reply in the ticket history is the obvious reference, and it is a trap. Staff reply under time pressure. Sometimes the answer was wrong, sometimes it was right for a customer with an exception the ticket does not show.
Anthropic’s test for a good task is that two domain experts would independently reach the same pass/fail verdict. Apply it here. Where your label and the historic reply disagree, have a second expert settle it and write the reasoning on the case. These disagreements are often the most useful entries in the whole set.
Step 5: Split the Set and Be Honest About Its Size
Divide the labelled tickets into a development part you tune against and a held-out part you run rarely. Husain’s judge guide suggests about 10 to 20% for training examples and 40 to 45% each for dev and test, with the test set run once at the end. We round that to 10/45/45 below for the sums. Without a split, you will fix the agent for the exact tickets you keep looking at.
Now the size question, where the sources disagree.
| Source | Suggested size | Purpose |
|---|---|---|
| Anthropic engineering | 20 to 50 simple tasks drawn from real failures | A start, without waiting for hundreds |
| Husain and Shankar, evals FAQ | Start with 100 diverse traces; label 100 to 200 examples per failure mode | Reading for failure discovery, then judge validation |
| Husain, judge guide | Roughly 30 to 50 Pass and 30 to 50 Fail examples each for dev and test | Validating an LLM judge |
This is a difference of purpose as much as of opinion. Neither side publishes outcome data for its numbers, so treat them as informed practice.
Here is our own arithmetic, offered as such. One hundred tickets with that split leave about 40 to 45 for development. Husain’s rule of 30 to 50 Pass and 30 to 50 Fail in dev alone would call for 60 to 100 labelled items, per failure mode you want to judge. So 100 tickets give you a smoke set that catches obvious regressions. They do not give you a validated judge. Be clear with your management about which one you built.
Step 6: Calibrate the Judge Before You Trust It
Grading hundreds of answers by hand before each release does not scale, so many teams use a second LLM as judge. The judge needs checking against people, since LLM judges show position, verbosity and self-enhancement biases, according to the 2023 MT-Bench paper by Zheng and colleagues.
Shopify’s engineering team describes a concrete calibration in its Sidekick post. Its two best annotators or product experts blindly label 25 random samples, and the team measures agreement with Cohen’s kappa. According to the post, a very low value, around 0.2, means the rubric is ambiguous and the team should meet again and iterate. Shopify does not define an acceptable level.
Read that carefully. A kappa near 0.2 is a signal to fix the rubric. It is not a threshold you aim for or a pass mark. Once your humans agree with each other, compare the judge against their labels.
Watch the class balance when you do. Husain’s example is a judge that always says Pass: with 5% of examples being real errors, it still agrees with the humans 95% of the time. He recommends scoring the judge on its true positive and true negative rates instead of raw agreement.
Step 7: Run It Before Every Change, Then Keep It Alive
The set earns its keep at one moment: before you change a prompt, swap a model or edit a business rule. Run it, record the pass rate per failure mode, make the change, run it again. A drop on a single mode tells you where to look. The same set is the prerequisite if you later weigh fine-tuning against prompting.
Maintenance is the part teams skip. Anthropic describes an eval suite as a living artifact that needs ongoing attention and clear ownership, and warns about saturation, where an agent passes all the solvable tasks and the suite stops teaching you anything. For how a feedback routine turns staff corrections into new cases, see self-learning AI systems.
In practice:
- Name an owner for the set, as you would for any other asset.
- Feed it monthly. Add new failures from live tickets and from staff corrections, and retire cases that no longer match your products or policies.
- Re-check the redaction whenever you add data.
An indicative effort table for the first build follows. These are our own illustrative estimates for 100 tickets and no measurement, and you should price them at your own internal rate.
| Task | Illustrative effort |
|---|---|
| Sample and export | 2 to 3 hours |
| Redaction script and spot-check of about 50 outputs | 4 to 8 hours of engineering |
| Labelling single-message tickets at 3 to 5 minutes each | 5 to 8 hours |
| Second labeller on 25 tickets for the kappa check | About 2 hours |
| Judge build and calibration | 1 to 2 days |
| Multi-turn threads at 20 to 40 minutes each, 100 threads | 33 to 67 hours |
The last row borrows its scale from the TRAIL paper, which reports about 30 to 40 minutes to annotate a single agent trace. Those were multi-step engineering traces labelled by expert annotators, so support threads should be quicker. Even so, multi-turn labelling will dominate your budget if you include many threads.
What Last Month’s Tickets Cannot Tell You
A ticket set records what customers asked, and misses what they never asked because the old process discouraged it. It also ages. A policy change, a new product or a seasonal peak makes part of it stale within a quarter.
It will not find unknown risks either. Tickets will not contain the prompt injection nobody has tried yet, so security testing needs its own cases. And a passing run does not prove the agent is good. It proves the agent has not got worse on the failures you already know about, which is a narrower and still valuable claim.
Build the set small, label it with care, and guard the data. Then every later conversation about a cheaper model or a new prompt can start from a number instead of an opinion.
Frequently asked questions
How many tickets do I need for an AI agent test set?
Start with 20 to 50 cases if your aim is a regression check, which is the range Anthropic's engineers suggest. Husain and Shankar advise reading about 100 traces and labelling 100 to 200 examples per failure mode if you want to validate an LLM judge. Both numbers are practitioner opinion, so pick by purpose and grow the set as failures appear.
Can I use real customer tickets to test an AI agent under Swiss data protection law?
Only with care. Under the FADP, whether data identifies a person depends on the means available to whoever holds it, so a redacted ticket can remain personal data. Anonymise before use, restrict access, and have a lawyer confirm the position before an external model sees the tickets. This article gives no legal advice.
Does redacting names and numbers from tickets make them safe to share?
Not on its own. Redaction tools are best-effort, and Microsoft states that there is no guarantee Presidio finds all sensitive information. Free text also leaks identity through context, such as a rare job title or a unique incident. Sample the redacted output by hand, keep access limited, and treat the set as sensitive.
What does a Cohen''s kappa of 0.2 mean for an LLM judge?
It means your two labellers agree only slightly more than chance would produce, so the rubric is probably ambiguous. Shopify describes a very low value, around 0.2, as the point to meet again and iterate on the rubric. It is a warning sign. Shopify does not define it as an acceptable level.
Should the human agent''s reply count as the correct answer?
No, treat it as evidence. Anthropic notes that a good task is one where two domain experts would independently reach the same verdict, and a rushed or mistaken agent reply fails that test. Have a second expert review each disagreement between the human reply and your label, then record the decision with the case.