Your accounts-payable agent codes an invoice to the wrong cost centre. Maria in finance fixes it and moves on. Next week the agent makes the same mistake, and this time Luca fixes it. Nobody told anybody, and the agent has no memory of either fix.
That is the normal state of an AI deployment without a feedback loop. Staff do the correcting, the agent does the repeating, and the knowledge of what “right” looks like stays distributed across people’s heads and chat histories. The cost shows up as a quiet tax on every task.
This article describes how to turn those corrections into shared rules that every agent follows, and how to run the routine with a small team. If you want help applying it to a live process, our process optimization work starts with exactly this kind of review.
An AI feedback loop is a supervised routine that moves corrections out of individual heads and into shared artefacts the agents read. The compounding comes from people deciding what to keep. The system does not teach itself.
Scope note: this piece covers the correction routine itself. For the metrics around it, see the KPIs that prove your AI agents are working, and for the wider operating discipline, managing AI agents in production.
Why Corrections Get Lost
Every correction contains three things: what the agent did, what it should have done, and why. A person fixing an output usually supplies the third without writing it down. That reasoning is the valuable part, and it is the part that evaporates.
Three habits cause most of the loss:
- Personal workarounds. One employee learns that a certain supplier always needs a particular phrasing and rewrites the prompt each time. The skill is personal, so it leaves when the person does.
- Chat-window fixes. A correction made inside a conversation lasts as long as that conversation. The next session starts from the same blank state.
- Silent edits. Staff accept the output, adjust it by hand and say nothing. The agent’s accuracy looks fine in any report, because the task was completed.
Rahul Garg, a principal engineer at Thoughtworks, wrote about the pattern in April 2026 on martinfowler.com. His framing is that teams diverge over time depending on whether personal learning becomes team capability. He describes this as the difference between personal fluency and team capability. His article is practice advice drawn from software teams, and he does not publish outcome data, so treat it as a well-reasoned method rather than a proven result. The structure translates well to operations work, though, and it is simple enough to test in a month.
The Four Signals Worth Capturing
Garg splits what staff learn into four kinds of signal. Each one has a natural home, and sorting by type keeps the pile of corrections from becoming one undifferentiated list.
| Signal | What it looks like | Where it should go |
|---|---|---|
| Context | The agent lacked a fact: a supplier code, a price list version, a policy exception | Knowledge base or the agent’s standing instructions |
| Instruction | A phrasing that produced noticeably better or worse output | A shared prompt or command that everyone uses |
| Workflow | A sequence of steps that worked, such as checking the contract before drafting the reply | A playbook that describes the process |
| Failure | A wrong output, with the root cause | A guardrail, a rule or a documented anti-pattern |
Garg is emphatic on the last row: the root cause matters more than the symptom. “The agent coded the invoice wrongly” is a symptom. “The agent has no rule for invoices from group companies that bill on behalf of a subsidiary” is a cause, and a cause can become a rule.
A worked example
Take the accounts-payable case above, with illustrative numbers. The agent codes 400 invoices a month and staff correct 24 of them. Reviewing the 24 shows that 15 share one cause: invoices from the group’s shared-services company belong to the cost centre of the subsidiary named in the body text. That is a failure signal with a context gap underneath it.
One rule goes into the agent’s instructions, one row goes into the supplier table, and two of the real invoices become test cases. Next month the same 15 invoices are coded correctly. The remaining nine are the long tail, and the review continues with them.
That result is invented for illustration. The mechanism is what carries over: a bucket of related corrections collapses into one fix.
Where Corrections Should Live
A rule that exists only in someone’s notes has the same problem as a fix that exists only in someone’s head. The destination has to be a place the agent actually reads, and one the team can see.
Three kinds of artefact cover most cases:
- Rules and instructions. A short, versioned document the agent loads at the start of every run. Keep it brief. A long instruction file competes for the model’s attention with the task itself.
- Knowledge entries. Facts that change over time and that the agent retrieves when relevant: price lists, supplier terms, product specifications. If you already run a retrieval setup, see AI agents and knowledge management for how to keep it current.
- Test cases. The real input, the wrong output and the corrected output, stored together. Every fixed mistake becomes a regression check, so a later change cannot silently bring the error back.
Garg’s own example keeps a learning log in the repository, which later sessions load as context. The same idea works outside code: a shared document or table with one line per lesson, a date, an owner and a link to the example.
One caution. Every rule you add has a cost, because it takes up space in the agent’s context and can conflict with other rules. A rule that fixed a model’s weakness last year may be unnecessary after a model upgrade. That is why the routine below includes deletion.
The Routine: Small Enough to Survive
Most feedback programmes fail because they ask for too much. A form to fill in after every task is abandoned within a fortnight. Garg designs for the opposite: the heaviest ritual he describes is a five-minute agenda item, and he writes that rhythm matters more than rigour.
A routine that fits a ten-person operations team looks like this:
- At the moment of correction (under 30 seconds). The person fixing the output flags it with one tag from the four signals and a short note on why. Anchor this to a step that already exists, such as the approval button or the ticket close.
- Weekly (15 minutes). The owner reads the flagged items, merges duplicates and assigns each one a destination: rule, knowledge entry, test case or discard. One-off cases get discarded on purpose.
- Monthly (30 minutes). Review the repeat corrections. If the same mistake appears after a rule was written, the rule is unclear, in the wrong place or being ignored by the agent.
- Quarterly. Delete rules that no longer apply, check that the artefacts are still being read and re-test the agent against the saved cases. Garg suggests the periodic review be triggered by quarterly timing or by a sense of drift.
The owner matters more than the schedule. One named person per agent or process should have the final say, Garg gives the final call to a tech lead or designated owner. Committees produce contradicting rules.
Measuring Whether It Works
Be honest about the state of the evidence. Garg himself advises against building a dashboard: “I would not recommend building a dashboard. I would recommend paying attention.” Published outcome data for correction loops is scarce, and a claim like “feedback loops cut errors by half” would not be supported by anything we could verify.
You can still watch a few numbers that you already have:
- First-pass acceptance rate. The share of outputs staff accept without editing. Garg lists it among his preferred indicators.
- Corrections per hundred tasks. Counted from the flags in the routine above.
- Repeat corrections. The same mistake corrected twice is the most telling number, because a working loop should drive it towards zero.
- Post-delivery rework. How often an accepted output comes back later to be fixed.
These sit comfortably next to the operational metrics in our guide to AI agent KPIs. Set a baseline before the first weekly review, then compare after eight weeks. If repeats do not fall, the routine is the problem, and the answer is usually that rules are not reaching the agent or nobody owns them.
What Stays With People, Even in an Advanced Pipeline
It is tempting to read “feedback loop” as a system that improves automatically. The best public example of a sophisticated loop shows how much human work sits underneath it.
In August 2026, Shopify’s engineers described how they improve their Sidekick assistant. The pipeline scores live traffic with an automated judge, replays failed conversations with repair hints, and, according to Shopify, updates the model weights daily. It is company-reported, so the results are claims rather than independent findings. The human parts are what matter here:
- The product team defines the rubric. A few scored criteria (completeness, execution, response quality and safety) decide what counts as good.
- People calibrate the judge. Two expert annotators independently label 25 random samples, and the team measures how well they agree. Shopify notes that a very low agreement score, around 0.2, means the rubric is ambiguous, and the team should meet again and iterate on it. To build such a test set from your own support tickets, see AI evals from real tickets.
- People handle what the machine cannot repair. Conversations that the automated critique could not fix go to human annotators.
Shopify also states that the surrounding setup is improved before weights change. Prompts, tool definitions and surrounding code come first, which matches the small-team approach in this article. The post estimates that serving the same traffic on a frontier model “could easily cost” $27 million a year versus about $1 million for the fine-tuned model. Those are Shopify’s own estimates, with no derivation shown, and the quality comparison relies on its own judge. Nothing in that pipeline is unsupervised improvement. It is a more expensive version of the same discipline: define good, check the checker, send the hard cases to people.
For a Swiss SME the lesson is scale-appropriate. You do not need reinforcement learning to start. You need the weekly slot, the owner and the habit of writing the lesson down. For what can improve without a person in the loop and what cannot, see self-learning AI systems.
Common Ways the Loop Goes Wrong
- Rule pile-up. Dozens of overlapping instructions slow the agent and create contradictions. Merge and prune every quarter.
- No owner. Flags accumulate in a spreadsheet nobody reads. Name the person before launching.
- Treating every edit as a lesson. Some corrections reflect personal taste and say nothing about error. The weekly triage exists to discard them.
- Sensitive data in the log. A learning log with customer names or personal data becomes a data-protection question. Record the pattern and an anonymised example. Check the rules in your sector before storing real inputs as test cases.
- No re-test. Rules that are never tested against saved cases can fix one problem and break another. Keep the cases and rerun them after each change, including after a model upgrade.
Starting Next Monday
Pick one agent with a visible error rate. Add a one-tag flag at the approval step, name an owner and book fifteen minutes weekly. After four sessions you will have a short list of recurring causes, a handful of rules and a first set of test cases built from your own work. That small collection is a company-specific asset: it encodes how your business wants things done, in a form that any future agent, or any future model, can read.
Frequently asked questions
What is an AI feedback loop?
An AI feedback loop is a routine that captures what your staff change or reject in an agent's output and turns it into something the agent reads next time, such as a rule, a knowledge entry or a test case. The model itself does not need to change. The improvement sits in the instructions, knowledge and checks around it.
Does an AI agent learn from corrections automatically?
Not by default. A correction made in a chat window usually disappears when the session ends. The agent improves only when someone, or a reviewed process, writes the lesson into a shared artefact such as a rule file, a knowledge base article or an evaluation case. Changing model weights is a separate and much rarer step.
How often should we review agent corrections?
Weekly works for most teams. A fifteen-minute slot is enough to triage the week's corrections, assign each one to a destination and name an owner. Add a quarterly review that deletes outdated rules and checks that the artefacts are still used. Rahul Garg describes the heaviest ritual as a five-minute agenda item.
Who should own the rules that come out of corrections?
One named person per agent or process, usually the operations lead who already owns the workflow. They decide what becomes a shared rule, what stays a one-off fix and what gets deleted. Without a single owner, rules pile up, contradict each other and quietly make the agent worse.
How do we know the feedback loop is working?
Track first-pass acceptance rate (the share of outputs staff accept without edits), the number of corrections per hundred tasks and repeat corrections of the same kind. Falling repeats are the clearest sign. Garg suggests paying attention to these signals rather than building a dashboard, and public outcome data on correction loops remains thin.