Your quoting assistant suggests the wrong lead time for a supplier, and your planner corrects it. Three weeks later it makes the same suggestion. You wonder what “self-learning” meant on the vendor’s slide.
The phrase covers very different things. Some systems write notes to themselves, some adapt prompts, and a few retrain model weights. If you are deciding what to buy or build, the useful question is where the learning is stored and who decides what gets kept. Our AI strategy work usually begins with exactly that question.
Self-learning AI in production is mostly captured feedback, rules, memory and tests kept outside the model, and it improves when people decide what to keep. Changing the model’s weights is rarer, and even there humans set the standard.
What “Learning” Usually Means in Practice
A language model does not rewrite itself while you chat with it. When you show it examples in a prompt, it adapts to them for that conversation only. The GPT-3 paper by Brown et al. describes this as working “without any gradient updates or fine-tuning”. When the context goes, so does the effect.
Persistent learning therefore has to live somewhere else. Anthropic’s documentation for its memory tool is a clear example. The model asks to read or write files, and your application carries out those requests. The memory sits in your systems, and a later conversation continues from it only when your application serves the same store.
This is how most products do it, and the description is generic. For the mechanics of what an agent can remember and for how long, see how AI agent memory works.
A practical consequence follows. If the lesson sits in a file, you can open it, correct it, version it and delete it. If it sits in model weights, you cannot read it directly, and undoing it means another training run. That difference in auditability is a good reason to start with the outside-the-model options.
The four places a lesson can land
- Memory files. Facts and preferences the agent reads at the start of a task, such as “supplier X delivers in 14 days”.
- Rules and prompts. Standing instructions that every run loads.
- Test cases. A real input with the right answer, rerun after every change.
- Model weights. Changed through fine-tuning or reinforcement learning. Costly, and handled as a project of its own.
Our reading, which is an inference, is that the first three cover nearly everything a small or mid-sized company needs. No published survey measures how many organisations retrain models in production, so treat “weight updates are rare” as an observation about product patterns rather than a statistic.
What Improves on Its Own and What Needs People
Once the lessons live outside the model, the division of labour gets easy to see. Machines are good at the mechanical parts. People hold the judgement.
| Task | Improves on its own | Needs people |
|---|---|---|
| Finding the relevant stored note for a task | Yes, once the notes are indexed and loaded | Someone checks the right notes are retrieved |
| Using a stored rule inside a conversation | Yes, once the rule is loaded | Someone wrote and approved the rule |
| Drafting a candidate rule from a correction | An agent can propose one | A named owner approves or rejects it |
| Defining what a good answer looks like (the rubric) | No | Yes, always |
| Pruning outdated or conflicting rules | Can be flagged automatically | A person decides what goes |
| Updating model weights | Pipelines can automate the run | People set the standard and release criteria |
| Sensitive decisions (credit, hiring, claims, health) | No | Yes, with review of individual outcomes |
The pattern: automation handles retrieval and drafting, while people own the standard, the approvals and the exceptions. A system with the left column and no right column drifts.
Rahul Garg, a principal engineer at Thoughtworks, makes the same point about software teams in his Feedback Flywheel article. He argues that teams need “an active practice of feeding learnings back in”, with a tech lead or designated owner making the final call on what goes into shared artefacts. An agent can draft the update, and a developer reviews it. His piece is practice advice and publishes no outcome data. Our companion article on the feedback flywheel shows how to run that routine with a small team.
A Public Case That Updates Weights
Shopify has published a detailed example of a loop that goes as far as weights. In an engineering post from 5 August 2026, its team describes how the Sidekick assistant improves. The following is company-reported, so read it as a claim rather than an audited result.
According to the post, the pipeline runs daily. It mines low-scoring production conversations, has a panel of frontier models critique each failure, replays the conversation with a repair hint and re-scores it with an automated judge. Passing replays become training data, and the model is then trained further. The post says the surrounding setup (prompts, tool definitions, orchestration code) is improved first, and parameter work starts “once harness improvements plateau”.
Failures enter the pipeline automatically, and the cycle runs without a person triggering it. Three human elements sit inside it:
- People define the rubric. The scored criteria decide what “good” means.
- People calibrate the judge. Two of the best annotators label 25 random samples blind, and the team measures their agreement with Cohen’s kappa. The post says a very low value, around 0.2, means the rubric is ambiguous: meet again and iterate.
- People handle what the machine cannot repair. Failures the automated critique cannot fix are not left to the loop.
The public description does not say whether a person approves each released model. Shopify also estimates that serving this traffic on a frontier model “could easily cost” about $27 million a year against closer to $1 million for the fine-tuned model. Those are the company’s own estimates with no derivation shown, and the quality comparison uses its own judge.
For a Swiss SME the takeaway is scale. You do not need reinforcement learning to start. You need a standard, a check on the checker, and somebody who looks at the hard cases.
When Learning From Users Goes Wrong
Systems that learn directly from user reactions have a documented record of going sideways. Two examples are worth knowing, with the caveat that both are the companies’ own accounts.
GPT-4o and the thumbs-up signal. In April 2025 OpenAI rolled back a GPT-4o update that made the model overly agreeable. In its postmortem, OpenAI said the update introduced an additional reward signal based on thumbs-up and thumbs-down data, and that this weakened the primary reward signal that had been holding sycophancy in check. It added that user feedback can favour more agreeable answers. That is OpenAI’s explanation rather than an independent audit.
Microsoft Tay. In 2016 Microsoft’s chatbot started posting offensive replies within its first 24 hours online. Peter Lee of Microsoft wrote in his post “Learning from Tay’s introduction” that a coordinated subset of people exploited a vulnerability in Tay. The common shorthand that Tay “learned racism from users” is looser than Microsoft’s own account.
The lesson for owners is simple. Feedback is a signal, and a signal can be gamed, biased or simply wrong. Somebody has to decide which feedback counts.
Where Swiss Law Draws a Line
If a self-learning system helps make decisions about people, Article 21 of the revised Federal Act on Data Protection (FADP) matters. It applies to decisions based exclusively on automated processing that have a legal consequence or a considerable adverse effect on the person. The controller must inform the person and, on request, let them state their view and have the decision reviewed by a natural person. The duties do not apply where the decision is directly tied to concluding or performing a contract and the person’s request is granted, or where the person explicitly consented to automated decisions. The revised act has been in force since 1 September 2023.
The word “exclusively” matters: a person who genuinely makes the decision with AI support falls outside the article, but a human who only confirms the machine’s output may not. Published Swiss guidance on applying Article 21 to adaptive or retrained models is limited, so coverage depends on the facts of your case. We cover the design side in automated decisions under FADP Art. 21.
Questions to Ask Before You Buy or Build
Use these with a vendor or your own team:
- Where is the learning stored, and can we read it?
- Who approves a new rule or memory before agents use it?
- What is the rubric, and who calibrated the judge against human labels?
- How do outdated rules get found and removed?
- Do model weights change at all, and who signs off on a release?
- Which decisions never run without a person?
If the answers are “the system handles it”, press for names. A learning system that nobody owns tends to degrade quietly, and the quality of a deployment a year from now depends on those owners more than on the model.
If you want to see how this plays out for a whole business, becoming an AI-native company looks at the wider organisational picture, and governing AI agents covers who is accountable for what.
Frequently asked questions
What does self-learning AI actually mean?
In most business systems it means the software records what happened and what staff corrected, then reads that record next time. The record sits in memory files, rules, prompts and test cases. The model itself usually stays unchanged. Improvement depends on people deciding what is worth keeping, so it should never be read as unsupervised improvement.
Do AI agents learn from every conversation?
Not by default. A fix made inside a chat usually ends with that session. An agent carries a lesson forward only when something stores it, such as a memory file or a rule, and the application loads it again. Anthropic documents its memory tool as running in your application, so you control what is saved and read back.
Is fine-tuning the same as an AI learning on its own?
No. Fine-tuning is a deliberate training run that changes model weights, with data, a quality standard and a decision to release the result. It costs more than editing a rule and needs evaluation. Many teams reach their goals with memory, prompts and tests first, and move to weight updates only when those plateau.
Can self-learning AI make things worse?
Yes. OpenAI said an April 2025 GPT-4o update added a reward signal from thumbs-up and thumbs-down data, which weakened the signal that had kept sycophancy in check, and it rolled the update back. Stored rules can also pile up, contradict each other or go stale, so someone has to prune them.
Does Swiss data protection law affect self-learning systems?
It can. Article 21 of the revised Federal Act on Data Protection covers decisions made exclusively by automated means that have a legal consequence or a considerable adverse effect. The person must be told and can state their view and ask for review by a human. Whether a given system is covered depends on the facts.