A 40-person industrial distributor had a working AI quoting agent within a week. It read an incoming purchase request, matched the line items against a price list, and drafted a quote that looked ready to send. The sales director was sold on the spot. Eight months later, the agent was still sitting in a test folder, and the team was quoting the same way it had for a decade.
Nothing about the model was wrong. The reasoning held up, and every sample request the developer fed it came back clean. What it hadn’t done yet was survive contact with the actual business: the ERP it needed to query, the pricing exceptions nobody had written down, and a sales team that wasn’t about to send a customer a number it couldn’t vouch for.
That gap has a name now: the ai last mile problem. It is the distance between a model that performs in a demo and a system a company runs, unsupervised, on real work. Most of the AI projects that quietly go nowhere have a model that does its job. Where they fail is everything that has to happen after that.
The Number Everyone Quotes, and What It Actually Measures
A widely cited MIT study put a figure on this pattern: roughly 95% of enterprise generative AI pilots showed no measurable impact on profit and loss. The report traced the shortfall to a “learning gap” in how tools got adopted and integrated, rather than to weak model quality.
The number spread fast, and so did the pushback. Most of it targeted the methodology: the research leaned on a few hundred interviews and a review of public deployments, terms like “pilot” and “P&L impact” were never tightly defined, and a study measuring only immediate financial return will understate progress in workflows where the payoff arrives as fewer errors or faster cycle times before it reaches an income statement. Treat the 95% as a rough signal about where AI projects get stuck rather than a number to cite in a board deck as established fact.
What survives the criticism is the direction of the finding, and it matches what shows up repeatedly in practice. The bottleneck rarely has much to do with whether the model can do the task in principle. It usually comes down to everything required to let the model do that task safely and repeatedly, inside systems that existed long before anyone bought an AI license.
Closing that gap is normally sustained engineering work embedded in how a business operates, rather than a single deployment event. That is the premise behind an embedded AI engineering service: a technical presence that stays inside the workflow until the system earns its keep, instead of a one-off build.
What “The Model Works” Doesn’t Tell You
A demo answers one question: can the model produce a good output when given clean, well-formed input. That is a real and useful signal. It is also a small fraction of what production requires.
Here is what sits between a working demo and software a business trusts to run unattended:
- Systems of record. The model needs live, authenticated access to whatever holds the truth: the ERP, the CRM, the pricing database, the document store. Most of these systems expose APIs, but coverage is uneven, rate limits are real, and “has an API” is not the same as “an agent can reliably read and write the data it needs.”
- The messy tail of real requests. Call it the messy 20 percent, a rough shorthand rather than a measured figure: the malformed inputs, ambiguous line items, and edge cases that a curated demo set never surfaces but that show up constantly once real volume arrives.
- An evaluation suite. Some documented, repeatable way to answer “is this still working” beyond a person’s gut feel after a few spot checks.
- Permissioning and security. A service account with enough access to do the job and no more, reviewed by whoever owns security, with a clear answer for what happens when the agent touches data it shouldn’t.
- Workflow fit. The agent has to slot into how people work today: who reviews its output, at what point in the process, and what happens when it’s wrong.
- Staff trust. The hardest one to schedule. People who have been burned by a bad number or a tool that overpromised will not hand over judgment calls just because engineering says the accuracy metric looks good.
Any one of these can stall a project on its own. In practice they compound: fixing the ERP integration surfaces data quality problems, which changes what the eval suite needs to check, which changes what the sales team is willing to trust without review.
Why Generic Chatbots and Off-the-Shelf Copilots Stall Here
This is also why buying a general-purpose chatbot license or an off-the-shelf copilot rarely closes the gap on its own. Those tools solve the model half of the equation and leave the other half entirely to you.
| Generic chatbot / copilot | A system that has closed the last mile | |
|---|---|---|
| Access to your systems of record | Usually none, or a shallow plugin | Authenticated, scoped access built for the specific workflow |
| Handling of edge cases | Falls back to a plausible-sounding guess | Flags what it doesn’t know and escalates |
| Evaluation | Vendor benchmarks on generic tasks | A test suite built from your own historical cases |
| Security and permissioning | Broad, generic scopes by default | Least-privilege access reviewed by your IT team |
| Staff trust | Earned nowhere in particular | Earned against your own data before go-live |
A general assistant can write a plausible-sounding quote. It has no way to check that quote against your price list, your current stock position, or the discount your finance team approved last quarter for that specific account. It cannot tell you when it’s guessing. The gap sits in what a generic product is built to do; reasoning quality has little to do with it.
A 40-Person Distributor Learns the Difference the Hard Way
Back to the quoting agent from the opening. The company is a composite, built from patterns that recur across similarly sized distributors automating quote intake, and it shows how the six-item list above plays out on a live project.
The demo took a developer about a week, and it was good: clean extraction of specs from a customer email, a correctly formatted draft, plausible pricing on the handful of sample requests used to build it.
Then the project met the actual business, in four separate ways.
The price list had exceptions nobody had documented. Roughly a fifth of real requests involved a discontinued SKU with an informal substitute, a regional freight surcharge applied inconsistently by whoever happened to process the order, or a bundle discount one sales rep remembered and another didn’t. None of that lived in the master price list the demo had been built against.
The ERP integration was the real project. Current prices and stock levels lived in an ERP system rather than the exported spreadsheet used for the demo. Connecting to it meant mapping fields, negotiating API rate limits, and handling the fact that a cached price could quietly drift from what the ERP showed a minute later.
The sales team didn’t trust it, and had a point. Reps had been burned before by a pricing tool that sent a wrong number to a client. They wanted to see and approve every draft before it reached a customer, and they wanted an easy way to override anything that looked off. The design had to treat that requirement as a legitimate control rather than friction to automate away.
The eval set came from 200 historical quotes. Rather than trust “it looks fine to me,” the team pulled 200 real past quotes, redacted the customer details, and used them as a baseline: does the agent reproduce the same pricing logic, catch the same exceptions, flag the same items a human would have flagged? That test set, more than the original demo, is what let the sales director eventually sign off on running the agent without a review step for the simplest quote category.
The agent that finally shipped, months later, ran on the same model as the week-one demo. The difference was everything wrapped around it: an ERP connection, a documented exception list, an approval workflow the sales team had asked for, and a test suite built from the company’s own history. The demo was never the hard part.
What Closing the Last Mile Actually Takes
A few practices show up consistently in the projects that make it through this gap.
Eval-first development. Build the test before the feature. Hamel Husain argues that teams that skip this step end up unable to tell whether a change made the system better or worse, and every subsequent decision becomes a guess. Pulling a representative sample of real historical cases and defining what a correct answer looks like, before writing the agent logic, changes the entire build process: you’re building toward a target you can measure rather than toward a demo that merely looks convincing.
Context engineering against real company data. Anthropic describes context engineering as curating the set of information available to a model at inference time, as distinct from just writing a good prompt. Applied to a business system, that means grounding the model in your own price lists, past correspondence, and account history rather than its general training knowledge, and being deliberate about what it sees versus what would just add noise.
Hybrid architectures. The most reliable systems don’t ask a language model to enforce hard business rules. A price floor, an approval threshold, a compliance requirement: those belong in deterministic code that either passes or fails, with no room for interpretation. The model’s job is the part that genuinely requires judgment, like reading an ambiguous request or drafting language for a human to review. Splitting the two clearly is what makes an agent’s behavior predictable enough to trust at volume.
Someone organizationally close to the work. Specs miss things. The person who processes freight surcharges every day knows an exception the requirements document never mentioned, and finding that out in week two is far cheaper than finding it out after go-live. Projects that succeed usually have someone, whether embedded staff or a vendor engineer working on-site with the team, close enough to the workflow itself to catch that kind of gap before it becomes a production incident.
None of this is exotic. It’s closer to disciplined software engineering practice than to anything specific to large language models. The difference is that AI projects often skip it because the demo already looked so convincing that skipping felt safe.
Why This Keeps Producing the Same Delivery Model
Look at where the vendors with the most enterprise AI deployment experience have put their money, and a pattern shows up. In June 2026, AWS committed a billion dollars to a forward-deployed engineering program, embedding engineers directly with customers and the partners that serve them, to close integration gaps like the ones above. Salesforce has committed to building a team of 1,000 dedicated engineers for its Agentforce line, for the same reason. OpenAI formalized a similar function inside its own organization in 2025.
That is not a coincidence. If the hard part of an AI project is systems integration, edge-case handling, and staff trust rather than model quality, then the delivery model has to keep engineers close to the business until that trust is earned, which takes far longer than shipping a demo. That is a different commercial arrangement than buying software off a shelf, and it’s worth understanding on its own terms: see What Is a Forward-Deployed Engineer? for how that role and delivery model actually work.
This diagnosis is also narrower than the broader question of why AI agent projects fail in the first place. Plenty of projects never even get a working model into a real demo, for reasons that have nothing to do with the last mile: undefined scope, no business-side owner, no agreed way to measure success before build. Why AI Agent Projects Fail covers that project-level risk and a checklist for avoiding it. This article assumes you’ve cleared that bar and asks the next question: your model works, so why isn’t it running the business yet. The answer, almost every time, is the six things sitting between a demo and production, and somebody has to actually do that work.
Frequently asked questions
What is the AI last-mile problem?
It is the gap between an AI model that performs well in a demo or proof of concept and a system that runs reliably inside a company's actual workflows. The model reasoning is usually fine by the time a demo ships; what is missing is the integration to real systems, the handling of messy edge cases, an evaluation suite, proper permissions, workflow fit, and staff trust. Closing that gap is a software and organizational project rather than a modeling problem.
Is it true that 95% of AI pilots fail?
A widely cited MIT study reported that roughly 95% of enterprise generative AI pilots showed no measurable impact on profit and loss, and attributed the shortfall mainly to integration gaps rather than model quality. The study's sample and definitions have been questioned by other researchers, so treat the number as a directional signal about where AI projects stall rather than a precise industry benchmark.
Why do generic AI chatbots or off-the-shelf copilots underperform at work?
They ship with no knowledge of a specific company's price lists, systems of record, exception handling, or approval rules. A generic assistant can draft plausible-sounding text, but it cannot verify a number against your ERP, flag a request that falls outside your discount policy, or earn the trust of a team that has been burned by a wrong answer before.
What does closing the AI last mile actually require?
In practice: an evaluation suite built from real historical cases before the agent is trusted with new ones, context engineering that grounds the model in your own data rather than generic knowledge, a hybrid architecture where deterministic code enforces hard rules and the model handles judgment calls, and someone close enough to the day-to-day work to catch what a spec document misses.
How is the AI last-mile problem different from an AI project simply being poorly scoped?
They overlap but are not the same failure. Poor scoping is an organizational risk that can sink a project before a single line of code ships. The last-mile problem assumes the scope and the model are both reasonable and asks a narrower question: why does a technically working model still take months to become software the business actually runs unattended.