Insight · Applied AI
An AI project doesn't die from a bad model, it dies from a bad scope
Enterprise AI projects almost never fail on model quality. They fail because the scope had no owner, no measurement, and no acceptable failure threshold. Here are the four tests I put a use case through before writing the first line.
Published August 13, 2026 · 9 min read · Sergio Nokam
I have watched enough artificial intelligence projects stall to recognise the shape their death takes. It is almost always the same, and it looks nothing like what people imagine. The model works. The demo impressed the executive committee. The prototype answers correctly in 90 % of cases. And yet six months later nobody uses it, the budget is spent, and the organisation concludes that “AI isn’t mature enough for us yet”. That diagnosis is nearly always wrong. What wasn’t mature was the scope.
Let me be precise about what I mean by scope, because the word is worn out. It is not the list of features. It is the answer to four questions that have nothing technical about them, and that most scoping exercises skip: who is accountable for the outcome, how you measure it, what error rate you accept, and what happens when the machine gets it wrong. A project that has not answered those four in writing is not a project, it is an intention.
The four tests, in one table
| Test | The question | What invalidates the use case |
|---|---|---|
| Owner | Who will answer, in 90 days, for actual usage? | A project manager accountable for delivery, not for outcome |
| Measurement | What is the starting value, today? | No measurement before deployment, or a measurement of opinion |
| Error threshold | Above what failure rate does this stop paying? | Threshold unquantified, or discovered at the first incident |
| Recourse | What does the user do when the answer is wrong? | No correction, no source, no escalation |
A use case that fails any one of these four is not ready, whatever the quality of the demo. The sections below take each in turn.
The owner of the outcome, not of the project
The first question is the one people believe they have settled by naming a project manager. It is not the same thing. A project manager is accountable for delivery; they have succeeded when the feature is live. The owner of the outcome is accountable for usage; they have succeeded only if someone uses it and the number being tracked has moved.
The absence of that second person is the most reliable predictor of failure I know. It produces a characteristic situation: the feature ships, the technical team considers the mandate fulfilled, the business team was never consulted on how it fits into the real workflow, and the tool sits in a tab nobody opens. Nobody lied, nobody did poor work, and the result is nevertheless nil.
In practice, I now refuse to start a scoping engagement unless a named person — not a department, a person — agrees in writing to be the one asked, in 90 days, whether the number moved.
The measurement must exist before, not after
The second condition seems obvious and is almost always violated. To know whether an AI feature produced value, you need to know the starting value. How many support tickets per week, before. What average handling time, before. What conversion rate on internal search, before.
That baseline is cheap to establish and impossible to reconstruct once deployment has happened. It is the first of the five refusals I document in The AI that does nothing (in French): a project with no measured starting point can never prove it produced anything. A project that begins without it condemns itself to an unfalsifiable argument: the technical team will assert it works, the business team will feel nothing changed, and neither will be able to prove it. The conversation then settles on impressions and on relative power, which is exactly how a budget gets cut.
One quality requirement on that measurement: it must be about a behaviour, not an opinion. The number of times a suggestion was accepted is a measurement. A satisfaction score from an internal survey is not — or more precisely, it is one that arrives too late and too noisy to steer anything.
A model that is wrong ten times out of a hundred is excellent or unacceptable depending on what an error costs. Until that cost is quantified, model performance means nothing.
The acceptable error threshold, agreed in writing
A probabilistic system gets things wrong. That is not a defect to fix, it is a property to manage. The operational question is therefore never “how do we reach a hundred per cent”, but “above what error rate does this feature stop paying for itself”.
That question has radically different answers case by case. A wrong product suggestion costs the user one click; a 20 % error rate is perfectly bearable there if the remaining 80 % lifts average order value. A wrong answer about a delivery date costs a support ticket and some erosion of trust; the tolerable threshold drops by an order of magnitude. An error on an accounting entry or a tax calculation costs a dispute; the threshold falls to zero, which means either systematic human validation or dropping the idea.
What I require at scoping is not a particular number. It is that the number be written down and signed before deployment. An organisation that discovers the threshold at the moment of the first incident will set it under the influence of that incident, and it will almost always be set to zero, which kills the feature.
Recourse, when the machine gets it wrong
The fourth condition is the one demos systematically hide, because a demo shows the happy path. In production, the question that determines adoption is the opposite: what does the user do when the answer is wrong.
If they have no recourse — no button to correct it, no access to a human, no way to understand where the answer came from — they will learn within 3 occurrences to stop trusting the system, and they will route around it permanently. Trust in a probabilistic tool is not built on its success rate; it is built on the predictability of its failure. A system that fails visibly, admits it, and offers a way out gets used far longer than a better-performing but opaque one.
Concretely, that means planning from the scoping stage: a one-click way to flag an error, direct access to the source of the information offered, and a human escalation with an announced response time. These are interface elements, not model elements, and they weigh more on adoption than the choice of vendor.
Should you start internal or public?
These four conditions have a practical consequence I want to state plainly, because it often runs against the client’s first instinct. They are far easier to satisfy on a narrow, internal use case aimed at an identifiable user than on a public feature aimed at all traffic.
An internal tool has an obvious owner, an observable measurement, a user you can question directly, and a bounded cost of error. Within 3 weeks you know whether it serves or has been routed around. A public feature dilutes each of those signals: the owner is diffuse, the measurement is drowned in traffic variance, the user is anonymous, and diagnosing failure takes a quarter.
I therefore systematically recommend starting internal — not out of caution, but because it is the ground on which you learn fastest. External visibility comes next, once you know how to measure. I have described elsewhere the projects that in practice repay that learning quickest1.
How to write a one-page AI brief
The conclusion is an inversion. AI projects are usually commissioned by describing what the system will do. I propose commissioning them by describing what you will be able to measure, who will answer for it, what you accept getting wrong, and what you will offer the user when it does go wrong.
That brief fits on one page, contains no technical terms, and eliminates before any investment the majority of use cases that would have failed. It is not a brilliant method; it is simply where the outcome is actually decided. The choice of model, meanwhile, has become one of the least structural decisions of the project — and it is nevertheless the one that absorbs almost the entire conversation.
Footnotes
-
On the AI projects that repay their investment on a short horizon, and how to sequence them, see Three mid-market AI projects that pay back within 90 days. On the opposite case — deployments that produce nothing — see The AI that does nothing (in French). ↩
Frequently asked questions
How do you know an AI use case is ready for production?
Four conditions must hold at once. A named person must be accountable for the outcome, not merely for the project. A measurement must exist before deployment, with a known starting value. An acceptable error rate must be quantified and agreed in writing. And a recourse must be planned for the user when the feature gets it wrong. A use case that fails any one of these four is not ready, however impressive the demo.
Should you start with an internal use case or a customer-facing one?
Internal, almost always, and for a reason that is about measurement rather than caution. An internal use case has an identifiable, reachable user whose actual behaviour you can observe. Within three weeks you know whether they use the tool or route around it. A public feature dilutes the signal in traffic noise and makes diagnosing failure far slower. External visibility is a reward, not a starting point.
What error rate is acceptable for an AI feature?
There is no universal answer: the threshold depends entirely on what an error costs and how easily it is corrected. A wrong product suggestion costs one click. A wrong answer about a delivery date costs a support ticket and some trust. An error on an accounting entry costs a dispute. What matters is not hitting a standard number, but that the threshold was stated and accepted before deployment rather than discovered after the first incident.
How long does it take to ship a first AI feature to production?
On a properly bounded scope — one use case, one type of user, one measurement — two weeks is enough to ship something genuinely used. That timeline is not a technical feat: it follows mechanically from how narrow the scope is. Projects that run six months are not more ambitious, they are badly carved up, and their duration is the symptom of a scoping exercise that never happened.