Why most AI pilots fail before they reach production
The gap between a promising demo and a useful production system is often a data, governance and operating-model problem.
A proof of concept that impresses a steering committee is not the same thing as a system people rely on every day. Across East Africa we see the same pattern in banks, SACCOs, insurers and NGOs: a pilot works on a curated dataset, the demo goes well, and then nothing happens for six months.
The model is rarely the problem. The work around it usually is.
1. The pilot was built on data nobody will maintain
Pilots are often trained on a one-off extract that a data analyst cleaned by hand. In production, data arrives every day from core systems, M-Pesa statements, spreadsheets and field forms, and it is messier than the extract ever was.
Before a pilot starts, agree where the data will come from in production, who owns each source, and what happens when a feed breaks. If those answers are missing, the pilot is a demo.
2. Nobody owns the decision the model supports
A credit score, a fraud alert or a demand forecast only creates value when it changes a decision. Ask early:
- Which decision will change, and who makes it today?
- What will they do differently when the model says something new?
- How will we know the new decision is better?
If the business owner cannot answer these, the model will sit in a dashboard that nobody opens.
3. Governance arrives too late
Risk, compliance and data protection teams are frequently brought in after the build. They then ask reasonable questions about explainability, consent and data retention, and the project stalls.
Invite them in week one. Under the Kenya Data Protection Act, automated decisions about people need a lawful basis and, in many cases, a data protection impact assessment. These are much easier to design in than to retrofit.
4. There is no plan to run it
A model in production needs monitoring for accuracy and drift, a retraining routine, access controls and someone on call when it fails. That is an operating model, not a technical detail.
A pilot answers "can this work?". Production answers "will this keep working, safely, without the team that built it?"
What we do differently
We scope pilots backwards from production. Every engagement starts with the decision, the data owner and the success measure, and ends with a runbook the client's team can operate. Pilots take a little longer to start, and far less time to deliver value.
If you have a promising pilot that has stalled, we are happy to review it with you.
