Pilots that prove something

Most AI pilots end in a slide deck. A good pilot ends in a decision: scale it, change it or stop it. That only happens if success is defined before the first line of code.

An engineer working at a computer workstation
Photo: ThisisEngineering on Unsplash

The phrase “pilot purgatory” exists for a reason. Organisations run proof after proof, each one technically impressive, and few of them ever reach production. When we look at why, the cause is rarely the technology. It is that nobody agreed in advance what the pilot was supposed to prove.

Choose one or two pilots that matter

Ambition is good. Scattering effort across ten use cases is not. We advise institutions to pick one or two pilots tied to a decision that is frequent, costly and owned by a specific leader. If the pilot succeeds, someone should be eager to scale it.

ProblemApproved dataSuccess criteriaPilotMeasured result
A pilot that proves something starts with criteria agreed up front.

Agree the measures first

Every pilot we run has its measures written down before work begins. They are always in the language of operations, not data science:

  • Government: answer accuracy, response time, time saved and audit-trail coverage.
  • Defence: alert accuracy, false alarms, time to detect, downtime and equipment availability.
  • Healthcare: records digitised, documentation time, reporting delay and data completeness.
  • Transport and logistics: forecast error, stock exceptions, vehicle availability and on-time delivery.

If you cannot say in one sentence what would make the pilot a failure, you are not ready to start it.

Work only on approved data

Pilots are often run on convenient sample data and then struggle with the real thing. We prefer the opposite: a smaller scope, but on approved production data, networks and hardware. The results are slower to get and far more believable.

Plan the scale-up on day one

Before a pilot begins, sketch what production would look like: who would use it, what it would connect to, how it would be secured and supported. Many pilots that “worked” were never designed to survive contact with real operations.

Want to explore how this applies to your organisation? We share our approach with qualified organisations in a private briefing.