An AI pilot can look busy and still deliver little

An AI pilot can generate plenty of visible activity: workshops, prompt libraries, demonstrations and enthusiastic updates. None of that tells you whether the pilot is useful. The real test is whether a defined piece of work becomes better enough to justify the cost, attention and risk of continuing.

Reading Room’s 2026 research offers a useful warning. Its survey of 150 senior digital and marketing leaders in medium to large UK organisations identified lack of effective measurement among the main barriers to AI having greater impact. This was not a charity survey, so its percentages should not be treated as evidence about the sector. The underlying question still travels well: what would convince you that your pilot deserves to continue?

Start before the tool

Take a charity testing AI to draft first versions of supporter email copy. Before the pilot, record a small baseline: how long a typical draft takes, how many review rounds it needs, and what quality problems reviewers commonly find. Without that starting point, “it feels quicker” is about as far as the evaluation can go.

Then measure the whole task, not the impressive moment when text appears. If a draft takes two minutes to generate but twenty minutes to correct, fact-check and reshape, those review minutes count. So does time spent writing prompts, handling exceptions or explaining why an unsuitable draft cannot be used.

Use four lenses

Time is only one lens. I would judge a small pilot across four:

• Net time: time saved after review and correction, compared with the baseline.
• Quality: whether the output meets agreed standards for accuracy, tone, accessibility and usefulness.
• Adoption: whether the intended people use it in real work, rather than only during organised testing.
• Risk: what data enters the tool, what errors could cause harm, and whether controls are workable.

Choose a modest review point in advance: perhaps after a fixed number of real tasks, not an open-ended “pilot period”. Agree who will assess the evidence and what minimum result would justify further investment.

Scale, change or stop

The decision is not simply success or failure. A pilot may save time but produce uneven quality: change the workflow or narrow the use case. It may deliver good drafts that nobody uses: investigate training, trust or poor fit. It may create little benefit once checking is included: stop.

Stopping is not wasted work if the pilot answered a clear question. The waste is allowing activity to substitute for evidence. Before your next AI experiment begins, write the scale, change and stop criteria on one page. If you cannot define them, the pilot is not ready.