An AI document-extraction pilot that had to earn its place

The problem

A finance operations team processed thousands of supplier invoices a month by hand. Leadership asked what the company’s “AI strategy” was, and the honest answer was that there wasn’t one. The team had a specific, repetitive, high-volume workflow — exactly the kind of candidate worth testing — but no definition of what success would look like.

Constraints

• Invoice data is sensitive; nothing could go into a tool without a clear data-handling answer.
• A wrong number posted to the ledger is expensive, so accuracy mattered and the acceptable threshold was not obvious.
• The team had been promised “AI magic” before and were rightly wary.

What we did

The first question was not which model to use. It was whether this workflow needed AI at all, and what “good enough” would mean. Week one produced the unglamorous but decisive artefacts:

• A measurable acceptance bar — field-level accuracy on the fields that actually matter (totals, dates, supplier, line items), agreed at 94% before a line was built.
• A labelled evaluation set drawn from real invoices, including the awkward ones.
• A data-handling assessment: what leaves the boundary, what is retained, and the vendor terms.
• The fallback: what happens when the model is unsure — it routes to a human, it does not guess.

Week two built the smallest prototype that ran the real workflow on real documents, with a person reviewing every extraction. Week three measured it against the bar: accuracy on the fields that matter, cost per transaction at realistic volume, latency, and the failure modes.

The outcome that mattered

The prototype cleared the accuracy bar on the common cases and flagged the awkward ones for review rather than guessing — which is the correct behaviour, not a shortfall. Just as important, the evaluation harness built in week one meant “is it good enough?” became a measurement rather than an argument, and a future model change becomes a re-run rather than a leap of faith.

For a different team on a different workflow, the honest recommendation has sometimes been the opposite: that deterministic software or a fixed process solves the problem better and cheaper than AI. Producing that answer in week three is worth far more than the sprint costs — and we would rather say it then than after a year of effort.

Lessons

The discipline is dull and it is the whole game: pick one measurable use case, set the acceptance bar before building, keep a human accountable, and design the fallback for when the model is uncertain. Novelty is not a business outcome; a workflow that is faster and still correct is.