Insights
How to measure whether an automation actually worked
Here's a question that sounds simple and mostly goes unanswered: did the automation work?
Not "is it running." Almost all of them run. The system is live, it processes what it's supposed to process, the dashboard is green. By the standard most companies actually apply, that's success — and it's the wrong standard, because "runs" and "helped" are entirely different claims, and the distance between them is exactly where the disappointment with AI automation has been accumulating.
An automation can run flawlessly for a year and have made the business slightly worse. That's not a paradox. It's the default outcome when nobody defined what "helped" would mean before building.
Measure against the process, not the promise
The pitch had a number in it. Save ten hours a week, cut turnaround by half, reduce errors by some percentage. After launch, that number rarely gets checked against reality — partly because checking is work, partly because the person who'd have to check is often the person who sold it internally.
The first discipline is boring and non-negotiable: baseline before you build. How long does this actually take today, how often does it go wrong today, what does it cost today — measured, not estimated. Without a real baseline you have nothing to compare against, and "it feels faster" will fill the vacuum. It usually feels faster whether or not it is, because the visible part got quicker even when the total got slower.
Count the work that moved, not just the work that vanished
The single most common measurement error: counting the time saved on the automated task and stopping there.
The automation removed ten hours of doing. Did it add checking time? Handling of edge cases? Maintenance? Those are real hours and they usually land on someone else, which is exactly why they're easy to leave out of the tally — different person, different team, different budget line, so they never get subtracted from the headline saving.
An honest measurement is net. Hours removed minus hours created, wherever the created hours landed. Sometimes still strongly positive — genuine win. Sometimes close to zero, and you've spent build money to relocate work sideways. You cannot tell which without counting both sides, and most measurement only counts one.
Watch the second-order effects
Some of the most important consequences never show up in the metric you set.
Did quality change? An automation that's faster and slightly worse can be a bad trade even at higher throughput, but "slightly worse" doesn't trip any alarm — it shows up much later as drift in an outcome nobody connected back to the automation. Did the human skill atrophy? When people stop doing the work themselves, they slowly lose the judgement that let them catch the automation's mistakes — so your error rate can be quietly rising at the same time as your ability to notice it is quietly falling. Did it make the process rigid? An automated process is harder to change, so you may have traded flexibility for efficiency without pricing the flexibility you gave up.
None of these appear on the dashboard. All of them determine whether the automation actually helped. They have to be looked for deliberately, on a longer cycle than the operational metrics, because they move slowly and quietly.
A measurement that's worth running
Putting it together, an honest post-automation review answers five questions:
- Against a real baseline, did the target metric move — measured, not felt?
- What's the net time effect — hours removed minus hours created, everywhere they landed?
- Did quality change in either direction, and how would we know?
- What new ongoing costs did we take on — verification, maintenance, edge-case handling?
- What did we lose — flexibility, skill, resilience — that didn't show up in the headline?
Run that at ninety days and again at a year. The ninety-day review catches the projects that are quietly underwater while you can still act. The one-year review catches the second-order effects that hadn't surfaced yet.
Most companies run neither, which is the real reason the conversation about AI automation swings between hype and disappointment. The disappointment isn't usually because the automations failed. It's because nobody measured, so the wins couldn't be proven and the quiet losses couldn't be caught — and in the absence of measurement, the loudest anecdote wins. Measurement is what turns automation from a matter of faith into a matter of fact, and it's astonishing how rarely anyone bothers.
Entrepreneur, Co-Founder & COO of BoostMySites, associated with BoostMySites Global (Hong Kong). He writes about AI automation, operations, scaling and working with early-stage founders. More about Darshan →