How to tell whether an AI tool is actually helping

Guy PavlovPublished

Write down the baseline before anyone turns a tool on: hours, errors, turnaround, or cost, over a period you can name. Agree the success measure in writing. Check it after people have used the workflow, not on the day you launch.

A few studies show what "helped" has looked like when someone measured it:

  • Customer support: AI assistance raised issues resolved per hour by 15% across 5,172 agents, with the biggest gains for less experienced staff (Brynjolfsson, Li and Raymond, QJE 2025).
  • Knowledge workers: in a randomized trial of 7,137 workers, Copilot access cut email time by 1.3 hours a week on average, and 3.6 hours among regular users, with no change in meeting time (Dillon et al., NBER 2025).
  • Online retail: randomized experiments found effects from no detectable change to 16.3% higher sales, depending on the use (Fang et al., 2026).

Hours saved free capacity. They do not, by themselves, cut payroll. Report hours recovered, costs removed, extra work delivered, and extra gross profit on separate lines.

To see which area to work on first, take the free AI readiness self-check.

Bring one task to a call.

Pick the one that takes your team too much time, and we'll work out whether a review makes sense. 30 minutes, free, no preparation needed. I'll tell you on the call if there's no opportunity.

Book a call

Guy Pavlov The call is with me.