Demo · AI strategy & economics

Did the AI pilot work, or did you just pick good teams?

Most AI pilots are judged with a before-and-after chart or a comparison against teams that didn't get the tool. Both are easy to fool: the volunteers were often your strongest teams, and other things changed while the pilot ran. This demo simulates a support organization where the tool's true effect is known, measures it three ways, and shows how far each answer moves the ROI.

What really happened

True effect of the AI tool Pilot teams were already ahead by Company-wide change at launch time Pilot teams were also improving faster (per month)

The pilot

Teams in the pilot (and as many without) Weeks before and after launch Random seed

The money

Annual labor cost of the teams using the tool ($M) Tool cost per seat per month ($) Seats One-time rollout and training cost ($) Share of freed-up time actually put to use

"Already ahead" is selection: volunteers or hand-picked teams. "Company-wide change" is anything else that moved at the same time, like a new knowledge base or a seasonal lull. Both are common in real pilots.

Three answers to "did it work?"

Check the assumption before you trust the answer

Difference-in-differences assumes the pilot and comparison teams were on parallel paths before launch. You can't prove that for the period after launch, but you can check the months before it.

How each method works

Before and after compares pilot teams' productivity before and after launch; it also counts anything else that changed at the same time. Pilot vs. everyone else compares the two groups after launch; it also counts however far ahead the pilot teams already were. Difference-in-differences takes the pilot teams' change and subtracts the comparison teams' change over the same weeks, which removes both, as long as the two groups were on parallel paths. Its 95% range comes from the spread across teams. First-year ROI counts the productivity gain as labor capacity worth that share of annual labor cost, times the share of freed-up time actually put to use (time saved that nobody redeploys isn't a saving), against the tool's annual seat cost plus the one-time rollout cost.

All data is simulated: each team has its own baseline, weekly noise and the effects set on the left; productivity is measured as tickets resolved per agent per week. Real pilots also need to watch for the tool changing what gets measured (for example, faster but lower-quality resolutions).