To measure ROI after deploying an AI coding cost tool, measure the decisions it caused, not the dashboard. Record a baseline before rollout. After rollout, count only savings you can trace to one specific action, priced as before versus after on the slice of work that action touched. Add them up each month and set them against the tool's price plus the time people spend reading it. Savings minus cost is the net per month, and the month your running total turns positive is payback.
As of November 2026. The company in this post is fictional, and its numbers are invented to show the arithmetic. Maestro AI makes VibeCheck, a tool of this kind.
Keep a decision log, starting with a baseline
A dashboard doesn't lower anything. Somebody has to notice something and change a default, fix a test or cancel a seat. So keep a decision log from day one. Each row is one change made because of what the tool showed: the date, the decision, the slice of work it touches, the unit cost before and after, the volume, and who checked it. The return is the sum of that log. If the log is empty at day 90, the return is zero, however good the charts look.
You can't price a change without a "before", so record these from the three to six months before rollout:
- Monthly agent spend from invoices. The invoice is the total. Session-level estimates are for splitting it, not replacing it.
- The spend trend. At least three months, so you can see the slope.
- A spend guardrail. Cost per merged PR is the easy one: invoiced agent spend divided by human-authored PRs merged that month. It measures spend intensity, not what the work was worth (where it misleads); here it only has to show whether spend per change creeps back up. A single month is noisy, because PRs merge after the sessions behind them, so watch the trend.
- Rework share, if you can get it. The share of agent spend in sessions with heavy rework. If the tool only sees sessions from install onward, take it from the first two weeks and label it that way.
Write the numbers down somewhere finance can see them. A baseline recorded after the fact tends to drift in the tool's favor.
What counts as a saving, and what doesn't
A saving counts when you can name the action, the slice it touched, and the before and after unit cost on that slice. Price it as after-volume × (before unit cost − after unit cost). Using after-volume matters: if the slice grew, a raw before-and-after comparison understates the saving, and if it shrank, it overstates it.
The usual decisions are a cheaper default model for routine work, a rework cause fixed in one repo (a flaky test, a missing repo instruction), subagent fan-out curbed on a command that reloads the same context many times, and flat-fee seats nobody has used in two months.
Total spend going down doesn't count; in a slow month the bill falls whether or not the tool did anything. Total spend rising more slowly doesn't count either, because a migration or a new hire moves the total more than any single good decision does. "Time saved" estimates are survey answers, not invoice lines; leave them out of the log.
Totals are still worth watching, against the trend rather than a flat line. Jellyfish's AI Engineering Trends report (October 2026) found that "the median company ended the third quarter at 98% of its first-quarter headcount while spending 3.8x as much per engineer." Spend that was rising before rollout will likely keep rising, and a tool judged against a flat line looks like a failure in a quarter where it did its job. Project the pre-rollout trend forward and keep it beside the actuals. It tells you when the totals need explaining.
Reading the sessions is the work
The cost side has two lines: the tool's price and the time people spend reading it. Price the time at a loaded hourly rate, fixed across quarters; leaving it out flatters the result.
That reading is also where the savings come from, and it's the slow part. A spend chart shows that the payments repo got expensive. It doesn't show that a third of its sessions were the agent rerunning the same failing fixture. You find that by opening the most expensive sessions each week and asking what each one was for and whether it went in circles. VibeCheck does the sorting and labeling for that read: sessions ranked by estimated cost, each tagged with the kind of work and how much rework it carried.
Worked example: Kestrel Ledger's first 90 days
Kestrel Ledger is a fictional company with 30 engineers. In the three months before rollout, its invoiced agent spend was $15,000, $16,500 and $18,000, a steady $1,500 a month of growth. The trend projects $19,500, $21,000 and $22,500 for months one to three. The last baseline month had 450 merged human-authored PRs, so cost per merged PR was $18,000 ÷ 450 = $40.
Say the tool costs $1,000 a month. The engineering lead reads it about 8 hours a month at a loaded $100 an hour, which is $800. Total cost: $1,800 a month.
The decision log at day 90:
| Decision | Slice | Before | After | Monthly value |
|---|---|---|---|---|
| Cheaper default model for small bugfixes and reviews (start of month 2) | 400 sessions/mo before, 440 after | $6/session | $3/session | 440 × $3 = $1,320 |
| Fixed a flaky fixture in the payments repo (start of month 3) | Repo spend $6,000 before, $6,300 after | 30% heavy rework | 10% heavy rework | $630 (below) |
| Narrowed a test-generation command's subagent fan-out (start of month 3) | 300 runs/mo after | $5/run | $2/run | 300 × $3 = $900 |
| Cancelled four flat-fee seats unused for 60 days (start of month 3) | 4 seats | $160/mo | $0 | $160 |
| Total, monthly run rate | $3,010 |
The model-default row shows why after-volume matters. The raw drop is $2,400 (400 × $6) to $1,320 (440 × $3), or $1,080, but the team ran 40 more sessions after the change. At the old price those 440 would have cost $2,640, so the saving is $2,640 − $1,320 = $1,320.
The payments row is priced conservatively. At the old 30% rework share, $6,300 of repo spend would have carried $1,890 in heavy-rework sessions. At 10%, it carried $630, which is $1,260 less. A sample of the old sessions showed they usually finished the task at about twice the cost of a clean one, so about half was redo; the rest still delivered something. Kestrel counts half of the $1,260: $630.
At the day-90 run rate, traced savings are $3,010 a month against $1,800 of cost, a net of $1,210 a month. Leave out the reading time and it looks like $2,010; report the $1,210.
The quarter itself came out behind, because the savings arrived late. Month 1 logged nothing, month 2 had the model default and month 3 had all four: $0 + $1,320 + $3,010 = $4,330 against 3 × $1,800 = $5,400, or $1,070 short. If the decisions hold, month 4 nets another $1,210 and the running total ends month 4 at $140, so payback lands in month 4. Say so in the review, and next quarter drop any row whose "after" has drifted back.
Against the trend, month 3 came in at $21,300, $1,200 under the $22,500 projection and less than the $3,010 in the log. A database migration started that month and ran $1,900 in agent sessions. Without the four decisions the month would have been $21,300 + $3,010 = $24,310, $1,810 over trend, almost all of it the migration. The totals reconcile roughly, which is all you should ask of them. Report the $3,010, not the $1,200 gap.
The guardrail held: cost per merged PR went from $40 to $21,300 ÷ 520 ≈ $41, roughly flat with a migration added. That says spend per change didn't creep back, not that the shipped work was worth more or less. PR counts can't tell you that; AI broke them as a proxy for output.
The tool should also stop bad cuts
That migration was the most expensive work in the org in month 3. A team looking at a total or a per-engineer average would see a spike and might cap it. A team reading sessions sees six expensive sessions that each led to a merged change. The tail of the spend distribution mixes work you want with work you don't (averages hide which is which), and a blunt cut hits the big legitimate projects first. Caps, alerts or coaching covers how to govern spend without doing that.
Log these decisions too, at $0. "Did not cap the migration; it produced six merged changes" is a real outcome. Don't price avoided harm; a made-up number there undermines every real one in the log.
A 30/60/90-day plan
Days 0 to 30: baseline and read. Record the baseline before install. Roll the tool out to everyone at once; if it only sees part of the spend, it can only find savings in that part. Each week, read the 20 most expensive sessions and note what each was for. Don't change anything yet. By day 30, write down up to three candidate decisions, each with its slice and its before unit cost.
Days 31 to 60: change one thing per slice. Finish by about day 60 so each change has four weeks of "after". Keep them separate: change the default model and the repo instructions in the same repo the same week and you can't price either. Tell the team what changed and why.
Days 61 to 90: price and report. Fill in the log, compute the monthly net and the payback month, and check that cost per merged PR hasn't risen. Then decide whether to renew, expand or drop the tool, and say which log rows the decision rests on.
If you're reporting upward, how to measure the ROI of AI coding tools covers whether the agents themselves pay off, and reporting engineering ROI and capacity allocation to the board shows where AI spend sits among the other engineering lines.
Where VibeCheck fits
VibeCheck captures Claude Code, Codex and Cursor sessions on each engineer's laptop and estimates each session's cost from its own tokens. Sort by highest cost for the weekly read. Filter by kind of work and model to price a model-default change, or by repo and heavy rework to price a rework fix. Costs are estimates at published reference prices, so use your invoices for the total. Pro adds spend by team and skill, and cost per authored PR with the GitHub App; that's authored, not merged. Free covers up to five engineers, enough to run the weekly read on one team. Get started for free.