parallax circle item parallax circle item parallax circle item
Insights & Blogs

We used AI to fix a problem AI created

At one point during this I said out loud that we were using AI to build an AI dashboard to track our AI costs. Ah, the irony. 

 

The problem AI created

AI spend doesn't behave like other software spend.

Most software you buy once a year. You agree on a price, multiply by the number of people who need it, and look at it again at renewal. AI is a little different. It turns up inside things you already pay for, the way Gemini appeared across our whole business the day Google put it in the Workspace bundle. Every vendor has different token, credit or usage-based pricing. And the billing model can change underneath you.

As a software development business that has adopted the latest in AI tooling across our entire organisation, that last part is what caught us. In June, GitHub moved Copilot from premium requests to token-based credits. Plenty of notice, published openly, and nothing underhanded about it. With over 120 Sandfield developers using Copilot daily, our cost still went up around six times, because the unit being counted changed. The price per seat never moved.

It also isn't a small-company problem. Uber spent its entire 2026 AI budget in four months after putting up an internal leaderboard ranking teams by usage, and their COO's public comment afterwards wasn't really about the money. It was that he couldn't draw a clear line between all that usage and the features actually reaching customers. Flexera surveyed 512 technology professionals this year and found only 31% had accurate visibility into what they spend on AI software. 

 

What we'd have done last year

We wouldn't have ignored the problem. We'd have done what most businesses are doing right now, which is open a spreadsheet.

A perfectly reasonable approach, and for a while it was ours. Someone pulls the invoices at month end, converts a few currencies, works out which slice of a mid-month subscription change belongs to which team, and pastes it together. It gives you a number but it’s out of date the moment it's finished, it tells you what happened rather than what's happening, and it quietly eats a day of someone's time every month.

The better answer was always a dashboard pulling live from the vendor APIs. We knew that but building one used to mean a business case, a project manager and a slot in the development queue, competing against work that earned revenue. For a piece of internal reporting, the effort and the cost never justified the return. It's just the maths, and every organisation has a list of good ideas sitting permanently below that line because the build was worth more than the problem.

The difference today is that the build got cheaper and faster thanks to AI. The problem stayed roughly the same size and the cost of solving it fell through the floor, so something that never used to add up suddenly paid for itself in weeks.

That's the opportunity that is present today. Not the enormous AI transformation programme. The small, irritating, genuinely expensive things you've been living with because they were never worth a project.

 

So we just built it

I'm not a developer. I had the first go at it myself in Google AI Studio, connected a couple of the vendor APIs, and had a working dashboard inside a couple of hours. I felt quite pleased with myself.

Then I spent considerably longer than a couple of hours failing to make the rest of it work.

Worth knowing before you get carried away. With AI, the first cut is the most impressive thing you'll produce. Someone who isn't a developer gets to roughly 70% surprisingly fast, then finds out the last 30% is the actual job.

So I handed it to our own developer team, who knew how to do this properly. They rebuilt it with Claude, hosted it in our own AWS environment, and got the rest of the connections working. The hard parts were never generating the code. Vendors bill in different currencies on cycles that don't line up, and not every vendor exposes full cost data through an API, so some of it still gets typed in by hand.

The bigger complication was making the total mean something. Knowing the total cost is easy, but it tells you almost nothing. What we wanted was cost by person, by team and by customer, and vendors have no idea who our customers are. They know about seats, API keys and usage. Mapping that back onto the way we actually organise work inside our job management system is the part that takes thought, particularly when a developer might spend a single week across three different clients.

Then there's everything that moves. People join partway through a month. People shift between subscription tiers when their usage changes. Both need pro-rating, so we built that into the dashboard against a start date rather than leaving it as a monthly manual correction. 

When that is right, the questions change. We could go from asking what the business spent on AI last month, and start asking whether a particular team is consuming what you assumed it would, which teams have genuinely adopted the tools and which haven't, and who's about to run past their limits.

What it was worth

The first thing it did was let us make a decision we'd been circling for months. We moved off Copilot onto Claude subscriptions, sized so most usage sits inside the subscription rather than spilling into extra credits, and our monthly AI cost dropped to a fraction of what it had been.

We'd have suspected that was the right move anyway. 

The rest is probably less interesting but more valuable. We know every vendor's billing cycle, so changes get planned in advance. We can see which teams are near their limits before they hit them. And we can start to provide clearer indicators of AI productivity gains against the cost of using AI.

It opens up the next set of decisions too, and those are the interesting ones. Monitoring individual agents and routing tasks by complexity, so code formatting, comment generation and documentation stop running on the frontier models.Those are for complex problem-solving and new platform development. Solutions like OpenRouter and LiteLMM make that routing straightforward once you know the complexity and consumption pattern of each type of work. Also enabling us to send tasks to our recently purchased hardware running an open-weight model locally, where the marginal cost of a request is electricity. Open models are cheaper, sometimes dramatically so, but require testing for capability and performance, plus vetting for your security and compliance requirements.

 

The real takeaway

I don't think the dashboard is the interesting thing here. It’s a common solution to a common data problem

The interesting thing is that a job which would have stayed in a monthly spreadsheet indefinitely got built properly in a few weeks, mostly by people fitting it around other work. Nobody had to make a case for it, because there was no case to make. The bar for what's worth building moved, and it moved a long way.

We pointed that at AI costs because it was a constantly changing environment where timely, accurate information was vital to make good decisions and bring our costs under control. The same maths applies to every other quiet, expensive annoyance sitting on your list.

profile picture of Justin Knight
Posted by Justin Knight

Justin Knight leads IT Operations at Sandfield, where his team manages security, availability and performance across customer systems.

Outside of work Justin is usually busy working with his family on their little farm, planting trees, raising animals and enjoying the produce from their land. Playtime often involves kayaking or swimming at the local beaches and estuary or walking and mountain biking through the forests around Northland.

Follow us for the latest insights