FinOps for AI workloads: what your CFO will start asking
AI spend is getting its own line in the budget, and finance will want to know what it buys. The habits that keep a cloud bill under control work on tokens too.
· 7 min read
For most of the last decade, FinOps meant cloud cost: right-sizing instances, buying reserved capacity, killing idle resources, and tagging everything so you could tell the marketing team’s QA environment from engineering’s load test cluster.
It now means AI cost as well, and the two are hard to pull apart. A product with AI features runs on cloud infrastructure and calls a model on every request, and both show up on the same invoice.
If you run a product team, your CFO has probably noticed already. The question used to be “why is the AWS bill up 12%?” Now it’s “why is the AI line up 90%, and what are we getting for it?” Have an answer ready before someone asks.
The Shape of an AI Bill
On the surface, cloud and AI costs look alike. Both are metered, both grow with usage, and both arrive monthly. What drives them is different.
Cloud costs are mostly provisioned. You pay for capacity whether you use it or not, so you save money by right-sizing, scheduling, reserving capacity, and shutting things down at night.
AI costs are mostly per call. You pay for every token in and every token out, which gives you a different set of levers:
- Which model you call. A 10x price gap between a small model and a frontier model is normal. Sending the easy cases to the cheap model and escalating only the hard ones is often the biggest saving available.
- How much context you send. You pay for every token in the prompt. Sending the whole chat history, the whole knowledge base, or an oversized system prompt on every call is a cost you can audit and cut.
- How long the answer is. Output tokens usually cost 3 to 5 times as much as input tokens, so asking for shorter, structured answers saves real money.
- How many tool calls the agent makes. Each round of “think, call a tool, read the result” is another round of tokens. An agent that finishes a task in 5 turns instead of 12 costs noticeably less.
- Whether you cache. Prompt caching, now offered by most providers, can cut the cost of repeated context by 50 to 90%. Most teams aren’t using it.
These levers multiply. Take a support assistant handling 2,000 conversations a day, with three model turns per conversation:
Naive build prompt: 6,500 tokens/turn (full history + full system prompt + 10 RAG chunks) output: 450 tokens/turn model: frontier ($3.00/M input, $15.00/M output)
daily input: 2,000 × 3 × 6,500 = 39.0M tokens → $117.00 daily output: 2,000 × 3 × 450 = 2.7M tokens → $40.50 ≈ $4,725/month
After three afternoons of work trimmed history + 4 RAG chunks: 6,500 → 2,200 prompt tokens cached system prompt (90% discount on 1,400 of those) structured outputs: 450 → 220 output tokens routing: 70% of turns on a small model at 1/10th the price
≈ $410/month, same eval scoresThe product and the users are the same, and the spend is about 91% lower. None of these changes is unusual; they are the AI version of turning off the dev cluster at night. The mistake we see most often is treating AI cost as fixed, when it responds to engineering the same way a slow query plan or an oversized container does.
Why this is reaching the CFO now
Three things changed in the last year that moved AI cost from engineering’s problem to finance’s.
1. The line item got big enough to notice. At a few thousand dollars a month, AI cost hides inside the cloud budget and nobody asks about it. At six figures a month, which is becoming common for products with heavy AI use, it gets its own row in the board deck.
2. Nobody can say which feature it pays for. Cloud costs map fairly cleanly to features: this database serves checkout, this cluster serves search. AI costs often don’t. One LLM call might serve five features, and splitting the cost between them takes instrumentation most teams haven’t built.
3. Unit economics are starting to matter. For two years the line was “AI is the new electricity.” Product owners are now asking a harder question: what does an active user cost us in tokens, and is it less than they pay us? For some products the answer is comfortable. For others it’s ruinous. You want to find out which before the CFO does.
Attribution: The Unglamorous Prerequisite
Every lever above depends on one dull capability: knowing what each call cost and what it was for. In practice, every model call records its token counts and its owner, ideally on the same traces your observability stack already collects:
def record_cost(response, *, tenant: str, feature: str, model: str): usage = response.usage # provider-reported counts are authoritative price = PRICE_TABLE[model] # your local price table gives the estimate cost = (usage.input_tokens * price.input + usage.output_tokens * price.output) / 1_000_000
span = trace.get_current_span() span.set_attributes({ "gen_ai.usage.input_tokens": usage.input_tokens, "gen_ai.usage.output_tokens": usage.output_tokens, "gen_ai.request.model": model, "app.cost_usd": round(cost, 6), "app.tenant": tenant, "app.feature": feature, })That’s fifteen lines per call site, or none if your client wrapper does it once for every call. It changes what a bad Monday looks like. “The bill went up 40%” needs a war room. “The summarizer went up 40% for these three tenants, starting with Thursday’s deploy” needs a Jira ticket.
Attribution is also what makes model routing safe. Try the cheap model first, check the result, and escalate only when the check fails:
def complete(request): draft = call("small-model", request) if passes_confidence_check(draft, request): # judge, self-eval, or heuristic return draft # ~70-85% of traffic stops here return call("frontier-model", request) # the hard cases earn the priceOn workloads we’ve measured, this cuts model spend by 40 to 70% with no quality change that the evals can detect. Without an eval harness you can’t make that claim, because nothing would tell you the cheap model had started giving worse answers. That’s why we treat cost control and evals as one piece of work.
One discipline for both bills
Controlling cloud cost and controlling AI cost take the same practices, applied to different meters:
- Tagging. Every cloud workload is tagged with its team, product, and feature, and every model call should carry the same tags. Without them, you can’t allocate cost.
- Dashboards. Spend per team and per feature, visible to the people who can change it. For cloud that’s a Cost Explorer view; for AI it’s a dashboard built from your provider’s usage API or your own span attributes.
- Anomaly alerts. A workload’s cost shouldn’t double overnight. When it does, someone changed something, and you want to know within hours instead of at month end.
- Architecture reviews. Large cost lines should be defensible. “Why is this workload on a managed database instead of self-hosted?” and “why is this prompt 8,000 tokens instead of 800?” are the same conversation.
The teams we’ve seen do this well run the cloud bill and the AI bill as one budget with two cost centers. Finance gets one number, and engineers get two sets of levers they already know how to pull.
What Good Looks Like
These practices separate teams that have AI cost under control from teams that don’t:
Cost as a CI metric
Every PR that touches an AI feature reports the change in cost per call next to the eval scores. A prompt change that improves quality 2% and triples token spend is a trade-off someone should make on purpose, in review, with the numbers in front of them, instead of finding it on the invoice.
Per-feature unit economics
Each AI feature has a tracked cost per active user, refreshed weekly and owned by the product manager rather than the CFO. If that number drifts toward what the user pays, the PM finds out before finance does.
Model routing by default
A small model handles the easy cases, with explicit rules for when to escalate. New features start on the cheap model and move up only when evals justify it. Defaulting to the frontier model "to be safe" is the AI version of over-provisioning.
Caching by default
Prompt caching for system prompts and shared context, result caching for repeated queries, and embedding caching where retrieval is stable. Cache hits are the cheapest tokens you will ever buy.
Budgets that enforce
Per-request and per-tenant budgets, soft at first (warn and continue) and hard later (fail with a graceful fallback) once the workload is sized. A hard budget from the start mostly produces mysterious failures; a soft one produces the data you need to set the hard one.
Quarterly cost reviews
The same cadence and audience as your cloud cost reviews, asking what grew, why, and whether it was worth it.
The questions your CFO is going to ask
Answer these before the meeting and it will be a short one:
- What is our AI spend this quarter, broken down by product, team, and feature?
- What is our AI cost per active user, and is it growing faster than revenue per user?
- What does one more user cost us, and does our pricing cover it?
- If our AI vendor raised prices 30% tomorrow, what would happen to our margin? What would we change, and how fast?
- Where are we paying for capacity (reserved instances, GPU clusters) and where are we paying per call (LLM APIs)? Is the capacity side sized right?
If you can answer these in five minutes, you have FinOps. If you can’t, that’s the work.
The pattern that keeps repeating
Every new category of infrastructure spend goes through the same arc. First, engineering teams grow it fast and expense it; later, finance and engineering govern it together. Cloud went through this between 2017 and 2020. AI is going through it now.
Teams that build cost visibility, model routing, caching, and unit economics in before the CFO asks for them keep the conversation in engineering’s hands, and they have the margin to show for it when budgets tighten.