The meter is always running
Contents
About Microsoft
Microsoft, founded in 1975 by Bill Gates and Paul Allen, headquartered in Redmond, Washington, and led by chairman and chief executive Satya Nadella since 2014: is one of the world's most valuable companies (NASDAQ: MSFT). The Microsoft Cloud spans Azure, Microsoft 365, Dynamics 365, Microsoft Security, Microsoft Fabric and GitHub. Volge delivers across that estate through six alliance offerings.
What Volge delivers on Microsoft: six offerings
The offering this book covers: marked ◆: is Cloud Infrastructure & Operations, delivered on Microsoft Azure.
| Volge offering | What it covers | Anchored on |
|---|---|---|
| Modern Workspace & E7 | Secure modern work on the E7 bundle; human-agent collaboration | Microsoft 365 · Copilot |
| Applied AI | Building, governing and running enterprise AI agents and apps | Azure AI Foundry |
| Zero Trust Security | Identity, threat protection and data governance | Microsoft Entra · Defender · Purview |
| Business Applications | ERP, CRM and low-code, with in-process agents | Dynamics 365 · Power Platform |
| Digital Product Engineering | Autonomous software engineering and CI/CD | GitHub · GitHub Copilot |
| Cloud Infrastructure & Operations ◆ | Cloud build, run and cost governance (FinOps) | Microsoft Azure |
Sources: Microsoft corporate and product documentation, 2025–26.
This book: Cloud Infrastructure & Operations on Microsoft Azure
Microsoft Azure is Microsoft's public cloud for compute, storage, data, networking and AI, billed on consumption: which makes cost governance a first-class operating concern.
In the field in 2026
- Microsoft Cost Management, cost analysis, budgets, alerts, anomaly detection and allocation for showback/chargeback
- Copilot in Azure for cost management: natural-language analysis and optimisation of cloud and AI spend
- Azure Advisor recommendations plus reservations, savings plans and Azure AI Foundry provisioned throughput (PTUs)
- The Microsoft FinOps toolkit and FinOps hubs, open-source, FOCUS-aligned (toolkit v14, April 2026)
Executive summary
An enterprise AI deployment no longer carries a price that a budget can fix in advance. The licence fee has been replaced by a meter, and the meter runs on two readings: every token a model processes, and every hour a graphics processing unit (GPU) sits reserved. What turned a predictable line item into a moving one is the operator. Autonomous agents, not human prompts, now drive the readings, and an agent consumes infrastructure in a pattern that a chatbot never did.
The pattern is the whole story. A chatbot exchanges one prompt for one completion. An agent breaks a goal into steps, re-sends its accumulated context at every step, calls tools, and loops until the task closes. Gartner's March 2026 analysis found that agentic AI models require 5 to 30 times more tokens per task than standard chatbots (Gartner, 2026). The cost driver therefore sits in the architecture, not in any single careless prompt, and it cannot be negotiated away with the vendor.
Place that multiplier against the size of the spend and the exposure becomes plain. Gartner forecasts worldwide AI spending will reach USD 2.59 trillion in 2026, a 47 per cent year-on-year increase, with AI infrastructure alone projected at USD 1.43 trillion, more than half the total (Gartner, 2026). The share of that spend that delivers nothing is the part leadership controls. The FinOps Foundation's 2026 State of FinOps report found that 73 per cent of enterprises reported their AI costs exceeded original projections (FinOps Foundation, 2026).
The waste is not an estimate hedged in adjectives; it is measured. GPU fleets have been observed running at utilisation as low as 5 per cent, which leaves 95 per cent of provisioned capacity paid for and untouched (Rack2Cloud, 2026). Across cloud generally, roughly 27 per cent of spend is waste, more than USD 100 billion globally in 2026, and that rate has barely moved since 2019 (Spendark, 2026). The failure mode has a face: one healthcare enterprise burned one trillion tokens over six months, more than USD 6 million unbudgeted, before its finance team could even name the workload responsible (FinOps Foundation, 2026).
The central finding of this paper is the following. Controlling agentic cost requires no new technology. It requires a discipline already proven on cloud spend, redirected at the token and the GPU-hour: usage visibility, model right-sizing, caching, provisioned-throughput planning, and resource optimisation. This is FinOps for AI. The enterprises that run it keep their deployments profitable. The enterprises that file the AI bill under the cost of innovation hand between a quarter and a half of their AI budget to waste, quarter after quarter.
Five conclusions are advanced, each developed in the chapters that follow.
First, agentic cost is volatile by construction, and the construction can be re-engineered. Both meters, tokens for inference, hours for GPU compute, read against usage rather than against value delivered. A multi-step agent multiplies token consumption by the number of steps it takes. Because the volatility is built into the architecture, it yields to architectural controls and to nothing else.
Second, the bill cannot be cut before it can be read, and most enterprises cannot read it. A token charge that cannot be attributed to a team, a workload, a model, and a use case has no accountable owner and no baseline to improve against. Most enterprises lack real-time cost visibility and therefore cannot measure cost per token or unit economics (FinOps Foundation, 2026). Tagging, showback, and token telemetry come before every lever that follows.
Third, the levers are known, sourced, and large, and they stack. Model right-sizing, prompt caching, batch processing, provisioned throughput, and retrieval efficiency each carry documented savings. The Batch application programming interface (API) applies a flat 50 per cent discount on qualifying workloads (Microsoft, 2026). Prompt caching cuts cached input-token cost by 50 to 90 per cent depending on deployment type (Microsoft Foundry documentation, 2026). Right-sizing from a large to a small language model can cut cost per million queries by more than 100 times on suitable tasks (Index.dev, 2026).
Fourth, the provisioned-throughput-versus-pay-as-you-go choice is the single largest structural lever, and it is a utilisation bet. Provisioned Throughput Units (PTUs) deliver up to 70 per cent savings on sustained workloads, but only once utilisation crosses a threshold (CloudZero, 2026). Below it, pay-as-you-go wins. The bet pays only when it is placed on 30 to 60 days of real telemetry, never on a forecast.
Fifth, FinOps for AI is an operating model, not a tool in the procurement basket. It needs named ownership, a tagging standard, a token-telemetry pipeline, a forecasting cadence, and a unit-economics framework that ties token spend to business outcomes. Most FinOps practices now manage AI costs, and the discipline has hardened into a defined practice with its own certification (FinOps Foundation, 2025).
Azure is the lens of this paper, reflecting its standing as the default cloud and AI estate of the large enterprise through Azure OpenAI and Microsoft Foundry. The levers travel across clouds; the pricing mechanics, the PTU model, and the cost-management tooling are stated in Azure terms. Pricing figures are stated as of June 2026 and change frequently; reconfirm them before any commitment.
Volge's position is plain. The cost of agentic AI is not a tax on ambition. It is a managed variable, and the means to manage it are mature, documented, and on the shelf. What follows sets out how to use them.
Chapter one: Why agentic costs are volatile: token and GPU economics
The takeaway
Agentic cost moves because two meters run at once, and both read against usage rather than against value: tokens for model inference, hours for GPU compute. A multi-step agent inflates the token meter by re-sending its full context at every step; a reserved GPU charges the hour whether it computes or idles. The volatility lives in the architecture, which is the good news of the rest of this paper: architectural and financial controls reach it directly.
2.1 The two cost axes, defined early
Loose language about "the AI bill" hides the fact that there is no single bill. There are two charges, billed on different bases and reached by different levers, and conflating them defeats optimisation before it starts.
The token axis prices model inference. A token is the atomic unit of text a language model processes, roughly three-quarters of a word. Azure OpenAI and Microsoft Foundry meter input tokens and output tokens separately, at different rates. Output is typically billed at four times the input rate, which makes a verbose model a quietly expensive one.
The GPU axis prices compute capacity. A GPU is the specialised processor behind model training and self-hosted inference, billed per hour of reservation regardless of whether it does useful work in that hour. Azure's ND H100 v5 instance is priced at approximately USD 12.29 per GPU-hour (Spendark, 2026).
These are not two ways of paying for the same thing. A managed Azure OpenAI estate pays mostly on the token axis. A self-hosted open-weight estate pays mostly on the GPU axis. Run at scale and an enterprise pays on both, which is why a FinOps discipline that covers only one of them leaves money on the table.
2.2 Why agents multiply the token bill
Token spend on an agentic workload does not track the number of tasks. It tracks the number of reasoning steps inside each task, and that is a different and steeper curve.
The mechanism is re-sending. A reasoning agent does not fire one prompt and take one completion. At every step it ships the full accumulated context, system prompt, conversation history, and every prior tool result, back to the model. Reach step 20 of a task and the agent has paid for that same context 20 times (LeanOps, 2026).
The multiplier is documented, not theoretical. Gartner's March 2026 analysis put agentic AI at 5 to 30 times the tokens per task of a standard chatbot (Gartner, 2026). Practitioner measurements run higher still, with agents burning tokens 10 to 100 times faster than chatbots because every reasoning step adds context that rides along on the next tool call (LeanOps, 2026).
Reasoning models steepen the curve again. A model that works through a problem step by step can demand 5 to 30 times the compute of a standard response, and a single query routed through a reasoning model can cost roughly six times the same query through a standard one (Index.dev, 2026; instinctools, 2026).
By the numbers: the agentic token multiplier
- 5–30× more tokens per task for agentic AI versus standard chatbots (Gartner, 2026).
- ~30× higher cost for a complex orchestrated interaction (USD 1.20) versus a simple 2023 workflow (EY / Exceeds.ai analysis, 2026).
- 24× projected increase in enterprise token consumption by 2030, driven largely by agentic workflows (Goldman Sachs, cited 2026).
- 483% growth in total enterprise AI spending from 2024 to 2026, despite per-token prices falling roughly 80 per cent since mid-2023 (industry analysis, 2026).
The last figure carries the chapter. Per-token prices have collapsed, and yet total enterprise AI spending has climbed 483 per cent over the same window (Optimum Partners, 2026). Falling unit prices cannot defend a budget when unit volumes rise faster, and agentic consumption is exactly the pattern that makes volumes rise faster. That is why cost discipline cannot be outsourced to vendor price competition, the competition has been fierce, and the bills have grown anyway.
2.3 Why GPU compute is volatile and wasteful
The GPU axis is volatile for the opposite reason. It charges the reserved hour, not the productive hour, and reserved GPUs idle far more often than they run.
The shortfall is large and measured consistently. GPU clusters have been observed averaging utilisation as low as 5 per cent, leaving 95 per cent of capacity dormant, though optimised fleets have held 49 per cent, proof the ceiling is a choice, not a hardware limit (Rack2Cloud, 2026). Broader studies place GPU utilisation between 30 and 50 per cent across cloud deployments, with many teams wasting 30 to 50 per cent of their GPU budget on capacity provisioned but never put to work (Spheron, 2026).
The idle cost is concrete enough to invoice. One high-end training instance left running across a 48-hour weekend costs approximately USD 1,573, and a month of casual overnight and weekend idling typically wastes USD 3,000 to USD 8,000 per instance (Spheron, 2026). Spread across a team of ten machine-learning engineers, idle notebook waste commonly reaches USD 500 to USD 2,000 per month in Azure Machine Learning environments (Spheron, 2026).
The root cause is hoarding. A team reserves capacity to guarantee it is there when needed, then under-uses it the rest of the time. The reservation is rational for the team and wasteful for the enterprise: the classic coordination failure that a FinOps discipline exists to resolve.
2.4 The cost-driver breakdown
The diagram below separates the token axis from the GPU axis and surfaces the charges that ride alongside both.
The surrounding charges are not rounding error. Azure adds an estimated 15 to 40 per cent on top of headline model pricing through support plans, data egress, fine-tune hosting, Private Link, and Log Analytics (CloudZero, 2026). An estimate built on headline token prices alone understates the real bill by that margin, and the gap can be extreme: one published analysis recorded a USD 4 estimate against a USD 1,906 actual for the same workload (Azure Noob, 2026).
2.5 The cost-driver reference table
The table below sets each cost driver against its pricing basis, its characteristic volatility, and the lever that controls it. The levers are developed in chapter three.
| Cost driver | Pricing basis | Volatility characteristic | Primary lever |
|---|---|---|---|
| Input tokens | Per 1M input tokens | Scales with prompt size and agent step count | Caching; retrieval efficiency; right-sizing |
| Output tokens | Per 1M output tokens (~4× input) | Scales with response verbosity | Output limits; right-sizing; structured outputs |
| Agent reasoning steps | Multiplies token volume 5–30× | Scales with task complexity and loop depth | Step caps; orchestration design; model routing |
| GPU reservation | Per GPU-hour | Bills whether utilised or idle | Auto-shutdown; serverless; utilisation monitoring |
| Idle GPU time | Per GPU-hour, no work done | 30–95% of reserved capacity commonly idle | Auto-shutdown; right-sizing; scheduling |
| Surrounding charges | Variable (support, egress, logging) | Adds 15–40% on top of model pricing | Architecture review; egress minimisation |
Sources: Gartner, 2026; CloudZero, 2026; Spheron, 2026; Rack2Cloud, 2026. Percentages are drawn from the cited analyses and apply to the workloads those analyses examined; they are indicative rather than guaranteed.
2.6 The implication for the leadership agenda
Agentic cost is volatile by construction: the token axis multiplies with agent steps, the GPU axis bills for idle hours, and the surrounding charges inflate the headline by a margin a naive estimate misses. None of this is a defect to be endured. Each is a variable to be managed, and management begins where chapter two does, with the ability to see the spend at all.
Chapter two: Visibility: you cannot optimise what you cannot see
The takeaway
A token bill that cannot be attributed to a team, a project, a model, and a use case cannot be cut, because nobody owns it and nothing baselines it. Visibility is therefore the precondition, not a nicety, and it has three parts: a consistent tagging standard, a token-telemetry pipeline that meters at the step level, and a showback or chargeback mechanism that puts the cost in front of the team that ran it. Every lever in chapter three depends on this instrumentation being in place first.
3.1 The visibility gap, quantified
Most AI cost overruns trace to a single root cause, and it is not the price of tokens. It is that nobody could see the spend accruing.
Most enterprises lack real-time cost visibility, which leaves them unable to measure cost per token or reason about their unit economics (FinOps Foundation, 2026). Costs are then discovered after the fact rather than steered as they accrue. The healthcare enterprise that ran through one trillion tokens and USD 6 million before its finance team identified the driver is the archetype, a failure of instrumentation, not of technology (FinOps Foundation, 2026).
The FinOps literature states the premise without hedging: consumption governance demands visibility across every cost layer, not the token bill alone, and absent that instrumentation, optimisation is guesswork (nOps, 2026). The phrase that recurs across the practice is this chapter's operating rule: you cannot optimise what you cannot see.
3.2 The tagging standard
Cost allocation rests on tagging, for a blunt reason: an untagged cost cannot be attributed, and an unattributed cost cannot be owned.
The practice has converged on a five-dimension schema that covers the large majority of chargeback cases. The five are team, the group that owns the workload; project, the specific initiative or product; environment, whether development, staging, or production; model-name, the specific model variant in use; and cost-centre, the finance mapping (Spheron, 2026). Applied consistently, those five dimensions cover roughly 95 per cent of chargeback use cases (Spheron, 2026).
Consistency is the hard part. The schema has to hold from the infrastructure layer through to the cost report, with no break in the chain. For self-hosted GPU workloads, that means carrying the same labels from container orchestration through GPU monitoring exporters into the metrics store, so the cost figures all derive from one taxonomy rather than several that disagree (Spheron, 2026).
3.3 Token telemetry
Tagging allocates cost; telemetry measures consumption. They are different instruments, and an AI FinOps practice needs both running.
Token telemetry captures, per request, the model invoked, the input-token count, the output-token count, the cached-token count, and the tags. It is the AI counterpart of a cloud cost dashboard, and the FinOps literature recommends standing up token-consumption monitoring on a par with the cloud cost dashboards already in place (FinOps Foundation, 2026).
The telemetry earns its keep three ways. It attributes cost to the tagged dimensions in real time. It lets token demand be forecast across teams and workloads. And it feeds the unit-economics framework that ties token spend to a business outcome: the measure that separates a profitable deployment from a costly one.
3.4 Showback and chargeback
Visibility only changes behaviour when it lands in front of the team that incurred the cost. Showback and chargeback are the two ways to make it land, and they differ in how hard they bite.
Showback shows a team what it would be charged, as a report, with no money crossing budgets (CloudZero, 2026). It builds awareness and accountability without administrative friction.
Chargeback moves the actual cost onto the team's budget or cost centre (CloudZero, 2026). It creates a direct financial incentive to optimise, at the price of more administrative machinery.
Microsoft's own FinOps guidance prescribes a sequence: open with showback for visibility, implement cost allocation to align cloud costs to the organisational reporting hierarchy, then layer chargeback on top (Microsoft FinOps guidance, 2026). Order matters, because chargeback imposed before allocation is accurate produces disputes instead of savings.
3.5 The token-flow through a multi-step agent
The diagram below traces how tokens pile up through a single multi-step agent run: the mechanism that makes the spend both large and hard to see. Each step re-sends the accumulated context, so the billed volume climbs with every loop even though the user issued one request.
The diagram fixes the requirement. A cost report that logs only the user request, and not the steps inside it, understates the bill by the full agent multiplier. Telemetry has to meter at the step level, not the request level, or the numbers will be wrong in the direction that hurts.
3.6 The implication for the leadership agenda
Visibility precedes optimisation, and skipping it is the most common way the programme fails. A tagging standard, a token-telemetry pipeline that meters at the step level, and a showback mechanism are the minimum instrumentation. Without them, every lever in the next chapter is pulled blind, and any saving it produces cannot be measured or held. With them, the levers become a programme rather than a run of one-off fixes.
Chapter three: The optimisation levers
The takeaway
The levers that cut the AI bill are documented, individually large, and additive. Model right-sizing handles the biggest share of inference cost by sending simple work to cheaper models. Prompt caching and batch processing apply documented percentage discounts to qualifying workloads. The provisioned-throughput-versus-pay-as-you-go choice is the largest single structural decision, and it turns on utilisation. Retrieval efficiency cuts token volume before any discount applies. Stacked together under the visibility of chapter two, these levers commonly recover between a quarter and a half of an unoptimised AI bill.
4.1 Model right-sizing: the largest lever
The single most consequential move is to stop spending large-model rates on work a small model handles. This is model right-sizing: the inference equivalent of picking the correct compute instance size instead of the largest one on the menu.
On suitable tasks the gap is not marginal, it is order-of-magnitude. Small language models (SLMs) can cut cost per million queries by more than 100 times on high-volume, narrow tasks (Index.dev, 2026). In money: processing one million conversations a month runs approximately USD 15,000 to USD 75,000 on a large language model, against USD 150 to USD 800 on a small one (Index.dev, 2026).
The capture mechanism is a router over a hybrid fleet. SLMs win six of eight major use cases on cost-efficiency grounds, with large models held back for the cross-domain reasoning that genuinely needs them (Index.dev, 2026). A router classifies each request and sends it to the cheapest model that can do the job, which is how the differential is captured at scale rather than admired in a benchmark.
The trade-off deserves stating plainly. SLMs frequently need fine-tuning to reach the required accuracy on a narrow task, whereas large models clear most tasks untouched (instinctools, 2026). Right-sizing therefore trades a one-time fine-tuning cost against recurring inference savings: a trade that favours right-sizing wherever query volume is high enough to amortise the tuning quickly.
By the numbers: model right-sizing
- >100× reduction in cost per million queries for suitable tasks (large model to small model) (Index.dev, 2026).
- USD 15,000–75,000 per million monthly conversations on a large model, versus USD 150–800 on a small model (Index.dev, 2026).
- 6 of 8 major use cases favour small models on cost-efficiency grounds (Index.dev, 2026).
- ~6× cost premium for a reasoning model over a standard model on the same query (Index.dev, 2026).
4.2 Prompt caching
Prompt caching discounts the cost of repeated context, which is precisely the cost an agent racks up most.
The mechanism is automatic on Azure OpenAI and Microsoft Foundry. On supported models, cached input tokens are billed at a discount for Standard deployments and at up to a 100 per cent discount for Provisioned deployments (Microsoft Foundry documentation, 2026). Third-party analyses of 2026 Azure pricing put the cached-input discount in the 50 to 90 per cent range (Amnic, 2026).
Eligibility is governed by the mechanics. Caching applies after the first 1,024 tokens of a prompt, with cache hits for every 128 additional identical tokens, provided the leading 1,024 tokens are identical (Microsoft Foundry documentation, 2026). The design consequence is direct: structure prompts with the stable, repeated content first, system prompt, tool definitions, retrieved context, so the largest possible share qualifies for the discount.
Agents and caching are made for each other. An agent re-sends the same system prompt and tool definitions at every step, and the repeated portion is exactly what caching discounts. The workload that suffers most from re-sending is the one caching rewards most.
4.3 Batch processing
For work that can wait, the Batch API hands over a flat discount in exchange for nothing more than tolerating delay.
The Batch API returns completions within 24 hours at a flat 50 per cent discount on global standard pricing (Microsoft, 2026), and the discount holds across models. A model priced at USD 4.40 per million tokens drops to USD 2.20 under the Batch API (TokenMix, 2026).
The qualifying workloads are easy to spot. Evaluations, bulk classification, content-generation pipelines, and any non-interactive processing all fit, because they tolerate the 24-hour window (Microsoft, 2026). Teams routing at least half their workload through the Batch API report saving 35 to 48 per cent of total monthly API spend (TokenMix, 2026).
The discounts compound. Prompt caching and batch pricing are independent and combine, so a cached input token processed through the Batch API takes both discounts at once (TokenMix, 2026). That stacking is why a disciplined programme recovers a large share of the bill rather than a marginal one: the savings multiply rather than merely add.
4.4 Provisioned throughput versus pay-as-you-go: the structural decision
The largest structural decision is whether to pay per token consumed or to reserve dedicated capacity. It turns on utilisation, and it is expensive to get wrong in either direction, too much committed capacity bleeds idle, too little leaves spikes uncapped.
Pay-as-you-go bills per million input and output tokens. It is flexible, suits variable and unpredictable load, and is the right call for development and early production where consumption patterns have not yet settled (DeployBase, 2026).
Provisioned Throughput Units (PTUs) reserve a fixed quantity of model-processing capacity at a fixed cost, starting around USD 2,448 per month per unit, with up to 70 per cent savings on sustained workloads (CloudZero, 2026). An annual commitment saves a further roughly 35 per cent against monthly (CloudZero, 2026). PTUs suit consistent, predictable, high-volume load.
The crossover is documented. The economics favour a switch from pay-as-you-go to PTU once monthly token volume for a model clears approximately 150 to 200 million tokens at 50 per cent or better sustained utilisation (CloudZero, 2026). A blunter test: once pay-as-you-go costs for a workload clear roughly USD 1,800 per month, a PTU reservation starts to pay for itself (CloudZero, 2026).
The discipline is to decide on evidence. Validate a PTU commitment against 30 to 60 days of real pay-as-you-go telemetry before placing it, because a PTU reserved below its utilisation threshold costs more than the pay-as-you-go it replaced (CloudZero, 2026). This is the point at which the visibility discipline of chapter two pays back directly, in avoided commitment error.
4.5 The PTU-versus-pay-as-you-go decision
The diagram below sets out the decision logic, and the table that follows summarises the two models.
| Dimension | Pay-as-you-go | Provisioned Throughput Units (PTU) |
|---|---|---|
| Billing basis | Per 1M input/output tokens | Per reserved capacity-hour |
| Entry cost | None; pay per use | ~USD 2,448/month per unit (CloudZero, 2026) |
| Best for | Variable, unpredictable, dev, early production | Sustained, predictable, high-volume production |
| Savings | None inherent; relies on caching/batch | Up to 70% on sustained workloads; +~35% on annual (CloudZero, 2026) |
| Switching signal | Below threshold or variable load | >150–200M tokens/month at ≥50% utilisation (CloudZero, 2026) |
| Risk | Cost spikes with usage | Wasted spend if utilisation falls below threshold |
Sources: CloudZero, 2026; DeployBase, 2026. Figures are as of June 2026 and change frequently; reconfirm current Azure pricing before committing.
4.6 GPU resource optimisation
On the GPU axis, self-hosted and training workloads, the levers go straight at idle time, because chapter one identified idle time as the dominant waste.
Auto-shutdown is the most direct of them. Automatic shutdown of idle GPU instances and notebooks typically saves 20 to 35 per cent of total GPU spend (Spheron, 2026), and it carries almost no risk, because it reclaims only capacity that is by definition doing nothing.
Serverless inference removes the idle cost entirely by billing per invocation rather than per reserved hour (Spheron, 2026). For inference with variable or intermittent demand, serverless converts a fixed idle cost into a variable usage cost and dissolves the hoarding problem at its source.
Utilisation monitoring and scheduling attack the coordination problem behind hoarding. Optimised fleets have held 49 per cent utilisation against a 5 per cent baseline, which establishes the ceiling as a management outcome rather than a technical limit (Rack2Cloud, 2026). The lever is to pool and schedule GPU capacity across teams instead of reserving it team by team.
4.7 Retrieval efficiency
The last lever cuts token volume at source, before any discount applies, by controlling how much context is retrieved and shipped to the model.
Retrieval-augmented generation injects retrieved documents into the prompt, and a sloppy retrieval step inflates the input-token count on every single request. The fix is to retrieve precisely: tighter chunking, relevance filtering, and a cap on the number of retrieved passages each cut the input tokens billed per call.
This lever reinforces caching rather than competing with it. Stable retrieved context placed at the front of the prompt both shrinks volume through tighter retrieval and qualifies for the cache discount, so the two levers pull in the same direction at once.
4.8 The optimisation-decision flow
The diagram below sequences the levers into one decision flow, applied per workload. The order follows a single principle: the cheapest token is the one never sent, and discounts come after for the tokens that survive.
4.9 The lever-comparison summary
The table below consolidates the levers and their documented savings, ordered by axis. Figures that are illustrative rather than sourced are labelled.
| Lever | Axis | Documented saving | Source |
|---|---|---|---|
| Model right-sizing (SLM) | Token | >100× cost/query for suitable tasks | Index.dev, 2026 |
| Prompt caching | Token | 50–90% on cached input tokens | Microsoft Foundry docs, 2026 |
| Batch API | Token | Flat 50% on qualifying tokens | Microsoft, 2026 |
| PTU (sustained workloads) | Token | Up to 70%; +~35% on annual | CloudZero, 2026 |
| GPU auto-shutdown | GPU | 20–35% of total GPU spend | Spheron, 2026 |
| Serverless inference | GPU | Eliminates idle GPU cost | Spheron, 2026 |
| Retrieval efficiency | Token | Reduces input tokens at source (illustrative) | Volge analysis (illustrative) |
Savings ranges apply to the workloads the cited analyses examined and are indicative. Full qualifying conditions appear in Appendix A. Figures are as of June 2026 and should be reconfirmed.
4.10 The implication for the leadership agenda
The AI bill is a managed cost, not a fixed one, and the levers that manage it are documented and large. They stack: a right-sized model, processing cached prompts through the Batch API on a correctly sized PTU, with idle GPUs shut down, recovers a large share of an unoptimised bill. None of them apply themselves, though. They need an operating model to own them, which is chapter four.
Chapter four: The FinOps-for-AI operating model and roadmap
The takeaway
FinOps for AI is an operating model, not a tool purchase. It carries established cloud financial-management discipline onto the token and GPU axes, and it requires named ownership, a tagging standard, a token-telemetry pipeline, a forecasting cadence, and a unit-economics framework. The discipline has hardened into a defined practice with its own framework and certification. The roadmap that installs it runs in order: visibility first, levers second, the loop institutionalised third, because a lever pulled before visibility exists produces savings nobody can measure or hold.
5.1 FinOps for AI, defined
FinOps brings financial accountability to variable cloud spend, letting teams trade off cost, speed, and quality deliberately rather than by accident. FinOps for AI extends that practice onto the token and GPU axes.
The discipline is no longer informal. The FinOps Foundation introduced Scopes into its 2025 Framework as a core element, letting organisations define context-specific boundaries for FinOps activity, with AI named a major new technology category meriting its own Scopes (FinOps Foundation, 2025). Most FinOps practices now manage AI costs (FinOps Foundation, 2025).
It is also credentialed. The FinOps Foundation launched a FinOps for AI Certification covering the financial dynamics of AI and machine-learning workloads, the monitoring and forecasting of GPU and accelerator usage, and the collaboration between FinOps and data-science teams to build financial accountability into the AI development lifecycle (FinOps Foundation, 2025).
5.2 The unit-economics framework
The measure that defines FinOps for AI is unit economics: the cost of an AI workload per unit of business value delivered, not as an absolute figure on an invoice.
The framework binds token spend to a specific outcome, a resolved support ticket, a processed invoice, a generated document, so cost is judged against value rather than against budget alone. The FinOps literature recommends building unit-economics frameworks that tie token spend to specific business outcomes, and standing up governance that requires return-on-investment justification before a new AI workload is approved (FinOps Foundation, 2026).
The measure reframes the cost conversation entirely. A workload that costs more in absolute terms but returns proportionally more value is not a candidate for the axe. A workload whose cost per unit of value exceeds the value itself is unprofitable no matter how small its absolute spend. Unit economics is what tells the two apart, and it is the figure a chief financial officer needs in order to govern AI spend rather than merely watch it climb.
5.3 The FinOps-for-AI operating loop
The diagram below sets out the continuous operating loop. It is a cycle, not a project, because consumption shifts continuously and optimisation has to be re-applied as it does.
The loop maps onto the three established FinOps phases. Inform builds the visibility of chapter two. Optimise applies the levers of chapter three. Operate enforces policy, forecasts demand, and governs new-workload approval. The cycle repeats on a set cadence, because a one-time optimisation decays as workloads and prices move under it.
5.4 The roadmap
The roadmap below stages the installation of FinOps for AI. It is sequenced so that visibility precedes optimisation, on the lesson of chapter two: optimisation without visibility is guesswork.
Stage one, the first 60 days: establish visibility. Define and apply the five-dimension tagging standard. Stand up token telemetry metering at the step level. Implement showback against the tagged dimensions. Name an accountable owner for AI cost. The objective is a single, trusted view of where AI spend is incurred and by whom.
Stage two, months two to four: apply the levers. Right-size models and introduce routing. Enable prompt caching and structure prompts to qualify. Route non-interactive workloads to the Batch API. Apply GPU auto-shutdown and serverless inference. Validate PTU decisions against the now-available telemetry. The objective is a measured reduction in cost per unit of value, baselined against stage one.
Stage three, months four to nine: institutionalise the loop. Move from showback to chargeback where allocation is accurate. Establish a forecasting cadence for token demand. Require unit-economics justification for new workloads. Embed quarterly utilisation reviews for PTU and GPU reservations. The objective is a self-sustaining loop rather than a one-time optimisation.
The cadence is fast on visibility and deliberate on institutionalisation by design. That inverts the common failure pattern, in which optimisation is attempted before visibility exists and the savings cannot be measured or held.
5.5 The cost-maturity model
The model below lets leadership locate the enterprise and name the next increment. It is the diagnostic counterpart to the roadmap.
| Level | Description | Visibility | Optimisation | Typical outcome |
|---|---|---|---|---|
| 0: Blind | AI spend unmanaged; no tagging | None; cost discovered after the fact | None | Overruns; 27%+ waste; surprise bills |
| 1: Visible | Tagging and showback in place | Cost attributed to teams/workloads | Ad hoc | Awareness; first waste identified |
| 2: Optimised | Levers applied per workload | Token telemetry at step level | Caching, batch, right-sizing live | Measurable cost-per-unit reduction |
| 3: Governed | Chargeback and forecasting live | Real-time, with budget alerts | PTU/GPU reviewed quarterly | Unit economics positive; predictable |
| 4: Optimising continuously | FinOps loop institutionalised | Unit economics per workload | Continuous re-optimisation | AI spend a managed, profitable variable |
Most enterprises sit between Level 0 and Level 1 as of mid-2026, consistent with the finding that 73 per cent reported AI costs exceeding projections (FinOps Foundation, 2026). The discipline in this paper moves an enterprise from Level 0 to Level 3 across the nine-month roadmap.
5.6 The capability comparison: unmanaged versus FinOps-for-AI estate
The table below contrasts the two operating states, to make concrete what the discipline changes.
| Capability | Unmanaged estate | FinOps-for-AI estate |
|---|---|---|
| Cost attribution | Aggregate bill; no owner | Tagged to team, project, model, use case |
| Token measurement | Invoice-level, after the fact | Step-level telemetry, real time |
| Model selection | Default to largest model | Routed; right-sized per request |
| Capacity | Reserved and hoarded; idle | Auto-shutdown; serverless; pooled |
| Commitment decisions | Forecast-based or absent | Validated on 30–60 days of telemetry |
| Cost conversation | Absolute spend versus budget | Cost per unit of business value |
| Governance | Reactive to overruns | ROI justification before approval |
5.7 The implication for the leadership agenda
FinOps for AI is the discipline that turns a volatile, surprise-driven AI bill into a managed, profitable variable. It is not a tool, though tools support it. It is an operating model, ownership, tagging, telemetry, the levers, forecasting, and unit economics, run as a continuous loop. Enterprises that install it keep their deployments profitable. Enterprises that do not surrender between a quarter and a half of their AI budget to waste, every quarter, until they do.
Chapter five: Implications for the IT and finance leadership agenda
Cost has moved from procurement footnote to first-order operating concern, and the evidence in this paper points the chief information officer, the chief financial officer, and the FinOps practitioner toward a short list of clear conclusions.
The AI bill is volatile by construction, consumption-driven on both the token and GPU axes, with agents multiplying token consumption by their step count, and that volatility is governable. The waste is large, between a quarter and a half of an unoptimised bill, and it is recoverable through documented levers. Recovering it is not a technology problem. It is a discipline problem, and the discipline is mature, framework-backed, and available now.
Six questions belong on the leadership agenda. First, is AI spend tagged to a team, a project, a model, and a use case, or is it one aggregate bill with no owner. Second, does token telemetry meter at the step level, so the agent multiplier is visible rather than buried. Third, are models right-sized and routed, or does every request default to the most expensive model on hand. Fourth, are prompt caching and the Batch API applied to every workload that qualifies, given that the discounts stack. Fifth, are PTU and GPU commitments validated against 30 to 60 days of real telemetry and reviewed quarterly, or set on forecast and left untouched. Sixth, is AI spend governed on unit economics, cost per unit of business value, or merely watched against an absolute budget.
Volge's contribution is in the execution. The platform publishes the pricing levers, and the FinOps Foundation publishes the framework. What decides whether a deployment stays profitable or bleeds is the discipline that combines them into a sustained loop, the tagging standard, the step-level telemetry, the lever programme, the commitment validation, and the unit-economics governance. That is where Volge concentrates its work, and where the difference between a managed bill and a surprise one is actually made. Volge looks forward to discussing the application of this framework to specific operating contexts in detail.
Appendices
Appendix A: Cost-lever comparison (with sourced savings where available)
| Lever | Axis | Documented saving | Qualifying condition | Risk if misapplied | Source |
|---|---|---|---|---|---|
| Model right-sizing (SLM) | Token | >100× cost/query for suitable tasks | High-volume, narrow tasks; fine-tuning may be needed | Accuracy loss on tasks needing a large model | Index.dev, 2026 |
| Prompt caching | Token | 50–90% on cached input tokens | Repeated leading context ≥1,024 identical tokens | None; automatic where eligible | Microsoft Foundry docs, 2026; Amnic, 2026 |
| Batch API | Token | Flat 50% on qualifying tokens | Non-interactive; 24-hour return acceptable | Unsuitable for real-time work | Microsoft, 2026 |
| PTU (sustained) | Token | Up to 70%; +~35% on annual commitment | >150–200M tokens/month at ≥50% utilisation | Wasted spend below utilisation threshold | CloudZero, 2026 |
| GPU auto-shutdown | GPU | 20–35% of total GPU spend | Idle-prone training/notebook workloads | Interrupted long-running jobs if mis-scoped | Spheron, 2026 |
| Serverless inference | GPU | Eliminates idle GPU cost | Variable/intermittent inference demand | Higher per-call cost at very high sustained load | Spheron, 2026 |
| Retrieval efficiency | Token | Workload-specific (illustrative) | Retrieval-augmented workloads | Relevance loss if over-trimmed | Volge analysis (illustrative) |
| Output limits / structured outputs | Token | Reduces output tokens (~4× input rate) (illustrative) | Verbose-response workloads | Truncation if limit too tight | Volge analysis (illustrative) |
Sources as cited. Savings ranges apply to the workloads the cited analyses examined and are indicative, not guaranteed. Pricing and discount figures are as of June 2026 and change frequently; reconfirm current Azure pricing before committing.
Appendix B: Cost-optimisation checklist
The following is a minimum control set, drawn from the FinOps Foundation 2025–2026 Framework guidance, Microsoft FinOps and Azure OpenAI documentation (2026), and the practitioner analyses cited in this paper.
Visibility (the precondition)
- A five-dimension tagging standard, team, project, environment, model-name, cost-centre, is defined and applied consistently from infrastructure to cost report.
- Token telemetry meters per request at the step level: model, input tokens, output tokens, cached tokens, tags.
- Showback is in place against the tagged dimensions before chargeback is attempted.
- An accountable owner for AI cost is named.
Token-axis optimisation
- Models are right-sized; a router directs each request to the cheapest capable model.
- Prompt caching is enabled and prompts are structured with stable content first to qualify.
- Non-interactive workloads are routed to the Batch API for the flat 50 per cent discount.
- Retrieval is tuned: tighter chunking, relevance filtering, capped passage counts.
- Output limits and structured outputs constrain the higher-rate output tokens.
GPU-axis optimisation
- Auto-shutdown policies are applied to idle GPU instances and notebooks.
- Serverless inference is used for variable or intermittent inference demand.
- GPU capacity is pooled and scheduled across teams rather than reserved per team.
- Utilisation is monitored against a target ceiling, not left to default.
Commitment and governance
- PTU and reserved-capacity decisions are validated against 30–60 days of real telemetry.
- Utilisation is reviewed quarterly, and commitments are resized as workloads change.
- Surrounding charges (support, egress, Private Link, logging) are reviewed; they add 15–40 per cent.
- New AI workloads require a unit-economics and ROI justification before approval.
- Cost is governed on cost per unit of business value, not absolute spend alone.
Appendix C: Glossary
- Batch API. An Azure OpenAI processing mode returning completions within 24 hours at a flat 50 per cent discount, suited to non-interactive workloads.
- Chargeback. A cost-allocation mechanism that transfers actual AI cost to the incurring team's budget or cost centre.
- FinOps. The operational practice bringing financial accountability to variable cloud spend through cross-functional collaboration on cost, speed, and quality.
- FinOps for AI. The extension of FinOps to token-driven and GPU-driven AI workloads, formalised in the FinOps Foundation 2025 Framework Scopes.
- GPU (graphics processing unit). The specialised processor running model training and self-hosted inference, billed per reserved hour.
- Idle GPU time. Reserved GPU capacity that bills while performing no useful work; commonly 30–95 per cent of reserved capacity.
- Prompt caching. Automatic discounting of repeated leading prompt content, billed at 50–90 per cent below the standard input rate where eligible.
- PTU (Provisioned Throughput Unit). A reserved unit of model-processing capacity at a fixed cost, delivering up to 70 per cent savings on sustained, high-utilisation workloads.
- Right-sizing. Selecting the smallest, cheapest model or instance that performs a task adequately, in place of a larger default.
- Showback. A reporting mechanism giving teams visibility of their AI cost without transferring budget.
- SLM (small language model). A smaller model offering dramatically lower inference cost for narrow, high-volume tasks, often after fine-tuning.
- Token. The atomic unit of text a language model processes (roughly three-quarters of a word), the basis of inference billing.
- Token telemetry. Instrumentation capturing per-request token consumption and tags, the AI equivalent of a cloud cost dashboard.
- Unit economics. The cost of an AI workload expressed per unit of business value delivered, the measure that distinguishes profitable from unprofitable deployments.
Appendix D: Sources
This paper draws on the following, accessed June 2026. Figures are attributed in-text to their primary source. Where a statistic appears across multiple secondary outlets, the most authoritative attribution available is given. Pricing and discount figures are as of June 2026 and are noted as subject to frequent change.
- FinOps Foundation: 2025 FinOps Framework (introduction of Scopes; AI as a technology category); FinOps for AI technology category and certification; State of FinOps 2026 (73 per cent of enterprises reported AI costs exceeding projections); Token Economics: The Atomic Unit of AI Value.
- Gartner: Worldwide AI spending to total USD 2.59 trillion in 2026 and 47 per cent growth forecast (USD 1.43 trillion AI infrastructure); March 2026 analysis on agentic AI token multipliers (5–30×).
- CloudZero: Azure OpenAI Pricing: Models, Real Costs, and How to Track AI Spend in 2026 (PTU pricing, up to 70 per cent savings, switching thresholds, surrounding charges).
- Microsoft: Azure OpenAI Service pricing; Microsoft Foundry prompt-caching documentation (cached-token discounts, caching mechanics); Batch API guidance (flat 50 per cent discount); Microsoft FinOps guidance (showback-to-chargeback sequence).
- Microsoft Foundry / Microsoft Learn: Prompt caching with Azure OpenAI in Microsoft Foundry Models (50–100 per cent cached-token discount; 1,024-token threshold).
- Amnic: Understanding the True Cost of Azure OpenAI (cached-input discount range; PTU and hidden costs).
- Index.dev: Small vs Large Language Models: The 2026 Reality Check (>100× cost reduction; per-conversation costs; reasoning-model premium; use-case split).
- instinctools: LLM vs SLM (fine-tuning trade-off; reasoning-model compute).
- Spheron: The GPU Cloud Cost Optimization Playbook and GPU Cloud FinOps for AI Teams (utilisation ranges; idle-cost figures; auto-shutdown and serverless savings; five-dimension tagging schema).
- Rack2Cloud: GPU Utilization: Why 95% of Enterprise Capacity Sits Idle (5 per cent average utilisation; 49 per cent optimised ceiling).
- Spendark: The State of Cloud Waste 2026 (27 per cent cloud waste; USD 100 billion-plus; H100 GPU-hour pricing).
- LeanOps: Agentic AI Cost Runaway (context re-send mechanism; token-burn multipliers).
- EY: Agentic AI Enterprise Token Cost; Exceeds.ai: AI Model Token Pricing Breakdown 2026 (per-interaction cost escalation).
- Optimum Partners: AI Token Costs: Why Enterprise AI Bills Keep Rising in 2026 (483 per cent spend growth; ~80 per cent per-token price decline).
- TokenMix: OpenAI Batch API 2026 and Azure OpenAI Cost 2026 (batch savings; discount stacking; 15–40 per cent surrounding fees).
- DeployBase: Azure OpenAI Pricing: PTU vs On-Demand Comparison (workload-fit guidance).
- Azure Noob: Azure OpenAI Pricing: Real Costs (estimate-versus-actual discrepancy).
- nOps: FinOps X 2026 Day 1 Keynote Recap (consumption-governance visibility across cost layers).
Addendum: How Volge's forward-deployed engineers shall help: Cloud Infrastructure & Operations
The Volge engagement team
Volge delivers through one accountable team that runs from the boardroom to the production branch. Four roles carry the engagement, and they hold their remit from the first workshop to operational handover rather than handing off between phases.
| Role | What they own |
|---|---|
| Principal | The client relationship and the account. Sets the engagement's business outcomes, the governance cadence, and the value tracking, and stays the single point of accountability throughout. |
| Architect | The target architecture and the transformation roadmap. Sequences the work, sets the standards, and makes the platform, build-versus-buy, and control-design calls. |
| Principals | The senior engineering leads. Each owns a workstream, makes the hard technical calls, sets the quality bar, and mentors the embedded engineers. |
| Forward-deployed engineers | The embedded build. They sit inside the client's team, build and ship in the client's own environment, and own the last mile to production. |
The Principal holds the outcome, the Architect holds the design, the Principals hold the workstreams, and the forward-deployed engineers hold the keyboard, one team, accountable from strategy to shipped software.
This paper has argued that controlling agentic AI cost is a discipline problem, not a technology one, the means to read and steer volatile token, compute, and GPU-hour spend on Azure are mature and on the shelf. Volge is a deep-tech engineering firm, and its Cloud Infrastructure & Operations practice is where that discipline is installed and run as engineering rather than admired as a slide. The same FinOps loop that turns a surprise cloud bill into a managed variable is the one that turns the agentic AI bill into a managed, profitable variable, and Volge operates both as one estate.
What Volge delivers
The capability set below maps Volge's Cloud Infrastructure & Operations practice to the outcomes a CIO, CFO, or platform lead reading this paper is trying to secure. Each capability is part of how Volge runs an Azure estate, not a product to be bought.
| Capability | Client outcome |
|---|---|
| FinOps operating model | Cost becomes a continuous engineering concern with named ownership: a CCoE that gives AI and cloud spend a single accountable home, not a quarterly finance review. |
| Cost and capacity management | The documented levers applied continuously, right-sizing against utilisation, scheduling non-production off out of hours, removing orphaned resources, so the bill reflects consumption, not habit. |
| Commitment planning (PTU and reservations) | Provisioned-throughput-versus-pay-as-you-go and reservation or savings-plan decisions validated against real telemetry, not forecast, so committed capacity is sized to utilisation rather than to hope. |
| Azure landing zones | A governed landing zone, management-group hierarchy, Azure Policy, role-based access, network topology, and a cost-management structure, in place and enforced before workloads land, with tagging that makes every cost attributable. |
| Reliability and SRE | Resilience exercised, not assumed: monitoring through Azure Monitor, recovery rehearsed against defined RTO and RPO, and incidents turned into corrections. |
| Observability and SLOs | Telemetry translated into a small set of service-level objectives, availability, latency, recovery, and cost per workload, with alerting tied to them, so the estate can be operated rather than merely watched. |
The Volge difference
Four points separate Volge's approach from a generic cloud managed-service engagement.
Token and GPU economics as a first-class discipline. Volge runs the token axis and the GPU axis as one cost estate. The right-sizing instinct that picks the correct VM family is the same one that routes a request to the cheapest capable model; the auto-shutdown reflex that reclaims an idle GPU is the same one that questions a reserved instance bleeding the weekend. The two axes are managed with one discipline rather than two disconnected teams.
Showback and chargeback that change behaviour. Visibility only moves a team when the cost lands in front of it. Volge installs tagging from infrastructure through to the cost report, stands up showback first to build accountability without friction, and layers chargeback on once allocation is accurate: in the sequence that produces savings rather than disputes.
Right-sizing and commitment validation on evidence. Every commitment, a PTU reservation, a savings plan, a reserved instance, is placed against real utilisation telemetry and reviewed on a cadence, because a commitment reserved below its threshold costs more than the consumption it replaced. Volge treats the commitment as a bet to be evidenced, never a forecast to be guessed.
Accelerators and a repeatable factory. Volge brings landing-zone templates, FinOps reference patterns, and a migration-factory discipline, standard runbooks, automation, and a measured cadence, so the controls in this paper are installed as a paved road, not re-solved on every workload.
Capability to outcome
The loop is continuous by design. Consumption shifts as workloads and prices move, so each pass re-applies the levers and re-validates the commitments rather than treating optimisation as a one-time event. A representative outcome, a quarter to a half of an unoptimised bill recovered, baselined and held, is illustrative only, drawn from the ranges cited in the body of this paper, and any figure should be validated against an enterprise's own telemetry.
A concrete first step. A short Volge assessment establishes where AI and cloud spend is actually incurred and by whom: a tagging and telemetry baseline across the token and GPU axes, a read on PTU-versus-pay-as-you-go and reservation fit against thirty to sixty days of real usage, and a costed shortlist of the levers that pay back first. It produces a single trusted view of the spend and a sequenced plan to install the FinOps loop, visibility first, levers second, the loop institutionalised third. Volge would welcome a conversation about applying this to a specific Azure estate.
Prepared by Volge · June 2026. This document is a point of view intended to inform IT, platform, and finance decision-making. It is not financial, investment, or tax advice. All pricing, discount, and savings figures are attributed to their sources and were accurate to the best of available knowledge as of June 2026. AI pricing changes frequently, and the savings ranges cited apply to the specific workloads the cited analyses examined; readers are advised to reconfirm current Azure pricing and to validate any commitment against their own telemetry before relying on these figures for budget or procurement decisions.
Voice of Volge: the firm's point of view on the technology it delivers. Sourced, year-stamped, and built to be handed to a board.
Volge is an independent deep-tech enterprise service provider and premier Microsoft delivery partner. That interest is disclosed; the analysis in this book is independently sourced.