>
The questions every board should answer before, during and after an enterprise AI rollout, one at a time.
FAQ is Volge's reference series. Each volume takes one subject a board is being asked to decide on and answers it a question at a time, in plain terms and with the working shown: what the thing is, what it costs, what it breaks, who has to own it, and what would have to be true before it is worth doing. It is written to be consulted rather than read through, and to be as useful to the person asking the question as to the person who has to answer it.
The questions a board is asked before, during and after an enterprise AI rollout, answered one at a time: what the technology is, what it costs, what it breaks, who has to own it, and what would have to be true before it is worth doing.
Figures carry their basis and their date. Where published estimates disagree, the range is given and the basis named rather than averaged away, and where a widely quoted number does not survive checking it is corrected here rather than repeated. Every chart plots a figure stated in the text, and nothing is projected beyond what a named source projects.
Before a single dollar is committed, a board needs to agree on what the thing is, why most attempts fail, and what success would even look like. Two chapters, eleven questions, no acronyms left unexplained.
What the technology actually is, what it is not, and who in the building should own it: the vocabulary a board needs before the first decision.
When people say "AI" today they almost always mean one recent breakthrough: software that picked up language and reasoning by reading most of the internet. It belongs to a much older family, and treating the two as the same thing is where most leadership teams stumble first.
Say "AI" to a board, a vendor or a newspaper in 2026 and they mean generative AI: the large language models that draft text, write code and answer questions when you ask in plain English. That is the technology that reset the conversation. It is also one branch of a family that reaches back to the 1950s. The system recommending your next video is AI. So is the model that flags a suspicious payment, and the scanner that reads an invoice. Most of it has run quietly inside large companies for years.
What is new is how differently this branch behaves. Older AI does a single narrow job it was trained for. Generative AI is general-purpose and conversational, and it will not give you the same answer twice; ask the same question again and the wording shifts. That last trait quietly wrecks budgets. Run it like the deterministic software you already own and the programme tends to fail.
When someone says "AI strategy" in 2026, they almost always mean the rightmost box, but the value often hides in combining it with the older ones.
The model can be the same in both places. What turns it into "enterprise" is the wrapper around it: your data boundary, your audit trail, your control over where it runs. That wrapper carries most of the cost and almost all of the risk.
A consumer chatbot and an enterprise system often call the identical model. The difference sits in the envelope. The consumer tool sends your words to another company's servers under consumer terms. It keeps no record you control, plugs into nothing in your business, and makes no promise about where the data rests or who can demand it. Tolerable for a recipe. Not tolerable for a patient record, a live deal model, or your source code.
Enterprise AI is the work of putting that same capability behind the controls a regulated business needs: a private boundary, access control, logging you can audit, a contract that keeps your data out of the next training run, and connections to the systems where the real work happens. The model is the engine. Everything else is the rest of the car.
Same engine, different vehicle. The "enterprise" premium buys the boundary, the record and the integration, not a smarter model.
A large language model predicts the next word. It learned to do that from an enormous amount of text, and at scale the guessing starts to resemble reasoning. What it is not: a database, a calculator, or a reliable source of truth.
Underneath, the model does one thing. You give it text and it predicts what should follow, fragment by fragment, drawing on more reading than a person could finish in a thousand lifetimes. That really is the whole mechanism. Yet at sufficient scale the output looks like understanding and synthesis. The useful stance for a
leader is to take the capability seriously without believing there is a mind behind it.
Its limits matter as much as its strengths. The model looks nothing up in a trusted store, so it can state something false as smoothly as something true. It knows nothing about your business until you tell it. And because it optimises for what sounds right rather than what is right, it occasionally produces confident, well-formed nonsense. The industry calls this a hallucination. A large part of enterprise design exists to contain that one weakness.
| It is | It is not |
|---|---|
| A general-purpose language and reasoning engine | A database of verified facts |
| Excellent at drafting, summarising, translating, coding | A reliable calculator or system of record |
| Able to follow nuanced, plain-language instructions | Aware of your business unless you supply the context |
| Fast, tireless and scalable | Deterministic: answers vary run to run |
| A probabilistic predictor of plausible text | A guarantor of truth: it can be fluently wrong |
A chatbot replies. An agent acts. Give it a goal and it will break the work into steps, use your systems to carry them out, and check its own results, which is what makes it both more useful and more dangerous than a chatbot.
A chatbot is a single exchange: you ask, it answers, the loop closes. An agent is handed an objective such as "reconcile these invoices," "clear this ticket queue," or "draft and file the report," and allowed to pursue it. It reads documents, calls other software, writes to systems, and chooses its next step from what it finds, looping until the job is done or a limit stops it.
That move from answering to doing is the real story of enterprise AI in 2026. The productivity sits there, because an agent finishes work rather than advising on it. So does the risk, because software that can act can act wrongly, and it does so quickly and at scale. Most of Part Three is about making agents capable without making them a hazard.
The board-level rule of thumb: a chatbot needs good answers; an agent needs guardrails.
It is a change programme dressed as a software purchase. The licence is the cheap part. Own it jointly, with a business sponsor accountable for the outcome and a technical owner running the platform, rather than handing it to IT as procurement.
Buying model access takes an afternoon. Getting value from it takes far longer, because it runs into messy data, processes built for people, nervous staff, and controls that never expected software to act on its own. The studies from 2025 and 2026 are blunt: about four-fifths of the effort to reach production goes into data, integration, governance and measurement, not the model. Budget and sponsor it as a transformation.
Ownership follows the same logic. Leave it with IT and you get capable infrastructure aimed at no clear problem. Leave it with the business and you get ungoverned sprawl. What works is a pairing: a business sponsor accountable for value and change, a technical owner accountable for the platform and controls, supported by a small platform team that supplies guardrails and reusable parts, with risk, security and legal setting the boundaries.
| Role | Owns | Accountable for |
|---|---|---|
| Business sponsor | The use case and the outcome | Value delivered, adoption, change |
| Technical owner (CTO/CIO) | The platform, models, integration | Reliability, security, cost |
| AI platform / CoE | Standards, guardrails, shared tooling | Consistency, reuse, governance |
| Risk / security / legal | The boundaries | Compliance, audit, liability |
AI is a team sport across four positions, and ~80% of the effort is data, integration and governance, not the model. Source: Volge synthesis of MIT, S&P Global, Deloitte studies, 2025–26.
The headline numbers are grim, and they agree with one another. MIT's much-quoted 2025 work found roughly 95% of generative-AI pilots made no measurable dent in profit and loss. S&P Global reported that 42% of companies walked away from most of their AI initiatives in 2025, more than double the year before. For agents the picture is starker: industry analyses put the share that never reach production near 88%.
The most important number in this book is the failure rate. Start here: with what goes wrong, what it costs, and what a realistic return looks like once the hype is stripped out.
Yes. The most-cited 2025 study found that about 95% of enterprise AI pilots produced no measurable profit. Most died in the gap between a demo that works and a business case that works.
None of this argues against doing AI. It argues against doing it casually. The failures cluster around the dull things, not the technology. Poor use-case choice. Weak data foundations. No governance, and no honest measure of success. The companies that win do that unglamorous work first.
Sources: MIT (2025); S&P Global (2025); industry agent-adoption analyses (2026). Figures are widely cited and directional, not audited.
A demo proves the model can do the task once, in ideal conditions. Production asks it to do the same task reliably, securely, affordably, ten thousand times over. That last mile is mostly data and plumbing, not intelligence.
A demo is a staged performance. Clean inputs, a friendly audience, no auditor, no cost ceiling, no furious customer when it slips. Production is none of that. Crossing from one to the other means wiring into real systems, cleaning the data, adding guardrails, proving compliance, and building the measurement that shows the thing actually helps. That is where the time, the money and the political will tend to run out.
There is a name for getting stuck here: pilot purgatory. The project demos forever and ships never. The way out is to build the pilot for graduation from the start, on the same data, the same security and the same success metric as production, rather than building it to impress a room.
Five failure modes, one pattern: the hard part was never the model.
Plan for three layers of cost: the model, the build, and the run that never stops. Then judge returns soberly. Studies cluster around $3–4 of value for every $1 spent on programmes that work, while a large minority return nothing.
Cost here is not a licence fee. It comes in three layers. The model cost: per-use API charges, or hardware if you self-host. The build cost: integration, data work, security, which is the biggest one-time line. And the run cost that catches everyone out: the meter that ticks with every query, every day, indefinitely. Chapter 4 takes all three apart.
Returns split in two. Microsoft and IDC put the average near $3.70 back for every $1 spent on generative AI, and other studies land in the same $3–4 range for programmes that work. IBM's 2025 CEO study tells the other half: only about a quarter of initiatives hit their expected return, and a meaningful slice were underwater at twelve months. The successful median is real and worth chasing. The average looks worse because of the failures this book exists to help you dodge.
Payback for successful programmes typically lands inside 12–18 months; failed ones never reach it. Sources: Microsoft/IDC (2026); IBM CEO study (2025).
Most enterprise risk has a familiar shape your controls already handle. A few do not. First, the confident falsehood: a system that can be wrong as fluently as it is
right, in places where wrong gets expensive. Second, prompt injection, a new kind of attack where instructions buried in ordinary text hijack the AI, much as phishing hijacks a person. Third, data leakage through prompts, logs or training. Fourth, and biggest, autonomous action: an agent that can act can act wrongly, at machine speed and scale.
Four risks are new in kind: confident falsehoods, an attack surface made of plain language, data leaking out through prompts, and software that takes real actions on its own. None of them fit neatly into the controls you already run.
Chapters 5 and 10 go deep on each. The board-level point is simple. These are not old IT risks with a new label, to be bolted onto the framework you already have. They need their own controls, their own audit trail, and increasingly their own regulatory compliance.
| New risk | What it is | Closest old analogue |
|---|---|---|
| Hallucination | Confident, well-formed falsehood | None: genuinely new |
| Prompt injection | Hidden instructions hijack the AI | Phishing / social engineering |
| Data leakage | Secrets escape via prompts, logs, training | Data loss / insider exfiltration |
| Autonomous action | Agent acts wrongly at speed and scale | Runaway automation, but smarter |
| Model dependence | Vendor change breaks the business | Supplier lock-in |
For most enterprise work in 2026 it augments far more than it replaces. It removes tasks, not usually whole jobs. Leaders who say that plainly and reinvest the freed time get adoption. Those who hint at quiet headcount cuts get quiet sabotage.
The evidence points to augmentation as the near-term pattern. AI takes the repetitive, document-heavy, first-draft parts of knowledge work and leaves judgement, relationships and accountability with people. Some roles do change, and a few shrink. But the usual unit of change is the task, not the job. The best results come from redesigning the work so people spend more time on what only people can do, which Chapter 5 picks up.
The communication test is candour. Staff can smell it when "augmentation" is a polite word for something else, and the moment they suspect they are training their own replacement, adoption dies. The leaders who land these programmes are honest about what is changing, clear that the freed-up time gets reinvested, and visible in the reskilling.
The honest line for a town hall: "It changes how the work gets done, and we are reinvesting the time, not cutting the team."
Once a board has agreed what AI is and why it is worth doing, the decisions begin: how the technology works well enough to judge it, where it should run, what it costs, and who keeps it under control. Three chapters, forty questions: the heart of the book for anyone who has to choose.
Enough of the mechanism to make a good decision: no more, no acronyms left standing. If you can follow these thirteen answers, you can hold your own with any vendor in the room.
It is strong wherever the job is to transform language: drafting, summarising, translating, explaining, coding. It is weak wherever the job needs guaranteed accuracy, current facts, exact arithmetic, or a single correct answer.
The quickest way to predict whether AI will help is to look at the kind of task. Work that is really about handling language or patterns (a long document into a short one, a spec into code, a rough draft into a clean memo, a question into a plain answer) plays to its strengths, often startlingly so. Work that needs a provably correct, single answer (the exact figure, the live price, the binding legal fact) does not. There the raw model is the wrong tool, and has to be paired with something that supplies and checks the truth.
Most disappointment comes from aiming it at the second kind of task and expecting first-kind performance. A good programme matches the tool to the work, then engineers around the weak spots. Much of this book is about that engineering.
Mechanically it only predicts the next likely word, so it has no internal sense of truth. That is exactly why it hallucinates, stating falsehoods as smoothly as facts. The fix is not a smarter model but grounding it in your real data and checking its work.
Whether a model "really understands" will be argued long after this book is forgotten, and for a decision-maker it is the wrong question anyway. The right one is what follows from how it works. Because it optimises for plausible continuation rather than verified truth, nothing inside it rings an alarm when it is wrong. It is not lying. Lying needs knowing the truth. It is producing the most likely-sounding text, usually right and occasionally, fluently, not.
A hallucination is that gap filled with something that fits the pattern but is false. It shows up most when the model is pushed past its training or asked about your private data. The fixes are architectural, not a bigger model. Retrieval hands it the real documents. Grounding and citation force a source. Verification has a second pass check the first. Human review catches the rest.
It predicts plausibility, not truth. So design as though it has no conscience about accuracy, because it does not.
By default it knows only the general patterns of its training data, up to a cut-off date. Nothing about your business, your customers, or yesterday. So everything specific and current has to be handed to it as you ask, inside a working space called the context window.
A model shows up knowing a lot about the world in general and nothing about you. It has never seen your contracts, your pricing, your policies, or last night's numbers. So for any answer that turns on your reality, you have to put the relevant information in front of it at question time: in the context window, the limited working memory it can hold for a single task.
That is why "just ask the AI" falls apart inside a company, and why retrieval matters so much. The craft, which Chapter 8 covers, is getting the right small slice of your information into that window at the right moment. Not everything, which swamps it. Just what the question needs.
| Ingredient | Where it comes from |
|---|---|
| General knowledge & language | Baked in during training (to a cut-off date) |
| Your instructions / the system prompt | Set by the people who built the application |
| Your relevant documents | Retrieved and inserted at question time (RAG) |
| The current conversation | Carried along as the exchange proceeds |
| Live data & tool results | Fetched from your systems mid-task |
The model reasons over whatever is in this window, and only this. Good answers are mostly a question of good ingredients.
Retrieval looks up the relevant pieces of your own information and hands them to the model just before it answers, so it responds from your facts instead of its memory. It is the single most important pattern in enterprise AI.
RAG (retrieval-augmented generation) is simpler than its acronym suggests. A question comes in. The system searches your knowledge (documents, databases, records) for the most relevant passages, drops those passages into the context window next to the question, and asks the model to answer using them. The model becomes a very capable reader of material you chose, rather than an unreliable rememberer of material it half-absorbed in training.
Vendors bring it up constantly because it is how you make a general model useful on your specifics, without
the cost and rigidity of retraining. In 2026 it is the default architecture for almost every enterprise assistant and question-answering use case.
No, and the confusion wastes millions. Feeding data through retrieval is cheap, instant, and easy to update. Training or fine-tuning bakes patterns into the model itself: expensive, slow, and the right answer far less often than people assume.
There are three ways to make a model work on your specifics, and people mix them up constantly. Context engineering shapes the instructions and information you give it each time: fast, free, no change to the model. Retrieval looks your facts up and feeds them in as it answers: cheap, and always current. Fine-tuning actually adjusts the model's internal weights on your examples. That is powerful for teaching a consistent style or a niche skill, but it is costly, slow to update, and easy to get wrong.
The expensive mistake is reaching for fine-tuning when retrieval would have done the job. The rule of thumb: if the problem is the model not knowing your facts, retrieve. If it is the model not behaving the way you want, consider fine-tuning. Chapter 8 lays out the full decision.
| Approach | What it changes | Cost & speed | Use when |
|---|---|---|---|
| Context engineering | The instructions, per task | Free, instant | Always: the first lever |
| Retrieval (RAG) | The facts supplied at answer time | Cheap, always current | It needs to know your information |
| Fine-tuning | The model's own weights | Expensive, slow to update | It needs to behave a specific way |
The harness is everything built around the model that turns it from a clever talker into a reliable worker: memory, tools, a sandbox, checks, limits. Since every
serious provider's model is now excellent, the harness is where the real gap between a demo and a dependable system opens up.
Picture the model as a brilliant mind with nothing attached. It can reason, but it has no hands, no memory between tasks, and no judgement about what it is allowed to touch. The harness (engineers also call it scaffolding) is the cockpit built around it. Memory, so it can carry state. Tools, so it can act. A sandbox, so the damage is bounded. Checkpoints, where a human approves. Budget caps, so it cannot run away with your money or your systems.
People say this so firmly in 2026 because the leading models have converged on quality. You can no longer win on the engine alone. The advantage now lives in the harness. The same model, well-harnessed, is trustworthy; badly harnessed, it is a liability. Chapter 9 takes it apart piece by piece.
| Harness part | What it gives the model |
|---|---|
| Memory | State across steps and sessions |
| Tools | The ability to act: search, read, write, call systems |
| Planning loop | A way to break a goal into steps and check progress |
| Sandbox & permissions | Limits on what it can touch or break |
| Human checkpoints | Approval at the points of consequence |
| Budget & step caps | A guarantee it cannot run away with cost or actions |
It runs a loop. Look at the goal and what has happened so far, decide the next action, take it with a tool, see what comes back, repeat, until the job is done or a limit stops it. That simple cycle, run well, is what produces autonomy.
Strip away the vocabulary and an agent is a loop. It plans (what is the next step toward the goal?), acts (uses a tool: searches a system, reads a file, sends a request), observes (what came back?), and repeats, each turn shaped by the last, until it decides the goal is met or hits a cap on steps, time or spend. The better agents add a verify step, checking their own work before they call it done.
Everything that makes agents powerful, and everything that makes them risky, lives in this loop. It is why they can finish multi-step work unattended, and why an ungoverned loop can burn a fortune or take a damaging action. Designing the loop well (what some now call loop engineering) is the subject of Chapter 9.
Repeat until the goal is met or a step / time / budget cap intervenes. The cap is not optional.
Tools are the connections that let an AI do things in your systems instead of just talking about them. MCP is the emerging standard plug that lets any model connect to any tool. Together they turn AI from an adviser into a worker.
On its own a model can only produce text. Give it tools (a way to search your knowledge base, query a database, send an email, update a record, call an API, even drive a web browser or operate a desktop application the way a person would when no API exists) and it can take real action in your business. That is the leap from a chatbot that suggests to an agent that does. MCP, the Model Context Protocol, matters because it standardises the connection. Instead of custom-wiring every model to every system, MCP is a common socket, so something you build once can be reused across models and applications.
Leverage and risk arrive together. Connecting AI to your systems is where almost all the value sits. And the moment it can act, it has to be governed: identity, permissions, audit, limits, all in Chapter 10.
Cost is charged per unit of text, and a single agent task can quietly run the model dozens of times around a long loop. So the way the loop is designed, far more than which model you pick, sets the bill.
Enterprise AI is metered by the token, roughly a word-fragment going in and coming out (Chapter 4 has the detail). A simple question costs a few fragments. But an agent on a multi-step task re-reads its growing context every loop, calls tools, retries, and verifies, so one "task" can mean twenty or fifty model calls, each paying for the whole context again. The per-token price is a small lever. The number and size of calls the design provokes is the big one.
That is why two teams on the same model can see costs differ tenfold. The real cost-control surface is the design (short contexts, cheap models for routine steps, expensive ones only where they earn it, hard caps), not the price list.
The price list is the lever everyone looks at; the design is the lever that matters.
Most mature deployments use several. A small, cheap model for routine steps, an expensive one only where the task is genuinely hard. Routing work to the right-sized model is the biggest cost-and-speed win on offer.
The instinct to standardise on one flagship is understandable and usually wrong. A flagship is overkill, and overpriced, for the routine majority of steps (classify this, extract that, format the other) where a small model does just as well at a fraction of the cost and latency. Keep the expensive model for the steps that actually need its reasoning.
This right-sized routing (sometimes called the reasoning sandwich, strong models to plan and check, cheap ones for the work in between) is one of the highest-return design choices in the book. It does add complexity: several models, and a router to choose between them. That is why it belongs to a maturing programme, not a first pilot.
Variation is built into how these models generate text. For creative or advisory work it is harmless, sometimes useful. For anything that needs a consistent, checkable result it is a real problem: managed by constraining the output and testing differently, not by expecting identical answers.
Ask a model the same thing twice and the wording shifts; sometimes the substance does too. This non-determinism is intrinsic. The system samples among plausible continuations rather than pulling back one fixed answer. For drafting an email or brainstorming, the variety helps. For deciding whether a transaction is fraud, or what a contract clause means, "it depends which time you ask" will not do.
The answer is not to demand identical outputs. It is to engineer for consistency where it counts: constrain the model to structured, checkable outputs, turn the randomness down for deterministic tasks, and (this is the important part) test the system statistically, across many runs, rather than expecting the single-pass repeatability of ordinary software. Chapter 9 covers testing the non-deterministic.
| Variation is acceptable | Variation is a problem |
|---|---|
| Drafting, brainstorming, ideation | Fraud and risk decisions |
| Summaries for a human to review | Anything feeding a system of record |
| Exploratory analysis | Regulated or audited outputs |
| Manage it by: structured outputs · lower randomness · statistical testing · human review at consequence points |
Ordinary software fails predictably, the same way every time. AI fails plausibly, differently each time, and sometimes invisibly. That is why it needs a different kind of testing, monitoring and control than anything else in your estate.
Conventional software has a comforting habit: same input, same output, and when it breaks it usually breaks loudly. AI breaks differently. It can produce a wrong answer that looks entirely right: the hallucination. It can pass testing and then drift over months as the world shifts around it. It can be steered by ordinary-looking text: prompt injection. And an agent can take a confident wrong action before anyone notices.
So you cannot govern AI with the assumptions you use for normal IT. It needs continuous evaluation rather than a one-time test, monitoring tuned for quality drift rather than just uptime, and human checkpoints wherever a plausible-but-wrong output would do real harm. None of that is paranoia. It is the price of admission, and the rest of the book builds it out.
| Failure mode | Unlike normal software because... | Demands |
|---|---|---|
| Plausible wrong answer | Looks correct; fails silently | Grounding, citation, review |
| Quality drift | Degrades as the world changes | Continuous evaluation |
| Prompt injection | Hijacked by ordinary text | Input controls, isolation |
| Non-reproducible bug | Doesn't repeat on demand | Statistical testing |
| Autonomous misstep | Acts before anyone sees | Sandboxes, caps, checkpoints |
Closes Chapter 3. Chapter 4 turns from how it works to where it runs, and what each choice costs.
Cloud, on-premises or hybrid: the choice every board eventually faces, and the money that follows it. Fourteen questions on where the technology runs and what it costs.
It is the classic rent-versus-own decision, applied to intelligence. Rent capability from a provider over the internet and trade control for convenience, or run models on hardware you own and trade convenience for control. Most large enterprises end up doing both.
Cut the jargon and the choice is about who runs the computer the model sits on. Cloud means renting. A provider hosts the model, you reach it over the internet, you pay per use, and your data leaves your walls under contract. On-premises means owning. You run the model on hardware in your own data centre, your data never leaves, and you carry the cost and effort of operating it.
Neither wins outright. The decision turns on how sensitive the data is, how heavily you will use the system, what your regulator demands, and whether you have the people to run infrastructure. The rest of this chapter takes those one at a time.
In the cloud you rent access to a model on someone else's computers, billed per use. On-premises you buy GPU servers and the team to run them, and download the model free. Either way, the model itself is the cheapest part.
"Cloud" is not one thing. You can call a model maker's API directly (fastest, frontier models), use a hyperscaler's managed service (AI inside the cloud governance you already have), or self-host an open model on rented cloud GPUs (cloud economics, your control). All are billed by use, and your data leaves your walls under contract.
On-premises is the opposite trade. You buy GPU servers (scarce and costly in 2026), the data-centre to run them, serving software, and (the bigger, quieter cost) a platform team to operate, secure and update it. The open-weight model on top, a Llama or a Qwen or a Gemma, is free to download. On-prem pays off on control and, above a certain volume, on cost. Never on simplicity.
| Line | Cloud (rent) | On-premises (own) |
|---|---|---|
| Compute | Included in the per-use price | GPU servers you buy (scarce, costly) |
| The model | Rented; closed frontier available | Open-weight, free to download |
| Serving & ops | Managed for you | Your serving software & platform team |
| Data | Leaves your walls (under contract) | Never leaves |
| Cost shape | Per-token, perpetual | Fixed; wins above heavy steady use |
Three cloud modes: direct API, managed platform, self-host on cloud GPUs. The model is the least of either bill.
The biggest fault line in the model market is open versus closed. Open-weight models are published for anyone to download and run on their own hardware. In 2026 the strongest come from Meta (Llama),
Alibaba (Qwen), DeepSeek, Mistral and Google (Gemma), with Chinese labs holding many of the top open spots. Closed models (OpenAI's GPT, Anthropic's Claude, Google's Gemini) stay with their makers and come only as a rented service.
You can self-host "open-weight" models whose makers publish them for download: Llama, Qwen, DeepSeek, Mistral, Gemma. The leading "closed" frontier models (GPT, Claude, Gemini) you can only rent, reaching them as a service. They are never handed over.
This decides what on-premises can even mean for you. If your strategy needs models running inside your own walls, you are choosing from the open set. The good news in 2026 is that the best open models have closed much of the gap to the frontier, which the appendix scorecards at the back lay out.
If on-premises is a requirement, your choice is the left column. See Appendices A and B for how they score. Source: open-weight leaderboards (BenchLM, Artificial Analysis), June 2026.
Below heavy, steady usage the cloud is cheaper, because you pay only for what you use. Above a crossover (commonly put around 2–3 million tokens a day at high utilisation) owning the hardware wins, and can be many times cheaper per unit.
Cost follows the rent-versus-own logic exactly. At low or spiky volumes, pay-per-use is hard to beat: no idle hardware, no team to staff. But the cloud charges for every token, forever, so as usage climbs the meter eventually overtakes the fixed cost of owning. Analyses in 2026 put the crossover around 2–3 million tokens a day at 70%-plus hardware utilisation. Past it, running an open model locally has been measured at up to roughly ten times cheaper per token.
So the question is not "which is cheaper" but "how much, and how steadily, will we use it?" Light and variable favours cloud. Heavy and predictable favours on-premises. The same logic points to hybrid: own the baseline load, rent the peaks.
Below the crossover, rent. Above it, owning wins, often by a wide margin. Source: on-prem vs cloud cost analyses, 2026 (TrueFoundry, PremAI, arXiv 2509.18101).
Hybrid means routing each piece of work to the right place: sensitive or high-volume tasks to models you control, everything else to the cloud's best-and-cheapest. Enterprises land there because it captures the control of on-premises and the reach of cloud at once.
Few large organisations are purely one or the other. Hybrid is the deliberate mix. A smaller, self-hosted model handles the sensitive data and the steady high-volume baseline inside your walls. The cloud's frontier models get called for the hard, occasional reasoning where they are worth the price and the data is not sensitive. A routing layer decides, task by task, which path to take.
The appeal is that it dissolves the false binary. You keep your crown-jewel data in-house, you control your baseline cost, and you still reach the best models in the world when a problem demands them. The price is complexity (two environments and a router to maintain) which is why hybrid is a place a programme grows into, not where it starts.
Sovereignty is the demand that your data stay under your own country's laws and reach. And yes: US-headquartered cloud providers can be compelled
under the US CLOUD Act to hand over data even when it sits abroad. That is exactly why so many enterprises in 2026 are rethinking where their AI runs.
Data sovereignty is the principle that information about your citizens, patients or customers stays governed by your jurisdiction, not someone else's. The complication is legal reach. Under the US CLOUD Act, American providers can be ordered to produce data they hold no matter which country's data-centre it physically sits in. Storing data "in region" with a US provider does not, on its own, put it beyond US legal reach.
This has become a board-level issue. Surveys in 2026 show a majority of firms trying to cut their dependence on US clouds for sovereignty reasons. For genuinely sovereign workloads the options narrow to non-US providers or on-premises deployment, which is one of the strongest non-cost arguments for running open models in your own walls.
The sovereignty test
If a foreign court could compel disclosure of this data, it is not sovereign, wherever it is "stored." For workloads where that is unacceptable, true sovereignty means non-US infrastructure or on-premises. Source: CLOUD Act analysis; enterprise sovereignty surveys, 2026.
They start from the data, not the technology. The most sensitive, regulated workloads go on-premises or to a sovereign cloud; the rest goes to mainstream cloud. The result is almost always hybrid, drawn along the line of what the regulator will tolerate leaving the building.
In regulated sectors the deployment decision sits downstream of the compliance regime. A hospital, a bank or a ministry classifies its data by sensitivity, then maps each class to an environment the regulator accepts. Patient records, core financial data and citizen information lean toward on-premises or certified sovereign clouds. General productivity, public information and internal drafting move freely to mainstream cloud.
The outcome is a tiered, hybrid estate drawn along regulatory lines. What separates the mature from the reckless is doing the classification first (deciding what may leave the building before choosing tools) rather than buying a platform and arguing compliance afterward.
| Data class | Typical home | Example |
|---|---|---|
| Most sensitive / regulated | On-prem or sovereign cloud | Patient records, core banking, citizen data |
| Internal / confidential | Private cloud or hybrid | Contracts, internal analysis |
| General / public | Mainstream cloud | Drafting, public info, productivity |
A token is a fragment of text, roughly three-quarters of a word, and you pay for every one going in and coming out. To budget, estimate tokens per task, multiply by volume and the per-million-token price, and remember that output costs several times more than input.
Models read and write in tokens, small pieces of words; a page runs roughly 500 to 800 of them. Providers charge per million tokens, and output is priced higher than input. In 2026 the spread is enormous. A frontier closed model like GPT-5-class runs about $2.50 in and $15 out per million; Claude Opus about $5 and $25; cheap-but-capable options like Gemini Flash-Lite or DeepSeek sit near $0.10 to $0.40. Prices have fallen roughly 80% in a year.
To budget, estimate the tokens a typical task burns (including the context you feed it and any agent looping) multiply by expected daily volume and the blended price, then project forward. The arithmetic is simple. The discipline is remembering that agentic loops and big contexts multiply the token count far beyond the visible question.
| Model class | Input | Output |
|---|---|---|
| GPT-5-class (frontier) | ~$2.50 | ~$15.00 |
| Claude Opus (frontier) | ~$5.00 | ~$25.00 |
| Claude Sonnet / Gemini Flash (mid) | ~$1.50–3.00 | ~$9–15 |
| Gemini Flash-Lite / DeepSeek (budget) | ~$0.10–0.44 | ~$0.28–0.90 |
Output costs several times input. Prices fell ~80% across 2025–26. Sources: BenchLM, pricepertoken.com, provider pricing, June 2026.
The build cost is one-time and visible. The run cost is the killer: every query paying its token bill, every day, forever, climbing as adoption grows. It breaks business cases because success itself makes it bigger.
Traditional software has a big build cost and a tiny cost to run. AI flips that intuition. The run cost is consumption-based and never-ending: every user, every query, every agent loop pays again. Worse, it scales with success. The more people adopt the tool, the higher the bill, so a triumphant pilot can produce a frightening invoice precisely because it worked.
This is the line most often missing from a business case. Fund the build, forget the run, and you are the board that cancels the project in month nine when the cloud bill lands. The cost-control disciplines in Chapter 7 (caching, routing to cheaper models, capping loops) exist to keep this meter survivable.
The cost curve that surprises CFOs: run cost grows with success. Model this before you scale, not after.
Default to buying or renting proven products for anything that is not your competitive edge. Reserve building for the narrow places where a custom system genuinely sets you apart. Most enterprises overbuild, and the maintenance never ends.
There are three routes. Rent a finished AI feature inside software you already use: the fastest, lowest-effort path. Buy a specialised AI product and configure it. Or build a custom system on models and infrastructure you assemble. The right answer is usually the least effort that meets the need, and most organisations reach for "build" far too readily, underestimating what it costs to maintain whatever they create.
The question that decides it is whether the capability is a differentiator. If AI in this workflow is part of what makes you distinctive, building can be worth it. If it is a commodity task a dozen vendors already solve, buying or renting is cheaper, faster and better maintained. Build where you compete. Buy where you do not.
Tie it to a specific, measurable outcome. Budget all three cost layers, including the perpetual run cost. State the baseline you will measure against. And be honest about the failure odds. A business case that pretends AI is free or guaranteed is the one that gets torn up.
A durable AI business case looks like any durable business case, with two extra disciplines. First, it names a concrete outcome (hours saved in a named process, cycle-time cut, error rate down), not "adopt AI." Second, it budgets the run cost out loud, because the CFO who gets surprised by the perpetual meter is the CFO who kills the programme.
Then it does what the 95% who fail did not. It sets the baseline and the metric up front, so success can be proven, and it is candid about risk, proposing a small, measured first step rather than a leap. A business case built this way survives scrutiny precisely because it does not oversell.
Measure the time saved against a baseline you captured before launch, then convert it honestly into redeployed capacity or avoided cost. Do not count saved hours as cash unless they actually turn into more output or fewer people.
The honest scorecard tracks the operational metric (time, throughput, error rate) and states the conversion assumption out loud, rather than quietly monetising every saved minute. It is less flattering and far more defensible. It is also the difference between a programme that keeps its funding and one that loses it at the first audit.
Most enterprise AI value shows up first as time: a task that took an hour now takes ten minutes. To make that credible to a CFO you need a before figure measured the same way as the after, which is why baselining before launch is non-negotiable. Then convert the saved time honestly. It is real value only if it is redeployed into higher-value work or removed as avoided cost. Saved hours that just evaporate into the day are not return.
| Step | What you do |
|---|---|
| 1 · Baseline | Measure the task's time/cost/error before launch |
| 2 · Measure after | Same metric, same method, post-launch |
| 3 · State conversion | Redeployed capacity? Avoided cost? Be explicit |
| 4 · Report honestly | Operational gain + the assumption behind any $ figure |
Rising cost with flat adoption, no agreed success metric, and a run-cost line nobody is watching. Together those three are the financial signature of a project that gets killed at the next budget review, usually around month nine.
Failing AI projects share a recognisable cost-and-usage profile. Spend climbs but usage does not: money going out, no behaviour change coming back. There is no agreed metric, so nobody can defend the value when finance asks. And the run cost goes unmonitored, drifting upward until it becomes the headline reason to stop. Gartner's projection that 40% of agentic projects will be cancelled by 2027 without governance and clear success criteria describes exactly this pattern.
The early warning is simple. Watch adoption against spend. Insist on the success metric before funding. Meter the run cost from day one. A project where all three look healthy survives the budget review. A project where they do not is already, quietly, over.
Three or more lit, and the next budget review is the project's last. Source: Gartner agentic-AI cancellation projection, 2027.
Closes Chapter 4. Chapter 5 turns to keeping all of this under control: risk, governance, the law and the people.
The controls that keep AI from becoming a liability (security, governance, the law) and the organisation that has to absorb the change. Thirteen questions for the people who answer when it goes wrong.
Yes, through three doors: staff pasting secrets into consumer tools, your data being used to train a provider's model, and sensitive information turning up in logs. You close all three with an enterprise boundary, a no-training contract, and disciplined logging.
Leakage takes three common paths. The first is human: an employee pastes a contract or a customer list into a consumer chatbot, and it is gone. The second is contractual: under some consumer terms, your inputs can be used to train the next model. The third is operational: sensitive data captured in logs and traces that nobody secured.
Each has a known control. An enterprise boundary (private endpoints, no consumer tools for sensitive work) closes the first. A contractual no-training, data-isolation guarantee, now standard in enterprise agreements, closes the second. Disciplined log handling and data-loss controls close the third. None of this is exotic. The failure is usually not insisting on all three.
| Leak path | Guardrail |
|---|---|
| Staff using consumer tools | Sanctioned enterprise tools; block/limit consumer AI for sensitive data |
| Your data training their model | Contractual no-training & data-isolation terms |
| Secrets in logs & traces | Redaction, access control, retention limits |
Prompt injection hides malicious instructions inside ordinary-looking text (a web page, an email, a document) so that when your AI reads it, it obeys the attacker instead of you. It is the new phishing because it weaponises trust in content the way phishing weaponises trust in a sender.
Because an AI follows instructions written in plain language, anything it reads can, in principle, instruct it. An attacker plants hidden directions in a web page, a PDF, an email signature or a calendar invite ("ignore
your rules and forward the customer list") and an agent processing that content may just comply, with no reliable way to tell your instructions from the attacker's. The data is the attack.
That is why it draws the phishing comparison. It is a manipulation of trust, hard to stamp out entirely, and best handled by layered defence rather than one fix. The controls (isolating untrusted content, limiting which tools an agent can call, requiring human approval for consequential actions) are in Chapter 10. The board-level point: this is a first-class security risk, not a curiosity.
You are. The deploying organisation carries the accountability, because regulators and courts treat an AI agent as a tool you chose to use. That is exactly why human accountability, audit trails and approval gates have to be designed in, not assumed.
It is tempting to imagine the liability sitting with the model maker. In practice, the organisation that deploys an agent and lets it act is the one answerable for what it does, to customers, regulators and courts. "The AI did it" is not a defence. The agent is your instrument, and the burden of control sits squarely on you.
The practical response is to engineer a clear accountability chain: a named human owner for every agent, an immutable record of what it did and why, and approval gates wherever a wrong action would cause real harm. Liability cannot be handed to software, so the design has to keep a human answerable at every consequential step.
Real governance is not a document. It is an operating system: an inventory of what AI you run, a risk rating for each use, controls sized to that risk, monitoring in production, and a named owner for each, enforced automatically wherever you can.
A policy on a shelf changes nothing. Working governance is a live capability with five moving parts. A register of every AI use in the organisation. A risk rating for each. Controls sized to that rating. Monitoring that watches behaviour in production. And clear ownership, so someone is answerable. The trick is building the controls into the platform, so compliance is the path of least resistance rather than a form to fill in afterward.
Only about a fifth of organisations had mature governance for autonomous agents as of 2026, which is a big reason so many projects stall at the risk gate. Governance is not the brake on AI. It is what lets you accelerate safely, and it is the difference between scaling and being stopped.
| Element | What it does |
|---|---|
| AI register | Know every AI use in the organisation |
| Risk rating | Classify each by potential harm |
| Proportionate controls | Heavier guardrails for higher risk |
| Production monitoring | Watch real behaviour, not just intentions |
| Named ownership | Someone accountable for each use |
Source: only ~21% of organisations had mature agent governance, 2026 industry surveys.
Shadow AI is employees using unsanctioned AI tools on company work, with no oversight. It is far bigger than most leaders think, because the tools are free, useful, and one browser tab away. The only real cure is a sensible sanctioned alternative.
When the organisation is slow to provide approved AI, staff provide their own, pasting work into whatever free tool gets them to the deadline. From the employee's side this is rational. From the organisation's it is dangerous: confidential data leaves through channels nobody monitors, under terms nobody vetted. The sanctioned usage leaders can see sits on top of a much larger mass they cannot.
Bans do not work, because the pull to use the tools is too strong. What works is making the sanctioned path the easy path: provide good enterprise tools, make them convenient, and explain plainly why the consumer ones are unsafe for company data. You cannot police your way out of shadow AI. You out-compete it.
The cure is not a ban but a better, sanctioned alternative that staff actually prefer.
The EU AI Act and its imitators regulate by risk tier: a few uses banned outright, heavy obligations on "high-risk" ones, and most uses barely touched. So your first compliance step is to classify each use case by tier, then design to the obligations that tier carries.
The emerging regulatory pattern, led by the EU AI Act, is risk-tiered. A small set of uses is prohibited. A defined "high-risk" set (typically employment, credit, healthcare, essential services) carries substantial obligations: risk management, data governance, transparency, human oversight, documentation. Everything else faces light or transparency-only duties. Other jurisdictions are converging on similar shapes, so building to this model travels reasonably well.
For builders, the consequence is that where you apply AI shapes your compliance burden as much as how. A drafting assistant and a credit-decision agent sit in different tiers and need different controls. Classify each use case early. Chapter 10 maps specific controls to specific obligations.
| Tier | Obligation | Typical uses |
|---|---|---|
| Prohibited | Not allowed | Social scoring, manipulative uses |
| High-risk | Heavy: oversight, docs, data governance | Hiring, credit, health, essential services |
| Limited | Transparency (tell users it's AI) | Chatbots, content generation |
| Minimal | Light / none | Most productivity uses |
Source: EU AI Act risk framework; convergent regimes, 2026.
Before any AI touches production: a defined boundary on what data and systems it can reach, human approval for consequential actions, an audit trail, monitoring, content filters, and hard caps on cost and
behaviour. Anything less is not a pilot. It is an incident waiting to happen.
There is a floor below which no enterprise AI should go live, however promising it looks. A clear data and tool boundary (what it may see and touch). Human approval at consequential steps. An audit trail of everything it does. Monitoring for quality and abuse. Content and safety filters. And hard caps on spend, steps and actions, so it cannot run away.
These are not gold-plating. They are the seatbelt and the brakes. A system without them can demo beautifully and still be unfit to operate, because the failure modes of AI (plausible-wrong outputs, injection, autonomous missteps) are exactly what these controls contain. Treat the list as the go-live gate.
Beyond the data and software people you already have, expect to need AI and ML engineers, a platform team, an AI-governance or risk lead, and (increasingly) people who design and maintain the context and the evaluations, plus business-side translators who turn problems into use cases.
An AI programme stretches the org chart. On the technical side you need engineers who can build and operate AI systems, a platform team that supplies shared tooling and guardrails, and people who own the unglamorous-but-decisive work of context, retrieval and evaluation. On the control side you need an AI-governance or risk lead accountable for safe, compliant use.
Just as important, and often missed, are the translators: the people who sit between the business and the technology and turn a vague "could AI help here?" into a scoped, measurable use case. The shortage is rarely raw modelling talent (you buy the models), but these connective and operational roles. Hire for the gaps, not the headlines.
| Role | Owns |
|---|---|
| AI/ML & agent engineers | Building and operating the systems |
| AI platform team | Shared tools, guardrails, reusable patterns |
| Context & evaluation owners | Retrieval quality, testing, reliability |
| AI governance / risk lead | Safe, compliant, accountable use |
| Business translators | Turning problems into scoped use cases |
Bolt AI onto an unchanged process and you get a faster version of the old way, with a fraction of the value. Redesign the process around what AI does well, and what people do well, and you get the step-change. That is the harder, rarer choice.
The disappointing AI deployments are usually the literal ones: take an existing task, add an AI button, change nothing else. The gain is marginal, because the process was built around human limits the AI no longer has. The transformative ones redesign the work: who does what, where the human judgement sits, what becomes possible when a step that took a day now takes a minute.
This is why AI is a change programme, not a purchase. The technology is necessary but not enough. The value comes from the redesign around it. Study after study finds adoption and impact driven more by how the work and the incentives are reshaped than by the tool itself.
Pick a use case that is high in value and high in feasibility, narrow enough to ship in a quarter, on data you already control, where a wrong answer is recoverable. And resist the urge to make your first project the hardest, most visible thing you do.
The best first project is chosen on two axes. Value: does it matter to the business? And feasibility: can we actually do it soon, on data we have, at tolerable risk? The sweet spot is high on both: a real problem that is also winnable. Document-heavy, high-volume,
low-complexity tasks often qualify, because they pay back fast and the before-and-after is easy to measure.
Size matters just as much. A first project should be narrow enough to deliver in a quarter and forgiving enough that an error is recoverable. Trying to prove the technology on the hardest, most prestigious problem is how programmes burn their credibility before they have any. Win a small, real thing first. Earn the right to the big one.
| High feasibility | Low feasibility | |
|---|---|---|
| High value | START HERE: do now | Plan & build toward |
| Low value | Quick wins (sparingly) | Avoid |
First project: top-left. Narrow, valuable, feasible, recoverable.
Build the pilot on the same data, the same security, and the same success metric as production, from day one, with the path to scale designed in. Then graduating is a decision, not a rebuild. The pilots that dazzle on a clean demo are exactly the ones that die at the production gate.
The reason roughly 88% of agent pilots never ship is that they were built to impress, not to graduate: clean sample data, no real security, no agreed metric, no integration. Crossing to production then means rebuilding everything that was faked, and the will runs out. A graduating pilot does the opposite. It uses real, if scoped, data, the actual security model, and the production success metric from the start.
Run it through explicit gates: does it work on real data, does it pass security, does it hit the metric, does it integrate. Then the move to production is an approval, not an excavation. The pilot's job is not to look good. It is to prove it is ready.
A demo shows the technology at its best, on the vendor's chosen example. The questions that matter are the ones a demo is designed to bury: data handling and training terms, how the system fails and how
accuracy is measured, the fully-loaded cost at your scale, the integration reality, and your exit if you want out. A vendor who answers those crisply is a very different proposition from one who keeps steering back to the demo.
Judge vendors on the things a demo hides: what happens to your data, how they handle failure and accuracy, what it costs at scale, how you would leave. And make them prove it on your data, not theirs. Every demo is built to look flawless.
The single most clarifying move is to insist on a proof of concept on your own data and your own task. Performance on a curated showcase tells you little. Performance on your messy reality tells you everything. Below is the short list of questions that separates substance from theatre.
In 90 days, stand up the guardrails, pick one valuable-and-feasible use case, and ship it to real users with a measured baseline. In year one, do not boil the ocean, do not build what you can buy, and do not scale anything you have not yet proven.
A credible first 90 days is deliberately modest. The first month sets the foundation: governance floor, sanctioned tools, the team, and a chosen use case with a measured baseline. The second builds and tests a graduating pilot on real data and security. The third ships it to real users and measures against the baseline, producing evidence rather than a demo. By day 90 you have one real thing working, and the proof to fund the next.
Restraint matters just as much. The year-one failures are predictable: trying everything at once, building commodity capabilities better bought, scaling on enthusiasm before proof, skipping the governance that blocks you later. A short do-not list protects a young programme from its own ambition.
Closes Part Two. Part Three goes under the hood: choosing, building, deploying, securing and scaling the technology, for the teams who do it.
For the people who build what the board approves. How a model is chosen on evidence, deployed on-premises and in the cloud, fed the right data, wrapped in a dependable harness, and run securely at scale. Five chapters, fifty-two questions, with worked examples and named models.
The landscape in mid-2026 and how to choose on evidence rather than hype: capability, cost, latency, licence and control, judged against your own work, not a public leaderboard.
Score candidates on five axes: capability on your task, cost per unit of work, latency, licence, and how much control you need. Then weight them by the use case. The "best" model on a leaderboard is rarely the best model for a given job.
Model choice is a multi-criteria decision, not a single ranking. Five axes matter. Capability: how well it does your task. Cost: fully loaded, per unit of work, not the headline token price. Latency: fast enough for the experience. Licence: what you are allowed to do, which bites hardest if you self-host. And control: open-weight you run, or closed you rent. A use case weights these differently: a customer-facing assistant cares about latency and cost; a sovereign workload cares about control and licence.
The discipline is to set the weights from the use case before you look at any benchmark, then score real candidates against them. Teams that start from "which
model is best?" buy a flagship and overpay. Teams that start from "best for this job, on these axes?" make choices they can defend.
| Axis | The question | Matters most for |
|---|---|---|
| Capability | How well does it do OUR task? | Hard reasoning, coding, analysis |
| Cost | Fully-loaded cost per unit of work? | High-volume, consumer-facing |
| Latency | Fast enough for the experience? | Real-time assistants, chat |
| Licence | What are we permitted to do? | Self-hosting, redistribution |
| Control | Run it ourselves, or rent it? | Sovereign & sensitive workloads |
Open-weight models give you control, sovereignty, no per-token fee, and freedom from a single vendor, but you have to run them yourself. Closed models give you convenience and often the peak capability of a managed frontier service, but you give up control and pay forever.
It is the control-versus-convenience trade again, restated for models. Open-weight ones (Llama, Qwen, DeepSeek, Mistral, Gemma) you can download, run in your walls, fine-tune freely, and use with no per-token charge. The price is that you operate them. Closed ones (GPT, Claude, Gemini) are managed services: often at or near the frontier, effortless to call, but rented forever and run on someone else's terms and hardware.
In 2026 the call is more balanced than it used to be, because the best open models have closed much of the capability gap. For sovereignty, heavy steady volume, or deep customisation, open wins. For peak capability with no operational burden, closed still leads. Plenty of enterprises run both: open for the controlled baseline, closed for the hard edge.
Here is the picture in June 2026. At the closed frontier, OpenAI's GPT-5 family, Anthropic's Claude Opus 4.8 (the practical pick for many outside the US after a stronger variant was pulled from non-US customers
under export controls in June 2026) and Google's Gemini 3 trade the lead on reasoning, coding and agentic work. Among open weights, DeepSeek, Alibaba's Qwen, Zhipu's GLM and Moonshot's Kimi lead, with Meta's Llama standing out for ultra-long context and Google's Gemma for small, edge-friendly sizes.
The closed frontier (GPT-5-class, Claude Opus 4.8, Gemini 3) still leads on the hardest tasks. But the best open models (DeepSeek, Qwen, GLM, Kimi, Llama) now sit just behind and lead outright on price-performance, with Chinese labs holding most of the top open spots. The chasm of 2024 is a single step in 2026.
The trajectory matters more than the snapshot. Open models have closed most of the practical gap and win decisively on cost, and benchmark leadership now changes hands every few months. The strategic takeaway, which Q57 develops, is to build for swappability rather than bet the firm on any one model.
| Tier | Closed (rent) | Open (run or rent) |
|---|---|---|
| Frontier | GPT-5-class, Claude Opus 4.8, Gemini 3 | DeepSeek V4, Qwen 3.5, GLM-5, Kimi K2.6 |
| Workhorse | Claude Sonnet, Gemini Flash, GPT-5 mini | Llama 4, Mistral Large, Qwen mid-size |
| Small / edge | Gemini Flash-Lite, Haiku | Gemma 3 (1–4B), Mistral Small, Qwen small |
A snapshot of a fast-moving market. Names illustrate categories, not endorsements. Sources: Artificial Analysis, BenchLM, LM Arena, June 2026.
Smaller than you think, for most tasks. A small or mid-size model handles classification, extraction, routine drafting and most retrieval-grounded answers at a fraction of the cost and latency. Save the big model for genuinely hard reasoning.
Bigger models are more capable, more expensive, and slower, so the skill is matching size to task. A great deal of enterprise work (categorising a ticket, pulling fields from a document, summarising a passage, answering from retrieved text) sits well within reach of a small or mid-size model. Reaching for a frontier model on those is paying first-class fare to ride one stop.
The frontier earns its price on the hard minority: multi-step reasoning, complex code, ambiguous judgement, long-context synthesis. The mature pattern (Q20, Q66) sends most steps to small models and escalates only the hard ones, which is often the single biggest cost-and-latency win in a deployment. Test the smallest model that could plausibly work before assuming you need the largest.
Cost and latency rise steeply with size; capability gains taper for routine work. Right-size, then route.
Build a small evaluation set from your own real tasks, with known-good answers, run every candidate through it, and score on the dimensions you care about. Public leaderboards measure generic ability, not performance on your job, and the two often disagree.
Public benchmarks are good for orientation and bad for decisions. A model that tops a coding leaderboard can underperform on your particular domain, data and prompts. The only test that counts is on your work. Pull together a representative set of real tasks with known-good outputs (an "eval set") run each candidate through it, and score against criteria you define: accuracy, format, tone, refusal behaviour, cost, latency.
That eval set becomes a lasting asset. It lets you compare models honestly, re-test the moment a new one appears, and catch regressions when something shifts. Building it is the unglamorous work that separates teams who pick models on evidence from teams who pick on reputation. Chapter 9 carries evals from selection through into production monitoring.
"Open-weight" does not mean "no rules." The most permissive licences (Apache-2.0 and MIT, used by Qwen, DeepSeek and GLM among others) let you
deploy commercially, modify, and redistribute with essentially no royalty or restriction. Others carry conditions: acceptable-use clauses, limits that trigger at very large scale, attribution requirements, or bans on using the outputs to train a competing model.
It varies sharply. The cleanest (Apache-2.0 and MIT, used by Qwen, DeepSeek and GLM) allow unrestricted commercial use. Others, like some Llama and Gemma terms, add conditions on scale, branding or permitted uses. So have legal read the licence before a model is built into a product, not after.
For an enterprise embedding a model into a product, this is not a footnote. A licence that forbids your intended use, or kicks in obligations at your scale, can sink an architecture after it is built. The rule is simple. Have legal read the licence of any model before it is designed in, and lean toward the permissive ones where the use is commercial.
| Licence | Used by (examples) | Commercial use |
|---|---|---|
| Apache-2.0 / MIT | Qwen, DeepSeek, GLM, Mistral (most) | Unrestricted, no royalty |
| Custom "community" | Some Llama, Gemma terms | Allowed, with conditions / scale limits |
| Research-only | Some released checkpoints | Not for production |
Always confirm the specific version's terms with legal. Licences change between releases.
Standardise enough to keep your platform simple and your costs legible, but never so hard that you cannot swap. The right posture is a default model for most work, plus a deliberate, abstracted ability to route to others. Single-vendor lock-in is a real risk in a market this fast.
Two failure modes bracket this. Over-fragmentation (a different model for every team) makes the platform unmanageable and the costs opaque. Over-consolidation (one model, hard-wired everywhere) is simple right up until that vendor raises prices, changes terms, or gets overtaken, and you cannot move. The healthy middle is a default model for the bulk of work, with the architecture abstracted so others can be routed to or swapped in without a rebuild.
This is both a cost lever (route cheap work to cheap models) and a resilience lever, with no single point of dependence. The price is the engineering of an abstraction layer and a router, which is why it is a maturity step rather than a day-one move. But in a market where leadership changes quarterly, that optionality is worth the effort.
If your use case involves more than text (scanned documents, images, audio, charts) you need a multimodal model, and that narrows the field. Not every model handles every modality equally well, and the gap between text-only and true multimodal performance is real.
Much enterprise data is not plain text: scanned invoices, photographs, diagrams, recorded calls, mixed-format documents. Multimodal models read and reason over these directly, which opens up use cases (document understanding, visual inspection, call analysis) that text-only models cannot touch. Most frontier closed models are strongly multimodal. Among open models, capability varies a lot more.
For selection, this adds modality coverage as a hard filter. If the task needs to read a document image or interpret a chart, models that cannot are simply out, whatever their text scores. And confirm the modalities you need against real performance, because claimed support and useful performance are not always the same thing.
| Your data includes | You need |
|---|---|
| Text only | Any capable model |
| Scanned docs & images | Vision-capable multimodal model |
| Audio / calls | Speech-capable model or pipeline |
| Charts, tables, mixed formats | Strong document-understanding model |
Build an abstraction layer so the model is a swappable component. Keep your prompts, evals and data independent of any one provider. Design so that changing
models is a configuration change, not a re-engineering project. In this market, today's best model will be overtaken.
The pace of change is itself a design constraint. A system hard-wired to one model's quirks (its exact prompt format, its specific API, its particular behaviours) turns into a liability the moment a better or cheaper model shows up. The defence is portability by design: an abstraction layer between your application and the model, prompts and eval sets kept provider-neutral, and your data and retrieval owned by you, not trapped in a vendor's format.
Done well, swapping models becomes a tested config change: point traffic at the new model, run your eval set, compare, switch. That protects you against obsolescence and keeps your negotiating leverage. The small ongoing tax of an abstraction layer buys a lot of strategic freedom.
Far more than the token price or the free download. The fully-loaded cost includes inference, the hardware or API spend, the engineering to integrate and maintain it, evaluation, monitoring, and the switching cost if you ever leave. Compare models on that total, not the sticker.
A model's headline price (a per-token rate, or "free to download") is the most visible and least complete part of its cost. The fully-loaded figure adds the run cost at your real volume, the infrastructure (cloud spend or on-prem hardware), the engineering to integrate, tune and maintain, and the ongoing evaluation and monitoring. For open models, "free" hides the biggest costs in hardware and people. For closed models, the low entry price hides the perpetual meter.
The honest comparison puts every candidate on the same total-cost basis at your expected scale. A cheap-per-token closed model can cost more than a "free" open one at high volume, or far less at low volume. Only the loaded total, modelled against your usage, settles it.
| Cost component | Closed (rented) | Open (self-hosted) |
|---|---|---|
| Model access | Per-token, perpetual | Free to download |
| Infrastructure | Included in price | GPU hardware / cloud GPUs |
| Engineering & ops | Lower: managed | Higher: you run it |
| Eval & monitoring | Required either way | Required either way |
| Switching cost | Lower if abstracted | Lower if abstracted |
Compare on the loaded total at YOUR volume: the crossover (Q26) decides which wins.
Closes Chapter 6. Chapter 7 turns selection into deployment: running these models on-premises and in the cloud, with the training-versus-inference distinction made plain.
Concrete reference builds, with named models and real numbers: running an open model like Gemma in your own data centre, standing up a cloud architecture, and the training-versus-inference distinction that underpins all of it.
Training is the one-time, hugely expensive job of building a model by learning from data. Inference is running the finished model to produce an answer. Because you pay for inference on every single query, forever, it (not training) is where almost all of an enterprise's AI cost and engineering effort goes.
The two words name two completely different activities, and mixing them up gets expensive. Training is construction: feeding a model vast data over weeks on thousands of GPUs to teach it language and reasoning. It happens once, with occasional updates, costs millions to hundreds of millions, and for almost every enterprise is something you buy the result of. You do not train frontier models. You use them. Inference is operation: every time you send a prompt and get an answer, the model runs once. That is inference, and you do it millions of times.
For an enterprise that distinction is the whole economic story. You rarely pay to train. You constantly pay to infer. Every token price in Q30, every GPU in Q60, every cost-control lever in Q66 is about inference. When this book or the market says "running cost," "serving," or "the meter," it means inference. Fine-tuning (Q15, Q73) sits in between (a smaller, optional training step on top of a bought model), but over a system's life it too is dwarfed by the cost of inference at scale.
Rule of thumb for an enterprise: you don't train, you fine-tune occasionally, and you infer constantly. Optimise for inference.
| Training | Inference | |
|---|---|---|
| What it is | Building the model by learning from data | Running the finished model to get an answer |
| How often | Once (plus rare updates) | Every single query: continuously |
| Who does it | Model makers (you buy the result) | You: every day, at scale |
| Cost shape | Huge, one-time | Per-query, perpetual: the real bill |
| Hardware | Thousands of GPUs for weeks | Enough GPUs (or API) to serve your traffic |
Five layers: GPU hardware, the model file, a serving engine to run inference efficiently, a gateway and supporting services around it, and a small team to operate the lot. Assembled, they make a private inference service that never sends your data outside your walls.
Running an open model like Google's Gemma in your own data centre is a well-trodden path in 2026. From the bottom up: GPU servers provide the compute; the model weights are downloaded and stored; a serving engine (Q62) loads the model and runs inference efficiently for many users at once; a gateway and supporting services add an API, caching, authentication, logging and monitoring; and a platform team operates, secures and updates it. The result behaves like a cloud API but lives entirely inside your boundary.
A typical mid-size starting point: a small open model (a few billion parameters, well quantised) on one or two GPU servers, serving an internal document-Q&A workload for a few hundred users. That is enough to prove the architecture and the sovereignty case before you scale. The reference stack below is the shape almost every on-prem deployment ends up taking.
| Layer | What it does | Example |
|---|---|---|
| Application / gateway | API, auth, rate limits, logging | Internal API gateway |
| Serving engine | Runs inference efficiently at concurrency | vLLM / TGI |
| Model | The open weights you downloaded | Gemma (quantised) |
| GPU hardware | The compute that runs it | 1–4 data-centre GPUs |
| Ops & monitoring | Patching, scaling, observability | Platform team + tooling |
Your data never leaves this stack. That is the whole point of on-premises.
The deciding spec is VRAM, the GPU's fast memory, which has to be big enough to hold the model plus its
working state. A small, quantised model of a few billion parameters fits comfortably on a single modern GPU. A mid-size model needs a high-end data-centre card or two. A frontier-size open model needs several costly GPUs working together. Sizing the GPU to the model is the first on-prem engineering decision.
Inference needs GPUs with enough high-speed memory, called VRAM, to hold the model. A small model fits on one accessible GPU; a frontier-size model needs several expensive data-centre GPUs. They are scarce because AI demand has outrun supply, which makes the hardware, not the model, the real constraint.
Scarcity is the second reality. Through 2025 and 2026, demand for AI accelerators outstripped supply, stretching lead times and pushing up prices, while Deloitte reported the great majority of enterprises expecting their AI infrastructure budgets to more than triple within three years. For many organisations the GPU (its cost and its availability) is the true constraint on an on-prem strategy, not the model or the software.
| Model size | Approx VRAM (quantised) | Typical hardware |
|---|---|---|
| Small (1–8B), e.g. Gemma 3 4B | ~4–10 GB | 1 accessible GPU |
| Mid (8–30B) | ~12–24 GB | 1 high-end data-centre GPU |
| Large (30–80B) | ~40–80 GB | 1–2 top GPUs (e.g. 80GB class) |
| Frontier-size open (100B+) | Multiple ×80 GB | A multi-GPU server |
Quantisation (Q63) cuts these figures sharply. Source: Deloitte AI Infrastructure Survey, 2026; vendor specs.
A serving layer is the software that turns a raw model file into an efficient, multi-user service. It manages memory, batches many requests together, and streams answers back fast. vLLM, TGI and Ollama are the common choices, from high-throughput production engines to simple single-machine tools.
The common options sit on a spectrum. vLLM and TGI (Text Generation Inference) are production-grade, high-throughput engines built to serve many users efficiently. Ollama is a simple tool for running models on a single machine: great for development and small workloads, not for serving the enterprise. Choosing
the serving layer is choosing your throughput, your latency, and how much operational complexity you take on.
A downloaded model is just a big file. Something has to load it onto the GPU and run inference well for many users at once. That is the serving layer. Its job is efficiency: keeping the GPU busy, batching concurrent requests so the hardware is fully used, managing memory so more users fit, and streaming tokens back with low latency. Without it you have a model that technically runs and practically cannot serve a department.
| Tool | Built for | Use when |
|---|---|---|
| vLLM | High-throughput production serving | Many concurrent users; efficiency matters |
| TGI | Production serving, broad model support | Production workloads on common models |
| Ollama | Simple single-machine running | Development, prototyping, small scale |
Quantisation shrinks a model by storing its numbers at lower precision, like saving an image at lower resolution. It cuts memory and cost by half or more, usually with small quality loss. That is what makes running capable models on affordable hardware practical.
A model is billions of numbers. By default they are stored at high precision (16-bit), which is accurate but memory-hungry. Quantisation stores them at lower precision (8-bit, or even 4-bit) roughly halving or quartering the memory and speeding inference up. Think of a high-resolution photo saved smaller: almost identical to the eye, far lighter to carry. An 8-bit model usually loses very little quality. 4-bit saves more memory for a more noticeable, often still-acceptable, trade.
The payoff is big. Quantisation is frequently what moves a model from "needs an expensive multi-GPU server" to "runs on one affordable card." It is a standard, near-mandatory step in on-prem deployment, and the precision level is a deliberate dial between cost and quality that you set per use case.
Bars show approximate memory footprint. Quality loss is usually small at 8-bit, modest at 4-bit; test on your eval set.
A single modern GPU running a small-to-mid model serves tens to low-hundreds of tokens a second across concurrent users: enough for a department. The cost crossover where owning beats renting commonly lands around 2–3 million tokens a day at high (70%+) utilisation.
Throughput depends on model size, hardware, quantisation and how well the serving layer batches, but the shape is knowable. A small, quantised model on one capable GPU comfortably serves a department's interactive workload. Scaling to thousands of users, or to bigger models, means adding GPUs. The metric that decides the economics is utilisation: a GPU you have paid for but leave idle is wasted money, which is why batching and steady load matter so much.
On the crossover: because on-prem is a fixed cost and cloud is per-token, owning wins above a usage threshold. Analyses in 2026 put it around 2–3 million tokens a day at 70%-plus utilisation. Below that, cloud's pay-per-use is cheaper. Above it, on-prem can be several times to ten times cheaper per token. Model your real volume against this line before you commit capital.
Sources: on-prem vs cloud cost analyses (TrueFoundry, PremAI, newline), 2026.
Beyond the model and serving engine you need a gateway, caching, a vector store for retrieval, authentication, logging and monitoring. And you need a lifecycle process to test, version and roll out updates safely, because an unpatched, unmonitored model is a liability however well it ran on launch day.
A real on-prem deployment is a small platform, not a single program. Around the model and serving layer sit a gateway (API, auth, rate limiting), a cache (so you do not recompute repeated answers), a vector store for retrieval (Chapter 8), and logging and monitoring for quality, cost and security. These supporting
services are most of the operational work, and most of where production reliability actually comes from.
The lifecycle matters just as much. Models improve, security patches arrive, and your own fine-tunes evolve, so you need a disciplined process: test new versions on your eval set, stage them, roll out with the ability to roll back, and keep versions under control. Treating the model as a static install is how a sound launch quietly decays into an unmaintained risk.
A cloud architecture wraps the model API in the same supporting cast (gateway, retrieval, caching, logging, guardrails), but rents the model instead of hosting it. It comes in three modes: calling a model maker's API directly, using a hyperscaler's managed AI service, or self-hosting an open model on rented cloud GPUs.
Cloud and on-prem architectures look strikingly similar above the model line (same gateway, retrieval, caching, monitoring, guardrails) because those concerns exist wherever inference runs. The difference is the model layer. Direct API calls a maker's endpoint (simplest, fastest to the frontier). Managed service uses a hyperscaler's platform (Azure OpenAI, AWS Bedrock, Google Vertex) to reach many models inside the cloud governance you already have. Self-host on cloud GPUs runs an open model on rented hardware: on-prem's control with cloud's elasticity.
Most enterprises pick the managed service for the bulk of work, because it folds AI into security, billing and compliance they already run, then add direct APIs or cloud self-hosting where a specific need justifies it. The architecture is mode-agnostic by design, so the choice can vary by workload.
| Mode | You manage | Best when |
|---|---|---|
| Direct API | Just your application | Speed to the frontier model |
| Managed service | App, inside cloud governance | AI within existing security & billing |
| Self-host on cloud GPUs | App + model + serving | Control of an open model, cloud elasticity |
Four levers cut the bill hard: cache repeated answers, route each step to the cheapest model that can do it, batch and trim context so you pay for fewer tokens, and save the expensive frontier model for the few steps that truly need it. That last one is the "reasoning sandwich."
Because cloud is metered per token (Q30), spend is an engineering problem with known levers. Caching stores answers to repeated or similar queries, so you do not pay twice. Model routing sends routine steps to cheap small models and only the hard steps to expensive ones: often the single biggest saving. Context discipline trims what you send each call, since you pay for every token of context. Batching groups work for efficiency where latency allows.
The pattern that ties routing together is the reasoning sandwich: a strong, expensive model to plan and to verify, cheap models for the routine work in between. Stacked, these levers routinely cut cloud spend several-fold against a naive "send everything to the flagship" design. That is why two teams on the same model can see wildly different bills.
Set an explicit latency budget for the whole task, then engineer to it: stream responses, use smaller and faster models where speed matters, cache, and run across redundant capacity for availability. An accurate answer that arrives too late, or a service that is down, fails the user just as surely as a wrong one.
Production AI has to meet experience targets, not just accuracy ones. Latency is budgeted across the whole task: retrieval, model calls, tool calls, any verification. The levers are streaming the answer so it appears as it generates, choosing faster (often smaller) models on latency-critical paths, caching common results, and cutting round-trips. A multi-step agent in particular has
to be designed so its loop does not leave the user waiting too long.
Availability is the familiar discipline of redundancy and graceful degradation, with one AI-specific twist: a portable, multi-model design (Q57) lets you fail over to another model or provider when one is down or rate-limited. Set the targets first, then engineer the budget and the redundancy to meet them.
| Stage | Budget | Lever |
|---|---|---|
| Retrieval | Fast | Indexed vector store, caching |
| Model call(s) | The bulk | Right-sized model, streaming |
| Tool calls | Variable | Parallelise; time-out and degrade |
| Total to first token | Set a target | Stream so the user sees progress |
Put a routing layer in front of both environments. It classifies each request by sensitivity and difficulty, then sends regulated or high-volume work to your on-prem models and non-sensitive hard reasoning to the cloud frontier. Govern it with an explicit, auditable policy, not ad-hoc choices.
Hybrid (Q27) becomes real through a router. Each incoming request is classified (is the data sensitive or regulated? is the task routine-bulk or hard?) and dispatched accordingly: sensitive and high-volume to the controlled on-prem model, non-sensitive and occasionally-hard work to the cloud frontier. The policy that governs this should be explicit, version-controlled and auditable, because it is also a compliance control: it is what guarantees regulated data never leaves your walls.
This delivers the prize of Chapter 4 (sovereignty and cost control from on-prem, peak capability from cloud) in one architecture. The price is the router and two environments to maintain, plus the discipline of keeping the routing policy right as use cases change. It is the architecture mature enterprises converge on.
| Request type | Routed to | Why |
|---|---|---|
| Sensitive / regulated data | On-prem model | Never leaves the boundary |
| High-volume routine | On-prem model | Cost: above the crossover |
| Non-sensitive, hard reasoning | Cloud frontier | Peak capability, worth the price |
| Overflow / failover | Either, by policy | Availability & elasticity |
The routing policy is also a compliance control: make it explicit and auditable.
Closes Chapter 7. Chapter 8 turns to what the model works from: context, retrieval and the data underneath every good answer.
The cheapest performance lever in AI, engineered properly. What the model works from (the context you give it and the retrieval that fills it) decides answer quality more than the model itself.
Context engineering is the deliberate practice of getting exactly the right information in front of the model at the right moment: no more, no less. It is the cheapest lever because it improves answers without changing the model, buying hardware, or paying to train anything.
Since a model reasons only over what is in its context window (Q13), the quality of that window largely sets the quality of the answer. Context engineering is the craft of curating it well. The guiding idea, repeated across the industry in 2026, is to find the smallest set of high-signal information that produces the result you want. A model has a finite attention budget, and stuffing the window with everything makes answers worse, not better.
It comes down to four moves. Offload information to external systems, so the model does not carry it all. Retrieve what is relevant when it is needed, rather than front-loading everything. Isolate context, so one task does not bleed into another. And reduce history as it grows, keeping only what still matters. None of it needs a new model or new hardware, which is exactly why it is the first lever to pull.
| Move | What it means |
|---|---|
| Offload | Keep information in external systems, fetch on demand |
| Retrieve | Pull what's relevant at the moment of need |
| Isolate | Stop one task's context contaminating another |
| Reduce | Trim history to what still matters (compaction) |
Source: Anthropic, "Effective context engineering for AI agents," and industry practice, 2026.
Five stages: ingest your documents, split them into sensible chunks, turn each into a searchable numerical form, retrieve the most relevant chunks for a question, then re-rank to keep only the best. The quality of each stage, especially chunking and ranking, decides whether the answer is grounded or garbage.
A working RAG pipeline is a sequence, and any stage can make or break it. Ingest brings documents in. Chunk splits them into passages small enough to be precise but whole enough to make sense (Q73). Embed turns each chunk into a vector (a numerical fingerprint of its meaning) stored for search. At query time, retrieve finds the chunks whose meaning is closest to the question, and re-rank orders them so only the most relevant reach the model.
The common failure is treating retrieval as a black box. "The AI gave a bad answer" usually means "retrieval fed it the wrong chunks," which traces back to poor chunking, weak embeddings, or no re-ranking. Investing in these unglamorous stages, and measuring retrieval quality directly, is what separates a RAG system that grounds answers from one that hallucinates over irrelevant text.
An embedding turns text into a list of numbers that captures its meaning, so similar ideas sit close together. A vector database stores millions of these for fast similarity search. You need one whenever you do retrieval at scale, though small cases can start simpler.
An embedding is the bridge between language and search. It turns a passage into a vector (a long list of numbers) placed so that passages with similar meaning sit near each other, even when they share no words. That is what lets retrieval find the chunk that answers a question rather than just matching its keywords. A vector database stores those vectors and finds the nearest ones to a query vector quickly, across millions of chunks.
Do you need one? For retrieval at any real scale, yes: it is the engine of the pipeline. For a small prototype over a handful of documents, simpler approaches work and you can put the database off. The choice among vector stores comes down to scale, latency, whether it has to run on-prem for sovereignty, and how it fits your existing stack.
| Decision factor | What to weigh |
|---|---|
| Scale | Thousands vs millions vs billions of chunks |
| Latency | Speed needed for the user experience |
| Deployment | Managed cloud vs on-prem (sovereignty) |
| Integration | Fit with your existing data stack |
| Filtering | Can it enforce per-user access at query time? |
Split documents along their natural structure (sections, paragraphs, logical units) into chunks large enough to stand alone but small enough to be precise, keep metadata with each, and test on real questions. Clumsy chunking is the most common reason a retrieval system underperforms.
Chunking sounds trivial and turns out to be decisive. Chunks that are too large dilute relevance and waste context budget. Too small, and they lose the surrounding meaning needed to answer. The better approach splits along the document's natural structure (headings, sections, paragraphs) so each chunk is a coherent unit, and attaches metadata (source, date, section, access level) for later filtering and citation.
There is no universal setting. The right chunking depends on your documents and your questions, which is why it has to be tested against real queries and tuned. Teams that treat chunking as a one-line default and move on are usually the ones puzzled by poor answers later. It is humble work with an outsized effect on quality.
| Strategy | Good for | Watch out for |
|---|---|---|
| By structure (sections/paras) | Most documents | Very uneven section sizes |
| Fixed size with overlap | Unstructured text | Splitting mid-thought |
| Semantic (by meaning) | Quality-critical retrieval | More compute to prepare |
| + Metadata on every chunk | Filtering, citation, access | Always do this |
Engineer context first, because it is free and instant. Add retrieval when the model needs to know your facts. Fine-tune only when you need to change how the model behaves (its style, format, or a specialised skill), not what it knows. Fine-tuning for knowledge is the expensive wrong tool.
This is the decision that wastes the most money when you get it wrong (Q15, restated for builders). Work up the ladder. Context engineering (better instructions and examples) is free, instant, and often enough. Retrieval is the answer when the gap is knowledge: the model needs your facts, which change and have to stay current. Fine-tuning is justified when the gap is behaviour: a consistent house style, a specific output format, a narrow skill the base model handles badly.
The classic error is reaching for fine-tuning to teach facts. It bakes a snapshot into the weights that is costly to produce, out of date the moment it ships, and beaten by retrieval that just looks the current answer up. Ask first: is this a knowing problem or a behaving problem? Retrieve for the first. Fine-tune, sparingly, for the second.
Compaction is summarising a long conversation or task into a compact form and continuing from that summary, so the model stays coherent without drowning in its own history. It is the main technique for keeping long-running agents both effective and affordable.
A long task piles up history, and history is tokens, which cost money and, past a point, degrade performance as the signal drowns in accumulated noise. Compaction handles this. When context grows large, summarise what has happened into a tight form and restart from the summary, keeping what the task still needs and dropping the bulk. Long-context models, some now reaching millions of tokens, raise the ceiling but do not remove the principle: more context is not automatically better.
For agents that run many steps, compaction is essential to both coherence and cost. The engineering art is summarising without losing what matters later: keeping the decisions and facts the task will still need while shedding the transcript. It is a big part of why some long-running agents stay sharp and cheap while others wander and overspend.
Compaction trades a full transcript for a faithful summary: cheaper, and often sharper.
Treat the knowledge base as a living system. Re-index when source documents change, stamp content with dates, retire what is obsolete, and enforce the user's own access permissions at retrieval time. A retrieval system that serves stale or unauthorised content is worse than none.
Retrieval is only as good as what it retrieves. Freshness means the index reflects current reality: when a policy, a price or a document changes, the system has to re-index so it stops citing the old version. Date-stamping content and retiring the obsolete prevents the common failure of confidently serving last year's answer.
Access control is the other half, and it is non-negotiable. The system has to enforce, at query time, the same permissions the user has everywhere else, so retrieval never surfaces a document the asker is not allowed to see. This is where the metadata attached during chunking (Q73) earns its keep. A retrieval layer that ignores permissions is a data-leak engine with a helpful face.
The same data discipline that always mattered, now load-bearing. Know what data you have, clean it, classify it by sensitivity, control access, and track its lineage. AI amplifies both the value of good data and the damage of bad, and the data work is the four-fifths of the effort that decides success.
Every chapter has pointed back here, because data is where AI programmes are won or lost. The model is bought. The data is yours, and its quality flows straight into every answer. The groundwork is unglamorous
and decisive: inventory what you hold, clean it, classify it by sensitivity (which drives the deployment and access decisions of Chapters 4 and 10), control access, and track lineage so any answer can be traced to its source.
This is the biggest part of the 80% of effort that is not the model. It is also the most deferred, because it is tedious and its payoff is indirect. But a retrieval system over dirty, unclassified, ungoverned data produces confident, well-formatted, unreliable answers, and no model upgrade fixes that. Data readiness is the foundation everything else stands on.
When the answer depends on precise facts, relationships or calculations (figures, entitlements, org structures, connected entities) query a structured source or knowledge graph directly rather than retrieving text about it. For exact, relational questions, a database beats searching prose every time.
Retrieval over documents is ideal for questions answered by passages of text. It is the wrong tool when the answer lives in structure: an exact current figure, who reports to whom, which products a customer owns, how entities connect. For those, the right design has the AI query the structured source (a database, an API, a knowledge graph) and reason over the precise result, rather than hunting for prose that mentions it.
A knowledge graph (a map of entities and how they relate) is especially good at questions that hop across connections ("which suppliers are exposed to this risk through which contracts?"), which plain retrieval handles poorly. Mature systems use both: structured queries for facts and relationships, document retrieval for explanation and nuance. Match the source to the shape of the question.
Best systems use both, and route each question to the right source.
Closes Chapter 8. Chapter 9 wraps the model in a dependable harness: the loop, the controls, and the operations that make it production-grade.
Turning a capable model into a system you can trust in production: the harness, the loop, the newer reasoning models, multi-agent designs, and the evaluation and operations that keep it dependable.
A production harness has seven working parts: the model, tools, a planning loop, memory, a sandbox, an orchestration layer and a serving layer. Getting those right, not picking the model, is what separates a reliable agent from a clever demo.
Chapter 3 introduced the harness as the cockpit. Here is its anatomy. The model reasons. Tools let it act. The planning loop breaks goals into steps and checks progress. Memory carries state within and across tasks. The sandbox bounds what it can touch. The orchestration layer coordinates multiple steps, tools and sometimes agents. The serving layer runs the whole thing reliably under load.
The lesson the field absorbed through 2026 is that engineering value moved from the model to this surrounding system. Every leading model is excellent. The difference between a trustworthy agent and a liability is the quality of these seven parts, which is why harness engineering became a named discipline.
The most mature harnesses in production are the coding agents: tools such as Cursor and Claude Code that engineers now use daily. They are worth watching because they show the pattern the rest of the enterprise will follow: much of their behaviour is steered not by retraining the model but by plain-text instruction files kept in the repository (a CLAUDE.md and its equivalents) that hold standing rules and memory the agent reads on every run. Governing an agent, in practice, increasingly means governing these files.
| Component | Role |
|---|---|
| Model | Reasoning and language |
| Tools | The ability to act in your systems |
| Planning loop | Break goals into steps; check progress |
| Memory | State within and across tasks |
| Sandbox & permissions | Bounds on what it can touch |
| Orchestration | Coordinate steps, tools, agents |
| Serving layer | Run it reliably under load |
Four patterns dominate: the simple act-observe loop (ReAct), the plan-execute-verify loop for reliability, the state machine for controlled workflows, and the graph for complex branching. The choice trades flexibility against predictability, which is exactly the dial an enterprise wants to set deliberately.
"Loop engineering" is the design of how the agent iterates, and a few patterns keep coming up. ReAct (reason–act–observe) is the simple, flexible default. Plan-Execute-Verify adds a planning step up front and a verification step at the end, trading some speed for much higher reliability: the workhorse for serious tasks. A state machine pins the agent to defined states and transitions, maximising predictability for regulated workflows. A graph handles complex branching where you need many paths and parallel steps.
The pattern is a deliberate choice along a flexibility-versus-control axis. A creative research task wants the freedom of ReAct. A regulated financial process wants the predictability of a state machine. Picking the loop to match the task's risk and complexity is core to making agents both capable and safe.
| Pattern | Character | Use for |
|---|---|---|
| ReAct (reason–act–observe) | Simple, flexible | Open-ended, exploratory tasks |
| Plan-Execute-Verify | Reliable, self-checking | Serious multi-step work |
| State machine | Predictable, constrained | Regulated, defined workflows |
| Graph / flow | Branching, parallel | Complex, multi-path processes |
Reasoning models spend extra computation "thinking" through a problem step by step before answering,
which sharply improves hard maths, coding and multi-step logic. But they cost more and run slower, so use them for genuinely hard tasks and cheaper standard models for everything else.
A big shift of 2025 and 2026 was the rise of reasoning models, also called "thinking" models. Instead of answering immediately, they generate an internal chain of reasoning first, spending more computation at the moment of inference to work the problem through. This "test-time compute" markedly improves performance on hard, multi-step problems (advanced maths, complex code, intricate logic) where a snap answer falls down.
The trade is cost and latency: thinking burns more tokens and takes longer. So they are a tool for the hard minority of tasks, not a default for everything. The mature pattern routes difficult steps to a reasoning model and routine steps to a fast standard one: the same right-sizing logic as Q52, applied to the thinking-versus-fast dimension. Test whether a task actually benefits before paying for it to think.
Route hard steps to thinking models; keep routine work on fast ones. Don't pay to think when you don't need to.
Hard caps: a maximum number of steps, a token and cost budget per task, a wall-clock timeout, and loop-detection that halts repetition. Enforce them in the harness, not by asking the model nicely. An agent that can act must have limits it cannot exceed.
An autonomous loop without limits is a runaway risk in both cost and behaviour. The controls are non-negotiable and belong in the harness, where the agent cannot override them: a cap on steps per task, a token and cost budget that halts when it is hit, a timeout, and loop-detection that notices the agent repeating itself and stops it. These turn "it might cost anything" into "it cannot cost more than X."
Trusting the model to police itself is the mistake. It has no reliable sense of when it is stuck or overspending. The limits have to be external and absolute. Well-set caps double as a budgeting tool, because they make per-task cost a known maximum, which is what lets finance sign off on scaling.
Place human approval only at the points of real consequence (irreversible or high-value actions) and let everything reversible run on its own. A human gate on every step kills the productivity that justified the agent. No gate on the dangerous steps kills trust.
Human oversight is essential and easy to get wrong in both directions. Gate everything, and you have rebuilt the manual process with extra steps. Gate nothing, and a confident error runs unchecked. The discipline is to map actions by consequence and reversibility, and put the human only where it matters: irreversible actions (sending money, deleting data, external communications) get approval; reversible, low-stakes ones (drafting, analysing, proposing) run free.
The pattern is "propose, then approve, then act" for consequential steps, and full autonomy for the rest. That keeps the speed that justifies automation while keeping a human answerable wherever harm is possible, which satisfies both the productivity case and the accountability requirement of Q38.
| Action type | Human gate? | Example |
|---|---|---|
| Irreversible / high-value | Yes: approve before acting | Payments, deletions, external sends |
| Reversible / consequential | Sometimes: review after | Internal record updates |
| Reversible / low-stakes | No: run autonomously | Drafting, analysis, proposals |
A sandbox is a bounded environment with least-privilege access, so an agent can only touch the specific systems and data its task needs and nothing more. It is the same principle that contains any powerful account, applied to software that now acts on its own.
If an agent can act, the question is "act on what?" A sandbox answers it by confining the agent to a defined environment with least-privilege access: it can
reach only the systems, data and actions its task genuinely needs, scoped tightly, with everything else out of reach. A document-Q&A agent can read the knowledge base and nothing else. It cannot send email or touch finance, because those were never in its sandbox.
This is classic security thinking applied to a new kind of actor. Combined with the permission tiers of Q83 and the identity of Q89, the sandbox makes sure that even a compromised or malfunctioning agent (one hit by prompt injection, say) can do only limited, bounded harm. The blast radius is decided in advance, by design.
| Tier | The agent may... |
|---|---|
| Read-only | See specific, scoped data: nothing else |
| Act-reversible | Take actions that can be undone |
| Act-consequential | Only with a human gate (Q83) |
| Out of scope | Everything not granted: denied by default |
Multi-agent systems split work across specialised agents that coordinate, increasingly across organisational boundaries via emerging agent-to-agent protocols. They are worth it when a task genuinely breaks into distinct expert roles, but they multiply cost and failure modes, so reach for a single well-built agent first.
A multi-agent design uses several agents, each specialised (a researcher, a writer, a checker) coordinated by an orchestrator, much like a team. The 2026 frontier of this is agent-to-agent (A2A) interoperability: emerging protocols, alongside MCP for tools, that let agents from different teams, vendors or even organisations discover and delegate to one another, pointing toward networks of cooperating agents.
The honest counsel is restraint. More agents mean more token cost, more latency, and more ways things break, and the coordination overhead can swamp the benefit. They earn their complexity when a task truly breaks into distinct expert roles that a single agent handles poorly. For most enterprise tasks in 2026, one well-engineered agent with good tools beats a committee of mediocre ones. Start single. Go multi only when the problem demands it.
Default to one good agent. Add agents (and A2A) only when the task truly decomposes.
Test statistically with a reusable evaluation set, before launch and continuously after, run under an LLMOps discipline that versions prompts, monitors quality and cost, and controls model rollout. A probabilistic system needs continuous evaluation and AI-specific operations, not one-time testing.
Non-determinism (Q21) breaks normal testing, so evaluation is rethought. You build an eval set of representative tasks with criteria for a good answer (Q53), run it many times and score the distribution, and set a quality bar as the gate to launch. The same evals then run in production (a pyramid of broad automated evals, targeted human review, and live spot-checks) because quality drifts as data, models and usage change.
LLMOps is the operations layer that makes this routine: treating prompts and context as versioned, tested assets; running continuous evaluation; monitoring quality and cost alongside uptime; controlling model versions and rollout; and catching drift before users do. It exists because AI fails in ways standard operations never caught: silent quality decay, cost creep, non-reproducible bugs. Budget for it as a distinct capability.
| Evaluation (the pyramid) | LLMOps adds on top |
|---|---|
| Broad automated evals, run statistically | Prompt & context versioning |
| Targeted human review of nuance | Continuous evaluation in production |
| Live spot-checks of real traffic | Quality & cost monitoring beyond uptime |
| A quality bar as the launch gate | Model version control & drift detection |
Watch four things continuously (output quality, cost, latency and safety signals) against baselines, with alerts when they move, plus running evals and sampled human review. AI degrades quietly, and the first sign of trouble usually shows up in the metrics, not a crash.
Unlike conventional software, an AI system can fail without breaking: still up, still responding, just gradually worse. So monitoring watches quality (through continuous evals and sampled review), cost (token spend per task and in total), latency (the experience), and safety (refusals, flagged content, anomalous behaviour), each against a baseline, with alerts on meaningful movement. The dashboard is your early-warning system.
Drift (slow quality decay as the world, the data or the model shifts) is the specific danger, and you catch it by trend, not by a single reading: a quality metric sliding over weeks, a cost line creeping up, refusal rates changing. Catching it early is the difference between a tuning adjustment and a public failure. Monitoring is not optional infrastructure. It is how you keep a probabilistic system trustworthy over time.
| Watch | For | Drift signal |
|---|---|---|
| Quality | Answer accuracy & usefulness | Eval scores sliding over time |
| Cost | Token spend per task & total | Creeping upward |
| Latency | Response time | Slowing |
| Safety | Refusals, flags, anomalies | Rates changing |
Closes Chapter 9. Chapter 10 covers securing, governing responsibly, integrating and scaling all of this, and the canonical build to standardise on.
Defending it, governing it responsibly, integrating it, scaling it, and the canonical build to standardise on. The chapter that turns everything before it into a working implementation.
Layered defence: treat all external content as untrusted, isolate it from instructions, limit which tools an agent can call, require human approval for consequential actions, and monitor for anomalies. Prompt injection cannot be fully eliminated, only contained.
Prompt injection (Q37) has no single cure, so you defend it in depth. Treat any content the AI takes in from outside as untrusted, and keep it separated from your trusted instructions so the model is less likely to obey it. Constrain the agent's tools, so that even if it is hijacked, it cannot reach dangerous actions. Require human approval at consequential steps (Q83). And monitor for the anomalous behaviour that signals an attack in progress.
Data exfiltration (secrets escaping through an agent's actions) is contained by the same layers plus tight output controls and the least-privilege sandbox of Q84.
The mindset is the security mindset: assume the attack will sometimes get through, and engineer so that when it does, the blast radius is small. This is a first-class security workstream, owned with your security team, not an afterthought.
| Attack | Defence layer |
|---|---|
| Hidden instructions in content | Isolate untrusted input from instructions |
| Hijacked agent takes action | Constrain tools; least-privilege sandbox |
| Consequential misuse | Human approval gate |
| Data exfiltration | Output controls, egress monitoring |
| Novel / evolving attacks | Anomaly monitoring, rapid response |
Treat each agent as a first-class identity with its own credentials and least-privilege access, and log every consequential action (input, context, model and version, action, approval) in a tamper-evident store. You cannot hold accountable, or prove compliant, an autonomous actor you cannot individually identify and record.
When software acts on its own, "who did this?" has to have an answer. That means each agent needs its own identity in your access system, with least-privilege permissions scoped to its task (not a shared key, not broad admin rights), so its actions are attributable and its reach is exactly as wide as its job. This is the foundation under the sandbox (Q84), and an increasing regulatory expectation.
Accountability (Q38) and governance (Q39) then depend on being able to reconstruct what happened. A proper audit trail captures, per consequential interaction, the input, the context retrieved, the model and version, the action taken, and any human approval, stored tamper-evidently so you can answer "why did the AI do that?" after the fact. Regulators require it, courts demand it, your teams need it to debug. Design it in from the start. Retrofitting it is often impossible.
| Identity controls | Audit-trail fields |
|---|---|
| Unique identity per agent | Input & user/agent |
| Least-privilege, scoped access | Context retrieved |
| Attribution of every action | Model & version |
| Provision, rotate, revoke | Output & action taken |
| A named human owner | Approval, if gated |
Q36 named the three leak paths for a board. Here is the engineering. For prompts, control and filter what
gets sent to external models, with automated detection of sensitive patterns (identifiers, secrets) before they leave. For logs, redact sensitive content, restrict access, and limit retention; logs are a common, overlooked leak. For training, get contractual and technical guarantees that your data is isolated and never used to train a provider's model.
Close all three doors on purpose: filter and control what users and agents can send, secure and redact logs, and contract for no-training data isolation. Add automated detection of sensitive data in transit. Leakage is rarely malicious: it is almost always a door someone forgot to close.
The control that ties it together is data classification (Q77) flowing through every layer, so the system knows what is sensitive and handles it accordingly. Most leaks are not attacks. They are sensitive data taking a path nobody secured. Closing the doors is methodical work, and it is cheaper than the breach.
| Door | Control |
|---|---|
| Prompts | Filtering, sensitive-data detection, egress rules |
| Logs & traces | Redaction, access control, retention limits |
| Training | Contractual no-training, data isolation |
| All | Classification flowing through every layer |
Make it operational. Test for biased or unfair outcomes the way you test for accuracy, be transparent that AI is in use and how, keep a human accountable for every consequential decision, and document it. Responsible AI is a set of engineering and governance practices, not a values statement, and regulators are increasingly treating it that way.
"Responsible AI" earns the name only when it is built in. Fairness and bias: because models learn from data that carries historical bias, test outcomes across groups for disparate impact (especially in high-stakes uses like hiring or credit) and fix what you find, as a measured part of evaluation. Transparency: tell users when they are dealing with AI and, where it matters, how a decision was reached. Accountability: a named human owns every consequential AI decision (Q38, Q89). Documentation: record the model, the data, the testing and the limitations.
This is where ethics meets engineering and law. The EU AI Act and comparable regimes are turning several of these from good practice into obligations for high-risk uses (Q92). The organisations that operationalise Responsible AI (bias testing in the eval pipeline, transparency in the product, accountability in the org chart) both reduce real harm and stay ahead of the regulator. Treating it as a poster on the wall does neither.
| Principle | The practice |
|---|---|
| Fairness / anti-bias | Test outcomes across groups; mitigate disparate impact |
| Transparency | Disclose AI use; explain consequential decisions |
| Accountability | A named human owner per decision |
| Safety | Guardrails, content filters, red-teaming |
| Documentation | Model, data, testing, limitations on record |
Map each regulatory obligation to a specific control you can show (risk management, data governance, human oversight, transparency, logging, accuracy testing) so compliance is demonstrable from the system itself, not asserted in a document. High-risk obligations need evidence, not intentions.
Compliance gets tractable when you translate obligations into controls already in this book. For a high-risk use, the EU AI Act's demands map straight across: risk management → your governance process (Q39); data governance → data readiness and lineage (Q77); human oversight → the human-in-the-loop design (Q83); transparency → disclosure and explanation (Q91); record-keeping → the audit trail (Q89); accuracy and robustness → your evaluation regime (Q86).
Built this way, the system produces its own evidence of compliance (logs, eval results, documented oversight) rather than leaning on a policy that describes good intentions. And since other jurisdictions are converging on similar requirements, a control-mapped approach travels. Start by classifying each use case's tier (Q41), then implement the controls that tier needs, and keep the mapping current as the rules evolve.
| Obligation | Control in this book |
|---|---|
| Risk management | Governance operating model (Q39) |
| Data governance | Data readiness & lineage (Q77) |
| Human oversight | Human-in-the-loop design (Q83) |
| Transparency | Disclosure & explanation (Q91) |
| Record-keeping | Audit trail (Q89) |
| Accuracy & robustness | Evaluation regime (Q86) |
Defend on two fronts. Verify inbound content (be sceptical of audio, video and documents that could be AI-forged, especially in payment and identity flows) and label your own AI-generated output using emerging provenance standards. Convincing synthetic media is cheap now, and both fraud risk and disclosure duties follow.
The same technology that drafts your reports can fake a convincing voice, video or document, and that cuts two ways. Inbound, deepfakes have pushed fraud past email: synthetic audio and video have been used to impersonate executives and authorise payments, so high-stakes identity and approval flows need verification that does not rest on "it looked and sounded real." Outbound, you increasingly have to label AI-generated content, both for ethics and to meet transparency rules.
The tooling is maturing: provenance and watermarking standards, such as content-credential schemes, that cryptographically attest where a piece of media came from. Adopt them for content you publish, and build verification into the processes where a forged input would be costly. This is a fast-moving area, but the principle (verify what comes in, attest what goes out) will outlast the specific standards.
Yes, increasingly. AI inference at scale uses real power and water, which shows up as cost, as a constraint on
where you can build, and as an ESG-reporting obligation. So efficiency (right-sized models, quantisation, utilisation) is now both a cost lever and a sustainability lever at once.
The compute behind AI has a physical footprint that has become a board concern. Running models at scale draws significant power, and for data-centre cooling, water, which surfaces three ways: as cost (energy is a real line in on-prem economics), as a siting constraint (power availability now limits where AI data centres can be built, and demand is straining some grids), and as an ESG and reporting matter as sustainability disclosure tightens.
The reassuring part is that the levers line up. The same efficiency moves that cut cost also cut energy: right-sizing models (Q52), quantisation (Q63), high utilisation (Q64), routing routine work to small models. So good engineering discipline serves the sustainability goal directly. Where it matters for reporting or values, factor the energy profile of cloud regions and on-prem facilities into the deployment decision.
| Shows up as | Mitigated by |
|---|---|
| Cost (power in on-prem economics) | Right-sizing, quantisation, utilisation |
| Siting constraint (grid/power limits) | Efficient models; region/facility choice |
| ESG & disclosure obligation | Measuring & reporting energy per workload |
The efficiency levers that save money save energy. Cost discipline and sustainability point the same way.
Through a governed integration layer: standard connectors (increasingly via MCP), scoped permissions, and the same audit and approval controls as any agent action. That way AI reaches your systems of record without becoming an ungoverned back door into them.
AI delivers value by acting on your real systems (reading the CRM, posting to the ERP, querying the warehouse), but those are exactly the systems you most need to protect. The safe pattern is a governed integration layer: standardised connectors (MCP is becoming the common standard, Q18) rather than bespoke wiring, each with scoped, least-privilege access (Q89), and every consequential action subject to the audit trail and approval gates you have already built.
The anti-pattern is handing an agent broad, direct credentials into a core system "to make it work." Fast, and a serious risk. Treat every integration as a controlled interface with defined permissions and full logging, so AI reaches your systems of record as a governed participant, not an unmonitored super-user. This is where the platform team (Q97) earns its place.
Build a shared platform that gives every team the approved models, tools and guardrails out of the box, so business units move fast on common foundations instead of each rebuilding security and governance. The alternative (a hundred independent projects) is the sprawl that becomes unmanageable and unsafe.
Scaling AI is where many programmes lose control. Let every team build independently and you get sprawl: duplicated effort, inconsistent security, ungoverned data flows, and a cost no one can see whole. The answer is a shared platform (Q97) that provides the common foundations (sanctioned models, reusable tools and connectors, guardrails, monitoring, the audit trail) so teams build their specific use cases on top rather than reinventing the base each time.
This is the "paved road" principle: make the governed path the easy path, so doing it right is also doing it fast. Teams keep their autonomy over what they build. The platform keeps consistency in how it is secured and run. That balance (central foundations, distributed innovation) is what lets AI go from one success to fifty without fifty separate risks.
The platform team, or centre of excellence, is the connective tissue of a scaled programme. It owns the sanctioned model access and abstraction layer (Q57), the reusable tools and connectors, the guardrails and
security controls, the evaluation and monitoring infrastructure, the standards and patterns, and the governance framework (Q39). It provides all of that as a service, so business teams can move quickly and safely.
It is the small central team that owns the shared foundations (approved models and access, reusable tools and guardrails, standards, monitoring and the governance framework), but not the business use cases. Its job is to enable many teams, not to become the bottleneck that builds everything.
Just as important is what it does not own: the use cases. The failure mode is a central team that insists on building everything and turns into a bottleneck that throttles the whole organisation. Its mandate is enablement (paving the road and policing its edges) while the business units drive on it. Get that boundary right and the platform speeds everyone up. Get it wrong and it becomes the thing teams route around.
| Owns | Does NOT own |
|---|---|
| Approved models & access | The business use cases |
| Reusable tools, connectors, guardrails | Domain decisions |
| Eval & monitoring infrastructure | The right to build everything |
| Standards, patterns, governance | A bottleneck role |
Model cost as unit economics (cost per task times volume) set per-team budgets and hard caps, attribute spend so each team sees its own bill, and keep a model-abstraction layer so you can move providers. Together these stop the run-cost surprise and the single-vendor trap.
At scale, inference cost (Q59) has to be managed like any unit economic. Establish the cost per task, multiply by forecast volume, and you have a budget you can defend and monitor. Attribute spend to teams so each one sees and owns its consumption; visibility changes behaviour. Set hard caps (Q82) so no team or runaway agent can produce an unbounded bill. And apply the cost-control levers of Q67 (caching, routing, right-sizing) as standard platform features.
Lock-in is the strategic cost to avoid alongside the financial one. The model-abstraction layer (Q57) that makes models swappable also keeps your negotiating leverage and protects you against a provider's price rise or withdrawal. In a market where leadership and pricing shift quarterly, the modest cost of portability buys both resilience and bargaining power.
Four stages (foundation, first production use case, platform, and scaled portfolio) each a prerequisite for the next. The organisation builds capability and credibility in sequence, rather than reaching for enterprise-wide AI before it can run one use case reliably.
The whole implementation arc of this book reduces to a staged roadmap. Stage 1, Foundation: stand up the governance floor, sanctioned tools, data readiness and a small team, and pick one use case. Stage 2, First production: ship that one use case properly (real data, security, evals, monitoring) and prove value against a baseline. Stage 3, Platform: turn what you built into shared foundations (Q97) so others can reuse it. Stage 4, Scaled portfolio: many use cases on the common platform, governed and measured.
The discipline is sequence. Each stage is a prerequisite for the next: no platform before a proven use case, no scaling before a platform. The programmes that fail try to start at Stage 4 (enterprise-wide ambition with no foundation) and collapse. The ones that succeed earn each stage, turning one win into the capability for the next.
Earn each stage. The failures start at Stage 4; the successes get there.
A layered stack: applications on top, an orchestration and agent layer, a model layer (abstracted, hybrid cloud-and-on-prem), a data and retrieval layer, and a governance-and-security layer running through all of it. It is the shape almost every successful enterprise AI deployment converges on.
For all the variety of use cases, the production architecture settles into a recognisable stack. At the top, applications serve users. Beneath them, an orchestration / agent layer runs the harnesses, loops and tools (Chapter 9). Below that, a model layer, abstracted so models are swappable (Q57) and hybrid across cloud and on-prem (Q69). Under that, a data & retrieval layer: vector stores, structured sources, the knowledge base (Chapter 8). And cutting vertically through every layer, governance & security: identity, guardrails, audit, monitoring.
Standardising on this shape is what keeps scaling sane. New use cases slot into known layers, the platform team owns the lower ones, business teams build at the top. It is the architecture this book has assembled piece by piece, drawn as a single diagram. Adopt it as the template, adapt the specifics, and you have a foundation that holds from the first use case to the fiftieth.
| Layer | What lives here |
|---|---|
| Applications | The user-facing AI products |
| Orchestration / agents | Harnesses, loops, tools, multi-agent |
| Model layer (abstracted, hybrid) | Cloud frontier + on-prem open, swappable |
| Data & retrieval | Vector stores, structured sources, knowledge |
| Governance & security (vertical) | Identity, guardrails, audit, monitoring, through all layers |
The shape successful deployments converge on. Standardise here; innovate at the top.
The recurring mistakes are predictable: chasing the model instead of the system, skipping data work, fine-tuning what should be retrieved, ignoring run cost, deploying without evals or guardrails, and scaling before proving. The ten principles below are the antidote, distilled from the whole book.
The failures cluster, and naming them is the cheapest insurance. Teams obsess over the model and neglect the harness, data and operations that actually decide
success. They skip the data work and wonder why answers are poor. They fine-tune for knowledge that retrieval would supply. They ignore run cost until the bill cancels the project. They deploy without evals, guardrails or monitoring and meet AI's failure modes in production. And they scale before proving, multiplying an unvalidated bet.
Against each, a principle. The ten below are the book compressed: the things a technical team should keep on the wall, because every one of them is a failure this guide has shown how to avoid. They are, in the end, the answer to the question the whole document set out to address: how to put AI to work in the enterprise and have it actually work.
Closes Part Three and the main text. The appendices that follow score the leading cloud and on-premises models and define the terms used throughout.
●●●●● best-in-class · directional, not audited. Sources: Artificial Analysis, LM Arena, BenchLM, provider pricing, June 2026.
The leading cloud and on-premises models, scored by where each one wins, plus a plain-English glossary and the questions to ask any vendor. A snapshot of a fast-moving market: built to be updated.
The leading rented, closed-frontier models as of June 2026, scored on the axes that matter to an enterprise. Ratings are directional (●●●●● = best-in-class), drawn from public benchmarks and pricing; they will move: treat this as a template to refresh, not a fixed table.
| Model | Reasoning | Coding / agentic | Speed | Cost-efficiency | Multimodal |
|---|---|---|---|---|---|
| GPT-5-class (OpenAI) | ●●●●● | ●●●●○ | ●●●○○ | ●●●○○ | ●●●●● |
| Claude Opus 4.8 (Anthropic) | ●●●●● | ●●●●● | ●●●○○ | ●●○○○ | ●●●●○ |
| Claude Sonnet (Anthropic) | ●●●●○ | ●●●●○ | ●●●●○ | ●●●●○ | ●●●●○ |
| Gemini 3 Pro (Google) | ●●●●● | ●●●●○ | ●●●●○ | ●●●●○ | ●●●●● |
| Gemini Flash / Lite (Google) | ●●●○○ | ●●●○○ | ●●●●● | ●●●●● | ●●●●○ |
| Grok 4 (xAI) | ●●●●○ | ●●●●○ | ●●●●○ | ●●●○○ | ●●●○○ |
| DeepSeek (via API) | ●●●●○ | ●●●●○ | ●●●●○ | ●●●●● | ●●●○○ |
| Model | Best for | Watch |
|---|---|---|
| GPT-5-class | General frontier; maths & reasoning; broad ecosystem | Premium cost on top tier |
| Claude Opus 4.8 | Coding & agentic work; the practical frontier pick outside the US | Most expensive; slower |
| Claude Sonnet | The balanced workhorse: quality at sensible cost | Not the absolute top tier |
| Gemini 3 Pro | Multimodal & long context; Google-ecosystem firms | Ecosystem gravity |
| Gemini Flash / Lite | Cheap, fast, high-volume routine work | Not for the hardest tasks |
| Grok 4 | Real-time data & X ecosystem | Narrower enterprise track record |
| DeepSeek (API) | Cheapest capable frontier-class; also self-hostable | Provenance / governance review |
A 2026 caveat: geopolitics now shapes the menu
Model availability is no longer purely commercial. A stronger Anthropic variant (Claude Fable 5) was withdrawn from non-US customers on 13 June 2026 under a US export-control directive, leaving Claude Opus 4.8 as the practical frontier pick for many outside the US. Export controls, sovereignty rules and provider
geography can change your options overnight; factor it into model strategy.
The leading open-weight models you can run in your own walls, as of June 2026. By this point Chinese labs hold most top open positions, and the best open models trail the closed frontier by a step, not a chasm, while leading decisively on cost and control.
| Model | Capability | Coding | Efficiency / fit | Licence | Context |
|---|---|---|---|---|---|
| DeepSeek V4 (DeepSeek) | ●●●●● | ●●●●● | ●●●○○ | MIT ●●●●● | ●●●●○ |
| Qwen 3.5 (Alibaba) | ●●●●○ | ●●●●○ | ●●●●○ | Apache-2.0 ●●●●● | ●●●●○ |
| GLM-5 (Zhipu) | ●●●●○ | ●●●●● | ●●●○○ | MIT ●●●●● | ●●●●○ |
| Kimi K2.6 (Moonshot) | ●●●●○ | ●●●●○ | ●●●○○ | Open ●●●●○ | ●●●●○ |
| Llama 4 (Meta) | ●●●●○ | ●●●○○ | ●●●○○ | Community ●●●○○ | ●●●●● (to 10M) |
| Mistral Large / Small (Mistral) | ●●●●○ / ●●●○○ | ●●●●○ | ●●●●● (Small) | Apache-2.0 ●●●●● | ●●●●○ |
| Gemma 3 1–4B (Google) | ●●●○○ | ●●●○○ | ●●●●● (edge) | Community ●●●○○ | ●●●○○ |
●●●●● best-in-class among open weights · directional. Sources: BenchLM, Artificial Analysis open leaderboard, Hugging Face, June 2026.
| Model | Best for | Typical footprint |
|---|---|---|
| DeepSeek V4 | Top open reasoning & agentic coding; near-frontier in your walls | Multi-GPU server |
| Qwen 3.5 | Multilingual (200+ languages); permissive licence; size range | Single GPU → multi |
| GLM-5 | Coding with the cleanest MIT licence | Multi-GPU |
| Kimi K2.6 | Highest open overall intelligence index | Multi-GPU |
| Llama 4 | Ultra-long context (to ~10M tokens); rich tooling | Varies by size |
| Mistral Small | Best efficient coder; fits a single GPU; European/sovereign | Single GPU |
| Gemma 3 (1–4B) | Edge & on-device; smallest footprint; the easy on-prem starter | 1 accessible GPU / edge |
The shortlist logic
For a first on-prem deployment, a small efficient model (Gemma, Mistral Small, a small Qwen) on one GPU proves the architecture. For near-frontier capability in your walls, DeepSeek, GLM or Kimi on a multi-GPU server. For permissive licensing, prefer Apache-2.0 / MIT (Qwen, DeepSeek, GLM, Mistral). Always confirm the specific version's licence with legal, and benchmark on your own work (Q53).
Thirty terms in one honest sentence each, climbing from foundation to expert (enough to hold your own in any room) followed by the questions to ask any vendor and the one-page health check.
| Term | In one sentence |
|---|---|
| Artificial intelligence (AI) | Software that performs tasks once needing human judgement. |
| Machine learning | AI that learns patterns from data rather than being hand-coded. |
| Large language model (LLM) | A model that predicts text, trained on vast data until it reasons and writes. |
| Generative AI | AI that produces new content (text, code, images) from instructions. |
| Agent | AI that pursues a goal across steps, using tools to take real actions. |
| Token | A fragment of text (~¾ of a word); the unit AI reads, writes and is billed by. |
| Context window | The limited working memory the model can attend to for one task. |
| Prompt | The instruction and information you give the model. |
| Context engineering | Curating exactly the right information into the window: the cheapest quality lever. |
| Retrieval (RAG) | Looking up your relevant facts and handing them to the model before it answers. |
| Embedding | Text converted to numbers that capture meaning, so similar ideas sit close. |
| Vector database | A store that finds the most meaning-similar passages fast, for retrieval. |
| Training | The one-time, costly process of building a model from data. |
| Inference | Running the finished model to get an answer: the perpetual cost you pay per query. |
| Fine-tuning | Adjusting a model's weights on your examples to change how it behaves. |
| Hallucination | A confident, plausible answer that is simply false. |
| Harness | The system around the model (memory, tools, sandbox, limits) that makes it reliable. |
| Agentic loop | The plan–act–observe–verify cycle an agent repeats until done. |
| Reasoning model | A model that "thinks" step by step before answering, stronger on hard problems. |
| MCP | Model Context Protocol: the emerging standard plug connecting models to tools. |
| Tool | A connection that lets AI do something in your systems, not just talk. |
| Sandbox | A bounded environment limiting what an agent can touch or break. |
| Guardrail | A control that constrains AI behaviour: filters, caps, approval gates. |
| Prompt injection | An attack hiding instructions in content to hijack the AI: the new phishing. |
| Human-in-the-loop | A person approving AI actions at the points of real consequence. |
| Eval | A test set scoring AI quality, run before launch and continuously after. |
| LLMOps | The operations discipline for running AI: prompts, evals, monitoring, versions. |
| Quantisation | Storing a model at lower precision to cut memory and cost with small quality loss. |
| Open-weight vs closed | Models you can download and run yourself vs models you can only rent as a service. |
| Hybrid & sovereignty | Running across on-prem and cloud; keeping data under your own jurisdiction's law. |
| Run it yourself (on-prem) | Rent it (cloud) | |
|---|---|---|
| Models | DeepSeek, Qwen, Llama, Mistral, Gemma, GLM | GPT, Claude, Gemini, Grok |
| Control | Full: data never leaves | Contractual |
| Cost shape | Fixed; wins above ~2–3M tokens/day | Per-token; wins below it |
| Sovereignty | Strongest | Depends on provider & law |
| Burden | You operate it | Managed for you |
| Start with | A small model on one GPU (Gemma, Mistral Small) | A managed platform endpoint |
A living document
This guide is a snapshot of a market that moves monthly. Model names, prices and benchmark leaders will date; the reasoning, the architecture and the principles are built to last. It is designed to be maintained (questions added, scorecards refreshed, figures re-stamped) as enterprise AI evolves. Last full revision: June 2026.
The climb is sequential in skill, not in spend. A first project rarely needs all eight stages at production strength: it needs stages 1, 2, 4 and 7 working, and the rest deliberately deferred. Use the roadmap to plan
the team's capability; use the chapters to decide which capability is worth paying for first.
The board decides; someone still has to build. This is the engineer's climb: the eight capabilities an implementation team acquires, in the order they tend to acquire them. It is a companion to the decisions in the chapters, not a substitute for them; each stage points back to the questions that frame it.
Foundations · stages 1–2 Build · stages 3–6 Operate · stages 7–8
Read top to bottom, left to right. The foundation is set at stage 8; the harder work (deciding what to build) is the subject of the chapters. Source: Volge, adapted and redrawn from a community practitioner roadmap. Vendor and tool names omitted by design; capabilities are named, not products.
How to read it against the book
FAQ is Volge's reference series. Each volume takes one subject a board is being asked to decide on and answers it a question at a time, in plain terms and with the working shown: what the thing is, what it costs, what it breaks, who has to own it, and what would have to be true before it is worth doing. It is written to be consulted rather than read through, and to be as useful to the person asking the question as to the person who has to answer it.
Where a figure is not published, the edition says so rather than estimating it. Where published sources disagree, both are given and the disagreement is stated rather than averaged away. A number that does not survive checking is corrected here rather than repeated.
The edition carries no recommendation of any supplier and no offer of service. Where a vendor, platform or deployment is named it is because it is part of the published record of the subject.
This edition may be shared freely. It may not be altered or republished without permission, and figures carry their stated basis and date. Names and marks appearing in it are the property of their respective owners.