Your AI Bill Is the New Roaming Charge: A Bank’s Guide to Cutting LLM Costs

You know the feeling. You signed off on a modest generative-AI pilot last year. It worked. Word got around. A few teams started using it, then a few more, then someone wired an “agent” into a back-office process. Nobody re-approved anything. Then the invoice landed — and it read less like a software subscription and more like the phone bill you get after two weeks abroad with data roaming left on.

If that scene feels familiar, you’re in good company. This is now one of the most common conversations we have with banks, and it cuts across the whole sector — from global names to community banks, credit unions and building societies with no ML engineers on staff.

Let’s be precise about the topic. This isn’t about using AI to cut your bank’s costs. It’s about cutting the cost of the AI itself: the token spend, the model API bills, the copilot seat licenses, the agent workloads that quietly burn compute. That’s a different discipline, and it has a name now — FinOps for AI.

IWhy falling prices don’t mean a falling bill

Here’s the paradox. The price of AI has collapsed. In March 2023, OpenAI’s GPT-4 launched at roughly $30 per million input tokens and $60 per million output. By 2026, models that beat that original GPT-4 on every benchmark cost a fraction of it — a “good enough” model can be 100–300x cheaper than the flagship of three years ago.

A quick translation for the non-engineers: a token is a chunk of text, about ¾ of a word. You pay per token going in — your prompt — and per token coming out — the answer. Input is cheap; output is several times dearer.

So prices are down. Why is your bill up? Three reasons, and they compound.

First, usage explodes to fill the lower price — the classic pattern where cheaper units drive far more consumption. J.P. Morgan’s private bank noted in July 2026 that “AI prices now resemble utility bills more than traditional software subscriptions,” with token prices themselves up over 60% since December 2025 as demand for data-center compute spiked (index data via Bloomberg; task figures via Wolfe Research).

Second, agents. A simple chatbot answers in one model call. An “agent” plans, retrieves documents, calls tools, checks its own work and retries — re-sending its full accumulated context at every step. Gartner’s March 2026 analysis found agentic workflows consume 5 to 30 times more tokens per task than a standard chatbot. J.P. Morgan’s own illustration put a code-analysis task at roughly 56,000 tokens versus under 1,000 for an email draft — a 50-plus-fold spread.

Third, reasoning models. The newer “thinking” models generate a long hidden chain of reasoning before they answer — and you pay for those invisible thinking tokens, often at 5–8x standard output rates. Left uncapped inside an agent loop, a single complex task can run into real money before the user sees a word.

Add seat licenses that people forget to use, and you have a bill that grows while the price list shrinks.

IIThe five places bank AI budgets leak

1. Seat licenses nobody opens (“AI shelfware”). Microsoft’s CEO Satya Nadella told the company’s Q2 FY2026 earnings call (January 28, 2026) that it had reached 16.1 million paid Microsoft 365 Copilot seats as of December 31, 2025. But independent surveys suggest only 20–30% of licensed seats are used weekly. You’re paying for the gym membership; the question is who’s actually going.

2. Shadow AI. Staff expensing AI tools on corporate cards outside procurement. One 2026 audit estimated shadow AI at 4–9% of enterprise software spend — two to three times the formal AI budget line. It’s also a data-governance problem: IBM’s 2025 Cost of a Data Breach Report found high-shadow-AI organizations incurred an extra $670,000 in breach costs on average, and that shadow AI featured in 20% of breaches.

3. Frontier models doing junior work. Research suggests 60–80% of enterprise LLM cost sits in high-volume, low-complexity tasks — classification, extraction, routine chat — where a frontier model is wildly over-specified.

4. Runaway agents and reasoning. No token budgets, no step caps, “thinking” left on for tasks that don’t need it. A misconfigured retry loop can burn a month’s credits overnight.

5. Pilots that never pay off. MIT’s Project NANDA reported in 2025 that despite $30–40 billion in enterprise spend, about 95% of integrated generative-AI pilots showed no measurable P&L impact. Gartner separately predicted (July 2024) that 30% of GenAI projects would be abandoned after proof-of-concept by the end of 2025. The compute still gets billed while everyone decides whether the project lives.

And the boardroom is noticing. A 2026 governance survey found 62% of organizations said an unexpected AI cost materially altered a business decision in the past year — with board-level escalations, emergency spending freezes and cancelled programs following.

IIIThe playbook: what actually moves the number

Here’s how we’d approach it. Split the work into quick wins you can land this quarter and structural fixes that take a little longer but change the shape of your spend.

Quick wins this quarter

Model routing (right-size the model to the task). This is the single biggest lever. Send each request to the cheapest model that can handle it well; reserve frontier models for the hard cases. The peer-reviewed RouteLLM work (Ong et al., ICLR 2025) achieved 85% cost savings on MT-Bench while keeping 95% of GPT-4 quality, needing the expensive model on only about 14% of queries. In practice, teams report 40–85% bill reductions. Think of it as triage: not every patient needs the consultant surgeon.

DBS in Singapore has been unusually candid about this. It built what it calls a “multi-plug” architecture that, per The Edge Singapore (April 2025), lets the bank “monitor model performance in real time and switch to more cost-effective alternatives as they become available” — explicitly to “manage and optimise AI costs.” JPMorgan’s LLM Suite takes a similar model-agnostic stance; its chief analytics officer has said the bank doesn’t “want to architect ourselves around one particular model,” giving teams flexibility to pick the best one for the job.

Prompt caching. If the same long system prompt or policy document is sent on every call, you can cache it so the model doesn’t re-read (and re-charge) it each time. Anthropic and OpenAI both now offer up to a 90% discount on cached input tokens. For a bank with a big fixed compliance preamble on every prompt, this is close to free money.

Batch / async APIs. Work that isn’t time-sensitive — overnight document processing, back-book analysis — can run on batch APIs at roughly 50% off. Stack batch with caching and the effective discount on repeated input can exceed 95%.

Cap outputs and cap agents. Set maximum output lengths, token budgets, and hard limits on how many steps an agent can take before it stops. Turn “thinking” off for tasks that don’t need it. These are one-line settings that prevent four-figure surprises.

Structural fixes

RAG done economically. Retrieval-augmented generation — letting the model look things up in your documents instead of memorizing them — is the right pattern for banks. But stuffing entire documents into the context window on every call is expensive. Retrieve the relevant passages, not the whole filing cabinet.

Fine-tune or distill a small model for high-volume narrow jobs. If one task runs millions of times a month — say, categorizing transactions — a small model trained for that job can match a frontier model at a tiny fraction of the cost.

Know when self-hosting pays — and when it doesn’t. Running an open-weight model on your own GPUs sounds cheaper. For most institutions it isn’t, until volume is very high. Honest 2026 analyses put the break-even for pure self-hosting closer to $50,000–80,000 a month of premium API spend once you load in idle GPU time, over-provisioning and a senior infra hire. For a smaller bank with no ML team, APIs (or your core provider’s AI modules) almost always win on cost — self-hosting’s real justification is data control, not price.

Governance: attribution, budgets, alerts, kill switches. You cannot manage what you cannot see. Only 11% of organizations can forecast AI spend within 10%, per a 2026 survey; 15% of those running agents can’t attribute agent costs at all. Tag every use case, show each business unit its own spend (showback/chargeback), set budgets and alerts, and keep a kill switch. This is the heart of FinOps for AI — a discipline the FinOps Foundation formalized in 2025 with a dedicated framework and certification.

Evaluate, so you don’t overpay for quality nobody needs. Test cheaper models on your real prompts. If a model at a fraction of the price passes your quality bar, you’re done. J.P. Morgan’s analysts made the same point: as “good enough” models proliferate, premium models get reserved for the slice of work that truly needs them.

Negotiate and consolidate. Committed-use discounts, enterprise pricing, and killing duplicate seat licenses all help — but negotiate from a base of measured usage, not vendor optimism.

IV“But we’re a bank”

Every lever above has to work inside guardrails other industries don’t have. This is where cost optimization gets bank-specific — and where doing it wrong costs far more than tokens.

Model risk management still applies. In the US, the Fed/OCC framework SR 11-7 — updated to SR 26-2 in April 2026, which notably leaves generative and agentic AI outside its formal scope pending a separate rulemaking — and the UK PRA’s SS1/23 mean any model informing a decision needs validation, monitoring and an audit trail. Those logs are non-negotiable, and they add cost. Build them in; don’t optimize them away.

In the EU, the AI Act’s high-risk obligations for uses like credit scoring were set to bite from 2 August 2026 (with parts subsequently pushed toward late 2027 under the Digital Omnibus). DORA has been fully in force since January 2025 and puts hard requirements on third-party ICT dependencies — which is exactly what a frontier-model API is. Add data residency and sovereignty rules, and you see the catch: the cheapest model, in the cheapest region, is often simply not available to you. Vendor concentration risk cuts the same way — routing 100% of your AI through one provider is a resilience problem before it’s a pricing one.

The takeaway for smaller institutions: your cheapest, safest path usually runs through your existing core provider’s governed AI features and a small number of vetted models — not a sprawl of tools bought on cards. Constraints actually narrow your choices in a helpful way.

VSigns your LLM spend is out of control

• You can’t say, within 10%, what you’ll spend on AI next month.
• No one can tell you the cost per resolved query, per document, or per use case.
• You’re paying frontier prices for high-volume, routine tasks.
• You bought copilot seats by the hundred and can’t show weekly active usage.
• Agents run with no token budget, step limit, or kill switch.
• AI tools are appearing on expense reports that procurement never saw.
• Your “temporary” pilot is now a production cost line nobody formally owns.

If three or more of those ring true, the good news is that the fixes are mostly fast and mostly boring — which is exactly what you want in a bank.

VIISources

1. J.P. Morgan Private Bank, “AI use is exploding. So are the bills.” (July 2026)
2. “The LLM Pricing Collapse of 2026”; AI Price Index
3. Gartner, “30% of GenAI Projects Will Be Abandoned After PoC by End of 2025” (July 29, 2024)
4. MIT Project NANDA, “The GenAI Divide: State of AI in Business 2025” (via Fortune)
5. Microsoft Copilot seat count, Q2 FY2026 (Jan 28, 2026) — AlphaStreet; adoption gap — Value Add VC
6. Shadow AI Spend Report 2026; IBM 2025 Cost of a Data Breach (shadow AI $670k) via Cloud Security Alliance
7. 2026 State of AI Cost Governance Report (Mavvrik/Benchmarkit) via Intelligence360
8. RouteLLM (ICLR 2025) and routing savings — TrueFoundry; Digital Applied
9. Prompt caching (90%) and batch (50%) — OpenAI prompt caching explained; Anthropic API pricing breakdown
10. Reasoning / “thinking” token costs — TokenFence
11. Gartner agentic token multiplier (5–30x, March 2026) — Cockroach Labs; Bigeye
12. Self-hosting vs API TCO — Particula; LeanLM
13. FinOps Foundation, FinOps for AI (2025) — FinOps X 2025 keynote; 2025 FinOps Framework
14. DBS “multi-plug” architecture — The Edge Singapore (April 2025)
15. JPMorgan LLM Suite — American Banker; CNBC (Sept 30, 2025)
16. NatWest / Cora — NatWest Group; Klarna AI assistant — Customer Experience Dive
17. Model risk: SR 11-7 / SR 26-2 — enz.ai; PRA SS1/23 — VerifyWise
18. EU AI Act and DORA for banks — Auriga; regulation-dora.eu; Digital Omnibus timeline — sprinklingact.com
19. Community-bank / credit-union GenAI adoption (Cornerstone 2026) — Bank Tech Intel