Why your LLM bill exploded overnight (and how to regain control)
TL;DR
LLM bills often spike overnight when retries, agent loops, and prompt bloat compound with no shared control point. Provider dashboards frequently miss these patterns until it's too late, and monthly budget alerts are far too slow. Route LLM traffic through one boundary, enforce retry budgets, token caps, and cost-aware routing there, and watch unit metrics (tokens per request, retry rate, cost per tenant) so you catch a spike in minutes, not on next month's invoice.
Organizations running LLMs in production face a harsh reality: bills can spike overnight. Unlike traditional cloud workloads, LLM costs hide behind architectural patterns, provider opacity, and cascading failures that compound fast.
Billing and usage dashboards can diverge in edge cases, especially with delayed aggregation, retries, or streaming responses.
There are anecdotal reports of discrepancies between dashboard usage and billing, though root causes are often unclear. For example, one OpenAI user was charged $6.68 while their dashboard showed $1.70 , with no clear explanation. Cases like this are rare, but they highlight why relying solely on provider dashboards is risky.
For platform teams: You cannot rely solely on provider dashboards for accurate cost attribution. Route LLM traffic through a shared boundary, such as a Router or federated graph. Log tokens and cost per request at that boundary. That is usually the most reliable place to measure usage and attribute cost.
The fastest route to a 100x bill is a retry loop.
Retry mechanisms turn minor hiccups into cost catastrophes. Google's SRE documentation warns that automatic retries can overload degraded backends and cause cascading failures. Most LLM retries are billable once they reach the model, so failures multiply cost fast. This results in what the SRE community calls "retry storms " turning into cost storms.
Agent architectures amplify costs through patterns commonly observed in production agent systems :
- Silent retry loops on tool failures that spike token usage
- Memory systems that pull entire conversation histories into every request
- Tool thrashing, where agents cycle between tools and generate redundant calls
- Context window creep: the same feature gets more expensive as conversations grow and memory expands. The tell is tokens per request rising week over week.
- RAG over-retrieval: too many chunks, chunks too large, or a citation for every answer
- Verbose output without caps: long answers streamed unchecked
- Auto-upgrades: silent routing to a newer, pricier model
In practice, cost spikes are often driven by edge-case failures rather than steady usage. Without per-call token budgets and retry limits, they escalate quickly.
Most spikes trace back to a small set of failure patterns. Each one has a signal you can watch for and a guardrail that stops it.
| Cost blow-up | What you’ll see | Guardrails to stop it | How WunderGraph helps |
|---|---|---|---|
| Retry storm | Retries per request spike, latency increases | Retry budgets and circuit breakers | Cosmo enforces retries, timeouts, and circuit breakers at a central boundary with request tracing (retries apply to query operations, not mutations) |
| Agent loop | Tool calls spike, long-running traces | Max step limits and tool allowlists | Cosmo provides visibility into tool-related operations; app layer enforces step limits and tool access |
| Prompt bloat | Tokens per request gradually increase | Prompt budgets and truncation | Cosmo provides a single visibility point; app layer enforces templates, truncation, and context limits |
| Over-retrieval (RAG) | Input tokens jump sharply | Top-k limits and chunk sizing | Cosmo centralizes retrieval entry points; app layer enforces retrieval limits and chunking rules |
| Verbose outputs | Output tokens dominate cost | Output caps and brief response modes | Cosmo ensures policy consistency; app layer enforces output limits and structured responses |
| Traffic surge | Spend spikes by endpoint or feature | Rate limits and per-tenant quotas | Cosmo provides rate limiting and centralized traffic control across tenants |
| Abuse | One tenant consumes disproportionate spend | Per-tenant caps and blocking | Cosmo enables throttling; app layer handles blocking, account-level controls, and escalation |
| Model drift | Increased usage of expensive models | Centralized routing policies | Route LLM calls as named GraphQL operations through Cosmo so model choice sits at one layer; app layer defines defaults and escalation rules |
| Embedding churn | High embedding volume, repeated indexing | Caching + deduplication + batching | Route embedding calls through Cosmo the same way, so they're visible at one boundary; app layer enforces caching and avoids duplicate embedding work |
Look closely and most of these come back to the same three roots: too many tokens, too many steps, or too little control over how the prompt gets assembled.
Inefficient prompts drain cost at every request. In controlled experiments, constraining chain-of-thought reasoning has reduced token usage by up to 67% with minimal accuracy loss, depending on the task.
In multi-tool agents, selective tool use and prompt trimming can significantly reduce token usage, in some cases by over 50% , depending on the setup. The fixes are straightforward: include only task-specific context, cap output length, limit retrieval aggressively, and treat prompt size as a budget.
You're not going to forecast this perfectly, and you don't need to. The goal is narrower: make a spike visible in minutes, and cap the damage when it happens.
Treat retries as a limited resource, not a default. Implement global retry budgets that cap total retries per process. Enforce per-request limits. Use exponential backoff with jitter to prevent synchronized retry storms.
Add circuit breakers. When error rates or latency exceed thresholds, the circuit opens and blocks failing dependencies. This stops degraded providers from becoming cost amplification engines.
If you model an LLM call as a GraphQL mutation, note that Cosmo's built-in retry only covers queries by default. Mutations aren't retried automatically, since they aren't guaranteed idempotent.
Token usage should be bounded. Enforce token limits, output caps, and prompt size controls at the Router so they apply everywhere. Don't rely on individual teams to implement discipline.
Log every request with input tokens, output tokens, model used, and cost. Aggregate hourly and daily. Track trends by feature, team, model, and tenant.
Reconcile internal cost calculations against provider invoices. Your Router is the most consistent place to handle attribution across teams and features.
Not every query needs a premium model. Route simple queries to cheaper models. Escalate only complex cases. A lightweight classifier trained on your patterns can make routing decisions in real time.
Research shows hierarchical routing can reduce costs significantly without degrading quality when tuned correctly. If services choose their own models, cost control becomes advisory. Central routing makes enforcement significantly easier.
Monthly budget alerts are too slow. Detection should use unit metrics and trigger within minutes.
Monitor:
- Cost per request
- Tokens per request
- Retry rate
- Cost per tenant
Unit metrics expose architectural drift before total spend spikes. Define clear ownership and response playbooks. Know when to trigger automated circuit breakers versus manual review.
If you can't answer these in a few minutes, you don't have control of your LLM spend. You have a bill and a hope.
When it spikes, don't guess. Check these first:
- Did tokens per request jump?
- Did retries spike?
- Did tool calls or agent steps increase?
- Did one tenant or feature spike?
- Did the model mix change?
The first hour is about containment, not root cause. In rough order:
- Switch non-critical traffic to cheaper models
- Tighten output token caps
- Disable agent tools or cut max steps
- Block the abusive tenant or endpoint
- Open circuit breakers for degraded providers
If your RAG bill is rising, run this check:
RAG is not always the cheapest or simplest option.
RAG is genuinely useful. It's also overused. Often a simpler pattern cuts both the cost and the operational risk.
Everything above is a control-plane problem, which is the kind of problem Cosmo is built for. Here's how the pieces map.
One place to see AI traffic. Cosmo provides request analytics and traces in Cosmo Studio. The Operations page shows which operations call LLMs, sortable by latency and error rate. Inspect operation content and see which clients use which features.
Enforce guardrails, not just warn. Cosmo Router has native traffic shaping for timeouts, retries, and circuit breakers. Redis-backed rate limiting enforces per-tenant quotas with key suffix expressions. Request-body and header limits prevent oversized requests.
Safer AI surface. The MCP Gateway exposes GraphQL operations as tools for AI models, drawn from a curated set of named operations rather than arbitrary query shapes.
Less wasted repeated work. Cosmo supports automatic persisted queries, persisted-operation caching, execution-plan caching, and deduplication of identical in-flight requests.
Custom cost logic when you need it. Custom modules intercept requests before routing, inspect query-plan stats before fetches, add custom logging, and inspect subgraph responses.
Put one boundary in front of AI requests. Represent LLM, RAG, and agent actions as named GraphQL operations behind the Cosmo Router. Stop letting services call providers directly.
Make each AI use case an explicit operation. Give each use case a clear name. Add descriptions directly in the GraphQL operation. The description becomes the tool description the model sees.
Attach business dimensions once. Send GraphQL-Client-Name and GraphQL-Client-Version (Cosmo's standard client-identification headers), plus your own custom headers like x-tenant-id, x-feature, and x-model. Map these into telemetry and access logs so every request carries the same dimensions.
Enforce guardrails centrally. Use traffic shaping for retry, circuit-breaker, and timeout logic. Use rate limiting with key_suffix_expression for per-tenant quotas. Use Cost Control (@cost/@listSize directives) and persisted-operation safelists to prevent unbounded query shapes.
Instead of letting services call OpenAI, Anthropic, or Google directly, make LLM providers subgraphs or data sources in your federated graph. Services call operations like generateProductDescription or summarizeCustomerFeedback rather than raw API calls. This creates a unified control plane for all LLM traffic.
Each LLM use case becomes an explicit operation in your graph schema with declared properties: token budget, acceptable latency, retry policy, routing rules, and owning team. This makes LLM usage visible in your API catalog and subject to the same governance as the rest of your platform.
3. Attach cross-cutting policies at the graph boundaryRetry budgets, token limits, rate limits, circuit breakers, and cost attribution become platform-level policies enforced by the graph layer. Instead of every service reimplementing retry logic, the graph handles it centrally. When a new cost control pattern works better, you deploy it once at the graph layer rather than updating dozens of services.
This shifts LLM cost management from a distributed problem to a centralized platform capability. Your platform team manages policies. Product teams consume LLM capabilities through well-defined operations. Cost discipline becomes automatic.
LLM cost management is often best handled as a control-plane problem.
If LLM calls bypass your shared Router, you don't control cost. You have distributed financial risk and hope nothing breaks on a bad day.
Retry budgets, token limits, routing rules, anomaly detection. These are most effective when enforced at a shared boundary. Guidance alone is often insufficient. Enforcement helps prevent cost spikes.
The goal is to shrink the financial blast radius. When something fails, detect it in minutes and contain it at the platform layer, not on next month's invoice.
If you see this as a router-level control problem, RAG cost control for AI agents goes deeper on where retrieval and agent spend drifts and how to measure it before you change anything. For the security side of putting AI behind a federated boundary, when prompt injection gets real covers how persisted operations and scoped access contain what a model is allowed to do.
Frequently Asked Questions (FAQ)
Many of the worst spikes come from edge-case failures, not steady usage. Retries multiply billable calls, agent loops cycle through tools, and prompts grow until tokens per request climb week over week. Without per-call budgets and retry limits, a $50/day pattern can become $5,000/day fast.
A retry storm is when automatic retries overload a degraded backend and trigger cascading failures. In most commercial LLM APIs, every model call is billable—including retries—so a small hiccup can multiply into a large, repeated charge. Retry budgets, exponential backoff with jitter, and circuit breakers stop it.
Billing and usage dashboards can diverge with delayed aggregation, retries, or streaming responses, and the root cause is often unclear. You should not rely on provider dashboards alone for cost attribution. Use them for totals, but route LLM traffic through a shared boundary that logs tokens and cost per request, then reconcile that data against the invoice.
Four layers: retry budgets and circuit breakers at the router, enforced token and output caps with prompt size limits, cost-aware model routing that sends simple queries to cheaper models, and alerts on unit metrics (cost per request, tokens per request, retry rate, cost per tenant) that fire in minutes.
RAG is not always the cheapest option. For a narrow domain with stable knowledge, a fine-tuned smaller model is often cheaper. For repetitive Q&A, caching answers frequently beats retrieving context on every request. Under strict governance, retrieval pipelines add complexity and require tight control.
Cosmo gives you one boundary in front of AI requests. The Router enforces timeouts, retries, and circuit breakers, with Redis-backed per-tenant rate limiting and request-body limits. Cosmo Studio shows request analytics and traces per operation, and the MCP Gateway exposes only a curated set of named operations as tools, the same safelist principle persisted operations use, to prevent unbounded queries.
Content Manager
Brendan Bondurant is the Content Manager at WunderGraph, owning technical content across GraphQL Federation, API tooling, and developer experience. He partners with leadership on product and company messaging and works cross functionally to align positioning, terminology, and content strategy across channels.
Data Architect: GTM & MI
Tanya brings cross-functional background in Data & MI, CMO, and BD director roles across SaaS/IaaS, data centers, and custom development in AMER, EMEA and APAC. Her work blends market intelligence, CRO and pragmatic LLM tooling teams actually adopts and analytics that move revenue.
