Your model bill may be rising, but cost is only the visible symptom. Unnecessary LLM calls also add latency, failure modes, privacy exposure, and behavior that can change when a prompt or model version changes.
The hard part is not finding the most expensive endpoint. It is separating calls that perform valuable semantic work from calls that compensate for missing product logic. You need an audit that can answer, at each call site: What decision is the model making, why is a model needed, and what evidence shows that the call improves the user outcome?
Find the decision behind each model call
Do not audit only by endpoint, provider, or model. Those are useful accounting views, but they are too coarse for a product decision. One user request can trigger a primary generation, retries, a moderation pass, an evaluator, a fallback model, and a background summary. A shared gateway can also serve unrelated jobs with very different quality and risk requirements.
The useful unit of analysis is a decision instance: a defined product task, its inputs, the output the model produces, and the downstream action that consumes that output. I treat every model call as a product decision with an operating cost, not merely an implementation detail.
Consider a feature described as “use AI to tag support tickets.” That label may hide several separate jobs: detect the language, retrieve account data, enforce routing policy, map the request to a known taxonomy, and write a short explanation. Those jobs should not inherit the same architecture simply because they happen in the same workflow. Retrieval belongs in a query. Stable policy belongs in code. Only the ambiguous classification or explanation may need a model.
For every call site, complete this sentence:
When this trigger occurs, the model uses these inputs to choose or create this output, which this consumer uses to take this action.
If you cannot complete the sentence without words such as “enhance,” “improve,” or “make intelligent,” the call does not yet have a testable contract. Clarify the job before debating models or prompts.
Then assign the call to one of four dispositions:
- Keep: The task requires interpretation, synthesis, or generation under genuine ambiguity, and the call produces a measurable benefit.
- Constrain: The model performs useful semantic work, but it receives excessive context, produces unnecessary prose, or has too much freedom over the result.
- Replace: A rule, query, parser, template, conventional classifier, or other bounded mechanism can satisfy the same contract more reliably.
- Remove: The output is unused, duplicates an earlier result, restates information already available, or has no observable effect on the workflow.
A cheap call is not automatically necessary, and an expensive call is not automatically wasteful. The deciding question is counterfactual: if you remove the model, can the product preserve the intended outcome with a simpler mechanism?
Build an inventory that connects calls to outcomes
Provider dashboards can show aggregate tokens, latency, and spend. They usually cannot tell you whether a call prevented a support escalation, produced a correct routing decision, or generated text nobody read. Your inventory has to join technical traces with product events.
Start with a static pass through the software. Include direct SDK usage, shared gateway clients, agent and tool wrappers, background workers, retry handlers, fallback chains, evaluator models, prompts stored in configuration, and AI features embedded through vendors. A repository-level scan can nominate call sites for review, including calls that may not be obvious from a cost dashboard. Treat the scan as discovery, not as the verdict; code alone does not reveal whether a call improves a product outcome.
Then verify the map with runtime traces. Dead code, conditional paths, retries, and indirect calls can make the static picture misleading. Give every production call site a stable identifier so that a model request can be traced back to its product job even when several jobs share the same client.
| Record for each call | Question it should answer |
|---|---|
| Call-site ID and accountable owner | Where can the behavior be changed, and who can approve the product tradeoff? |
| Product job and trigger | What user or system event caused the call? |
| Input fields and data origins | Which inputs are essential, redundant, sensitive, or available from a system of record? |
| Model, prompt, tool, and configuration versions | Which implementation produced the observed result? |
| Input tokens, output tokens, latency, and direct cost | What does one execution consume? |
| Retries, fallbacks, cache status, and termination reason | Is the visible call hiding additional work or an uncontrolled loop? |
| Output schema and downstream consumer | Does the next step need prose, a label, a field, a score, or nothing at all? |
| Outcome, override, escalation, and error signals | Did the call help complete the task, and what happened when it was wrong? |
Instrument the workflow around the call, not just the model request. If the model returns a classification, record whether the subsequent route was accepted, overridden, or escalated. If it drafts a response, record whether a person sends it, rewrites it, discards it, or has to repair a factual error. A technically successful response is not necessarily a successful product outcome.
Do not solve observability by copying every raw prompt and response into an unrestricted log. That can duplicate customer data, credentials, or regulated information into another system. Prefer references, approved redacted fields, and narrowly scoped samples where they can answer the audit question. Apply access controls and retention rules appropriate to the underlying data.
The completed inventory should produce three separate queues:
- Eliminate: Remove calls whose outputs have no necessary semantic contribution.
- Reduce: Deduplicate equivalent requests, reuse safe and sufficiently fresh results, or prevent repeated calls inside a workflow.
- Right-size: Keep the capability while shortening context, narrowing the output, or selecting a model proportionate to the task.
Do not combine these into one generic “AI cost optimization” backlog. Eliminating a decision, reducing its frequency, and changing its model have different evidence requirements and different failure modes.
Apply a replacement ladder before tuning the prompt
Prompt tuning is often the first response to a weak or expensive call. It should come later. First ask for the least complex mechanism that can satisfy the product contract. Move up the ladder only when the lower rung fails a representative evaluation.
| Mechanism | Use it when | Evidence required |
|---|---|---|
| No operation | The output is unused, duplicated, decorative, or produced for a path that no longer exists. | Trace the downstream workflow and confirm that removal does not change an intended outcome. |
| Configuration or template | The output is approved copy, a stable policy value, or a fixed response assembled from known fields. | Verify coverage for supported states, localization, and product variants. |
| Deterministic code | The task is calculation, schema validation, permission enforcement, exact mapping, or another explicit invariant. | Unit tests, boundary cases, and clear handling for invalid inputs. |
| Database query or retrieval | The answer already exists in an authoritative system and does not need interpretation to be identified. | Freshness, access-control, completeness, and no-result behavior. |
| Bounded classifier or scoring model | The task is fuzzy but has a stable label set, representative examples, and a defined abstention path. | Validated performance by important segment, especially where errors have different consequences. |
| LLM with constrained output | Semantic interpretation is needed, but the downstream system requires a small schema or controlled action set. | Schema validation, tool authorization, task-level evaluation, and explicit fallback behavior. |
| Open-ended LLM generation | The user actually benefits from synthesis, explanation, or creation that cannot be reduced to a bounded output. | Human or calibrated evaluation of usefulness, correctness, and risk at the point of use. |
This ladder prevents several recurring design mistakes:
- Generating facts that can be retrieved: Query the system of record first. Use a model only if the user needs an explanation or synthesis of the retrieved material.
- Generating prose for a machine: If the next component needs a status code, record ID, or boolean, request or compute that object. Do not generate paragraphs and then ask another model to interpret them.
- Using a model as a validator: Enforce permissions, required fields, limits, and business invariants in code. A model may explain a validation failure, but it should not be the sole authority for the rule.
- Summarizing before knowing the consumer: A dashboard may need extracted fields and links to evidence rather than a narrative summary. Build what the decision-maker actually uses.
- Letting agents rediscover completed work: Persist tool results, define explicit completion conditions, detect equivalent requests, and stop the loop when no new state has been created.
Caching deserves particular care. A cache can reduce calls without proving that the underlying call is necessary. It can also return stale or cross-user information if identity, permissions, and freshness are not part of the key. Use caching only when semantic equivalence is defined and the data policy permits reuse.
Do not replace every probabilistic decision with a growing pile of fragile conditions. A model may remain the simplest responsible choice when inputs are genuinely varied, acceptable answers depend on context, and maintaining explicit rules would reproduce the same ambiguity less transparently. In that case, keep the model but reduce its authority: narrow the task, expose uncertainty, validate the output, and provide an abstention or review path.
Prove the replacement before changing production
Evaluate the product contract, not similarity to the old output
A replacement should not be judged by how closely it imitates the current model. The current behavior may be the problem. Judge both implementations against the product contract you defined for the decision instance.
Build a replay set from permitted, representative workflow inputs. Include ordinary cases, boundary conditions, missing data, malformed inputs, important customer segments, and cases in which the correct behavior is to abstain or escalate. Remove or protect sensitive data before moving examples into an evaluation environment.
Use the same sequence for each candidate:
- Write the acceptance contract before reviewing candidate results. Define the required output, allowed variation, hard constraints, and fallback behavior.
- Run the current implementation and the proposed replacement on the same inputs.
- Score task success, severity of errors, coverage, abstentions, latency, direct cost, retries, and human review effort.
- Inspect results by meaningful segment. An overall average can conceal a replacement that works on routine traffic but fails on a high-consequence path.
- Investigate disagreements instead of automatically treating the current model as correct.
- Approve the change only when the replacement satisfies the predefined contract and the remaining failure mode has an acceptable recovery path.
There is no universal quality threshold for removal. The threshold belongs to the job. A cosmetic wording choice and a permission decision cannot share the same acceptance rule. For billing, authorization, safety controls, or irreversible actions, keep an authoritative deterministic check even if a model helps interpret the request.
Measure economics at the accepted-outcome level. A useful internal equation is:
Cost per accepted outcome = total operating cost divided by successful, accepted outcomes.
Total operating cost can include inference, retries, fallback calls, orchestration, monitoring, human review, and the burden of recoverable failures. This prevents a cheaper component from looking efficient when it increases corrections or reduces successful completion. Keep assumptions visible when some terms cannot yet be measured.
An LLM grader can help process open-ended evaluations, but it should not be the sole judge of its own architectural replacement. Calibrate it against human-reviewed examples, use deterministic checks wherever possible, and inspect disagreements. Otherwise, the audit can merely transfer an unverified model decision from production into the evaluation layer.
Ship the change as a reversible product decision
Run the replacement in shadow mode when you need production-shaped inputs but cannot safely let the candidate act. Compare its proposed result with the live path without creating side effects. When the risk is low enough for live exposure, place the replacement behind a feature flag and preserve a known fallback.
Choose rollback conditions before rollout. Monitor the user outcome, error severity, abstention or escalation rate, latency, retries, and total operating cost. Keep the observation window long enough to cover the workflow’s normal variation rather than declaring success after only the easiest traffic.
Prioritize candidates using avoidable operating cost, call volume, confidence in the replacement, reversibility, and failure consequence. I would start with a high-confidence, reversible change that teaches the organization how to run the audit. The single largest model bill may be attached to a task whose replacement is much harder to prove.
Record the final decision with the call-site ID, owner, product contract, evaluation set, accepted tradeoffs, implementation versions, and rollback path. That record matters when a future model launch, pricing change, incident, or product redesign reopens the decision.
Key takeaways
- Audit the product decision behind a call, not just the endpoint, provider, or token count.
- Join runtime traces to downstream actions so technical success can be separated from user success.
- Try removal, deterministic code, retrieval, and bounded classification before tuning an LLM prompt.
- Keep security, permissions, billing rules, and other hard invariants under authoritative deterministic control.
- Compare alternatives on representative inputs using task success, error consequence, latency, cost, and review effort.
- Roll out replacements behind reversible controls, with fallback and rollback conditions defined in advance.
Pick one workflow in which nobody can describe the model’s decision in a single sentence. Trace it from trigger to downstream action, name the simplest plausible replacement, and run both against the same acceptance contract. If the replacement preserves the outcome, ship it under a flag. If it does not, you will still have a clearer reason for keeping the call and a better contract for improving it.
References
