Your AI Agents Could Be Burning Money: How to See Every LLM Call

Your provider dashboard says token usage jumped 240 percent overnight. The invoice is climbing, but the dashboard cannot tell you whether the cause was a new release, a looping research agent, an oversized conversation history, or a retry policy that quietly sent the same request five times.

This is the visibility gap facing teams that move from a single chatbot to autonomous agents, tool-calling workflows, reasoning models, and multi-model applications. Aggregate provider reports are useful for finance, but developers often need request-level answers: Which agent made the call? Which model did it select? How many input and output tokens were used? How long did the request take? Why did it fail? Was it retried?

An LLM proxy can provide that missing engineering layer. It sits between applications and model providers, creating one place to apply routing, authentication, logging, budgets, and security policies. The result is practical LLM observability without rewriting every agent whenever a provider, model, or policy changes.

Why Provider Dashboards Are Not Enough for AI Agent Debugging

OpenAI API usage tracking and Anthropic API cost tracking can show organization-level consumption, billing, and sometimes project or key breakdowns. Those views rarely contain all the application context needed to debug a distributed agent system. A provider sees an API request; your team sees a planner agent, a summarizer, a customer tenant, a deployment version, and a business task.

Direct integrations also scatter evidence across services. One agent may call OpenAI, another may use Anthropic, and a third may fail over to a different model. Engineers must reconcile incompatible logs, pricing structures, request identifiers, and token fields. Centralized LLM API management normalizes enough of that data to answer questions such as:

• Which agent, user, tenant, or workflow generated the request?
• Which model and provider handled it, including fallback attempts?
• How many tokens were reported, and what was the estimated cost?
• Did latency come from the application, proxy, or upstream provider?
• Are repeated calls evidence of a useful retry or an expensive loop?

How an LLM Proxy Works

An LLM proxy is an application-facing endpoint that forwards requests to one or more model providers. Agents authenticate to the proxy instead of holding a long-lived provider credential. The proxy validates the caller, applies budget and routing rules, translates or forwards the request, and records available response metadata.

Agent or application
        |
        | OpenAI-compatible request + virtual key
        v
Self-hosted LLM proxy / AI gateway
  - Authentication and access policy
  - Model aliases and provider routing
  - Rate limits, budgets, retries, fallbacks
  - Request status, latency, tokens, cost metadata
        |
        +------> OpenAI
        +------> Anthropic
        +------> Other approved providers

This OpenAI Anthropic API proxy pattern gives applications a stable contract. An alias such as support-fast can point to one model today and another after an evaluation, outage, or pricing change. That is multi-model AI routing without updating credentials and SDK configuration in every repository.

The proxy should capture request time, caller identity, model alias, resolved provider, status, latency, reported token counts, retry attempts, and an estimated cost. Attach application metadata such as agent name, tenant ID, environment, trace ID, and release version, but avoid recording prompt content by default.

Hands-On LiteLLM Proxy Setup

LiteLLM is a widely used open-source LLM gateway that exposes an OpenAI-compatible interface across multiple providers. The following LiteLLM proxy tutorial follows the project’s documented proxy pattern. Check the official LiteLLM Proxy quick start before production deployment because configuration options can change between releases.

1. Install the Proxy

python -m venv .venv
source .venv/bin/activate
pip install 'litellm[proxy]'

Pin and test a specific package version in production rather than automatically installing the newest release. Place provider secrets in a secret manager or injected environment variables, not in the configuration file.

2. Configure a Provider and Model Alias

Export the required secrets, including a strong proxy master key and a PostgreSQL connection when using persistent virtual keys and spend tracking. Then create config.yaml:

model_list:
  - model_name: support-fast
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY

  - model_name: analysis-agent
    litellm_params:
      model: anthropic/claude-sonnet-4-5
      api_key: os.environ/ANTHROPIC_API_KEY

general_settings:
  master_key: os.environ/LITELLM_MASTER_KEY
  database_url: os.environ/DATABASE_URL

Model identifiers evolve, so use identifiers currently available to your provider account. Aliases keep those provider-specific names out of application code.

3. Start the Self-Hosted AI Gateway

litellm --config config.yaml --port 4000

For local testing, inspect standard output for requests and errors. A production service should export structured logs and metrics to the team’s existing observability platform.

4. Generate a Virtual API Key

Use the administrative master key only for administration. Generate constrained virtual API keys for AI agents, services, or teams:

curl -X POST 'http://localhost:4000/key/generate' 
  -H 'Authorization: Bearer sk-proxy-master-key' 
  -H 'Content-Type: application/json' 
  -d '{"models":["support-fast"],"max_budget":25,"budget_duration":"30d"}'

The response includes a virtual key. Store it as a secret and never print it in CI output. See the official virtual key documentation for the fields supported by your installed version.

5. Send and Inspect a Test Request

curl 'http://localhost:4000/v1/chat/completions' 
  -H 'Authorization: Bearer YOUR_VIRTUAL_KEY' 
  -H 'Content-Type: application/json' 
  -d '{
    "model":"support-fast",
    "messages":[{"role":"user","content":"Summarize this incident in three bullets."}]
  }'

Inspect the proxy’s structured logs and, for a configured database-backed deployment, its administrative usage views. Confirm that you can identify the virtual key or team, model alias, resolved model, response status, latency, and available token or spend data. Integrations with tracing and metrics systems can connect an LLM request to the surrounding agent run.

How a Proxy Finds Expensive Agent Loops

Effective LLM request logging makes patterns visible. Group calls by agent run or trace ID, then flag unusually high request counts, repeated prompt hashes, rising context size, identical tool failures, or rapid fallback chains. A planner that calls a search tool 30 times may look like ordinary traffic in a provider dashboard; in an agent trace, it is an obvious termination bug.

This also improves AI agent token usage analysis. Teams can compare cost per completed task rather than cost per API call, detect agents that repeatedly resend full histories, and identify workloads using premium models when a smaller model would meet quality targets. Budget limits and rate limits provide guardrails, but alerts should fire before a hard limit disrupts users.

Security Controls an AI Gateway Should Enforce

An LLM proxy reduces the need to distribute provider credentials, but it becomes a sensitive security boundary. Apply these controls before routing production traffic:

• Secret management: Load provider keys from a managed vault, rotate them, and prevent exposure in configuration files, logs, or container images.
• Access control: Issue scoped virtual keys per service, team, environment, or tenant. Separate administrative and inference permissions.
• Log redaction: Remove authorization headers, personal data, tool credentials, and sensitive fields. Store prompts only when there is a defined need, lawful basis, retention period, and access policy.
• Tenant isolation: Prevent one customer from viewing another customer’s usage, metadata, cached responses, or request bodies.
• Rate and budget limits: Limit calls, tokens, concurrency, and spend by key or team. Treat limits as protection against bugs and compromised credentials.
• Network protection: Use TLS, restricted ingress, authenticated administration, dependency scanning, audit logs, and tightly controlled outbound destinations.

LLM Proxy vs Direct API Calls and Managed Gateways

Direct provider integrations have the fewest moving parts. They work well for one application, one provider, and modest governance requirements. However, routing logic, credentials, telemetry, and retry behavior become duplicated as the system grows.

A self-hosted LLM proxy offers the most control over routing, data handling, deployment region, retention, and custom integrations. Software may be open source, but infrastructure, databases, upgrades, on-call coverage, and security engineering are not free.

Managed AI gateways reduce operational work and may offer polished analytics, policy controls, and global scaling. The trade-offs can include platform fees, less deployment control, vendor dependency, and sending request metadata—or prompt content—to another processor. Review data use, retention, regional hosting, export options, and failure behavior before adoption.

The right LLM proxy vs direct API calls decision depends on scale and risk. A proxy becomes compelling when several agents share providers, costs must be attributed, credentials need central control, or provider changes are frequent.

The Trade-Off: Your Proxy Is Now Critical Infrastructure

A proxy adds a network hop and another service to deploy, patch, secure, monitor, and scale. A poorly designed gateway can become a single point of failure, increase latency, leak sensitive prompts, or block every AI feature during an outage.

Decision Checklist

• Do multiple agents, teams, tenants, or providers need one policy layer?
• Can current logs attribute cost and failures to a specific workflow?
• Would virtual keys materially reduce provider credential exposure?
• Does the team have capacity to operate a highly available service?
• Are data retention and privacy requirements clearly defined?
• Is the expected control worth the infrastructure and on-call cost?

Production Deployment Plan

• Implement liveness and readiness checks that do not depend on a billable model call.
• Set explicit connection, request, and streaming timeouts; test long-running responses.
• Retry only appropriate transient failures with exponential backoff, jitter, and strict limits. Avoid stacked application and proxy retries that multiply spend.
• Run redundant instances across failure zones behind a load balancer, with a resilient database and tested recovery process.
• Alert on error rate, p95 latency, saturation, database health, provider failures, retry volume, token spikes, and spend velocity.
• Load-test routing, streaming, key validation, and log pipelines before moving all agents to the gateway.

A Note About LLM API Cost Monitoring

Proxy costs are engineering estimates, not the final invoice. Tokenization differences, cached input discounts, reasoning or hidden-token accounting, batch pricing, tool-related charges, rounding, and provider-specific billing rules can create discrepancies. Reconcile proxy records with provider invoices and update pricing data regularly. Use proxy estimates for rapid detection and attribution; use provider billing as the financial source of truth.

Frequently Asked Questions

Does an LLM proxy store every prompt?

It should not have to. A privacy-conscious deployment can log metadata while excluding request and response bodies. If prompt storage is necessary for debugging, use redaction, encryption, short retention, strict access controls, and tenant-aware policies.

Will a proxy reduce model costs automatically?

Not automatically. It creates the evidence and controls needed to reduce costs through model routing, loop detection, budgets, caching policies, and smaller contexts. Poor retry or fallback rules can increase costs instead.

Can LiteLLM route both OpenAI and Anthropic requests?

Yes. LiteLLM supports multiple providers behind model aliases and an OpenAI-compatible proxy interface. Test feature compatibility because provider-specific capabilities, streaming behavior, tool schemas, and error responses are not always identical.

Should every startup self-host an LLM proxy?

No. A small application with one provider may be better served by direct integration and disciplined telemetry. Self-hosting becomes more valuable as agent count, provider diversity, governance requirements, and the cost of unexplained behavior increase.

Make Every Agent Call Accountable

An AI gateway for developers is not merely a billing dashboard. It is a control point for understanding what autonomous software is doing with expensive external APIs. When every call carries an identity, trace, route, latency, token count, and policy decision, runaway behavior becomes easier to find—and much harder to hide in the monthly bill.