The LLM switchboard: Routing 20 billion+ tokens a year through Octus’ data science models

The Tuesday-morning problem

A new frontier model dropped this morning. The benchmarks look strong, the price per token is lower, and we already want to experiment with it and, if it holds up, roll it out. Now count what stands in the way. Across our production data science pipelines, an LLM can be a routine step in the workflow, whether that is extracting fields from filings, classifying documents, summarizing disclosures or running inference over unstructured credit text. Between them, these pipelines make roughly 10 million+ LLM calls and move about 20 billion+ tokens a year across three major vendors: Azure OpenAI, AWS Bedrock and Google Vertex AI.

If every pipeline talks to its vendor’s software development kit (SDK) directly, “try the new model” means a separate migration for each one, each with its own retry logic, its own token accounting, and its own idea of what a failure looks like. At that volume, the interesting engineering problem stops being “which model is best?” It becomes: How do you make the model a swappable component?

Our answer is a single internal library, UnifiedLLMClient, that every pipeline in our platform imports instead of a vendor SDK. This post breaks down what it does and why it mattered far more than we expected.

The problem: Three SDKs, three of everything

Before the unified client, each project talked to its provider directly. That sounds manageable until you notice that the three vendors disagree about nearly every detail that matters in production:

Every single row above was, at one point, duplicated inside individual project directories. The real cost wasn’t just code duplication. It was that each project had its own private answer to retry policies, cost math and determining whether a call actually succeeded. Just computing what we spend on inference per project involves extra code, overhead and setup, different for every project, to pull the numbers together.

The design: One contract, adapters underneath

The entire architecture rests on one core decision: every call, across every provider and code path, returns the exact same three-element response:

• Response text: The generated output, cleanly unwrapped from vendor-specific content-block structures or a parsed, schema-validated dictionary when an output schema is supplied.
• Status: Explicitly “SUCCESS” or “FAILED.” Never an unhandled exception. A batch pipeline processing 40,000 documents should not crash because item #39,001 tripped a content filter.
• Stats payload: A standardized data dictionary containing input tokens, output tokens, reasoning tokens, cache-read tokens, itemized costs and overall latency.

Sync calls, async execution, document uploads, cache warm-ups and cross-provider fallbacks all honor this structure. It’s a small guarantee, but it’s the reason a project can swap its underlying foundation model by editing a single line in a configuration file.

The contract extends to structured outputs. Passing an output_schema argument, a Pydantic model, returns a parsed dictionary guaranteed to conform to that schema instead of a raw string. Each adapter engages its provider’s native structured-output mechanism, so the guarantee holds across different LLMs without per-project glue code: No regexes over model text, no json.loads wrapped in try/except, no silent drift when a model update changes its formatting habits.

Just as important, three operational responsibilities live entirely inside the client, so no project team ever re-implements them: Retries and backoff (transient-versus-permanent error classification, exponential backoff with jitter, configurable attempt counts), token accounting (four token types parsed out of three unrelated vendor payloads into one schema), and automated cost evaluation (real-time pricing from a central registry, including long-context tier breaks and separate cache-read rates).

A pipeline author focuses purely on prompt logic. They never write a backoff loop or multiply token counts by prices looked up in cloud documentation. Underneath, thin provider adapters handle vendor-specific mechanics (payload structures, retry taxonomies, capacity strategies, usage parsing). The client core manages model selection, fallback routing, cache normalization and telemetry.

1. Models are data, not code

You won’t find conditional statements, such as, if model == “gpt-5.4,” anywhere in a project codebase. Models exist strictly as structured data in a central registry file. Each entry defines the internal model ID, full vendor model identifiers, environment variable names for account resource ARNs, display names and allowed regional deployment lists.

A companion pricing file stores real-time rates for input, output, cache writes, cache reads and tier-break token thresholds. When a vendor releases a new model on Tuesday, we update a JSON object, and all our LLM based production models can leverage it within minutes, without code reviews, business logic changes or per-project patches.

2. When a vendor has a bad day

Retries handle a flaky connection; they don’t protect against a provider having an outage. The client accepts primary and secondary model configurations across entirely different cloud providers (such as pairing Anthropic’s Claude on AWS Bedrock as primary with OpenAI’s GPT models on Azure as secondary) alongside logging metadata.

The primary adapter exhausts retries with exponential backoff and jitter before crossing over to the secondary vendor. A throttling surge on AWS Bedrock degrades latency slightly; it never halts the pipeline.

The same mechanism runs in reverse deliberately: Passing a flag on individual calls routes requests directly to the secondary tier. A pipeline can send bulk classification to a light mini model and reserve frontier models for high-judgment steps without instantiating or configuring a second client.

3. Scaling past vendor bottlenecks

Rate limits are the operating tax at scale, and providers relieve pressure differently. On Azure OpenAI, the adapter balances across a dynamic pool of regional deployments, with projects able to restrict regional scope via registry settings. On Google Vertex AI, the client rebuilds itself against an alternate region on transient failures, tracking burned locations to avoid loops.

Then there’s the gotcha: Gemini prompt caches are region-scoped, so hopping regions invalidates the paid server-side cache object you just created. To prevent orphaned caches, the registry supports a location-switch disable flag, pinning the region so the client backs off in place and keeps the cache intact. Cheap flag, real money saved.

4. Normalized prompt caching

Caching offers massive savings; cache reads run ~90% cheaper than fresh input tokens. The client abstracts vendor differences into one core concept: a static prefix set once.

Behind this single configuration argument, three distinct provider paths execute:

• Claude writes ephemeral cache checkpoints automatically on the first request.
• Gemini provisions an explicit server-side cache object and threads the returned cache ID onto subsequent requests.
• GPT caches implicitly above 1,024 tokens; if a prompt prefix is too short, the client proactively logs a warning indicating that prompt caching will not engage rather than letting developers assume savings.

Prompt resolution logic keeps the cached prefix byte-stable, folding call-specific variations strictly into the dynamic prompt tail. A single varying character at the front silently destroys cache hits. Solving this once centrally protects every team.

5. Every call leaves a receipt

No pipeline manually counts tokens or computes costs. Every response automatically yields an identical telemetry payload detailing input tokens, output tokens, reasoning tokens, cache-read input tokens, itemized costs and execution latency.

Enabling usage logging streams these records to a central metrics table tagged with project ID, project name, project version and a transaction UUID. We can answer financial and operational questions directly in SQL:

• What is the exact unit cost to generate a tear sheet summary?
• Did v2.3 of the deal terms extraction pipeline become cheaper or just faster?
• Which project accounts for last month’s AWS Bedrock spike?
• What percentage of input volume is served via prompt cache hits?

Cost shifted from a monthly cloud invoice surprise into a metric with real-time dashboards. Model selection evolved from qualitative debates into empirical price-performance benchmarks.

6. Concurrency without the footguns

Batch processing demands concurrency, but SDK async mechanisms vary. Enabling asynchronous mode allows the client to route execution properly: Awaiting native async surfaces on Gemini while executing blocking SDK calls on worker thread pools, backed by lock-guarded stats aggregation so a hundred concurrent calls still yield precise cost totals.

What it actually bought us

• Model migrations became config changes: Updating or swapping models requires changing the target model ID in a config file and re-running benchmarks; pipeline code remains completely untouched.
• Compounding system reliability: Retries, backoff, region failover and multi-vendor fallback were hardened once and tested across millions of production calls. A fix discovered in one model protects all models and pipelines.
• Complete spend visibility: Automatic telemetry gave us per-project, per-version and per-call attribution across every team without requiring manual instrumentation.
• Faster time-to-production: Integrating LLMs into new projects requires ~12 lines of setup instead of days of SDK archaeology.
• Honest model selection: Uniform cost and performance telemetry allows us to evaluate model trade-offs empirically in an afternoon.

What we learned

Standardize the return contract first. Everything else like fallback, telemetry, caching, async composes on top of one consistent response shape. And never raise exceptions in batch workflows: An explicit failed-status tuple beats crashing on document 39,001 of 40,000.

Keep models and pricing as data, not code. The JSON registry is the highest-leverage file in the entire library. It is the difference between adopting a new model in a day and adopting it in a quarter.

Expose vendor differences honestly, and meter from day one. A leaky abstraction that hides region-scoped caches or missing async support will bite someone downstream. Emit clear warnings instead. And centralized telemetry is far easier to establish at the start than to retrofit across live pipelines.

At roughly 1,500 lines of Python, UnifiedLLMClient, our LLM switchboard, sits between Octus applications and the major language model providers. It is the least glamorous utility in our platform, and the exact reason our high-value engineering and modeling stays focused on the work itself.

The models will keep changing, and so will the prices. What we expect to keep is the part that doesn’t: one return contract, a registry that treats models as data, and a meter that was running from day one. The next model that matters will be a one-line change for us, and the work that matters will already be underway.

The switchboard is part of our AI Framework, the principles and practices behind how we build and run AI at Octus. It puts that framework into practice with failover across vendors and a receipt for every call. The registry and routing logic are ours. The lessons are yours to use.

More AI & Data Science Insights

Blog Post

Oct 28, 2026

The LLM switchboard: Routing 20 billion+ tokens a year through Octus’ data science models

Blog Post

Sep 9, 2026

Building a model to read deal emails

Blog Post

Aug 21, 2026

Inside the document extraction pipeline behind CreditAI

Blog Post

Aug 13, 2026

Finding the comparable: How Octus built covenant similarity search