LLM Gateway Architecture

When three teams each integrate OpenAI with their own API key, retry logic, and logging format, you do not have three features - you have three reliability incidents waiting to happen. I built my first LLM gateway after the second team shipped a hardcoded API key in a frontend bundle.

The gateway is not middleware for its own sake. It is the single place where provider abstraction, tenant isolation, cost control, and observability converge. Every inference call in the organization should pass through it.

Core Responsibilities

A production LLM gateway does six things: authenticate the caller, validate and normalize the request, enforce rate and spend limits, route to the correct provider/model, execute the call with retries and circuit breaking, and emit structured telemetry for billing and debugging.

                    ┌─────────────────────────────────┐
  App Services ────▶│         LLM Gateway               │
                    │  ┌─────┐ ┌─────┐ ┌─────┐ ┌─────┐ │
                    │  │Auth │→│Limit│→│Route│→│Call │ │
                    │  └─────┘ └─────┘ └─────┘ └─────┘ │
                    └──────────┬──────────────────────┘
                               │
              ┌────────────────┼────────────────┐
              ▼                ▼                ▼
         OpenAI API      Anthropic API     Self-hosted
                                              (vLLM/TGI)
              
Sidecars: Redis (limits) · MySQL (config) · OTel (traces)

Request Normalization and Provider Abstraction

Applications send a provider-agnostic request: model, messages, temperature, max_tokens, and optional tools. The gateway translates to provider-specific formats. This means swapping GPT-4 for Claude or a fine-tuned Llama model is a routing config change, not an application redeploy.

// Unified request schema (gateway input)
{
  "model": "gpt-4o",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "Summarize this document." }
  ],
  "max_tokens": 1024,
  "temperature": 0.3,
  "stream": true,
  "metadata": {
    "tenant_id": "org_abc",
    "feature": "doc-summary",
    "trace_id": "abc-123"
  }
}

The metadata block is critical. It drives rate limiting, cost attribution, and trace correlation without polluting the model prompt.

Routing and Fallback Strategy

Routing rules live in MySQL and are cached in Redis with a 60-second TTL. Rules match on tenant, feature tag, model name, or cost tier. I support primary/fallback chains: if GPT-4o returns 503 twice, route to GPT-4o-mini; if that fails, return a structured 503 with retry-after header.

For self-hosted models, health checks run every 30 seconds against a /health endpoint on the inference server. Unhealthy backends are removed from the pool automatically.

Rate Limiting, Cost Accounting, and Caching

Rate limits are enforced in Redis before any provider call. I track requests-per-minute and tokens-per-minute per tenant, with burst allowance for enterprise tiers. Hard spend caps reject requests synchronously; soft caps trigger Slack alerts.

Response caching is underrated. For deterministic prompts (temperature=0, same system prompt), I cache completions in Redis keyed by hash(model + messages) with a configurable TTL. Cache hits skip the provider entirely - zero cost, sub-10ms latency.

Token counting happens pre-call (estimated input) and post-call (actual from provider response). Discrepancies above 20% trigger a warning - usually a tokenizer mismatch between gateway estimate and provider billing.

Streaming and Connection Management

Streaming responses (SSE) are the default for chat interfaces. The gateway opens a streaming connection to the provider, forwards chunks to the client, and aggregates the full response for logging and token counting on stream end.

ALB idle timeout must exceed your longest expected stream. I set ALB idle timeout to 120 seconds and send SSE comment heartbeats (`: heartbeat\n\n`) every 15 seconds to keep connections alive during slow token generation.

If the client disconnects mid-stream, the gateway aborts the provider call to avoid paying for tokens nobody reads.

Engineering Takeaways

• One gateway, one API key per provider, one place for retries and observability.
• Use a provider-agnostic request schema so model swaps are config changes, not code changes.
• Enforce rate and spend limits in Redis before the provider call.
• Cache deterministic completions (temperature=0) to cut cost and latency.
• Abort provider streams when the client disconnects - do not pay for unread tokens.
• Attach tenant_id, feature, and trace_id in metadata for every request.

Trade-offs I Would Consider

Option: Centralized gateway service (Node.js/Go)

• Pros: Full control over routing, caching, streaming, and custom business logic.
• Cons: You operate another service; becomes a single point of failure without HA.

Option: Managed gateway (LiteLLM, Portkey, cloud provider proxy)

• Pros: Faster to ship, built-in provider abstraction, community-maintained adapters.
• Cons: Less customization for tenant billing, harder to integrate internal auth patterns.

What I Would Do in Production

Deploy the gateway behind an internal ALB first. Only expose externally after auth, rate limiting, and WAF rules are proven.

Run at least two gateway instances across AZs. Redis and MySQL should be Multi-AZ. Provider outages should not take down your gateway - only the affected route.

Emit OpenTelemetry spans for every request phase: auth, limit check, route resolve, provider call, response normalize. Dashboard p99 per phase to find bottlenecks.

Rotate provider API keys quarterly via Secrets Manager. Gateway reads keys at startup and on SIGHUP - no redeploy needed.

Maintain a provider status page monitor and auto-disable routes when a provider declares an incident.

FAQ

Q: Should the gateway be synchronous or support async jobs?

Both. Synchronous for chat and quick completions under 30 seconds. Async (202 + queue) for batch inference, document processing, and agent runs. Route by expected latency at the gateway.

Q: How do I handle multiple model providers with different APIs?

Adapter pattern: one module per provider translating unified schema to provider format. Test adapters independently with recorded fixtures.

Q: What about prompt injection at the gateway level?

The gateway can scan for known injection patterns and block or flag requests, but it is not a substitute for tool sandboxing. Combine input scanning with output validation downstream.

Q: How do I bill tenants accurately?

Log input_tokens, output_tokens, model, and provider cost per request to MySQL. Aggregate nightly into billing records. Reconcile weekly against provider invoices.

Q: Node.js or Go for the gateway?

Node.js if your team already runs it and you need fast iteration with streaming SSE. Go if you need higher concurrency per pod with lower memory. I have run both; pick based on team strength, not benchmark blogs.