Designing Reliable AI API Infrastructure

The first time we put an LLM endpoint behind the same ALB rules as our REST API, p99 latency jumped from 400ms to 18 seconds and our on-call got paged for "database connection exhaustion." The database was fine. The Node.js event loop was blocked waiting on upstream inference.

AI APIs are not just slower CRUD. They have provider rate limits, per-token billing, context window constraints, and failure modes that look like success (200 OK with an empty or truncated response). Reliability engineering for inference starts with accepting that the model is an unreliable dependency you must isolate, not a function call.

Treat the Model Provider as an External Dependency

I wrap every provider call behind an internal gateway service - not because I love extra hops, but because isolation is non-negotiable. The gateway owns retries, timeouts, circuit breaking, request normalization, and response validation. Application services send a structured prompt request; they never hold raw provider API keys or implement retry logic themselves.

Client → ALB → API Service → LLM Gateway → Provider API
                      │                    │
                      │                    ├─ rate limiter (Redis)
                      │                    ├─ circuit breaker
                      │                    └─ token/cost accounting
                      ▼
                   MySQL (metadata, not prompts in prod logs)

This mirrors how I treat payment gateways. You would not let every microservice call Stripe directly with its own retry policy. Same rule applies to OpenAI, Anthropic, or a self-hosted vLLM cluster.

Timeouts, Retries, and the Idempotency Problem

Inference requests are expensive to retry blindly. A timeout at 30 seconds might mean the provider is still processing - retrying doubles your cost and can produce duplicate side effects if the model triggers tools or writes data.

My default policy: set client-facing timeouts shorter than provider timeouts (e.g., 25s client, 60s provider). Use a request ID stored in Redis with TTL to detect duplicate submissions. Retry only on explicit 429 and 503 with exponential backoff and jitter. Never retry on 400-class errors.

// Gateway retry policy (conceptual)
const RETRYABLE = new Set([429, 503, 502]);
async function callProvider(req, attempt = 0) {
  const controller = new AbortController();
  const timer = setTimeout(() => controller.abort(), 60_000);
  try {
    const res = await fetch(PROVIDER_URL, {
      method: "POST",
      body: JSON.stringify(req.body),
      signal: controller.signal,
      headers: { "X-Request-Id": req.id },
    });
    if (RETRYABLE.has(res.status) && attempt < 3) {
      await sleep(jitter(2 ** attempt * 500));
      return callProvider(req, attempt + 1);
    }
    return res;
  } finally {
    clearTimeout(timer);
  }
}

Rate Limiting and Token Budgets

Provider rate limits are per API key and per model. Internal rate limiting must happen before the provider call, not after you get a 429. I use Redis sliding-window counters keyed by tenant_id + model, with separate buckets for requests-per-minute and tokens-per-minute.

Token budgets prevent runaway costs. Every request gets a max_output_tokens cap at the gateway. Per-tenant daily spend limits live in MySQL and are checked synchronously for hard stops, asynchronously for alerting at 80% threshold.

Async Patterns for Long-Running Inference

Anything expected to exceed 10 seconds should not hold an HTTP connection open. I return 202 Accepted with a job_id, push the work to SQS or Redis Streams, and let clients poll or subscribe via SSE/WebSocket for completion.

This pattern saved us during a document summarization feature launch. Synchronous endpoints would have exhausted our ALB idle timeout and Node.js connection pool. Async with a 30-second poll interval kept p99 gateway latency under 200ms while inference ran for minutes in worker pods.

Workers scale independently on queue depth. API pods scale on RPS. That separation is the single biggest win for AI API reliability on Kubernetes.

AWS Infrastructure Layout

My typical production layout: ALB → ECS Fargate or EKS for the gateway and API tiers, ElastiCache Redis for rate limiting and idempotency keys, RDS MySQL for tenant config and billing metadata, SQS for async inference jobs, and Secrets Manager for provider API keys with automatic rotation where supported.

For self-hosted models, I add a separate GPU node pool (EKS with g5 instances or SageMaker endpoints) behind an internal NLB. The gateway routes by model name - cloud API for GPT-4 class, internal endpoint for fine-tuned models.

CloudWatch alarms on gateway 5xx rate, queue age, Redis memory, and provider error ratio. I alert on error budget burn, not every 429.

Engineering Takeaways

• Never call model providers directly from application services - centralize in a gateway with retries, limits, and accounting.
• Set client timeouts shorter than provider timeouts to avoid cascading event-loop blocking.
• Retry only 429/502/503 with backoff; use request IDs to prevent duplicate expensive calls.
• Enforce RPM, TPM, and spend caps before the provider call, not after a 429.
• Use async job patterns for inference expected to exceed 10 seconds.
• Scale API pods on RPS and worker pods on queue depth - never conflate the two.

Trade-offs I Would Consider

Option: Synchronous inference with aggressive autoscaling

• Pros: Simpler client integration, no poll/webhook machinery, lower operational surface.
• Cons: Expensive at scale, ALB/connection pool pressure, poor UX during provider outages.

Option: Async job queue with 202 + polling/SSE

• Pros: Stable gateway latency, independent worker scaling, natural retry and DLQ handling.
• Cons: More client complexity, harder to debug end-to-end without good trace IDs.

What I Would Do in Production

Deploy the gateway as a separate service with its own deployment cadence and SLO. Gateway bugs should not require redeploying every AI feature.

Log request_id, tenant_id, model, input_tokens, output_tokens, latency_ms, and provider_status - never log raw prompts in production unless compliance requires it and you have retention controls.

Run game days that simulate provider 429 storms and partial outages. Verify circuit breakers open and queued work drains without duplicate charges.

Set per-environment API keys with different spend limits. Staging keys should hard-stop at $50/day.

Keep a fallback model configured (cheaper, faster) that the gateway routes to when the primary circuit is open.

FAQ

Q: Should I use API Gateway or a custom Node.js gateway?

AWS API Gateway handles throttling and auth well but struggles with streaming SSE responses and custom token accounting. I use ALB + a Node.js gateway for flexibility, and API Gateway only for public-facing non-streaming endpoints if WAF integration is mandatory.

Q: How do I handle streaming responses reliably?

Stream from provider through gateway to client with heartbeat chunks every 15 seconds to keep ALB connections alive. If the provider stream breaks mid-response, send a terminal error event and log partial output tokens for billing reconciliation.

Q: What SLO should I set for AI APIs?

Separate availability SLO (gateway returns a valid response or structured error) from quality SLO (response meets evaluation criteria). I target 99.5% availability on the gateway and track quality with offline eval pipelines, not uptime monitors.

Q: How do I prevent one tenant from consuming all capacity?

Weighted fair queuing at the gateway: per-tenant concurrency limits, token budgets, and priority tiers. Enterprise tenants get higher concurrency; free tier gets queued behind paid.

Q: Self-hosted models or cloud APIs for production?

Cloud APIs for general capability and fast iteration. Self-hosted for fine-tuned models with predictable volume where unit economics justify GPU ops. Most teams I work with run hybrid - gateway routes by model name.