LoopShield LLM Gateway
LoopShield is personal research on ultra-low-overhead LLM gateway patterns: one control plane for auth, spend caps, provider routing, streaming passthrough, and observability. Applications send provider-agnostic requests; the gateway owns retries, circuit breaking, and token accounting - the same discipline as payment gateway middleware.
Problem
When multiple teams integrate LLM providers directly, API keys leak into repos, retry policies diverge, token costs run unbounded, and streaming responses break ALB idle timeouts - each team reinvents the same reliability incidents.
Before and After
Before
- Hardcoded provider keys in application services
- Per-service retry and timeout logic
- No org-wide token accounting or spend caps
- Streaming SSE handled inconsistently
- No circuit breaker when provider returns 429 storms
After
- Central gateway: auth, validate, route, stream, account
- Per-tenant rate limits and daily spend caps in Redis
- Provider abstraction - swap models via routing config
- Circuit breaker with fallback model tier
- OpenTelemetry spans for every inference call
Low-Latency LLM Gateway
Centralized gateway between applications and model providers: auth, rate limits, routing, circuit breaking, streaming SSE passthrough, and token accounting - designed for minimal per-request overhead.
Clients
Gateway
Control
Providers
What I Built
Gateway hot path
- Provider-agnostic request schema translated inside the gateway only.
- SSE streaming with 15s heartbeat chunks to keep ALB connections alive.
- Abort upstream provider call when client disconnects mid-stream.
Cost and safety controls
- Pre-call token estimate vs post-call billing reconcile.
- Hard reject when tenant exceeds daily spend cap.
- Input scanning for known injection patterns at the edge.
Reliability patterns
- Retry with jitter on 429/5xx; circuit opens to fallback model.
- Sync path for chat; async 202 + queue for batch inference.
- Hybrid routing: cloud APIs vs self-hosted endpoints by model name.
Engineering Decisions
- Gateway as product - same rule as payment gateways: apps never hold raw provider keys.
- Redis for limits - synchronous hard stops need sub-ms reads on hot path.
- Separate availability vs quality SLOs - gateway uptime != model output quality.
- Minimal middleware overhead - hot path designed for low per-request latency.
By the Numbers
Pattern source
Production LLM gateway writing
Routing
Multi-provider + self-hosted
Streaming
SSE with heartbeat
Observability
OpenTelemetry spans
Cost control
Per-tenant caps
Project type
Personal R&D
Reliability and Ops
- Two+ gateway instances across AZs; Redis Multi-AZ for limit store.
- Fallback model configured when primary circuit is open.
- Partial stream logging for billing reconciliation on provider disconnect.
Testing
- Load test streaming connections through ALB with concurrent tenants.
- Chaos: provider 429 storm triggers circuit breaker and fallback route.
- Token estimate vs actual discrepancy alerts above 20% threshold.
What I Did
- 01Designed centralized LLM routing with per-tenant rate limits and daily spend caps backed by Redis.
- 02Implemented SSE streaming with ALB-safe heartbeats and upstream abort on client disconnect.
- 03Added circuit breaker fallback routing and OpenTelemetry spans for org-wide inference visibility.