Designing a Highly Available API Platform

Part of AWS

AWS
System Design
APIs

Highly available does not mean nothing ever breaks. It means when something breaks, users still get served within your SLO. For an API platform that means designing for failure at every layer: AZ loss, bad deploy, dependency timeout, and traffic spikes.

At Edmingle our API tier sits behind AWS ALB, runs on Kubernetes across multiple AZs, talks to MySQL primary/replica, Redis for cache and streams, and several internal services. This post is how I think about HA as explicit failure modes with mitigations-not as a checkbox on an architecture slide.

Define Availability in Numbers

99.9% allows ~43 minutes downtime per month. 99.95% allows ~22 minutes. Pick a target per tier: auth and payments tighter than internal admin APIs.

Measure availability from the client perspective (synthetic checks + real traffic success rate), not just pod Running status. A pod can be Running and useless if readiness is wrong.

Multi-AZ Is the Baseline

Region: ap-south-1
├── AZ-a: ALB targets, EKS nodes, RDS subnet
├── AZ-b: ALB targets, EKS nodes, RDS subnet
└── AZ-c: ALB targets, EKS nodes, RDS subnet

Single AZ loss → reduced capacity, not total outage

Spread Pod replicas with topologySpreadConstraints or pod anti-affinity. ALB health checks should hit endpoints that validate critical dependencies lightly-not a deep DB query on every probe.

Failure Matrix

Every HA design should be a table you can walk through in a review.

Failure	Impact	Mitigation	Residual risk
Single pod crash	Minimal if replicas > 1	Deployment replicas, restart policy	Brief error blip during rollout
AZ outage	Capacity drop ~33%	Multi-AZ nodes + PDB + over-provision	Latency spike if not enough headroom
Bad deploy	Errors or latency on new version	Rolling update + readiness + quick rollback	Bad config passes probes
Primary DB failure	Write outage	RDS Multi-AZ failover (~60-120s)	In-flight transactions fail
Replica lag spike	Stale reads	Route critical reads to primary	Primary load increase
Redis unavailable	Cache miss storm	Graceful degrade, circuit breaker	DB load spike
Downstream API down	Feature degradation	Timeouts, fallbacks, async queue	Partial functionality loss
Region loss	Total outage	DR region (cold/warm), DNS failover	RTO/RPO trade-offs

Load Balancing and Health

ALB with cross-zone load balancing. Target type IP for pod direct routing on EKS. Deregistration delay aligned with connection draining.

Readiness probe should match real traffic readiness: can the app serve requests without blocking on a cold cache warm-up? Liveness should be dumb-only restart if deadlocked.

readinessProbe:
  httpGet:
    path: /health/ready
    port: 8080
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet:
    path: /health/live
    port: 8080
  periodSeconds: 10
  failureThreshold: 5

Data Layer HA

RDS Multi-AZ for automatic primary failover. Test failover quarterly-DNS and connection string behavior surprises teams who never practiced.

Redis: use ElastiCache with Multi-AZ and automatic failover for cluster mode or replica promotion. Treat cache as optional; never as source of truth.

For Redis Streams consumers at Edmingle, run consumer groups with enough workers and monitor pending message age-lag is an availability concern when events drive user-visible state.

Graceful Degradation

Not every dependency deserves hard failure. Return cached data, stub non-critical fields, or queue work for async processing when billing enrichment fails but core LMS read should succeed.

Feature flags let you disable expensive code paths under load without a full rollback.

Engineering Takeaways

HA is an SLO with measured client-side success-not 'we run three replicas.'
Multi-AZ across compute, load balancer, and database is table stakes for production APIs.
Maintain a failure matrix and game-day each row at least once a year.
Readiness probes must reflect real serving ability; liveness should not restart during slow dependency.
Cache and async patterns are availability tools when dependencies fail or lag.
Region-level DR is a business decision with explicit RTO/RPO-do not pretend multi-AZ covers it.

Trade-offs I Would Consider

Option: Active-passive DR region

Pros: Lower cost, simpler data replication story, clear failover procedure.
Cons: Higher RTO (hours), cold capacity, DR drills often neglected.

Option: Active-active multi-region

Pros: Lower RTO/RPO, geographic latency wins for global users.
Cons: Conflict resolution, data replication cost, operational complexity 3-5x.

What I Would Do in Production

Run synthetic canaries every minute from outside the VPC hitting critical API flows.

Keep N+2 capacity headroom in normal conditions so one AZ loss does not breach latency SLO.

Automate rollback on error rate or latency SLO burn during deploy window.

Document and rehearse RDS failover: app reconnect behavior, connection pool flush, expected error window.

Queue critical side effects (email, webhooks) so transient dependency failure does not lose work.

FAQ

Q: How many replicas minimum?

At least two across different AZs for stateless APIs. Three if you want rolling updates with zero capacity drop during one-replica surge.

Q: Is Kubernetes required for HA?

No. EC2 ASG + ALB achieves similar compute HA. K8s adds deploy velocity and standardized ops at complexity cost.

Q: What about DDoS?

AWS Shield, WAF rate limits, CloudFront for static-availability includes abuse scenarios. Separate from app HA but related in incidents.

Q: Should we use a service mesh for retries?

Retries can amplify load during outages. Prefer bounded retries with jitter in app code or mesh with strict budgets.

Q: How do we measure HA?

SLI: successful requests / total requests over rolling window. SLO: 99.9% monthly. Error budget drives release aggressiveness.

Get in Touch More on AWS ← All Posts