Designing Zero-Downtime Deployments

Zero downtime deployment does not mean zero risk. It means users should not see errors or forced disconnects while you ship new code. That requires compatible rollouts, correct health checks, and a strategy matched to your failure tolerance-not just maxUnavailable: 0 in a YAML file.

At Edmingle we deploy API services on Kubernetes dozens of times per week. Most use rolling updates; high-risk changes get canary analysis. Here is how I choose and implement each pattern.

Prerequisites Every Strategy Needs

• Backward-compatible API changes or versioned endpoints during transition.
• Readiness probes that pass only when the app can serve traffic (DB pool warm, migrations done).
• Graceful shutdown: handle SIGTERM, drain in-flight requests, preStop sleep if needed.
• Database migrations that expand-contract-never drop column before old code is gone.

Skip these and rolling vs blue-green vs canary all fail the same way.

Strategy Comparison

Rolling Update on Kubernetes

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  replicas: 3

Deploy v2 rolling (maxUnavailable: 0, maxSurge: 1)

[v1][v1][v1]  →  [v1][v1][v2]  →  [v1][v2][v2]  →  [v2][v2][v2]
                      ↑ new pod must pass readiness before old removed

maxUnavailable: 0 keeps capacity during deploy. maxSurge: 1 adds temporary pod-needs cluster headroom. PodDisruptionBudget protects against voluntary evictions during node maintenance.

Blue-Green

Run green environment alongside blue. Switch Service selector or Ingress weight from blue to green in one action. Keep blue warm for instant rollback.

On AWS you might use separate target groups behind ALB listener rules. On K8s, flip label selector or use Argo Rollouts with preview Service.

Cost is double capacity during cutover window. Worth it when migration risk is high or you need a human approval gate before traffic switch.

Canary

Route 5% traffic to new version, watch error rate and latency against baseline, promote or abort. Tools: Argo Rollouts, Flagger, or mesh traffic split.

Define promotion criteria objectively: 'canary error rate < 0.1% and p99 within 10% of stable for 15 minutes.'

Canary catches bugs that only appear under real traffic shape-load tests rarely replicate prod query patterns.

Database and Async Workers

API rolling deploy is useless if workers consume messages the new schema cannot read. Deploy order: expand schema → deploy consumers that handle both → deploy producers → contract schema.

For Redis Streams consumers, use consumer groups and deploy with enough overlap that pending messages drain on old and new code paths during dual-write window.

Engineering Takeaways

• Zero downtime requires compatible code, readiness probes, and graceful shutdown-not just a deploy strategy name.
• Rolling update is the default for stateless K8s services with maxUnavailable: 0 and tested probes.
• Blue-green trades 2x resource cost for instant rollback at traffic switch.
• Canary limits blast radius; define automated promotion/abort metrics before enabling.
• Database expand-contract migrations are part of the deploy design, not a DBA side quest.
• Rollback must be rehearsed; an undo button nobody trusts is not a strategy.

Trade-offs I Would Consider

Option: Rolling update only for everything

• Pros: Simple, no extra tooling, works for most API changes.
• Cons: Mixed versions during rollout; bad binary can touch all pods before you notice.

Option: Canary with Argo Rollouts for critical services

• Pros: Metric-driven promotion, small blast radius, automated abort.
• Cons: Tooling complexity, needs reliable metrics, longer deploy duration.

What I Would Do in Production

Default Deployment template: maxUnavailable 0, maxSurge 1, PDB minAvailable 2 for three-replica services.

Implement SIGTERM handler with server.close() and 30s preStop hook for in-flight HTTP drain.

Wire deploy markers on dashboards; alert on error rate SLO burn for 10 minutes post-deploy.

Use expand-contract migrations in CI checklist-block merge if migration is breaking without dual-read code.

Keep rollout undo documented in runbook; GitOps revert for manifest changes.

FAQ

Q: Is maxUnavailable: 0 always required?

For user-facing prod APIs, yes unless you have surplus capacity and accept brief reduction. Batch workers may tolerate 1.

Q: Does blue-green need two clusters?

No. Two Deployments or Rollouts in same cluster with traffic switch is sufficient.

Q: How long should canary run?

Long enough to hit representative traffic-often 15-30 minutes for B2B, longer for low-traffic services.

Q: What about WebSocket connections?

They survive rolling update if clients reconnect. Plan for reconnect storm; use sticky sessions or graceful connection drain.

Q: Can GitOps do canary?

Argo Rollouts integrates with GitOps. Plain Argo CD sync deploys whole manifest-use Rollouts CRD for progressive delivery.