Chaos Engineering for Platform Reliability

Reliability is not proven by uptime dashboards during quiet weeks. It is proven when the AZ you did not know you depended on disappears and checkout still completes. Chaos engineering is how platform teams turn 'we think we are resilient' into evidence.

The goal is not breaking prod for sport. It is controlled experimentation that exposes unknown dependencies, validates runbooks, and trains on-call muscle memory before customers find the gaps.

Chaos Maturity: Where Most Teams Stall

Teams typically progress through four stages. Most stall at stage two because they lack a hypothesis format and executive cover for induced failure.

If you cannot drain a node in staging without paging someone, do not start with prod AZ failure injection. Fix observability and runbooks first.

Hypothesis-Driven Experiments

Every experiment needs a written hypothesis before execution. 'Let's see what breaks' is not chaos engineering-it is an incident with extra steps.

Hypothesis template
──────────────────
Steady state:  checkout P99 < 800ms, error rate < 0.1%
Experiment:    terminate 1 of 3 api pods in ap-south-1a
Expected:      HPA replaces pod within 90s, no user-visible errors
Abort if:      error rate > 1% for 2 min OR checkout P99 > 2s
Rollback:      stop experiment, verify pod count, file learnings

Store hypotheses in the same repo as runbooks. Link experiment IDs to postmortems when hypotheses fail-that is the valuable outcome.

Tooling: Litmus, Chaos Mesh, and AWS FIS

Kubernetes-native teams often use Litmus or Chaos Mesh for pod, network, and IO chaos. AWS Fault Injection Simulator covers AZ impairment, RDS failover, and EC2 stop-closer to the metal than pod deletion.

# Litmus ChaosEngine - pod delete experiment
apiVersion: litmuschaos.io/v1alpha1
kind: ChaosEngine
metadata:
  name: api-pod-delete
spec:
  appinfo:
    appns: production
    applabel: "app=api"
    appkind: deployment
  chaosServiceAccount: litmus-admin
  experiments:
    - name: pod-delete
      spec:
        components:
          env:
            - name: TOTAL_CHAOS_DURATION
              value: "60"
            - name: CHAOS_INTERVAL
              value: "10"

• Litmus/Chaos Mesh: best for application-level and K8s resource failures.
• AWS FIS: best for AZ, subnet, and managed service failure simulation.
• Toxiproxy/Lambda: latency and partition injection at integration boundaries.
• Gremlin: commercial option with strong governance UI for larger orgs.

Blast Radius and Safety Controls

Production chaos without guardrails is negligence. Define blast radius explicitly: one namespace, one AZ, one tenant tier, one percentage of traffic.

• Run during business hours with full team availability-not Friday evening.
• Automated abort on SLO breach via Prometheus Alertmanager webhook to experiment controller.
• Feature flag kill switch for experiments independent of the chaos tool.
• Notify customer success before enterprise-tenant adjacent experiments.
• Never run competing experiments simultaneously until you understand interaction effects.

Blast radius checklist
───────────────────────
[ ] Single region / single AZ scope defined
[ ] maxAffectedPods or percentage capped
[ ] Abort thresholds wired to metrics
[ ] On-call engineer not the same person running experiment
[ ] Rollback tested in staging within last 30 days

From Findings to Platform Fixes

The experiment ends when the learning is merged, not when the chaos stops. Typical findings from our game days:

• HPA too slow because readiness probe passed before JVM warmup completed.
• Cross-AZ RDS dependency not documented; AZ failure caused 3x latency.
• Retry storms from workers without jitter saturated downstream Redis.
• Runbook assumed kubectl access pattern blocked by new RBAC policy.

A failed hypothesis is success. It means you found a fix cheaper than a customer-discovered outage.

Engineering Takeaways

• Chaos engineering validates resilience hypotheses-it is not random prod breakage.
• Progress through maturity stages; table-top and staging injection precede prod experiments.
• Every experiment needs steady-state metrics, abort thresholds, and a written rollback plan.
• Match tooling to failure domain: K8s chaos for pods/network, FIS for AZ and managed services.
• Blast radius caps and automated abort hooks are non-negotiable for production runs.
• Failed hypotheses are valuable-track findings to merged fixes, not just experiment count.
• Schedule chaos during staffed hours with on-call coverage, never as unattended automation initially.

Trade-offs I Would Consider

Option: Staging-only chaos on a schedule

• Pros: Low risk, builds team familiarity, catches obvious config gaps.
• Cons: Staging rarely matches prod topology, traffic patterns, or data volume-false confidence is common.

Option: Production experiments with strict blast radius

• Pros: Tests real dependencies, traffic, and on-call response; findings transfer directly to SLO improvements.
• Cons: Requires executive trust, mature observability, and disciplined abort automation.

What I Would Do in Production

Start with quarterly game days: table-top first, then staging injection with the same scenarios.

Maintain a chaos experiment catalog in Git with hypothesis, results, and links to remediation PRs.

Wire Litmus or FIS abort to the same Alertmanager routes as prod alerts-if abort fires, treat it seriously.

Include chaos validation in platform change review: 'What happens if this AZ or pod pool disappears?'

Run one small prod experiment per month after staging parity reaches 80% on critical paths.

Share learnings in a blameless postmortem format even when nothing broke-document why resilience held.

FAQ

Q: Does chaos engineering replace load testing?

No. Load testing finds capacity limits under expected traffic. Chaos finds failure modes under degraded infrastructure. You need both.

Q: How do we get management approval for prod chaos?

Frame it as risk reduction with bounded blast radius. Show staging gaps, cite industry precedents (Netflix, Amazon), and start with AZ-independent pod kills before AZ failure.

Q: What if an experiment causes a real outage?

Abort immediately, execute rollback, run a blameless postmortem. If abort hooks failed, fix tooling before the next experiment. Do not cancel the program-fix the safety controls.

Q: Can we automate chaos in CI?

Yes for staging and ephemeral preview environments. Pre-merge chaos on PR environments catches regressions in HPA, probes, and circuit breakers.

Q: How is this different from draining nodes for upgrades?

Node drains test one failure mode predictably. Chaos tests combinations-network partition during scale-up, DB failover during traffic spike-that drains alone never surface.