Hemal Buha

About Projects Blog Expertise Contact Get in Touch

22 Apr 20259 min readIntermediate

Kubernetes CrashLoopBackOff: Production Debugging Guide

Kubernetes
Debugging

CrashLoopBackOff is Kubernetes telling you the container started, exited, and the kubelet is backing off before trying again. It is not a root cause-it is a symptom. In production the clock is ticking because replicas are down, HPA may be scaling into the same wall, and someone is asking if we should rollback.

I have cleared hundreds of these across Edmingle services. The fix is almost always in logs, events, or a config diff-not in restarting the pod harder.

Read the Pod State First

Start with describe. It shows restart count, last exit code, OOMKilled, probe failures, and events.

kubectl get pods -n production -l app=payment-api
kubectl describe pod payment-api-7f8b9c-xyz -n production

Exit code 137 often means OOMKilled. Exit code 1 is application error. Exit code 0 with restart means your process exited cleanly-usually missing long-running foreground process or wrong CMD.

Events:
  Warning  BackOff    kubelet  Back-off restarting failed container
  Warning  Unhealthy  kubelet  Readiness probe failed: HTTP 503
  Normal   Pulled     kubelet  Container image already present

Logs: Current and Previous

The crashing instance may be gone by the time you look. Use --previous.

kubectl logs payment-api-7f8b9c-xyz -n production --tail=100
kubectl logs payment-api-7f8b9c-xyz -n production --previous --tail=100

If the container never starts (bad entrypoint), logs may be empty-check describe and image inspect.

For multi-container pods, specify the container name with -c.

Common Root Causes

Symptom	Likely cause	Where to look
OOMKilled in describe	Memory limit too low or leak	kubectl top pod, heap dump, raise limit temporarily
Error: config X missing	Missing ConfigMap/Secret mount	describe pod volumes, compare env from working pod
Connection refused DB	Wrong DATABASE_URL or network policy	env diff, NetworkPolicy, security group
Crash on boot only	Migration failure, bad seed	logs --previous, init container logs
Probe failed then restart	Readiness too aggressive	probe timing, /health implementation
ImagePullBackOff adjacent	Wrong tag or registry auth	describe events, imagePullSecrets

Compare Working vs Broken

If only new pods crash, diff the Deployment.

kubectl rollout history deployment/payment-api -n production
kubectl diff -f deployment.yaml
# Or compare live spec
kubectl get deployment payment-api -n production -o yaml | diff - previous.yaml

Recent changes to env, secrets, resource limits, or command are the usual suspects after a deploy.

Interactive Debug When Logs Are Not Enough

Ephemeral debug container (K8s 1.23+) without changing the image:

kubectl debug payment-api-7f8b9c-xyz -n production \
  -it --image=busybox --target=payment-api -- sh

Or run a one-off pod with the same image and env:

kubectl run debug-payment --rm -it \
  --image=123456789.dkr.ecr.ap-south-1.amazonaws.com/payment-api:abc123 \
  --env-from=deployment/payment-api -n production -- sh

Verify DNS, curl dependencies, check file mounts. Do not debug in prod longer than necessary.

Stopping the Bleeding

Rollback if the crash started with a deploy:

kubectl rollout undo deployment/payment-api -n production
kubectl rollout status deployment/payment-api -n production

Scale to zero only if traffic must stop-prefer rollback. Pause GitOps auto-sync if a bad commit keeps re-applying.

Document exit code and log snippet in the incident channel before deleting pods-evidence disappears fast.

Engineering Takeaways

CrashLoopBackOff is a symptom; describe pod + logs --previous find the cause in minutes.
Exit codes matter: 137 OOM, 0 clean exit with restart means wrong process model.
Post-deploy crashes → diff Deployment env, secrets, limits, and image tag first.
Readiness probe failures can look like crash loops if the app never becomes ready.
Rollback is a valid debug step when user impact is active-not admission of defeat.
Capture logs and events before pod deletion or rollout overwrites evidence.

Trade-offs I Would Consider

Option: Immediate rollout undo

Pros: Fastest user impact reduction, buys time for proper root cause analysis.
Cons: May revert a needed fix; does not explain why without follow-up.

Option: Debug forward with fix deploy

Pros: Keeps intended change if fix is trivial (typo in env var).
Cons: Slower recovery, risky under pressure if fix is wrong twice.

What I Would Do in Production

Set PodDisruptionBudget minAvailable so crash loops do not drain all capacity during bad rollout.

Use preStop hooks and reasonable terminationGracePeriodSeconds for clean shutdown.

Alert on restart rate and CrashLoopBackOff events via Prometheus kube-state-metrics, not only on pod Not Ready.

Store last-known-good Deployment revision in GitOps for one-click revert.

Run smoke tests in CI that start the container and hit /health before image reaches prod.

FAQ

Q: Why does kubectl logs show nothing?

Container crashed before writing logs, or wrong container selected. Use --previous and check init container logs.

Q: Can liveness probe cause CrashLoopBackOff?

Liveness failure kills the container, which triggers restart and backoff. Tune probes or fix slow startup.

Q: Pod runs locally but crashes in K8s?

Different env vars, secrets, architecture (arm vs amd), or file paths. Reproduce with same env-from in debug pod.

Q: How long is BackOff delay?

Exponential up to 5 minutes between restarts. Waiting does not fix config errors.

Q: Should I increase memory limit when OOMKilled?

Temporarily to restore service, then profile for leak or inefficient allocation. Limits are not a substitute for fixing memory use.

Get in Touch More on Kubernetes ← All Posts