Cost-Aware Autoscaling on Kubernetes

Most teams treat Kubernetes autoscaling as a latency problem. You set a CPU target of 70%, enable the Cluster Autoscaler, and call it done. Then finance asks why the EKS bill jumped 40% after a traffic spike that lasted twelve minutes.

Cost-aware autoscaling is not about running fewer pods during peak. It is about aligning scaling signals, instance economics, and scheduling constraints so you pay for headroom only when revenue or SLO risk justifies it. At Edmingle, we learned this after Black Friday prep exposed how HPA, VPA, and CA can fight each other while the bill climbs.

The Three Controllers and Where They Disagree

Kubernetes gives you three scaling levers that operate at different layers. The Horizontal Pod Autoscaler (HPA) changes replica count. The Vertical Pod Autoscaler (VPA) changes CPU and memory requests. The Cluster Autoscaler (CA) adds or removes nodes.

Traffic spike
    │
    ▼
┌─────────┐    scale out     ┌─────────┐    insufficient    ┌─────────┐
│   HPA   │ ───────────────► │  Pods   │ ──── capacity ───► │   CA    │
│ replicas│                  │ pending │                    │  nodes  │
└─────────┘                  └─────────┘                    └─────────┘
    │                              │
    │                              ▼
    │                        ┌─────────┐
    └──── CPU still high ──► │   VPA   │ (may recommend larger requests)
                             └─────────┘

The failure mode is circular: HPA scales out, pods go Pending, CA adds a m5.2xlarge, VPA recommends bigger requests, fewer pods fit per node, HPA scales out again. You end up with expensive nodes running half-empty because requests-not actual usage-drive scheduling.

Choosing Metrics That Reflect Cost, Not Just CPU

CPU utilization is a proxy. For request-driven APIs, custom metrics tied to business load often scale more predictably and cheaply than raw CPU.

apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: api-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  minReplicas: 3
  maxReplicas: 40
  metrics:
    - type: Resource
      resource:
        name: cpu
        target:
          type: Utilization
          averageUtilization: 65
    - type: Pods
      pods:
        metric:
          name: http_requests_per_second
        target:
          type: AverageValue
          averageValue: "120"

We run dual metrics on user-facing services: CPU as a safety net, RPS as the primary driver. Workers scale on queue lag with a hard maxReplicas cap tied to downstream DB connection limits.

Instance Mix and the Bin-Packing Problem

Cluster Autoscaler is node-group aware but not cost-optimal by default. If you only run one instance type, you pay for the shape that fits your largest pod.

A practical approach on AWS: combine a baseline node group of smaller instances (e.g., m6i.large) for steady state with a burst group of larger spot instances for scale-out. Use taints and tolerations or Karpenter provisioners to route burst workloads.

Steady state (on-demand, small)     Burst (spot, large)
─────────────────────────────────     ─────────────────────
api, web, sidecars                    batch jobs, spike replicas
70% avg utilization target            scale-to-zero friendly
pinned by PDB + minReplicas           max node lifetime: 24h

Karpenter often beats CA on bin-packing because it selects instance types at provisioning time rather than filling fixed ASGs. The trade-off is another controller to operate and tighter IAM requirements.

Rightsizing Requests Before You Autoscale

Autoscaling magnifies bad requests. If every pod requests 2 CPU but uses 200m, HPA thinks you are at 10% utilization and never scales-or scales late after throttling.

• Run VPA in recommendation-only mode for four weeks before applying.
• Set requests at P95 usage, limits at P99 or drop limits for CPU entirely on latency-sensitive services.
• Separate bursty batch workloads into their own node pool so they do not trigger API scaling.
• Use the Vertical Pod Autoscaler checkpoint API or Goldilocks dashboards for visibility without auto-apply risk.

# VPA recommendation-only - review before mutate
apiVersion: autoscaling.k8s.io/v1
kind: VerticalPodAutoscaler
metadata:
  name: api-vpa
spec:
  targetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: api
  updatePolicy:
    updateMode: "Off"

At Edmingle, rightsizing alone cut node count by 22% before we changed a single HPA target.

Scale-Down Discipline and Cooldown Tuning

Scale-up is fast; scale-down is where money leaks. Default HPA stabilization windows and CA scan intervals often leave extra nodes running for 10-15 minutes after a spike.

Tune scaleDown stabilization aggressively for non-prod. For prod, accept slower scale-down in exchange for fewer user-facing latency blips during flappy traffic.

• HPA behavior.scaleDown.stabilizationWindowSeconds: 300-600 for prod APIs.
• CA skip-nodes-with-local-storage and skip-nodes-with-system-pods flags-know what blocks removal.
• PodDisruptionBudgets that allow at least one node drain per pool during scale-down.
• Scheduled scale-down for known quiet windows (nights, weekends) via Keda cron scalers or cluster scale schedules.

Document the cost of aggressive scale-down: a 60-second blip during node consolidation may be cheaper than an hour of idle m5.xlarge capacity.

Engineering Takeaways

• HPA, VPA, and Cluster Autoscaler optimize different layers-misaligned settings inflate cost without improving SLOs.
• Custom metrics (RPS, queue depth) often produce cheaper scaling than CPU alone for request-driven services.
• Rightsize requests before tuning HPA targets; bad requests make every autoscaler lie.
• Mix instance types and purchase options: steady on-demand baseline, burst on spot with taints/tolerations.
• Scale-down policy is where savings live; tune stabilization windows with explicit latency vs cost trade-offs.
• Cap maxReplicas to downstream limits (DB connections, partner rate limits) to prevent runaway scale-out bills.
• Review autoscaling dashboards monthly alongside the AWS Cost Explorer EKS line item-not just during incidents.

Trade-offs I Would Consider

Option: Cluster Autoscaler with fixed node groups

• Pros: Well understood, stable, works with standard EKS managed node groups and ASGs.
• Cons: Poor bin-packing across heterogeneous workloads; slow to adopt cheaper instance SKUs without infra changes.

Option: Karpenter with flexible instance selection

• Pros: Better packing, faster provisioning, native spot/on-demand blending, strong cost optimization potential.
• Cons: Additional operational surface, requires careful IAM and interruption handling, team learning curve.

What I Would Do in Production

Start every new service with VPA in Off mode. Review recommendations weekly; apply request changes via Git, not auto-mutate.

Define HPA with at least one business metric plus CPU fallback. Set maxReplicas from connection pool math, not guesswork.

Split node pools: default for steady workloads, burst for spot-eligible stateless replicas. Never run stateful data stores on spot without explicit acceptance.

Wire CloudWatch Container Insights or Prometheus recording rules for cost-per-request dashboards. Alert when cost per 1k requests deviates 30% from baseline.

Run game days that simulate traffic spikes and measure time-to-scale and time-to-scale-down. Compare bill impact against SLO breach count.

Revisit autoscaling config quarterly when AWS releases new instance generations-m7g or c7g often beats older types on price-performance.

FAQ

Q: Should I enable VPA in Auto mode?

Rarely on production Deployments without extensive testing. Auto mode recreates pods and can cause disruption. Use Off or Initial mode, apply recommendations through your GitOps pipeline.

Q: What HPA CPU target should I use?

There is no universal number. Start at 65-70% after rightsizing, load test, and adjust based on scale-up latency and P99 during spikes. Lower targets cost more; higher targets risk throttling.

Q: Does Cluster Autoscaler respect AWS Savings Plans?

CA adds nodes from configured groups-it does not optimize for your reservation coverage. Karpenter can prefer instance types you have reserved, but you still need to align node shapes with your commitment portfolio.

Q: How do I prevent autoscaling during deploys?

Use HPA behavior policies, ensure readiness probes are accurate, and consider temporarily pausing HPA during risky migrations. Flapping during rollouts often indicates probe or startup probe misconfiguration.

Q: Is spot always cheaper for Kubernetes?

Only for fault-tolerant, horizontally scaled workloads with graceful interruption handling. Mix spot with on-demand baseline and never assume spot availability during regional capacity crunches.