EC2 Warm Pool Autoscaler
A production SaaS API fleet uses a fixed warm pool of pre-provisioned EC2 instances behind an Application Load Balancer. Native Auto Scaling Groups terminated instances despite protection, and evening capacity depended on brittle manual attach/detach cron jobs. I built a desired-state autoscaler as a single AWS Lambda (Node.js 24, reserved concurrency 1). Every trigger - scheduled evening capacity (18:30 / 20:45 IST), five-minute CPU evaluation, or manual scale commands - only updates a DesiredCapacity tag on the target group. A reconciliation engine runs every three minutes to converge the live fleet: start stopped warm-pool instances, register them to the ALB, sync public IPs to a regional WAF allowlist, and gracefully drain before scale-in. The system never terminates instances; it only start/stop/register/drain.
Problem
A production SaaS API platform runs a pool of pre-provisioned EC2 servers behind an Application Load Balancer. Edtech traffic has predictable evening peaks (IST) plus unpredictable CPU spikes. Native Auto Scaling Groups were a poor fit: instances were terminated despite protection policies, and evening capacity relied on fragile manual attach/detach scripts for specific servers.
Before and After
Before
- Manual evening attach/detach cron for specific servers
- Native ASG terminated instances (even with protection)
- No unified desired capacity across triggers
- Scaling tied to one metric or manual ops
- Risky deploys without soak or offline validation
After
- EventBridge schedules + 3-minute reconcile loop
- Warm pool: stop/start only, never terminate
- Single source of truth: DesiredCapacity target group tag
- CPU ladder + schedules + healing + guard rails
- Dry-run default, 271+ offline tests, CI quality gate
How Scaling Decisions Flow
Every trigger only updates DesiredCapacity on the ALB target group. A reconciliation loop converges the warm pool every three minutes - start, register, drain, stop, and WAF sync. Pick any block to see how traffic, metrics, and decisions connect.
Triggers
Control
Decision
Engine
AWS
Ops
What I Built
Desired-state reconciliation engine
- Reads DesiredCapacity from target group tags; computes delta = desired - (serving + pending).
- Scale-out: rescue draining -> register running -> start stopped pool members -> WAF-add.
- Scale-in: deregister -> PendingStop -> drain -> stop -> WAF-remove.
- Idempotent across duplicate EventBridge events, crashes, and manual stops (~3 min convergence).
- Bootstrap safety: first run adopts current serving count instead of scaling to MIN.
Scheduled capacity and CPU dynamic scaling
- 18:30 IST Mon-Sat sets evening desired capacity; 20:45 IST scales to night floor.
- Every 3 min: heal + reconcile. Every 5 min: CPU ladder + reconcile.
- Fleet average CPU: >=80% emergency +3, >=65% +1, <=25% for 3x5 min bins -> -1.
- Cooldowns: 10 min up / 20 min down. Peak window (18:00-20:45 IST) blocks CPU scale-in only.
- Ghost-bin protection skips stale CloudWatch metrics (>10 min).
Self-healing state machine
- detect -> replacing -> draining -> rebooting -> reregistering -> restoring (~15-25 min per heal).
- Starts spare capacity before touching the sick instance.
- Circuit breaker if >50% unhealthy; mass reboot playbook for fleet-wide failures.
- NeedsAttention tag + Slack ERROR on heal failure.
WAF sync, tag state store, and safety
- Add public IP on register; remove on stop with optimistic-lock retries (WafSyncPending tag).
- No DynamoDB: desired state lives in target group and EC2 tags.
- NODE_ENV !== production -> dry-run: reads AWS, logs [DRY RUN], no mutations.
- Kill switches: MONITOR_ENABLED, HEALING_ENABLED. Dead-man CloudWatch alarms -> SNS/PagerDuty.
Engineering Decisions
- Warm pool over ASG - ASG terminated instances; this system preserves EBS/stateful boxes via stop/start only.
- Single writer (reservedConcurrency: 1) - no distributed locks; TG tags are safe with one Lambda writer.
- Separation of concerns - decision/ is pure math (no AWS I/O); enforced by dependency-cruiser.
- Two tick frequencies - 3 min for heal/reconcile; 5 min for CPU (matches basic monitoring period).
- Graceful scale-in - always drain before stop; ScalingProtected instances never stopped by normal scale-in.
- Dry-run by default in serverless.yml - production requires explicit NODE_ENV=production.
By the Numbers
Offline tests (L1-L5)
~271
End-to-end feature scenarios
45
Live parameter matrix variants
141
Warm pool ceiling (configurable)
Up to 9 EC2 (prod: 7)
Reconcile frequency
Every 3 minutes
Lambda timeout
600s (healing spans ticks)
Worst-case drift recovery
<=3 min (design goal)
Reliability and Ops
- Production rollout: dry-run soak, bootstrap adopt, documented deploy checklist and ops playbook.
- Decision IDs in structured Pino logs and Slack for traceability.
- POC instances outside pool (AutoScalingPool=false) serve traffic but stay excluded from capacity/CPU/healing.
Testing
- L1-L5 run fully offline with in-memory fakes in test/fakes.js - zero AWS credentials in CI.
- npm run quality: ESLint, Prettier, knip, dependency-cruiser, unit, functional, feature, and matrix tests.
- Optional L6 Floci + Docker integration tests against a local AWS emulator.
What I Did
- 01Designed and deployed a serverless EC2 autoscaler (Node.js 24 / Lambda) for a warm pool behind ALB, replacing manual cron-based scaling and unreliable ASG behavior.
- 02Implemented desired-state reconciliation: DesiredCapacity target-group tag drives start/stop/register/drain with 3-minute convergence and idempotent EventBridge triggers.
- 03Built CPU scaling ladder (+1/+3/-1), IST evening schedules, peak-window scale-in guard, and WAFv2 IP-set sync with deferred retry (WafSyncPending).
- 04Authored self-healing state machine (replace -> drain -> reboot -> restore) with circuit breaker and mass-reboot playbook for fleet-wide failures.
- 05Delivered 271+ automated tests (unit/functional/feature/matrix), CI quality gate (lint, knip, dependency-cruiser), and dry-run mode for safe production rollout.