API Load Benchmarking for Platform Teams

A product team asked "how many users can we handle?" and the previous answer was a Swagger load test screenshot showing 2,000 RPS with zero errors. What it did not show: p99 latency was 4 seconds, Redis was at 98% memory, and MySQL connection pool was exhausted. The system was failing gracefully on paper and failing in reality.

Platform teams own the answer to capacity questions. That answer must come from benchmarks designed to find limits, not confirm optimism.

Benchmarks Should Answer Specific Questions

Every benchmark run should state its question upfront. "What is the max sustained RPS at p99 < 500ms?" is a good question. "Can we handle 10x traffic?" is not - 10x of what, with what latency budget, with what error tolerance?

I write a one-page benchmark brief before every run: target endpoint, expected production traffic shape, latency SLO, error budget, infrastructure under test, and what decision the results will inform (right-sizing, autoscaling policy, launch go/no-go).

Benchmark Brief
───────────────
Question:    Max RPS at p99 < 500ms for POST /api/v1/enrollments
Traffic:     70% read / 30% write, ramp over 15 min, hold 30 min
SLO:         p99 < 500ms, error rate < 0.1%
Infra:       3x t3.medium API, RDS db.r6g.large, ElastiCache r6g.large
Decision:    Launch capacity for Q1 enrollment spike

Tooling: k6 for Most API Benchmarks

I use k6 for the majority of API load tests. It runs well in CI, scripts in JavaScript, and exports metrics to Grafana Cloud or InfluxDB. For Node.js APIs behind ALB, k6 running from EC2 in the same VPC gives realistic network conditions.

// k6 benchmark script (simplified)
import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  stages: [
    { duration: "5m", target: 100 },
    { duration: "10m", target: 500 },
    { duration: "30m", target: 500 },
    { duration: "5m", target: 0 },
  ],
  thresholds: {
    http_req_duration: ["p(99)<500"],
    http_req_failed: ["rate<0.001"],
  },
};

export default function () {
  const res = http.post(
    "https://api.internal/enrollments",
    JSON.stringify({ course_id: "c_123", user_id: "u_456" }),
    { headers: { "Content-Type": "application/json", Authorization: "Bearer ..." } },
  );
  check(res, { "status is 201": (r) => r.status === 201 });
  sleep(0.1);
}

Thresholds in k6 fail the run automatically if SLOs are breached. This makes benchmarks gateable in CI for regression detection.

Traffic Shapes That Match Production

Flat RPS load tests lie. Production traffic ramps, spikes, and has mixed read/write ratios. I model traffic shapes from CloudWatch ALB metrics: extract p50/p95 RPS over the last 30 days, identify peak hours, and replay that shape in k6 stages.

For seasonal spikes (enrollment periods, sale events), I take the peak day from last year, add 30% headroom, and benchmark against that. If the system survives peak + 30% at SLO, I sign off on capacity.

What to Watch During a Run

Client-side metrics (k6 latency, error rate) tell you when users would feel pain. Server-side metrics tell you why. I watch both simultaneously on a Grafana dashboard with k6 metrics on the left and AWS metrics on the right.

Server-side checklist: ALB target response time, API pod CPU and memory, Node.js event loop lag (if applicable), MySQL active connections and slow query count, Redis memory usage and connected clients, and SQS queue depth if async paths exist.

The breaking point is where server-side saturation causes client-side SLO breach. That is your capacity ceiling. Document the first resource to saturate - it drives your scaling recommendation.

From Results to Recommendations

Benchmark results should produce a capacity report, not a CSV dump. My template: max RPS at SLO, first bottleneck resource, recommended instance count, estimated monthly cost, and headroom percentage above expected peak traffic.

If the bottleneck is MySQL connections, the recommendation is connection pool tuning or RDS Proxy - not "add more API pods." Platform teams add value by translating benchmark data into infrastructure actions, not by running the test and forwarding graphs.

Engineering Takeaways

• Write a benchmark brief with a specific question, SLO, and decision before running any load test.
• Use k6 with thresholds that fail the run when SLOs are breached - make benchmarks gateable.
• Model traffic shapes from production CloudWatch data, not flat RPS.
• Watch server-side saturation (DB connections, Redis memory, event loop lag) alongside client-side latency.
• The breaking point is where server saturation causes client SLO breach - that is your capacity ceiling.
• Deliver capacity reports with recommendations, not raw metric exports.

Trade-offs I Would Consider

Option: Benchmark in staging with production-like infra

• Pros: Safe, repeatable, no production risk, can run in CI.
• Cons: Staging rarely matches production data volume and cache warmth; results may be optimistic.

Option: Benchmark in production during low-traffic windows

• Pros: Real data, real cache state, real network paths - most accurate results.
• Cons: Risk of user impact, harder to schedule, requires feature flags and circuit breakers.

What I Would Do in Production

Run a baseline benchmark after every major infrastructure change (instance type, DB upgrade, cache cluster resize).

Store k6 scripts in Git alongside the services they test. Benchmarks are code - review and version them.

Automate a weekly staging benchmark in GitHub Actions. Alert on > 10% latency regression vs last week's baseline.

Include soak tests (4+ hours at 70% capacity) in pre-launch benchmarks. Memory leaks and connection pool exhaustion show up here, not in 30-minute runs.

Share capacity reports with product and leadership before launch decisions. "We can handle 3x peak at SLO" is more useful than "we ran a load test."

FAQ

Q: How often should I run load benchmarks?

Weekly automated regression on staging, full capacity benchmark before every major launch or traffic event, and ad-hoc after infrastructure changes.

Q: k6 vs Locust vs Artillery?

k6 for API benchmarks in CI - best Grafana integration and threshold support. Locust if you need Python and complex user behavior simulation. Artillery for quick YAML-based tests. I default to k6.

Q: How do I benchmark APIs with authentication?

Pre-generate JWT tokens or API keys in setup() stage. For OAuth flows, use k6 browser module or a setup script that authenticates once and shares tokens across VUs.

Q: What if staging data is too small to be realistic?

Seed staging with anonymized production data snapshots. Cache warmth matters - cold caches make benchmarks pessimistic for reads and optimistic for writes.

Q: Should I benchmark individual endpoints or full user flows?

Both. Endpoint benchmarks find per-route capacity. Flow benchmarks (login → browse → checkout) find interaction effects and shared resource contention. Start with the hottest endpoint.