Designing Benchmark Suites for Platform Changes

You upgraded Kubernetes minor versions, swapped the ingress controller, or enabled cgroup v2 on node images. CI is green. Production P99 latency climbed 18% for two weeks before anyone connected the dots.

Platform changes need benchmark suites that measure what application CI ignores: node-level behavior, DNS latency, storage IO patterns, and control plane overhead. Generic microbenchmarks are not enough-you need workloads that resemble your tenants' actual traffic shapes.

What Platform Benchmarks Must Capture

Application benchmarks answer 'is this endpoint fast?' Platform benchmarks answer 'did the substrate get slower or noisier for every service?'

Suite Architecture: Synthetic Plus Representative

Combine synthetic micro-workloads with a small set of production-derived scenarios. Synthetic gives reproducibility; representative catches real-world surprises.

Benchmark suite layers
──────────────────────
L1  Synthetic   - iperf, fio, hey, kube-bench
L2  Service     - golden path API load test (k6)
L3  Workflow    - checkout + webhook + worker pipeline
L4  Soak        - 4-hour run at 70% peak with chaos off

Run L1-L3 on every platform PR; L4 weekly and pre-prod cutover.

Pin workload versions and dataset sizes in Git. A benchmark that changes request payload size every run is not a benchmark-it is noise.

Implementation with k6 and Cluster-Level Fixtures

We run k6 scenarios against a dedicated benchmark namespace that mirrors production topology at 10% scale-same ingress class, same Redis cluster mode, same RDS parameter group family.

// k6 platform benchmark - golden path
import http from "k6/http";
import { check, sleep } from "k6";

export const options = {
  stages: [
    { duration: "2m", target: 50 },
    { duration: "5m", target: 200 },
    { duration: "2m", target: 0 },
  ],
  thresholds: {
    http_req_duration: ["p(99)<800"],
    http_req_failed: ["rate<0.01"],
  },
};

export default function () {
  const res = http.get(`${__ENV.BASE_URL}/api/v1/courses`, {
    headers: { Authorization: `Bearer ${__ENV.TOKEN}` },
  });
  check(res, { "status 200": (r) => r.status === 200 });
  sleep(0.1);
}

# GitHub Actions: run benchmarks against staging cluster post-change
- name: Run k6 platform suite
  uses: grafana/k6-action@v0.3.1
  with:
    filename: benchmarks/golden-path.js
  env:
    BASE_URL: https://bench.staging.edmingle.internal
    K6_CLOUD_TOKEN: ${{ secrets.K6_CLOUD_TOKEN }}

Baselines, Regression Gates, and Statistical Discipline

Store baseline results per platform version in object storage. Compare PR runs with statistical thresholds-not single-run pass/fail.

• Require 3 consecutive runs within 5% of baseline P99 before promoting node image.
• Track coefficient of variation; high variance means benchmark environment is unstable-fix before tuning thresholds.
• Separate regressions by dimension: 10% DNS latency increase differs from 10% API latency.
• Alert on gradual drift across weekly soak tests even when PR gates pass.

Regression decision tree
─────────────────────────
Δ P99 < 5% AND error rate stable  → pass
Δ P99 5-10%                       → investigate, may waive with ticket
Δ P99 > 10% OR error rate up      → block promotion
Variance > 15% across 3 runs      → fix benchmark env, not platform

Benchmark Environment Hygiene

Shared staging cluttered with feature branches produces lying benchmarks. Dedicate a benchmark cluster or namespace with frozen config during test windows.

• No auto-deploys to benchmark env during test runs-Argo CD sync freeze annotation.
• Same node count and instance types as production tier being validated.
• Warm caches before measurement or document cold-start as separate scenario.
• Record platform version, git SHA, and node image ID in every result artifact.

A benchmark environment that diverges from production topology validates nothing- it just generates confident spreadsheets.

Engineering Takeaways

• Platform benchmarks measure substrate changes that application CI cannot see.
• Combine synthetic (fio, iperf) and representative (k6 golden paths) workloads in layered suites.
• Pin workload definitions in Git; unstable inputs produce false regressions and false confidence.
• Use statistical baselines across multiple runs-never single-shot pass/fail on noisy metrics.
• Dedicate benchmark environments with sync freezes during measurement windows.
• Track P99, error rate, and variance; block promotion on >10% P99 regression or error rate increase.
• Run short suites on every platform PR and soak tests weekly plus pre-cutover.

Trade-offs I Would Consider

Option: Synthetic-only benchmarks (fio, iperf, hey)

• Pros: Fast, reproducible, easy to automate, good for isolating hardware and driver changes.
• Cons: Miss application-level interactions-DNS, connection pools, JVM warmup under real APIs.

Option: Production-derived golden path suites (k6 at scale)

• Pros: Captures real traffic shapes and service dependencies; catches regressions app teams feel.
• Cons: Slower, more expensive to run, requires dedicated env and careful baseline maintenance.

What I Would Do in Production

Maintain benchmarks/ directory with L1-L3 suites versioned alongside Terraform and Helm.

Run platform benchmark workflow on PRs touching node groups, CNI, ingress, or K8s version.

Store results in S3 with tags: cluster_version, node_ami, git_sha; visualize in Grafana.

Set promotion gate: three consecutive runs within 5% baseline P99 for production node rollouts.

Execute 4-hour soak with weekly cron; compare against rolling 30-day baseline for drift.

Document waiver process for accepted regressions with ticket, expiry, and compensating control.

FAQ

Q: How is this different from application load tests?

Application load tests validate code changes on stable infrastructure. Platform benchmarks validate infrastructure changes across all applications.

Q: Can we run benchmarks in production?

Read-only synthetic traffic at negligible volume during low-traffic windows can validate prod-specific effects. Default to dedicated benchmark env for mutate tests.

Q: What if baseline drifts seasonally?

Maintain seasonal baselines or normalize against a stable reference workload running on every node pool.

Q: Do we need k6 Cloud?

No. Self-hosted k6 with artifact upload works. Cloud helps with dashboards and distributed load generation at scale.

Q: How many golden paths are enough?

Three to five covering highest revenue and highest traffic services. Add paths when new services exceed 15% of platform traffic.