How to Debug a Slow API in Production

P99 latency doubled overnight. The dashboard is green because error rate is fine. Users are complaining. Welcome to the most common production incident that does not page you until it is bad.

Slow APIs are rarely one bug. They are a stack of small sins: a missing index, a connection pool at capacity, a downstream timeout set too high, or a deploy that increased payload size. At Edmingle I debug these by narrowing the blast radius first, then following the request path with data-not guesses.

Confirm the Problem Is Real

Before diving into flame graphs, validate scope. Is it one endpoint, one region, one tenant, or global? Compare p50 vs p95 vs p99. A p99 spike with flat p50 often means a subset of requests (large payloads, specific accounts, cache misses).

Check deploy timeline, feature flags, and infra changes in the same window. Correlation is not causation but it is a good start.

The Request Path Map

Client → ALB → Ingress → Pod → App → Redis/MySQL → External API
         │                              │
         └── access logs                └── APM spans + DB slow query log

Draw the path once per service and keep it in the runbook. Every slow request walks this chain. Your job is binary search with instrumentation.

Start at the Edge

ALB target response time vs application-reported latency tells you if slowness is before or inside the app. High ALB time with low app time suggests connection queuing, insufficient pods, or readiness flapping.

# Quick pod resource check during incident
kubectl top pods -n production -l app=api-service
kubectl get hpa -n production

If CPU throttling or memory pressure shows up, scale or fix limits before chasing SQL.

Traces Beat Logs for Latency

Open a trace for a slow request. Sort spans by duration. The longest span is usually your culprit: a SELECT without index, a Redis blocking call, or an HTTP client waiting 30s on a dead dependency.

// Ensure outbound calls create spans
const span = tracer.startSpan('call.billing-service');
try {
  await fetch(BILLING_URL, { signal: AbortSignal.timeout(3000) });
} finally {
  span.end();
}

If you lack traces, enable them before the next incident. Logs alone will not show parallel wait time correctly.

Database Is the Usual Suspect

Check connection pool saturation: waiting threads, pool size vs max connections on MySQL. A pool exhausted under load looks like app slowness.

-- Find running slow queries (MySQL 8)
SELECT * FROM performance_schema.events_statements_summary_by_digest
ORDER BY AVG_TIMER_WAIT DESC LIMIT 10;

Enable slow query log temporarily with threshold 500ms if needed. Look for full table scans, rows examined >> rows sent, and lock wait time.

Replica lag routing bugs can also look like slowness if apps retry or block waiting for consistency that never arrives.

Dependencies and Timeouts

A downstream service at 2s latency with 30s client timeout means your API absorbs all of it. Fail fast: set timeouts slightly above downstream p99, use circuit breakers, return degraded responses where product allows.

Redis: watch for hot keys, blocking commands (KEYS, unbounded SMEMBERS), and connection count. Stream consumer lag can backlog if processors fall behind.

The Fix and Verify Loop

Fix one variable, deploy or toggle, watch p99 for 15-30 minutes. Avoid changing indexes, pool size, and cache TTL simultaneously-you will not know what worked.

Write a short post-incident note: symptom, root cause, detection gap, preventive metric or alert.

Engineering Takeaways

• Compare p50 and p99 before blaming 'the database'-tail latency has different causes than median slowdown.
• Binary search the request path: edge timing vs in-app timing narrows scope fast.
• Distributed traces sorted by span duration find the bottleneck faster than grep-ing logs.
• Connection pool exhaustion mimics slow queries; check pool metrics alongside SQL.
• Aggressive client timeouts turn dependency stalls into your API's latency problem.
• Change one knob per fix cycle; verify with metrics before closing the incident.

Trade-offs I Would Consider

Option: Enable detailed profiling in prod during incident

• Pros: CPU flame graphs and heap snapshots pinpoint hot code paths immediately.
• Cons: Overhead risk, security exposure, requires tooling and practice.

Option: Reproduce in staging with load test + trace

• Pros: Safer experimentation, can inject faults, no prod impact.
• Cons: Staging may lack prod data shape; some bugs only appear at scale.

What I Would Do in Production

Maintain RED dashboards (Rate, Errors, Duration) per endpoint with p50/p95/p99 and deploy markers.

Alert on SLO burn rate, not just static latency thresholds-slow burns matter.

Standardize 3s outbound timeouts and circuit breakers in shared HTTP client libraries.

Run monthly 'slow API drills': pick a random endpoint and trace one request end-to-end.

Keep a living dependency map with owner contacts for parallel escalation.

FAQ

Q: Should I scale pods first when latency spikes?

If CPU/memory saturated or HPA maxed, yes. If resources are idle, scaling adds cost without fixing root cause.

Q: How do I debug without distributed tracing?

Structured logs with correlation ID and per-segment timestamps. Painful but workable. Prioritize adding tracing.

Q: Can CDN or WAF cause this?

Yes for edge-cached routes. Compare origin vs edge timing. Usually not for API JSON endpoints.

Q: What about GC pauses in Node.js?

Check event loop lag metric. Large heap + spike can stall requests. Use --heapsnapshot sparingly and review memory leaks.

Q: When is it not worth deep debugging?

If impact is below SLO and trending down post-rollback, document and schedule proper fix. Not every blip needs a 4-hour war room.