Logs vs Metrics vs Traces

Every observability conversation eventually lands here: should we log it, metric it, or trace it? The answer is often all three-but each signal has a cost profile and a question it answers best. Using logs for alerting on everything is expensive. Using metrics alone for root cause is incomplete.

At Edmingle we standardize on structured logs to Loki/CloudWatch, Prometheus metrics, and OpenTelemetry traces on critical API paths. This post is the decision framework I give teams when they ask what to instrument and where.

The Three Pillars in One Sentence Each

• Logs: discrete events with context-what happened on this request at this timestamp.
• Metrics: aggregated numbers over time-how many, how fast, how full.
• Traces: causal paths across services-where did time go in this request chain.

Request enters API
    │
    ├── Trace: span timeline across services (WHERE time went)
    ├── Metrics: http_requests_total++, latency histogram (HOW MUCH, HOW FAST)
    └── Logs: "payment failed: insufficient_funds user=123" (WHAT happened)

When to Use Logs

Logs excel at debugging specific failures: stack traces, validation errors, business logic decisions, audit trails. They carry high cardinality detail (user IDs, order IDs) that metrics should not.

logger.info({
  event: 'enrollment_created',
  userId,
  courseId,
  correlationId: req.headers['x-request-id'],
  durationMs: Date.now() - start,
});

Rules: structured JSON always, correlation ID on every line, consistent field names, appropriate log levels. ERROR means someone may page; INFO is normal business events.

Avoid logging secrets, full PII, or high-cardinality fields as metric labels-logs are the right place for request-scoped detail.

When to Use Metrics

Metrics answer trend and threshold questions cheaply at scale: RPS, error rate, p99 latency, queue depth, CPU, connection pool usage. They aggregate away individual request detail.

// Prometheus-style counters and histograms
httpRequestsTotal.inc({ method: 'POST', route: '/enroll', status: '500' });
httpRequestDuration.observe({ route: '/enroll' }, durationSeconds);

Alert on metrics: SLO burn, error rate spike, saturation. Dashboards for ops review. Metrics cardinality discipline is critical-never put userId as a label.

RED method for services: Rate, Errors, Duration. USE for resources: Utilization, Saturation, Errors.

When to Use Traces

Traces connect spans across services, queues, and databases. They show parallel work, slow downstream calls, and N+1 patterns logs cannot reveal efficiently.

Sample in production-100% tracing on high-traffic APIs is costly. Tail-based sampling (keep errors and slow traces) gives better ROI.

Propagate W3C trace context (traceparent header) from edge to database client. Broken propagation makes traces useless fragments.

// OpenTelemetry auto-instrumentation + manual spans for business steps
const span = tracer.startSpan('validate_enrollment');
try {
  await validateEnrollment(userId, courseId);
} finally {
  span.end();
}

Comparison Table

Correlating All Three

The magic is correlation ID and trace ID in logs, exemplars linking metrics to traces (where supported), and consistent service.name resource attributes in OpenTelemetry.

Incident workflow: alert fires on metric → open dashboard → identify spike window → find trace exemplar or search logs by correlation ID → root cause span or log line.

Without correlation, you have three silos and three tabs of pain.

OpenTelemetry as Unifier

OpenTelemetry provides SDKs for traces, metrics, and logs (logs bridge still maturing). One instrumentation effort exports to vendor backends: Grafana Cloud, Datadog, Honeycomb, AWS X-Ray.

Start with auto-instrumentation for HTTP and database drivers. Add manual spans only for critical business steps that auto-instrument misses.

Engineering Takeaways

• Logs for discrete events and forensics; metrics for trends and alerts; traces for latency path analysis.
• Never use unbounded cardinality in metric labels-put request-scoped detail in logs.
• Correlation ID in every log line and trace propagation across services is non-negotiable.
• Sample traces in prod; keep errors and slow requests via tail-based sampling.
• Alert on metrics, debug with logs and traces-not the other way around.
• OpenTelemetry reduces triple-instrumentation toil when adopted consistently.

Trade-offs I Would Consider

Option: Logs-only observability (ELK stack, grep alerts)

• Pros: Rich detail, familiar tooling, good for audit.
• Cons: Expensive at scale, slow queries, alerting on log patterns is brittle.

Option: Metrics + traces + structured logs (three pillars)

• Pros: Right tool per question, cheaper alerting, faster incident response.
• Cons: More instrumentation discipline, multiple backends or unified vendor cost.

What I Would Do in Production

Enforce structured JSON logging library in shared Node.js package; block console.log in CI for services.

Standard RED dashboards per service with deploy annotations and SLO lines.

Enable OTel auto-instrumentation in base Docker image; document manual span examples.

Retention: logs 14-30 days hot, metrics 13 months, traces 7 days with long-term sample archive for errors.

Run quarterly observability audits: random incident replay using only telemetry-find gaps.

FAQ

Q: Do I need all three from day one?

Start with structured logs and basic metrics (RED). Add traces when cross-service latency debugging hurts-usually after 3+ services.

Q: What about events vs logs?

Events are often structured logs or audit stream entries. Same rules: schema, correlation, retention policy.

Q: Can traces replace logs?

No. Traces lack business context and error messages spans do not capture. Complement, not replace.

Q: How much trace sampling?

Head-based 1-10% for volume; tail-based keep 100% of errors and p99+ requests. Adjust by traffic and budget.

Q: Prometheus vs CloudWatch metrics?

Prometheus model (pull, PromQL) dominates K8s. CloudWatch fine on AWS-native stacks. OTel exporter abstracts either.

Q: What is log level guidance?

DEBUG off in prod default. INFO business events. WARN recoverable issues. ERROR needs human attention or auto-ticket.