Observability for LLM Applications

We had a dashboard showing 99.9% uptime on our AI summarization endpoint. Users were complaining the summaries were wrong. Uptime and quality are different problems, and traditional observability conflates them.

LLM observability needs three layers: infrastructure telemetry (latency, errors, cost), request-level tracing (prompt, model, tokens, provider), and quality evaluation (did the output meet the task?). Most teams stop at layer one and wonder why on-call cannot diagnose "the AI feels worse today."

The Three-Layer Model

Layer 1 is standard APM: request rate, error rate, latency percentiles, saturation. Layer 2 is LLM-specific tracing: model name, prompt version, input/output token counts, provider latency, cache hit/miss, and retrieval context IDs. Layer 3 is quality evaluation: task completion scores, hallucination flags, user feedback signals, and regression detection against golden datasets.

Layer 1: Infrastructure        Layer 2: LLM Trace           Layer 3: Quality
─────────────────────        ──────────────────           ─────────────────
p50/p99 latency              model, prompt_version        eval score
error rate (5xx)             input/output tokens          user thumbs up/down
throughput (RPM)             provider, cache_hit          hallucination flag
cost per hour                retrieval doc IDs            regression vs golden set
CPU/memory (workers)         tool calls (agents)          A/B variant comparison

All three layers feed the same OpenTelemetry pipeline. The difference is attributes and downstream analysis, not separate tooling.

OpenTelemetry Instrumentation

I instrument the LLM gateway and application services with OTel SDK. Every inference call creates a span with standard and custom attributes. Custom attributes are what make LLM traces useful.

// Span attributes for LLM calls
span.setAttributes({
  "llm.model": "gpt-4o",
  "llm.prompt_version": "v2.3",
  "llm.input_tokens": 847,
  "llm.output_tokens": 256,
  "llm.total_tokens": 1103,
  "llm.provider": "openai",
  "llm.cache_hit": false,
  "llm.latency_ms": 2340,
  "llm.cost_usd": 0.0124,
  "llm.tenant_id": "org_abc",
  "llm.feature": "doc-summary",
  "llm.finish_reason": "stop",
});

Prompt content itself is not stored in spans by default - too much PII risk and cardinality explosion. I store a prompt_hash and prompt_version, with full prompts available in a separate secure store for debugging sessions initiated by authorized engineers.

Metrics That Actually Matter

Beyond the RED method (Rate, Errors, Duration), I track LLM-specific metrics that drive business decisions and on-call actions.

Cost metrics are not optional. I have seen teams discover a prompt change doubled token usage two weeks after deploy because nobody was watching llm_cost_usd_total broken down by prompt_version.

Quality Evaluation in Production

Quality cannot be a pre-deploy-only concern. I run a continuous eval pipeline: sample 1-5% of production requests, score them asynchronously against task-specific criteria, and alert on score drops.

Scoring methods vary by use case. Classification tasks use exact match or F1. Summarization uses ROUGE or LLM-as-judge with a fixed evaluation prompt. Agent tasks check tool audit logs against claimed actions.

User feedback (thumbs up/down) is the cheapest quality signal. I capture it with the trace_id so feedback correlates to exact prompt version, model, and retrieval context. A spike in negative feedback on prompt v2.4 is actionable; a generic "AI is bad" complaint is not.

Dashboards and On-Call Runbooks

My LLM on-call dashboard has four rows: health (error rate, latency), cost (spend rate, token volume), quality (eval score, feedback ratio), and capacity (queue depth, worker utilization). During an incident, the on-call engineer can determine in 60 seconds whether this is a provider outage, a cost runaway, a quality regression, or a capacity problem.

Runbooks map alert types to actions. Provider 503 spike → check provider status page, enable fallback route. Eval score drop → compare prompt versions, check recent deploys, roll back prompt if correlated. Cost spike → identify tenant/feature, check for agent loops.

I store prompt versions in Git and tag every deploy with the version hash. Correlating a quality regression to a prompt change should take minutes, not a forensic investigation.

Engineering Takeaways

• LLM observability needs three layers: infrastructure, request tracing, and quality evaluation.
• Use OpenTelemetry with LLM-specific span attributes - model, tokens, cost, prompt_version.
• Never log raw prompts in production traces; use prompt_hash and version with secure debug access.
• Track cost metrics (llm_cost_usd_total) broken down by tenant, feature, and prompt_version.
• Run continuous eval on a sample of production traffic - quality regressions happen post-deploy.
• Correlate user feedback to trace_id for actionable quality signals.

Trade-offs I Would Consider

Option: Full prompt logging in traces (sampled)

• Pros: Fastest debugging, complete replay capability, easy root-cause analysis.
• Cons: PII risk, storage cost, compliance burden, high cardinality in trace backends.

Option: Prompt hash + version only, full content in secure debug store

• Pros: Lower PII exposure, manageable cardinality, debug access on demand with audit trail.
• Cons: Extra infrastructure for secure store, slight delay when debugging unfamiliar issues.

What I Would Do in Production

Instrument the LLM gateway first - it sees every call and gives org-wide visibility before individual services are instrumented.

Set up cost dashboards before launch, not after the first billing surprise. Daily spend by tenant is the minimum.

Deploy the eval pipeline sampling 1% of traffic within the first month. Increase to 5% once scoring is stable.

Add prompt_version as a required field in every inference request metadata. Block deploys that omit it.

Run monthly "observability drills" - inject a bad prompt version in staging and verify alerts fire within 15 minutes.

FAQ

Q: Should I use a dedicated LLM observability tool (LangSmith, Helicone, etc.)?

They accelerate time-to-value for layers 2 and 3. I use them for eval and prompt management while keeping infrastructure metrics in our existing Grafana/Datadog stack. Avoid two on-call dashboards.

Q: How do I handle PII in LLM traces?

Redact at ingestion: strip email, phone, and name patterns from span attributes. Store full content only in an encrypted debug store with role-based access and 7-day retention.

Q: What eval sample rate should I use?

Start at 1% for cost-sensitive scoring (LLM-as-judge). Increase for high-stakes features (medical, financial). User feedback captures 100% of explicit ratings at zero marginal cost.

Q: How do I trace agent runs with many steps?

Parent span for the run, child spans for each step (llm_call, tool_call). Aggregate cost and tokens at the parent span level. Dashboard on run-level metrics, drill down to step spans.

Q: Can I reuse existing APM for LLM apps?

For layer 1, yes. Layers 2 and 3 need LLM-specific attributes and eval pipelines. Bolt them onto your existing OTel collector rather than adopting a separate agent.