AI Agent Infrastructure and Failure Modes

An agent that can call APIs, query databases, and send emails is not a chatbot with extra steps. It is an autonomous process with non-deterministic control flow running in your production environment. I learned this when a staging agent looped for forty minutes calling the same internal API because the model kept "confirming" a step it had already completed.

Traditional reliability patterns - timeouts, retries, circuit breakers - still apply, but agents add new failure classes: runaway loops, tool misuse, context rot, privilege escalation through chained tool calls, and silent partial failures where the agent reports success but did not execute the critical step.

The Agent Execution Model

I model every agent run as a state machine, not a single LLM call. Each step is: read state → call model → parse tool invocation → execute tool → write state → repeat until terminal condition. Infrastructure must enforce bounds on every transition.

┌──────────┐    ┌──────────┐    ┌──────────┐    ┌──────────┐
│  State   │───▶│   LLM    │───▶│   Tool   │───▶│  State   │
│  Store   │    │  Router  │    │ Executor │    │  Update  │
└──────────┘    └──────────┘    └──────────┘    └──────────┘
     ▲                                               │
     └───────────────── loop (max N) ────────────────┘
     
Hard limits: max_steps, max_tokens, max_wall_clock, tool_allowlist

State lives in Redis for active runs (fast reads/writes, TTL) and MySQL for audit and replay. Never rely on conversation history alone - compress and store structured state the agent can reason over.

Failure Mode 1: Runaway Agent Loops

The most common production incident: the agent enters a loop calling the same tool with slightly different arguments. Without hard step limits, it burns tokens, hammers internal APIs, and triggers downstream rate limits.

My defenses: max_steps (typically 10-25 depending on task complexity), max_wall_clock_seconds (hard kill at 5-15 minutes), duplicate tool call detection (same tool + same args hash within a run triggers abort), and exponential cost alerting per run.

// Loop guard (conceptual)
function shouldAbort(run) {
  if (run.stepCount >= run.config.maxSteps) return "max_steps";
  if (Date.now() - run.startedAt > run.config.maxWallClockMs) return "timeout";
  const hash = toolCallHash(run.lastToolCall);
  if (run.recentHashes.filter(h => h === hash).length >= 2) return "duplicate_loop";
  return null;
}

Failure Mode 2: Tool Misuse and Over-Privileged Agents

Agents inherit the permissions of the tools you give them. If the agent can call an internal admin API, the model effectively has admin access - including to prompt injection attacks embedded in user content or retrieved documents.

I implement tool allowlists per agent type, scoped credentials (short-lived tokens with minimal scope), and a human-in-the-loop gate for destructive actions (delete, send external email, financial transactions).

Failure Mode 3: Context Rot and State Drift

Long agent runs accumulate context until the model loses track of the original goal. Symptoms: the agent summarizes instead of acting, repeats completed steps, or hallucinates tool results.

I combat context rot with structured state summaries injected each step (not full history), periodic "goal check" prompts that compare current state to the original objective, and compaction - replacing raw tool outputs with extracted facts stored in MySQL.

When state drift is detected (goal check fails twice), the run aborts gracefully and returns partial results with a clear "needs human review" flag.

Failure Mode 4: Silent Partial Failures

The agent returns a confident natural-language summary, but one tool call failed silently three steps ago. The user acts on incorrect information. This is worse than a 500 error because it erodes trust slowly.

Every tool execution returns a structured result with success/failure, error code, and retryability. The final response must reference tool results - I validate that claimed actions have matching successful tool records in the run log before returning to the user.

For customer-facing agents, I add a verification step: a lightweight second model call or rule engine that checks the response against the tool audit log.

Observability for Agent Runs

Standard APM is insufficient. I trace each agent run as a span tree: run → step → llm_call → tool_call. OpenTelemetry attributes include agent_id, run_id, step_number, tool_name, tokens_used, and latency_ms.

Dashboards track: runs per hour, abort reason breakdown, average steps to completion, cost per run, tool error rate, and human-escalation rate. Alerts fire on abort rate spikes and cost-per-run anomalies - not on individual slow runs.

Engineering Takeaways

• Model agents as bounded state machines with hard limits on steps, wall clock, and token spend.
• Detect runaway loops with duplicate tool-call hashing, not just step counts.
• Scope tool credentials minimally - the model is an untrusted operator.
• Require human approval for destructive or irreversible tool actions.
• Validate final responses against tool audit logs to catch silent partial failures.
• Trace agent runs as span trees with per-step cost and latency, not as single LLM requests.

Trade-offs I Would Consider

Option: Single powerful agent with many tools

• Pros: Flexible, handles open-ended tasks, simpler orchestration code.
• Cons: Large attack surface, harder to debug, higher loop and misuse risk.

Option: Specialized sub-agents with narrow tool sets

• Pros: Easier to test, tighter permissions, clearer failure boundaries.
• Cons: Orchestration overhead, handoff latency, more infrastructure to maintain.

What I Would Do in Production

Start with read-only tools in production. Add write tools one at a time with approval gates and rollback plans.

Run agent workloads on dedicated worker pools - never co-locate with user-facing API pods.

Store full run logs (state snapshots, tool I/O, model responses) in MySQL with 30-day retention for incident replay.

Build a "run inspector" internal tool that replays agent steps - your on-call will use it weekly.

Load-test agents with adversarial prompts (injection, loop bait, privilege escalation) before every major tool addition.

FAQ

Q: How many steps should I allow per agent run?

Start with 10 for task-specific agents and 25 for research-style agents. Measure p95 steps to completion in staging and set max_steps at 2x p95. Adjust monthly as tasks evolve.

Q: Should agent state live in Redis or a database?

Redis for active run state (fast, TTL-based cleanup). MySQL for audit logs and completed run records. Never lose audit data when Redis evicts.

Q: How do I handle agent failures gracefully for end users?

Return partial results with explicit status: completed, partial (with what succeeded), or failed (with retry option). Never fabricate a success narrative when tools failed.

Q: Can I use the same LLM gateway for agents and simple completions?

Share the gateway for rate limiting and provider management, but route agent calls to a separate queue with higher timeout and cost tracking. Agent runs have different SLOs and cost profiles.

Q: What is the biggest security mistake with agents?

Passing production database credentials into the model context so the agent can "query directly." Use a read-only proxy API with query allowlists instead.

Q: How do I test agents in CI?

Golden-path tests with mocked tools for regression, plus a nightly eval suite with real model calls scoring task completion rate. Block deploys on golden-path failures; alert on eval score drops.