Production Agent Systems

Production Agent Systems

Running agents in production, for Principal & Staff Engineers (L6/L7)

This book assumes you already know how to build and evaluate an agent — see Building & Evaluating Agents first. If this were a book, this page is the table of contents. It is organized as a learning journey — from the runtime substrate through observability, reliability/security/governance, performance/cost engineering, and the internal platform other teams build agents on top of. Each chapter links out to the framework, platform, and SRE notes that already exist elsewhere in this wiki instead of duplicating them. Unwritten chapters are listed as stub rows, not empty files.

Parts

00 — Production Infrastructure

The runtime substrate underneath every agent in production. See 4-archive/h-aiops/02-playbooks/*.yaml for real declarative playbook/orchestration artifacts.

• Chapter 1. Agent Runtime — (stub) — The execution substrate hosting an agent’s reasoning loop — process model, container vs. serverless, cold-start latency, max-iteration/timeout enforcement.
• Chapter 2. Session Management — (stub) — Session ID generation, TTL/idle-timeout policy, session affinity, and resuming a stale session on a different instance.
• Chapter 3. State Persistence — (stub) — Durable stores for message history and working memory, write-ahead patterns, and snapshotting vs. event-log replay.
• Chapter 4. Event Streaming — (stub) — Publishing agent lifecycle events onto a stream (Kafka, Kinesis) so observability/billing/audit can react without coupling to the request path.
• Chapter 5. Message Queues — (stub) — Decoupling long-running agent tasks from the synchronous request path — delivery semantics, idempotency keys, dead-letter queues.
• Chapter 6. Workflow Engines — (stub) — Orchestrating multi-step, long-running agent workflows with durable execution engines (Temporal, Step Functions) that resume after a crash.
• Chapter 7. Distributed Execution — (stub) — Running agent workloads across multiple nodes — sharding by session/tenant, coordinating shared state.
• Chapter 8. Scheduling — (stub) — Placing agent workloads onto compute — priority queues, autoscaling tied to queue depth/token throughput, preemption policy.
• Chapter 9. Scaling Strategies — Stateless vs. stateful agent design, offloading long-running work to queues, horizontal scaling, and rate limiting against the LLM provider.
• Chapter 10. Multi-Tenant Architectures — (stub) — Per-tenant rate limits and token budgets, noisy-neighbor containment, and the pool-vs-silo tradeoff for provider capacity.
• Chapter 11. High Availability — (stub) — Redundant LLM provider routing with failover, degraded-endpoint health checks, and graceful degradation under partial outage.
• Chapter 12. Disaster Recovery — (stub) — RTO/RPO targets for stateful components, cross-region failover, and the recovery drill that validates a region loss doesn’t corrupt in-flight tool calls.

01 — Observability

Watching the agent itself, not just the systems it touches. This Part covers the telemetry substrate — metrics, traces, logs — while Building & Evaluating Agents’ Part 02 covers what you do with the scores that substrate feeds. Builds on observability’s Instrumentation Part; see AIOps / Agentic RCA for the conceptual frame this Part’s chapters implement.

• Chapter 1. AI Observability Fundamentals — (stub) — Building the metrics, logs, traces, dashboards, and alerting an agent needs so its operators can tell when it is misbehaving, not just when it is down.
• Chapter 2. Agent Tracing — (stub) — What AI observability adds on top of standard OTel instrumentation — spans around LLM and tool calls, tracing full agent execution, token usage as a first-class attribute.
• Chapter 3. Token Metrics — (stub) — Treating input/output/cached token counts as first-class SLIs, and alerting on anomalies as an early signal of prompt drift or a runaway loop.
• Chapter 4. Prompt Observability — (stub) — Capturing and versioning the exact prompt sent on every call, with redaction rules for what’s safe to log.
• Chapter 5. Memory Observability — (stub) — Instrumenting what an agent actually retrieved from long-term memory — hit rate, relevance score distributions, embedding staleness.
• Chapter 6. Tool Invocation Metrics — (stub) — Per-tool latency, error rate, and call-volume dashboards, plus argument-validation failure tracking.
• Chapter 7. AI Logging — (stub) — Structured logging conventions for an agent’s reasoning trace, balancing debuggability against cost and privacy risk.
• Chapter 8. AI SLOs — (stub) — SLOs, error budgets, incident response, and cost optimization applied to an agent workload — token cost as a first-class SLI and degrade-to-human-handoff as an error-budget policy.

02 — Reliability, Security & Governance

Threat model and safety net once an agent has real tool access. Closes with the two review-flagged gaps: the specific failure-containment mechanics (circuit breakers, timeouts, runaway-loop prevention) that a general “failure recovery” chapter doesn’t fully cover, and the human-factors side of trust — calibration and explainability — as a sibling to the existing approval-workflow chapter rather than a duplicate of it.

• Chapter 1. Guardrails — Input/output validation layers that constrain what an agent can say or do — schema constraints, content-safety classifiers.
• Chapter 2. Prompt Injection — Authentication, authorization, and secrets management for an agent, plus the genuinely agent-specific threat: prompt injection defense and data privacy in a tool-calling loop.
• Chapter 3. Jailbreak Prevention — Prompt-injection-resistant system prompt structuring, instruction-hierarchy techniques, and red-teaming against known jailbreak corpora.
• Chapter 4. Sandboxing — Isolating code-execution and shell-access tools — container/VM isolation, egress restrictions, resource limits.
• Chapter 5. Identity & Authentication — Service-to-service identity versus delegated user identity when an agent acts on a user’s behalf.
• Chapter 6. Authorization & Permissions — Least-privilege tool permissions, per-tool RBAC/ABAC policy, and what the LLM can request versus what the runtime authorizes.
• Chapter 7. Secrets Management — Vault-backed secret injection at call time, rotation without redeploying the agent.
• Chapter 8. Human Approval Systems — Approval UI/API contract design, timeout/escalation behavior, and audit-trail requirements.
• Chapter 9. Compliance — Data residency, retention/deletion policy, and the audit evidence a compliance review actually asks for.
• Chapter 10. AI Governance — Organizational policy for what agents may be built, what models they may use, and who signs off before production.
• Chapter 11. Failure Recovery — Partial-completion checkpointing, retry-with-backoff versus fail-fast policy, and distinguishing transient errors from genuine failures.
• Chapter 12. Rollback Strategies — Versioned prompt/model artifacts as deploy units, canary/shadow rollout, and rollback triggers tied to evaluation regressions.
• Chapter 13. Circuit Breakers & Timeout Strategies — The failure-containment mechanics one level below Failure Recovery: cascading-failure prevention, timeout budgets across a multi-hop tool chain, deadlock and oscillation detection between cooperating agents, and runaway-loop circuit breakers.
• Chapter 14. Trust & Explainability — The human side of reliability that Human Approval Systems (Chapter 8) doesn’t cover on its own: calibrating user trust to actual agent competence, confidence calibration, and explaining a decision after the fact well enough that a human can actually evaluate it, not just rubber-stamp it.

03 — Performance & Cost Engineering

Making an agent fast and affordable at scale.

• Chapter 1. Latency Optimization — (stub) — Model selection tradeoffs, reducing tool-call round trips, and time-to-first-token versus total completion time.
• Chapter 2. Parallel Execution — (stub) — Running independent tool calls and sub-agent tasks concurrently — fan-out/fan-in, bounding concurrency against rate limits.
• Chapter 3. Streaming Optimization — (stub) — Chunking strategy for tool-call detection mid-stream and buffering tradeoffs.
• Chapter 4. Token Optimization — (stub) — Prompt compression, few-shot example pruning, and summarize-vs-truncate decisions.
• Chapter 5. Context Optimization — (stub) — Relevance-ranked retrieval over raw dump, and the accuracy cost of over- versus under-stuffing context.
• Chapter 6. Semantic Caching — (stub) — Caching by semantic similarity rather than exact match, and similarity-threshold tuning to avoid serving a wrong-but-close cached answer.
• Chapter 7. Response Caching — (stub) — Exact-match and prefix caching, provider-level prompt caching versus application-level response caching.
• Chapter 8. Cost Engineering — The engineering levers for agent cost — caching, batching, model routing/tiering, and inference optimization — attributed by tenant/feature with budget alerts on spend. Paired with the executive ROI framing in AI Economics & ROI (Part 01 of Agentic AI: Projects & Engineering Mastery), which this chapter feeds rather than duplicates.
• Chapter 9. Capacity Planning — (stub) — Translating request volume into token throughput requirements and provider rate-limit headroom planning.
• Chapter 10. Performance Benchmarking — (stub) — Synthetic load testing that mimics real tool-call patterns and identifies the actual bottleneck under load.

04 — AI Platform Engineering

Building the internal platform other teams build agents on top of — golden paths and paved roads, applied to agent workloads specifically.

• Chapter 1. Designing Internal AI Platforms — (stub) — The reference architecture for an internal AI platform team — shared inference layer, tool/agent registry, paved-road SDKs.
• Chapter 2. Agent SDKs — (stub) — Building a first-party agent SDK versus adopting a vendor SDK — API stability, versioning, abstraction leakage.
• Chapter 3. Agent APIs — (stub) — The contract layer for exposing agents as internal APIs — schemas, streaming vs. synchronous invocation, idempotency, versioning.
• Chapter 4. Plugin Ecosystems — (stub) — Designing a plugin/extension model — tool manifests, capability declarations, sandboxed execution — so third-party tools register without a platform code change.
• Chapter 5. Agent Registries — (stub) — A central registry of agents and tools with ownership metadata, capability tags, and discovery APIs.
• Chapter 6. AI Gateways — (stub) — A single ingress for model routing, rate limiting, cost attribution, and prompt/response logging across providers.
• Chapter 7. Multi-Model Infrastructure — (stub) — Routing and fallback across foundation models by cost/latency/capability tier, with circuit breakers and shadow-testing.
• Chapter 8. Deployment Strategies — (stub) — Canary, blue-green, and shadow-deployment patterns for agent releases, where a bad deploy means bad tool calls, not just bad HTTP responses.
• Chapter 9. Platform Operations — (stub) — The day-2 operating model — on-call ownership between platform and product teams, cost governance, platform SLOs.

Metadata

Author	Amit Singh
Scope	production-agent-systems