# Production Agent Systems
All Production Agent Systems notes →1. Guardrails
Input and output validation layers that constrain what an agent can say or do — schema-constrained outputs, content-safety classifiers on both directions, and where guardrail checks sit in the request path so they add acceptable latency without becoming a bypassable afterthought.
2. Prompt Injection
Authentication, authorization, and secrets management for an agent, plus the genuinely agent-specific threat: prompt injection defense and data privacy in a tool-calling loop.
3. Jailbreak Prevention
Defending against adversarial prompts designed to override system instructions — prompt-injection-resistant system prompt structuring, delimiter and instruction-hierarchy techniques, and red-teaming the agent against known jailbreak corpora before it ships.
4. Sandboxing
Isolating code-execution and shell-access tools from the host and from each other — container and VM-level isolation, filesystem and network egress restrictions, and resource limits that stop a single tool call from taking down the runtime it shares with other tenants.
5. Identity & Authentication
How an agent authenticates itself to downstream systems and how end users authenticate to the agent — service-to-service identity (mTLS, workload identity) versus delegated user identity (OAuth token pass-through) when the agent acts on a user's behalf.
6. Authorization & Permissions
Scoping what actions an agent is allowed to take on a user's or tenant's behalf — least-privilege tool permissions, per-tool RBAC/ABAC policy, and the distinction between what the LLM is capable of requesting versus what the runtime will actually authorize.
7. Secrets Management
Keeping API keys, database credentials, and provider tokens out of prompts and tool code — vault-backed secret injection at call time, rotation without redeploying the agent, and why a secret ever appearing in a logged prompt is a sev-1, not a nit.
8. Human Approval Systems
Human-in-the-loop gates for high-stakes agent actions — designing the approval UI/API contract, timeout and escalation behavior when no human responds, and audit-trail requirements for what was approved and by whom.
9. Compliance
Mapping agent behavior to regulatory obligations — data residency for prompts and logs, retention and deletion policy for conversation history, and the audit evidence a compliance review will actually ask for (GDPR, SOC 2, industry-specific rules).
10. AI Governance
Organizational policy for what agents are allowed to be built, what models they may use, and who signs off before an agent goes to production — model risk review, an internal registry of approved agents and tools, and escalation paths when a team wants an exception.
11. Failure Recovery
How an agent detects and recovers from failure mid-task — partial-completion checkpointing, retry-with-backoff versus fail-fast policy per failure class, and distinguishing a transient provider error from a genuine task failure that needs human escalation.
12. Rollback Strategies
Reverting a bad prompt, model, or tool-schema change quickly — versioned prompt and model artifacts as first-class deploy units, canary and shadow rollout patterns for agent changes, and the rollback trigger thresholds tied to online-evaluation regressions.
1. Agent Runtime
The execution substrate that hosts an agent's reasoning loop — process model, container vs. serverless tradeoffs, cold-start latency, and how the runtime enforces max-iteration and timeout limits so a stuck agent doesn't run (and bill) forever.
2. Session Management
How an agent tracks conversation identity across turns and channels — session ID generation, TTL and idle-timeout policy, session affinity in a load-balanced fleet, and the handoff problem when a user resumes a stale session on a different agent instance.
3. State Persistence
Where agent state lives between turns and after a crash — durable stores for message history and working memory (Redis, Postgres, DynamoDB), write-ahead patterns for mid-tool-call failures, and the tradeoff between snapshotting full state vs. replaying an event log.
4. Event Streaming
Publishing agent lifecycle events (tool calls, state transitions, token usage) onto a stream like Kafka or Kinesis so downstream consumers — observability, billing, audit — can react without coupling to the agent's request path.
5. Message Queues
Decoupling long-running agent tasks from the synchronous request path using queues (SQS, RabbitMQ, Celery) — at-least-once delivery semantics, idempotency keys for tool calls, dead-letter queues for tasks that exhaust retries, and backpressure when the LLM provider rate-limits.
6. Workflow Engines
Orchestrating multi-step, long-running agent workflows with durable execution engines (Temporal, AWS Step Functions, LangGraph persistence) so a workflow survives process restarts and resumes exactly where it left off after a crash.
7. Distributed Execution
Running agent workloads across multiple nodes — sharding by session or tenant, coordinating shared state without a single point of failure, and the consistency tradeoffs when two agent instances could act on the same conversation concurrently.
8. Scheduling
Placing agent workloads onto compute — priority queues for interactive vs. batch agent runs, autoscaling triggers tied to queue depth or token throughput rather than CPU, and preemption policy when a high-priority request needs capacity held by a long-running agent.
9. Scaling Strategies
Stateless vs. stateful agent design, offloading long-running work to queues and background tasks, horizontal scaling, and rate limiting against the LLM provider.
10. Multi-Tenant Architectures
Isolating tenants sharing an agent platform — per-tenant rate limits and token budgets, noisy-neighbor containment, data isolation for prompts, memory, and logs, and the pool-vs-silo tradeoff for LLM provider capacity.
11. High Availability
Designing an agent platform to survive component failure — redundant LLM provider routing with failover, health checks that account for degraded (slow but not down) model endpoints, and graceful degradation to a smaller model or cached response under partial outage.
12. Disaster Recovery
RTO and RPO targets for an agent platform's stateful components — conversation history, vector memory, prompt and model version registry — cross-region failover for the control plane, and the recovery drill that validates a full region loss doesn't silently corrupt in-flight tool calls.
1. AI Observability Fundamentals
Building the metrics, logs, traces, dashboards, and alerting an agent needs so its own operators can tell when it is misbehaving, not just when it is down.
2. Agent Tracing
What AI observability adds on top of standard OTel instrumentation — spans around LLM calls and tool calls, tracing full agent execution, and capturing token usage as a first-class attribute.
3. Token Metrics
Treating input, output, and cached token counts as first-class SLIs — per-request, per-tenant, and per-model dashboards, the token-to-cost conversion, and alerting on token-count anomalies as an early signal of prompt drift or a runaway loop.
4. Prompt Observability
Capturing and versioning the exact prompt (system, few-shot, and injected context) sent on every call so a regression can be traced to a specific prompt-template change, with redaction rules for what's safe to log versus what must be hashed or dropped.
5. Memory Observability
Instrumenting what an agent actually retrieved from long-term memory on each turn — retrieval hit rate, relevance and similarity score distributions, and staleness of cached embeddings — so memory-driven hallucinations can be traced to a specific bad retrieval instead of guessed at.
6. Tool Invocation Metrics
Per-tool latency, error rate, and call-volume dashboards plus argument-validation failure tracking, so a misbehaving tool integration shows up as a metrics anomaly before it shows up as a user-facing failure.
7. AI Logging
Structured logging conventions for an agent's reasoning trace (thought, action, observation) that balance debuggability against the cost and privacy risk of logging full prompts and completions at high volume.
12. AI SLOs
SLOs, error budgets, incident response, and cost optimization applied to an agent workload — including token cost as a first-class SLI and degrade-to-human-handoff as an error-budget policy.
13. Circuit Breakers & Timeout Strategies
The containment mechanics one level below failure-recovery policy: circuit breakers that stop one failing tool or sub-agent call from cascading through a run, timeout budgets allocated across a multi-hop tool chain, deadlock/oscillation detection between cooperating agents, and the runaway-loop breakers that cap a retry-or-replan cycle before it becomes a cost incident.
14. Trust & Explainability
The human-factors problem underneath every approval gate — why users either over-trust an agent past its actual competence or route around it entirely, what it takes for an agent's confidence signal to mean something, and the difference between a post-hoc justification and a real causal trace a reviewer can actually evaluate.
1. Latency Optimization
Reducing end-to-end agent response time — model selection tradeoffs (smaller/faster vs. larger/better), reducing tool-call round trips, and where time-to-first-token versus total completion time matters for perceived responsiveness.
2. Parallel Execution
Running independent tool calls and sub-agent tasks concurrently instead of sequentially — fan-out/fan-in patterns, bounding concurrency against provider rate limits, and the correctness hazards of parallel writes to shared agent state.
3. Streaming Optimization
Streaming partial LLM output to the user as tokens generate — chunking strategy for tool-call detection mid-stream, and buffering tradeoffs between responsiveness and being able to cancel or rewrite a bad partial response.
4. Token Optimization
Reducing token consumption without losing task quality — prompt compression, few-shot example pruning, and choosing when to summarize versus truncate conversation history before it's sent to the model.
5. Context Optimization
Deciding what actually belongs in the context window on a given turn — relevance-ranked retrieval over raw dump, context window budget allocation across system prompt, history, and retrieved documents, and the accuracy cost of over-stuffing context versus under-providing it.
6. Semantic Caching
Caching LLM responses by semantic similarity of the input rather than exact match — embedding-based cache key generation, similarity threshold tuning to avoid serving a wrong-but-close cached answer, and cache invalidation when underlying data changes.
7. Response Caching
Exact-match and prefix caching for repeated agent requests — provider-level prompt caching (e.g., cached system prompts) versus application-level response caching, and TTL policy for cache entries that reference time-sensitive data.
8. Cost Engineering
The engineering levers that actually move agent spend — caching strategy, batching, model routing/tiering, and inference optimization — plus cost attribution by tenant/feature and budget alerting as a first-class signal, feeding the executive ROI numbers in Part 01 of Agentic AI: Projects & Engineering Mastery rather than duplicating them.
9. Capacity Planning
Forecasting compute and provider-quota needs for an agent platform — translating expected request volume into token throughput requirements, provider rate-limit headroom planning, and scaling lead time for a traffic spike that can't be absorbed instantly.
10. Performance Benchmarking
Establishing repeatable load tests for an agent platform — synthetic traffic generation that mimics real tool-call patterns, identifying the actual bottleneck (LLM provider latency vs. tool execution vs. queueing) under load, and regression-testing performance across releases.
1. Designing Internal AI Platforms
Covers the reference architecture for an internal AI platform team - shared inference layer, tool/agent registry, and paved-road SDKs - so product teams build agents without re-solving auth, observability, and deployment each time.
2. Agent SDKs
Compares the design trade-offs of building a first-party agent SDK (LangGraph, custom Python/TypeScript wrappers) against adopting a vendor SDK, focused on API stability, versioning, and abstraction leakage at scale.
3. Agent APIs
Defines the contract layer for exposing agents as internal APIs - request/response schemas, streaming vs synchronous invocation, idempotency keys, and versioning strategy for breaking prompt or tool changes.
4. Plugin Ecosystems
Examines how to design a plugin/extension model for agents (tool manifests, capability declarations, sandboxed execution) so third-party or team-owned tools can be registered without a platform team code change.
5. Agent Registries
Covers building a central registry of agents and tools with ownership metadata, capability tags, and discovery APIs, mirroring a service catalog but for autonomous and semi-autonomous agents.
6. AI Gateways
Explains the AI gateway pattern - a single ingress for model routing, rate limiting, cost attribution, and prompt/response logging across multiple LLM providers - and how it differs from a traditional API gateway.
7. Multi-Model Infrastructure
Covers routing and fallback strategy across multiple foundation models (cost tier, latency tier, capability tier), including circuit breakers when a provider degrades and shadow-testing a model swap before cutover.
8. Deployment Strategies
Applies canary, blue-green, and shadow-deployment patterns specifically to agent releases, where a bad deploy can mean bad tool calls or unsafe actions rather than just bad HTTP responses.
9. Platform Operations
Covers the day-2 operating model for an AI platform - on-call ownership boundaries between platform and product teams, cost governance, and the SLOs that keep a shared agent platform reliable.
Production Agent Systems
A book-shaped table of contents for Production Agent Systems: the runtime substrate, observability, reliability/security/governance, performance/cost engineering, and platform engineering underneath every agent in production. Book 4 of the AI Systems Engineering series.