# Production Agent Systems
All Production Agent Systems notes →1. AI Observability Fundamentals
Building the metrics, logs, traces, dashboards, and alerting an agent needs so its own operators can tell when it is misbehaving, not just when it is down.
2. Agent Tracing
What AI observability adds on top of standard OTel instrumentation — spans around LLM calls and tool calls, tracing full agent execution, and capturing token usage as a first-class attribute.
3. Token Metrics
Treating input, output, and cached token counts as first-class SLIs — per-request, per-tenant, and per-model dashboards, the token-to-cost conversion, and alerting on token-count anomalies as an early signal of prompt drift or a runaway loop.
4. Prompt Observability
Capturing and versioning the exact prompt (system, few-shot, and injected context) sent on every call so a regression can be traced to a specific prompt-template change, with redaction rules for what's safe to log versus what must be hashed or dropped.
5. Memory Observability
Instrumenting what an agent actually retrieved from long-term memory on each turn — retrieval hit rate, relevance and similarity score distributions, and staleness of cached embeddings — so memory-driven hallucinations can be traced to a specific bad retrieval instead of guessed at.
6. Tool Invocation Metrics
Per-tool latency, error rate, and call-volume dashboards plus argument-validation failure tracking, so a misbehaving tool integration shows up as a metrics anomaly before it shows up as a user-facing failure.
7. AI Logging
Structured logging conventions for an agent's reasoning trace (thought, action, observation) that balance debuggability against the cost and privacy risk of logging full prompts and completions at high volume.
12. AI SLOs
SLOs, error budgets, incident response, and cost optimization applied to an agent workload — including token cost as a first-class SLI and degrade-to-human-handoff as an error-budget policy.