Envoy is a high-performance L7 (application-layer) proxy originally built at Lyft, now a CNCF graduated project. It’s the piece of infrastructure most service meshes are built on top of — Istio’s data plane is Envoy sidecars; most “service mesh” behavior you observe in practice (retries, circuit breaking, mTLS) is Envoy doing the work under a mesh control plane’s configuration.
Core proxy capabilities
| Capability | Why it matters |
|---|---|
| xDS APIs | Dynamic configuration — routes, clusters, and listeners update without a restart, pushed by a control plane |
| Load balancing | Round robin, least-request, ring hash, and more — chosen per upstream cluster |
| Circuit breaking | Stops sending requests to an upstream that’s already failing, before it cascades |
| Retries / timeouts | Bounded retry budgets and deadlines per route — see 8 — Deadline Propagation |
| HTTP/2 & gRPC native | First-class support, not bolted on |
| Native observability | Emits stats (compatible with Prometheus scraping), access logs, and distributed tracing headers out of the box |
| WASM extensibility | Custom filters without recompiling Envoy itself |
Client ──▶ Envoy (sidecar or edge)
│
├── Apply routing rules (xDS-configured)
├── Load balance across healthy upstreams
├── Enforce timeout / retry policy
├── Emit stats + trace spans
└── Forward to upstream service
Sidecar pattern (service mesh)
┌─────────────┐ ┌─────────────┐
│ Service A │ │ Service B │
│ + Envoy │◀──────▶│ + Envoy │
│ sidecar │ mTLS │ sidecar │
└─────────────┘ └─────────────┘
Every service gets an Envoy sidecar; application code talks to localhost, and the sidecar handles
mTLS, retries, and telemetry transparently. This is why Envoy is the concrete implementation behind
a lot of what 2 — Tail Latency describes in the abstract — hedged requests, retry budgets, and load
shedding are Envoy config, not application code, in a mesh deployment.
2026 development: Envoy AI Gateway
The newer, directly relevant piece for an AI-agent pipeline: Envoy AI Gateway reached v1.0 in June 2026 — the first open-source AI gateway built on Envoy Gateway, with production users including Bloomberg (who initiated the effort), Nutanix, and LY Corporation.
Agents / apps
│
▼
Envoy AI Gateway (v1.0)
├── Unified API across LLM providers (route by provider, not custom code per SDK)
├── Native MCP Gateway ← governs MCP traffic the way Envoy governs HTTP traffic
├── Token-aware traffic management
├── Centralized credential management
└── AI-native observability
│
▼
LLM providers / MCP servers ([[grafana-mcp]], [[mcp-toolbox]], ...)
The native MCP Gateway is the piece that connects directly back to the rest of this list: if an agent pipeline is calling multiple MCP servers (What is Grafana MCP, What is MCP Toolbox, Playwright MCP), Envoy AI Gateway is the infrastructure layer for centralizing auth, rate limits, and observability across all of those calls, rather than each MCP client managing that per-connection. Google has flagged deeper MCP security and spend-based governance as the next area of focus.
Where it fits
| Concern | Classic Envoy (service mesh) | Envoy AI Gateway |
|---|---|---|
| What it fronts | Service-to-service HTTP/gRPC traffic | LLM provider calls + MCP server traffic |
| Config model | xDS from a mesh control plane (Istio) | Envoy Gateway CRDs, AI-specific extensions |
| Observability payoff | Uniform stats/tracing across all services | Token-aware, AI-native traffic observability |
Why it’s on the backlog: it’s the traffic-governance layer missing from the rest of this list — every other tool here is a client or a server for agent/MCP traffic, and Envoy AI Gateway is where you’d centralize auth, rate limiting, and observability once more than one agent is calling more than one MCP server.
Local graph
Linked from 17 notes
What is Istio
CNCF-graduated (July 2023) service mesh — sidecar model plus the newer sidecar-less ambient mode (stable since 1.24), now extending into AI traffic via the Gateway API Inference Extension and 2026's Ambient Multicluster beta.
What is a Reverse Proxy (NGINX)
Forward vs reverse proxy, and NGINX as the canonical implementation — event-driven architecture, core proxy capabilities, its config model, and the 2026 shift toward NGINX Gateway Fabric as a Kubernetes Gateway API implementation.
What is FluxCD
CNCF-graduated GitOps toolkit for Kubernetes, built as a set of composable controllers (source, kustomize, helm, notification, image-automation) rather than one monolithic app — Flagger is its progressive-delivery counterpart to Argo Rollouts.
OSI Layer Model (L1-L7)
What L1 through L7 actually mean, why 'L7 gateway' and 'L4 load balancer' are load-bearing terms in system design interviews, and why this numbering is unrelated to the pipeline's own Layer 1/2/3 architecture labels.
TLS Offload
Terminating TLS at the ingestion frontier instead of in every backend pod — why it's a Layer 1 responsibility, what it costs in defense-in-depth, and how mTLS re-encryption closes the gap.
4 — Auto vs. Manual Instrumentation
Four ways a span gets created — hand-written, framework-level auto-instrumentation, eBPF, and service-mesh sidecar capture — and the trade-off between code changes and business context each one makes.
What is eBPF
Extended Berkeley Packet Filter — sandboxed, verified bytecode run inside the Linux kernel without a module or a restart. The foundation under Cilium, Grafana Beyla, and Pyroscope: zero-instrumentation traces, metrics, and continuous profiling.
02 — Strangler Fig
Incrementally replace a legacy system by routing new functionality to a new implementation while the old system continues to run. Named after the fig tree that grows around and eventually replaces its host.
07 — Circuit Breaker
A state machine (Closed → Open → Half-Open) that stops calls to a failing dependency before they cascade. The foundational resilience pattern for distributed systems.
08 — Retry with Exponential Backoff and Jitter
Retry transient failures with exponentially increasing wait times and randomised jitter to prevent thundering-herd recovery storms. The foundational pattern for resilient RPC.
09 — Bulkhead
Partition resources (thread pools, connection pools, semaphores) so that a failure or overload in one partition cannot exhaust resources for others. Limits blast radius.
01 — Sidecar
Co-locate a helper container with the application container to handle cross-cutting concerns — TLS, observability, auth, retries — without modifying application code.
Related notes
What is Istio
CNCF-graduated (July 2023) service mesh — sidecar model plus the newer sidecar-less ambient mode (stable since 1.24), now extending into AI traffic via the Gateway API Inference Extension and 2026's Ambient Multicluster beta.
What is Cortex (cortexproject)
CNCF Incubating, horizontally-scalable multi-tenant long-term storage for Prometheus — the project Grafana Mimir forked from in 2022, still maintained as the vendor-neutral, community-governed alternative once Grafana Labs redirected engineering effort to Mimir.
What is HolmesGPT
Robusta.dev's open-source SRE agent (CNCF Sandbox) for investigating production incidents across Kubernetes, VMs, cloud services, and databases — an agentic tool-calling loop over 70+ toolsets, not a chatbot or RAG system, with a proactive 'operator mode' that monitors and opens fix PRs without a human trigger.
What is Fluent Bit
CNCF-graduated, C-written log/metrics/trace forwarder — Fluentd's lightweight sibling, the de facto node-level log-collection DaemonSet in most Kubernetes clusters, and Grafana Alloy's main incumbent competitor for that slot.