8. Quick-Reference Cheat Sheet

Appears in: Telemetry Ingestion Pipeline — this is §8 of the full design, split into its own file so the root stays a table of contents.

8. Quick-Reference Cheat Sheet

Fan-in at scale          → stateless gateway + persistent gRPC connections + connection pooling
Backpressure             → gRPC RESOURCE_EXHAUSTED (429) → agent WAL absorbs burst
Burst absorption         → Kafka between gateway and processor; 2-4h retention
Cardinality enforcement  → approximate counting (HyperLogLog) at processor; reject early
Tail sampling            → hash-partition by trace_id in Kafka; assemble per-partition
Multi-tenancy            → enforce at: network / auth / gateway / processor / storage (all layers)
Deduplication            → at-least-once + TSDB fingerprint dedup (metrics), hash dedup (logs)
Out-of-order samples     → Mimir out-of-order ingestion window (up to 1h)
End-to-end SLO           → synthetic canary batch every 60s through the full pipeline
Self-observability       → consumer lag (Kafka) is the single most important operational metric
Delta vs cumulative      → enforce cumulative at agent SDK; deltaToCumulative processor = stateful + restarts cause counter resets
k8s enrichment           → enrich at agent (DaemonSet downward API); avoid per-span k8s API calls
Data tiering             → ingester (hot, 2h) → object store (cold) → compactor (merge + dedup RF=3 blocks)
Global topology          → regional writes + async replication → global query tier; never single global cluster at 10M agents
Kafka producer gotcha    → max.message.bytes=1MB default; split large trace batches at gateway before produce