Appears in: Telemetry Ingestion Pipeline — §3, Deep Dives — this is §3.6.
3.6 Multi-Tenancy
This is the principal-level differentiator question: “how do you add a new tenant without affecting existing ones?”
Isolation layers:
flowchart LR
NET["Network\nSNI routing\n/v1/{tenant} path prefix"]
AUTH["Auth\nPer-tenant API key\nor mTLS cert"]
GW["Gateway\nExtract tenant ID\nadd as label/attribute"]
BUF["Buffer\nTenant ID in message header\nper-tenant topic or consumer-filtered"]
PROC["Processing\nPer-tenant cardinality budget\nsampling policy · retention override"]
STORE["Storage\nMimir: X-Scope-OrgID\nLoki: X-Scope-OrgID"]
NET --> AUTH --> GW --> BUF --> PROC --> STORE
Quota enforcement points:
- Gateway — connection-level and request-rate limits (coarse)
- Processor — cardinality budget, series-per-minute cap (fine-grained)
- Storage — Mimir/Loki/Cortex all have per-tenant limits (bytes-per-second, active series cap) as a safety net
Always enforce at multiple layers. Never rely on a single enforcement point.
Local graph
Linked from 10 notes
7 — Multi-Tenancy
Two separate guarantees hiding under one name — data isolation and performance fairness — and the tenant identification, quota enforcement, and selective backpressure that make both hold under shared infrastructure.
3 — Prometheus in the Observability Ecosystem
Where Prometheus sits in the CNCF landscape — its pull-based cloud-native origins, its companion projects, and where this book does (and doesn't yet) connect it to the wider stack.
2 — Long-Term Storage
Why single-node Prometheus has no built-in long-term-storage or HA story, and how remote_write receivers like Thanos, Cortex, Mimir, and VictoriaMetrics fill that gap at a horizontally-scaled, multi-tenant layer.
6. Interview Anchor Points (What to Say Out Loud)
The sentences that signal principal-level thinking for the telemetry ingestion pipeline design — ready to say unprompted in an interview.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q12: Exactly-Once for One Billing Tenant While Others Stay At-Least-Once
Full principal-level solution: support exactly-once ingestion for a single billing-critical tenant in a shared pipeline that is at-least-once everywhere else, and account for what it costs.
Q5: Adding Continuous Profiling to an Existing MELT Pipeline
Full principal-level solution: extend an existing metrics + logs + traces pipeline with continuous profiling as a fourth signal, without a full redesign.
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Q11: Redesigning the Ingestion Frontier for a Compromised-Agent Threat Model
Full principal-level solution: redesign the telemetry ingestion frontier assuming a compromised agent sends malformed and adversarial payloads — oversized batches, spoofed tenant IDs, garbage label values.
Tenant Identification and Routing at the Ingestion Frontier
How the gateway decides whose data this is — cert/API-key-derived tenant identity, propagation as a Kafka header and X-Scope-OrgID, and the shared-topic-with-filter vs per-tenant-topic vs shuffle-sharded-pool routing trade-off.
Related notes
Tenant Identification and Routing at the Ingestion Frontier
How the gateway decides whose data this is — cert/API-key-derived tenant identity, propagation as a Kafka header and X-Scope-OrgID, and the shared-topic-with-filter vs per-tenant-topic vs shuffle-sharded-pool routing trade-off.
Chapter 1 — Telemetry Ingestion Pipeline
Principal/Staff-level design of a high-throughput telemetry ingestion pipeline — requirements, architecture, deep dives, and trade-offs at 10x scale.
Q1: 500M Samples/Sec, Zero Drop on Rolling Deploy
Full principal-level solution: design a telemetry ingestion pipeline for 500M metric samples/sec from 100K services globally with a zero-drop guarantee during rolling deployment of the ingestion tier.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.