7 — Multi-Tenancy
“Multi-tenant” gets used as if it’s one property. It’s actually two, and conflating them hides real gaps: isolation (tenant A can never see tenant B’s data) and fairness (tenant A’s traffic spike can never degrade tenant B’s experience). A platform can have airtight isolation and terrible fairness, or vice versa — they fail independently, and get diagnosed and fixed differently.
Tenant identification: stamped as early as possible
Every downstream partitioning decision — which tenant’s storage this belongs to, which tenant’s quota this counts against, which tenant’s query is allowed to read it — depends on a tenant identifier being attached at the earliest possible point, usually at the ingestion edge before the collector pipeline does anything else. See Tenant Identification & Routing for the concrete mechanics — header-based, mTLS-cert-based, or token-based identification, and how routing keys off it once assigned. Get this wrong at the edge, and every guarantee downstream is built on a value that was never reliably there.
Isolation: a spectrum, not a binary
- Logical isolation — one shared binary, tenants isolated by the tenant ID as a partition key. Mimir, Loki, and Tempo are all natively multi-tenant this way: cheap (one fleet serves everyone), but a bug in the isolation logic is a cross-tenant data leak, not a degraded-performance incident.
- Physical isolation — dedicated infrastructure per tenant (or per regulated tenant class). Expensive, but the isolation guarantee no longer depends on partitioning logic being bug-free — there’s no shared process for a bug to leak across.
Most real platforms land somewhere on this spectrum deliberately: logical isolation as the default, physical isolation reserved for tenants whose compliance requirements make a shared-process guarantee insufficient regardless of how well-tested the partitioning logic is.
Fairness: the noisy-neighbor problem
Logical isolation solves visibility but does nothing about impact. One tenant’s cardinality mistake — see What is Cardinality (in observability) and 5 — Label & Attribute Schema Design — or traffic spike can exhaust shared ingester memory or query concurrency, degrading every other tenant on the same infrastructure, even though none of their data was ever exposed. This is a fairness failure wearing an isolation incident’s symptoms, and it needs a different fix.
Quota enforcement is that fix: per-tenant ceilings on ingest rate, active series, and query concurrency, applied at the edge before one tenant’s traffic can consume a shared resource pool meant for everyone. This is the same rate-limiting mechanic Rate-Limiting Architecture covers in general, applied per-tenant rather than globally — and 05 — Backpressure applied selectively: the well-designed response to a noisy tenant throttles that tenant specifically, not the whole platform, which is the entire difference between a contained incident and a platform-wide outage caused by one tenant’s mistake. Pipeline Multi-Tenancy walks through this at the ingestion-pipeline layer specifically.
Why this matters for an Observability Architect
A platform that only tested isolation (can tenant A read tenant B’s data — no) and never load-tested fairness (can tenant A’s mistake slow down tenant B’s queries — untested) has a gap that won’t show up until a real tenant hits it in production, usually via a cardinality mistake nobody caught in review. Reviewing multi-tenant readiness means testing both properties separately, with separate scenarios — a permissions test for isolation, a noisy-neighbor load test for fairness — not assuming one implies the other.
Metadata
| Dimension | Detail |
|---|---|
| Author | Amit Singh |
| Scope | observability |
Local graph
Linked from 16 notes
2 — Multi Tenancy
The isolation half of multi-tenancy as a security property — the tenant ID as a trust boundary, why every read needs an enforced tenant filter, the leak surfaces around the backend rather than in it, and proving isolation with negative tests.
5 — Security & Compliance
Why PII ends up in telemetry by accident rather than by design, the pipeline-layer scrubbing that catches what application discipline misses, tenant-scoped access control, and why the query audit log is itself security-relevant telemetry.
What is Cardinality (in observability)
The number of unique time series (or unique log/trace label combinations) a metric produces — the single biggest driver of ingest cost and query latency in Prometheus-family backends (Mimir, Cortex, Thanos), and the reason unbounded labels are a production incident waiting to happen.
What is Loki
Grafana Labs' log aggregation system — 'like Prometheus, but for logs': index only labels, store compressed chunks in object storage, query with LogQL. Shares its distributor/ingester/compactor architecture with Mimir and Tempo.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
Observability Architecture: Questions to Ask
A chronological sequence of 216 questions an architect asks when designing a production-grade observability platform — from business context through multi-tenancy, SLOs, onboarding, and validation.
2 — Security
Securing the exporter-to-Prometheus link with TLS and basic auth — self-signed certs, bcrypt password hashing, tls_server_config, and end-to-end curl verification, plus an honest look at what this setup doesn't cover.
1. Clarify Requirements First
The first-5-minutes clarifying questions for the telemetry ingestion pipeline design — signal types, scale envelope, consistency/durability, multi-tenancy, and protocol — whose answers change the entire architecture.
3.6 Multi-Tenancy
Multi-tenancy isolation layers and quota enforcement points across the telemetry ingestion pipeline, from network inbound to the storage write path.
Tenant Identification and Routing at the Ingestion Frontier
How the gateway decides whose data this is — cert/API-key-derived tenant identity, propagation as a Kafka header and X-Scope-OrgID, and the shared-topic-with-filter vs per-tenant-topic vs shuffle-sharded-pool routing trade-off.
Q5: Adding Continuous Profiling to an Existing MELT Pipeline
Full principal-level solution: extend an existing metrics + logs + traces pipeline with continuous profiling as a fourth signal, without a full redesign.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Related notes
5 — Security & Compliance
Why PII ends up in telemetry by accident rather than by design, the pipeline-layer scrubbing that catches what application discipline misses, tenant-scoped access control, and why the query audit log is itself security-relevant telemetry.
8 — Self-Observability
The bootstrapping problem — a platform can't fully trust itself to tell you it's failing — and the two mechanisms that get around it: an independent out-of-band health path, and a synthetic canary that catches silent stalls no internal metric surfaces.
2 — Multi Tenancy
The isolation half of multi-tenancy as a security property — the tenant ID as a trust boundary, why every read needs an enforced tenant filter, the leak surfaces around the backend rather than in it, and proving isolation with negative tests.
1 — RBAC
Why a role grants an action but never a data scope, the three independent layers every telemetry query passes through, and why a small role set plus label-based scoping beats a sprawl of fine-grained roles.