7 — Multi-Tenancy
“Multi-tenant” gets used as if it’s one property. It’s actually two, and conflating them hides real gaps: isolation (tenant A can never see tenant B’s data) and fairness (tenant A’s traffic spike can never degrade tenant B’s experience). A platform can have airtight isolation and terrible fairness, or vice versa — they fail independently, and get diagnosed and fixed differently.
Tenant identification: stamped as early as possible
Every downstream partitioning decision — which tenant’s storage this belongs to, which tenant’s quota this counts against, which tenant’s query is allowed to read it — depends on a tenant identifier being attached at the earliest possible point, usually at the ingestion edge before the collector pipeline does anything else. See Tenant Identification & Routing for the concrete mechanics — header-based, mTLS-cert-based, or token-based identification, and how routing keys off it once assigned. Get this wrong at the edge, and every guarantee downstream is built on a value that was never reliably there.
Isolation: a spectrum, not a binary
- Logical isolation — one shared binary, tenants isolated by the tenant ID as a partition key. Mimir, Loki, and Tempo are all natively multi-tenant this way: cheap (one fleet serves everyone), but a bug in the isolation logic is a cross-tenant data leak, not a degraded-performance incident.
- Physical isolation — dedicated infrastructure per tenant (or per regulated tenant class). Expensive, but the isolation guarantee no longer depends on partitioning logic being bug-free — there’s no shared process for a bug to leak across.
Most real platforms land somewhere on this spectrum deliberately: logical isolation as the default, physical isolation reserved for tenants whose compliance requirements make a shared-process guarantee insufficient regardless of how well-tested the partitioning logic is.
Fairness: the noisy-neighbor problem
Logical isolation solves visibility but does nothing about impact. One tenant’s cardinality mistake — see What is Cardinality (in observability) and 5 — Label & Attribute Schema Design — or traffic spike can exhaust shared ingester memory or query concurrency, degrading every other tenant on the same infrastructure, even though none of their data was ever exposed. This is a fairness failure wearing an isolation incident’s symptoms, and it needs a different fix.
Quota enforcement is that fix: per-tenant ceilings on ingest rate, active series, and query concurrency, applied at the edge before one tenant’s traffic can consume a shared resource pool meant for everyone. This is the same rate-limiting mechanic Rate-Limiting Architecture covers in general, applied per-tenant rather than globally — and 05 — Backpressure applied selectively: the well-designed response to a noisy tenant throttles that tenant specifically, not the whole platform, which is the entire difference between a contained incident and a platform-wide outage caused by one tenant’s mistake. Pipeline Multi-Tenancy walks through this at the ingestion-pipeline layer specifically.
Why this matters for an Observability Architect
A platform that only tested isolation (can tenant A read tenant B’s data — no) and never load-tested fairness (can tenant A’s mistake slow down tenant B’s queries — untested) has a gap that won’t show up until a real tenant hits it in production, usually via a cardinality mistake nobody caught in review. Reviewing multi-tenant readiness means testing both properties separately, with separate scenarios — a permissions test for isolation, a noisy-neighbor load test for fairness — not assuming one implies the other.
Metadata
| Dimension | Detail |
|---|---|
| Author | Amit Singh |
| Scope | observability |
Local graph
Linked from 15 notes
5 — Security & Compliance
Why PII ends up in telemetry by accident rather than by design, the pipeline-layer scrubbing that catches what application discipline misses, tenant-scoped access control, and why the query audit log is itself security-relevant telemetry.
What is Cardinality (in observability)
The number of unique time series (or unique log/trace label combinations) a metric produces — the single biggest driver of ingest cost and query latency in Prometheus-family backends (Mimir, Cortex, Thanos), and the reason unbounded labels are a production incident waiting to happen.
What is Loki
Grafana Labs' log aggregation system — 'like Prometheus, but for logs': index only labels, store compressed chunks in object storage, query with LogQL. Shares its distributor/ingester/compactor architecture with Mimir and Tempo.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
Observability Architecture: Questions to Ask
A chronological sequence of 216 questions an architect asks when designing a production-grade observability platform — from business context through multi-tenancy, SLOs, onboarding, and validation.
2 — Security
Securing the exporter-to-Prometheus link with TLS and basic auth — self-signed certs, bcrypt password hashing, tls_server_config, and end-to-end curl verification, plus an honest look at what this setup doesn't cover.
1. Clarify Requirements First
The first-5-minutes clarifying questions for the telemetry ingestion pipeline design — signal types, scale envelope, consistency/durability, multi-tenancy, and protocol — whose answers change the entire architecture.
3.6 Multi-Tenancy
Multi-tenancy isolation layers and quota enforcement points across the telemetry ingestion pipeline, from network inbound to the storage write path.
Q10: Self-Service Tenant Onboarding With Zero Platform-Team Involvement
Full principal-level solution: design a self-service tenant onboarding API for a telemetry pipeline that protects shared infrastructure from a misbehaving new tenant on day one.
Q5: Adding Continuous Profiling to an Existing MELT Pipeline
Full principal-level solution: extend an existing metrics + logs + traces pipeline with continuous profiling as a fourth signal, without a full redesign.
Tenant Identification and Routing at the Ingestion Frontier
How the gateway decides whose data this is — cert/API-key-derived tenant identity, propagation as a Kafka header and X-Scope-OrgID, and the shared-topic-with-filter vs per-tenant-topic vs shuffle-sharded-pool routing trade-off.
Grafana Cloud
A book-shaped table of contents for Grafana Cloud: platform foundations through telemetry collection, Mimir/Loki/Tempo/Pyroscope, visualization, application observability, reliability tooling, developer experience, governance, and enterprise reference architectures — cross-linking existing notes instead of duplicating them.
Related notes
5 — Security & Compliance
Why PII ends up in telemetry by accident rather than by design, the pipeline-layer scrubbing that catches what application discipline misses, tenant-scoped access control, and why the query audit log is itself security-relevant telemetry.
8 — Self-Observability
The bootstrapping problem — a platform can't fully trust itself to tell you it's failing — and the two mechanisms that get around it: an independent out-of-band health path, and a synthetic canary that catches silent stalls no internal metric surfaces.
2 — The Signals
Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.
1 — Dashboard Design
The three-question test for a vanity panel, why the same underlying data needs a different dashboard for different audiences, and the top-down layout that mirrors how an investigation actually drills down.