5 — Security & Compliance
Nobody deliberately designs a system to store a customer’s email address in a log line. It happens anyway, constantly, because logging a whole request object is one line of code and reviewing every field of every object anyone ever logs is not. Telemetry is a data-exfiltration surface precisely because it’s exempt from the scrutiny a database schema gets — nobody added a column for it, so nobody’s data-classification review ever looked at it.
Where PII actually gets in
Logs’ whole value is that they carry arbitrary developer-written content — that same flexibility is the exact mechanism PII leaks in through: a debug log that prints a full request body, an error message that interpolates a user object, a custom span attribute that captures a query parameter with a session token in it. None of these are semconv-standard attributes with a known shape a scanner could anticipate — they’re free text, by the same property that makes logs useful for debugging in the first place.
Pipeline-layer scrubbing is a backstop, not the fix
A processor in the collector chain can redact or hash known-sensitive field patterns before export — a real, worthwhile backstop. But it’s a backstop specifically because a pattern-based scrubber can only catch content shaped the way it expects; it can’t recognize a customer’s email address embedded in a free-text sentence it wasn’t written to match. Pipeline scrubbing catches the mistakes discipline missed; it doesn’t substitute for not logging PII in the first place, and treating it as a substitute is how a platform ends up “protected” against exactly the leaks it already knows about and blind to the ones it doesn’t.
For what disciplined logging actually looks like as platform policy, see Logging Guidelines and Logging.
Access control: scoped by tenant, not just by login
Authentication answers “who is this.” Label-based access control (LBAC) answers the question that actually matters for a multi-tenant observability platform: which tenant’s data — which label scope — is this authenticated user’s query allowed to touch at all. A query that’s technically authenticated but scoped to the wrong tenant is exactly the isolation failure 7 — Multi-Tenancy is about, just surfaced through the access-control layer instead of a storage bug. See Label-Based Access Control for what this looks like as a real, enforced policy layer rather than a convention someone has to remember.
The audit trail is itself security-relevant telemetry
Who queried which tenant’s data, and when, is usually the fastest way to reconstruct what an engineer actually saw during a security review — which makes the observability platform’s own query log a security artifact in its own right, not just an operational nice-to-have. That query audit log has to survive independently of whatever it’s auditing, which is the same bootstrapping problem Self-Observability covers for the platform’s health signals in general, applied here to its access records specifically.
For a worked example of the threat model on the other side of this — what happens if the collector or agent producing telemetry is itself compromised, rather than the query layer — see Compromised Agent Threat Model.
Why this matters for an Observability Architect
Reviewing a new service’s instrumentation for security means asking “what happens when someone logs a full object here” as a certainty, not a hypothetical — it will happen, usually without malice, usually from someone debugging under pressure who didn’t stop to think about what was in the object. The platform’s job is to have a backstop in place before that happens, not to rely on every engineer remembering PII discipline on the day the discipline mattered most.
Metadata
| Dimension | Detail |
|---|---|
| Author | Amit Singh |
| Scope | observability |
Local graph
Linked from 9 notes
6 — Audit Logging
The query audit log as a first-class security artifact — what a query-audit event must record (including the resolved scope, not just the query text), why it has to live apart from everything it audits, tamper-evidence, and compliance-driven retention.
1 — RBAC
Why a role grants an action but never a data scope, the three independent layers every telemetry query passes through, and why a small role set plus label-based scoping beats a sprawl of fine-grained roles.
2 — Multi Tenancy
The isolation half of multi-tenancy as a security property — the tenant ID as a trust boundary, why every read needs an enforced tenant filter, the leak surfaces around the backend rather than in it, and proving isolation with negative tests.
3 — Data Privacy
The privacy obligations that attach to telemetry as a dataset in its own right — why it's a shadow copy of personal data that skips classification, which data-subject rights an append-only store can't honor after the fact, and why retention minimization is a privacy control.
4 — PII Redaction
The pipeline-layer backstop that catches what source discipline misses — where redaction can sit and what each placement catches, drop vs mask vs hash vs tokenize, allowlist over blocklist, and why a regex list bolted onto the collector fails quietly.
7 — Secret Management
Two failures with one name — credentials leaking into telemetry payloads, and the observability plane's own auth tokens sitting in plaintext config. Detection by entropy and known prefixes, secrets in an external store referenced at runtime, and why a leaked write token is not a read-only problem.
8 — Observability as Policy
Why organizations shift from 'does this have dashboards' to 'does this satisfy our requirements before production' — what a tiered policy contains, compliance scoring, rolling it out across an existing estate without a big-bang block, and treating observability as a property of the workload rather than a feature of the platform.
RFC Template
- **RFC ID**: rfc-YYYY-MM-<slug> - **Authors**: [Name(s), Role(s)]
Observability Engineering
A book-shaped table of contents for observability engineering: foundations through architecture, metrics, logging, tracing, profiling, OpenTelemetry, instrumentation, Kubernetes/cloud, data platforms, visualization, alerting, SRE integration, cost, security, platform engineering, AI-driven operations, and MAANG interview preparation — cross-linking existing prometheus/grafana-cloud/kubernetes/sre/platform-engineering notes instead of duplicating them.
Related notes
7 — Multi-Tenancy
Two separate guarantees hiding under one name — data isolation and performance fairness — and the tenant identification, quota enforcement, and selective backpressure that make both hold under shared infrastructure.
8 — Self-Observability
The bootstrapping problem — a platform can't fully trust itself to tell you it's failing — and the two mechanisms that get around it: an independent out-of-band health path, and a synthetic canary that catches silent stalls no internal metric surfaces.
2 — Multi Tenancy
The isolation half of multi-tenancy as a security property — the tenant ID as a trust boundary, why every read needs an enforced tenant filter, the leak surfaces around the backend rather than in it, and proving isolation with negative tests.
1 — RBAC
Why a role grants an action but never a data scope, the three independent layers every telemetry query passes through, and why a small role set plus label-based scoping beats a sprawl of fine-grained roles.