Loki is Grafana Labs’ log aggregation system, built on a deliberate simplification: index only the labels attached to a log stream, never the log content itself. Full-text indexing is what makes Elasticsearch-based log stacks expensive to run at scale; Loki’s bet is that most log queries start from a small, known label set (service, namespace, pod) and only need full-text search within that already-narrowed stream — so that’s the only thing it indexes.
Streams, not documents
Log stream = a unique set of label values
{app="checkout", env="prod", pod="checkout-7f9c-x2k1"}
│
├── 14:02:01.001 {"level":"info","msg":"order placed","order_id":"9931"}
├── 14:02:01.045 {"level":"error","msg":"payment timeout"}
└── 14:02:01.203 {"level":"info","msg":"order shipped"}
Everything inside the curly braces is indexed; everything after it is compressed into chunks and pushed to object storage, unindexed. A LogQL query always starts with a label selector (cheap, index-backed) and only then applies a log-content filter or parser (the expensive part, but scoped to just that stream’s chunks):
{app="checkout", env="prod"} |= "payment timeout" | json | duration > 500ms
Architecture: same shape as Mimir
Write path Read path
Promtail/Alloy/Fluent Bit Grafana / API client
│ │
▼ ▼
Distributor Query-frontend
│ │
▼ ▼
Ingester Querier
│ │
▼ ▼
Object storage (chunks) ◀── Store-gateway
│
▼
Compactor
Loki actually predates What is Mimir — Mimir’s architecture was modeled directly on Loki’s, which is why operating one gives you a working mental model of the other: distributor validates and shards, ingester holds recent data in memory, object storage is the durable tier, compactor merges blocks in the background.
The cardinality trap, one layer up
Because Loki indexes labels the same way Prometheus indexes metric labels, the identical failure
mode applies: a label sourced from a high-churn or unbounded field (a raw request_id, a full URL
path, a user ID used as a label instead of a log field) doesn’t blow up a metric series count — it
fragments log streams and bloats the label index, with the same operational symptom (memory pressure
on ingesters, slow queries). See What is Cardinality (in observability) for the general mitigation hierarchy; for Loki
specifically the fix is the same: keep labels bounded (service, env, region), and put anything
high-cardinality into the log line itself where LogQL’s | json | ... pipeline can filter on it
without it ever becoming an index dimension.
Ingestion agents
| Agent | Status |
|---|---|
| Promtail | Original Loki-specific agent; in maintenance mode, feature development moved to Alloy |
| Grafana Alloy | loki.source.* + loki.write components — current default |
| What is Fluent Bit | Native Loki output plugin — common where Fluent Bit is already the platform-default DaemonSet |
| Fluentd | Loki output plugin also available |
Why it matters here: Loki is the log-storage tier alongside Mimir and Tempo in the ShipSolid
Grafana Cloud stack — label-schema discipline for Loki streams matters exactly as much as it does
for Mimir metric labels, and the same What is Cardinality (in observability) budget check applies before a new label gets
added to an Alloy loki.write or Fluent Bit kubernetes filter config, not after stream counts
spike.
Local graph
Linked from 17 notes
8 — Query Sharding
Splitting a single logical query into N independently-executable sub-queries that run in parallel and merge into one result — how Grafana Mimir and Loki answer high-cardinality queries within tight SLOs without adding more data shards.
What is Fluent Bit
CNCF-graduated, C-written log/metrics/trace forwarder — Fluentd's lightweight sibling, the de facto node-level log-collection DaemonSet in most Kubernetes clusters, and Grafana Alloy's main incumbent competitor for that slot.
What is Tempo
Grafana Labs' distributed tracing backend — the radical simplification vs. Jaeger's classic architecture: no dedicated index, just object storage and a trace-ID lookup, queried with TraceQL and linked from metrics via exemplars.
7. Component Map (What Exists in the Wild)
OSS and managed-SaaS options for every layer of the telemetry ingestion pipeline, mapped against ShipSolid's own production experience.
Telemetry Gateways: Protocol-Specific Ingestion Points
The ingestion frontier is a fleet of protocol-specific gateways — OTLP, Prometheus remote-write, Syslog, Kafka, and legacy tracing/metrics protocols — each terminating a different producer's wire format before a shared auth/rate-limit/tenant-routing layer.
Agentic AI: Projects & Engineering Mastery
A book-shaped table of contents for Agentic AI: Projects & Engineering Mastery: hands-on practitioner builds, Principal/Staff-level technical leadership, and the lookup appendices and vendor/framework reference notes for the whole series. Book 6 of the AI Systems Engineering series.
Grafana Cloud
A book-shaped table of contents for Grafana Cloud: platform foundations through telemetry collection, Mimir/Loki/Tempo/Pyroscope, visualization, application observability, reliability tooling, developer experience, governance, and enterprise reference architectures — cross-linking existing notes instead of duplicating them.
2 — The Signals
Metrics, logs, traces, profiles, and events — what each is built to capture, what it costs, and which question it actually answers vs. which one people mistakenly ask it.
7 — Multi-Tenancy
Two separate guarantees hiding under one name — data isolation and performance fairness — and the tenant identification, quota enforcement, and selective backpressure that make both hold under shared infrastructure.
8 — Log Aggregation
Schema-on-write vs. schema-on-read as competing bets about when to pay indexing cost, and the two different deduplication problems a log pipeline actually has to solve.
1 — OpenTelemetry SDKs & Semantic Conventions
OpenTelemetry is a specification and an API/SDK, not a backend — the pieces that make it up, and the semantic-convention vocabulary that lets two unrelated teams' telemetry be queried the same way.
9 — OTel Collector Pipeline Design
Receivers, processors, and exporters chained into a pipeline; why a platform runs more than one; and the agent/gateway topology that tail sampling specifically forces on that design.
Related notes
What is Fluent Bit
CNCF-graduated, C-written log/metrics/trace forwarder — Fluentd's lightweight sibling, the de facto node-level log-collection DaemonSet in most Kubernetes clusters, and Grafana Alloy's main incumbent competitor for that slot.
What is Mimir
Grafana Labs' horizontally-scalable, multi-tenant long-term storage for Prometheus metrics — the 2022 successor to Cortex, and the actual system serving every PromQL query and remote-write in a Grafana Cloud metrics stack.
What is Tempo
Grafana Labs' distributed tracing backend — the radical simplification vs. Jaeger's classic architecture: no dedicated index, just object storage and a trace-ID lookup, queried with TraceQL and linked from metrics via exemplars.
What is Cortex (cortexproject)
CNCF Incubating, horizontally-scalable multi-tenant long-term storage for Prometheus — the project Grafana Mimir forked from in 2022, still maintained as the vendor-neutral, community-governed alternative once Grafana Labs redirected engineering effort to Mimir.