Signal Forge ADR-004: Helm-managed Alloy stack (grafana/k8s-monitoring)
Status: Accepted
Decision: The production collector stack uses the grafana/k8s-monitoring v3.8.4 Helm chart
(five specialised Alloy roles). The hand-rolled DaemonSet in k8s/monitoring/grafana/ is kept as a
reference artifact but is not deployed. The split between cloud and local collector configs within
this stack is covered separately in
ADR-005.
Rationale:
- Running two Alloy instances receiving the same OTLP traffic caused duplicate spans, duplicate metric samples, version mismatches, and CrashLoopBackOff.
- The Helm chart manages RBAC, ServiceAccounts, and River configs with versioned upgrades. The hand-rolled version required manual maintenance of all these.
- The five-role split (metrics, logs, singleton, receiver, profiles) mirrors production AKS configuration, providing parity for validation.
The five roles:
| Role | Kind | Purpose |
|---|---|---|
alloy-receiver | DaemonSet | OTLP push receiver — app telemetry |
alloy-logs | DaemonSet | Pod + node log tailing → Loki |
alloy-metrics | StatefulSet | kubelet, cAdvisor, KSM → Prometheus |
alloy-singleton | Deployment | Cluster events, KSM API → Loki/Prometheus |
alloy-profiles | DaemonSet | Continuous profiling (disabled locally) |
Alternative considered: Single hand-rolled DaemonSet — rejected due to operational complexity and the duplicate-collector problem.
Local graph
Linked from 3 notes
Helm Monitoring Stack
How the grafana/k8s-monitoring Helm chart deploys Alloy roles for collecting and exporting telemetry, and why it replaced the legacy Makefile flow.
Signal Forge ADR-009: K8s attribute enrichment at collector (not in SDK)
Adds Kubernetes pod/namespace/deployment attributes via Alloy's k8sattributes processor at the collector, keeping application SDKs Kubernetes-agnostic.
Signal Forge ADR-005: Separate collector configmaps per deployment mode
Keeps cloud and local Alloy collector configs in separate files rather than one conditional configmap, so each mode's exporters stay explicit and uncoupled.
Related notes
Signal Forge ADR-005: Separate collector configmaps per deployment mode
Keeps cloud and local Alloy collector configs in separate files rather than one conditional configmap, so each mode's exporters stay explicit and uncoupled.
Signal Forge ADR-003: Span metrics generated before tail sampling
Places the spanmetrics connector ahead of tail_sampling so RED metrics reflect all traffic instead of only the ~25% of traces that survive sampling.
Signal Forge ADR-002: SpanLink for async RabbitMQ propagation (not parent-child)
Uses a SpanLink, not a parent-child span relationship, to connect RabbitMQ consumer spans back to the producer span across async, retry-prone delivery.
Architecture Overview
Signal Forge's topology, service communication, trace propagation, and per-signal pipeline flow across local and Grafana Cloud deployment modes.