Signal Forge ADR-001: Log tailing instead of OTLP log export
Status: Accepted
Decision: Set OTEL_LOGS_EXPORTER=none on all services. Ship logs via node-level Alloy tailing
(alloy-logs DaemonSet), not via the OTLP SDK.
Rationale:
- At production scale, log volume spikes must not consume SDK/process memory or CPU. Node-level agents absorb backpressure independently.
- Applications write structured JSON to stdout — the simplest possible contract. No log SDK configuration in service code.
- The tailing pattern explicitly validates log-to-trace correlation via metadata extraction
(
stage.json→stage.structured_metadata), which is a distinct OTel pattern from OTLP log push. This correlation stage is authored once and shared across both collector modes — see ADR-005. - Log tailing survives SDK crashes and OOM kills; OTLP log export does not.
Trade-off: A small delay (seconds) between log emission and Loki ingestion. Acceptable for all known use cases.
Alternative considered: Direct OTLP log export — rejected because it couples log delivery reliability to application health and adds SDK complexity.
Local graph
Linked from 6 notes
Log-to-Trace Correlation
How SignalForge correlates logs to traces via node-level tailing and Loki structured metadata, across both local and cloud monitoring modes.
Signal Forge ADR-005: Separate collector configmaps per deployment mode
Keeps cloud and local Alloy collector configs in separate files rather than one conditional configmap, so each mode's exporters stay explicit and uncoupled.
Architecture Overview
Signal Forge's topology, service communication, trace propagation, and per-signal pipeline flow across local and Grafana Cloud deployment modes.
SignalForge Documentation
Documentation hub for the SignalForge OTel Microservices Validation Lab — architecture, services, API, deployment, observability, and operations.
Replication Guides: Instrumenting Your Own Project
Step-by-step, copy-paste guides for replicating SignalForge's OpenTelemetry instrumentation pattern in a new .NET/Python/Angular/RabbitMQ/K8s project.
Guide: Collector & Pipeline Setup
Step-by-step: stand up a Grafana Alloy + grafana/k8s-monitoring Helm chart pipeline that receives OTLP traces/metrics/logs from your services and exports to Grafana Cloud or a self-hosted backend.
Related notes
Signal Forge ADR-005: Separate collector configmaps per deployment mode
Keeps cloud and local Alloy collector configs in separate files rather than one conditional configmap, so each mode's exporters stay explicit and uncoupled.
Signal Forge ADR-003: Span metrics generated before tail sampling
Places the spanmetrics connector ahead of tail_sampling so RED metrics reflect all traffic instead of only the ~25% of traces that survive sampling.
Signal Forge ADR-002: SpanLink for async RabbitMQ propagation (not parent-child)
Uses a SpanLink, not a parent-child span relationship, to connect RabbitMQ consumer spans back to the producer span across async, retry-prone delivery.
Architecture Overview
Signal Forge's topology, service communication, trace propagation, and per-signal pipeline flow across local and Grafana Cloud deployment modes.