Signal Forge ADR-003: Span metrics generated before tail sampling
Status: Accepted
Decision: The spanmetrics connector is placed in the pipeline before tail_sampling. See
the instrumentation reference for the full pipeline
walkthrough and PromQL examples built on this ordering.
Rationale:
- If span metrics were generated after sampling, only ~25% of traces would contribute to rate and error counters. A “request rate” metric reading 25% of actual traffic would be operationally useless.
- Placing
spanmetricsbefore sampling means every span contributes to RED metrics, regardless of whether the trace is kept. The sampled traces are for debugging; the span metrics are for SLO dashboards.
Pipeline order:
filter(healthz) → spanmetrics (ALL spans)
↘
tail_sampling (25% + errors + slow)
↓
batch
Alternative considered: After sampling — rejected because it produces misleading metrics.
Local graph
Linked from 2 notes
Tail-Based Sampling
How SignalForge's tail-based sampling policies decide which traces to keep, and why span metrics are generated before sampling runs.
Architecture Overview
Signal Forge's topology, service communication, trace propagation, and per-signal pipeline flow across local and Grafana Cloud deployment modes.
Related notes
Signal Forge ADR-005: Separate collector configmaps per deployment mode
Keeps cloud and local Alloy collector configs in separate files rather than one conditional configmap, so each mode's exporters stay explicit and uncoupled.
Signal Forge ADR-002: SpanLink for async RabbitMQ propagation (not parent-child)
Uses a SpanLink, not a parent-child span relationship, to connect RabbitMQ consumer spans back to the producer span across async, retry-prone delivery.
Architecture Overview
Signal Forge's topology, service communication, trace propagation, and per-signal pipeline flow across local and Grafana Cloud deployment modes.
Signal Forge ADR-008: Dead Letter Queue for poison message handling
Routes messages that exceed retry limits or are NACKed without requeue to a RabbitMQ DLQ, preventing infinite retry loops from starving the consumer.