Signal Forge ADR-008: Dead Letter Queue for poison message handling
Status: Accepted
Decision: The RabbitMQ notifications queue is declared with x-dead-letter-exchange pointing
to orders.dlq (fanout exchange). Messages that exceed x-max-retries or are explicitly NACKed
without requeue are routed to a notifications.dlq queue.
Rationale:
- Without a DLQ, a consistently failing message causes an infinite retry loop that starves processing of other messages and spikes CPU.
- The dead-letter pattern is built into RabbitMQ — no additional application code is needed in the
NACK path. The same notification-svc consumer that carries the
SpanLink from the RabbitMQ producer
NACKs with
requeue=False; the broker handles routing. - DLQ messages can be inspected via the RabbitMQ Management UI and reprocessed manually or via a separate consumer once the underlying bug is fixed.
Alternative considered: Manual retry counter in Redis with re-publish — rejected as unnecessary complexity when RabbitMQ provides this natively.
Local graph
Linked from 3 notes
Resilience patterns
Application-level failure handling in Signal Forge: retries, circuit breakers, backoff, and delivery-safety patterns for downstream dependency failures.
Service: notification-svc
notification-svc's RabbitMQ consumer architecture, dead-letter routing, idempotency handling, and OTel instrumentation.
Signal Forge ADR-002: SpanLink for async RabbitMQ propagation (not parent-child)
Uses a SpanLink, not a parent-child span relationship, to connect RabbitMQ consumer spans back to the producer span across async, retry-prone delivery.
Related notes
Signal Forge ADR-005: Separate collector configmaps per deployment mode
Keeps cloud and local Alloy collector configs in separate files rather than one conditional configmap, so each mode's exporters stay explicit and uncoupled.
Signal Forge ADR-003: Span metrics generated before tail sampling
Places the spanmetrics connector ahead of tail_sampling so RED metrics reflect all traffic instead of only the ~25% of traces that survive sampling.
Signal Forge ADR-002: SpanLink for async RabbitMQ propagation (not parent-child)
Uses a SpanLink, not a parent-child span relationship, to connect RabbitMQ consumer spans back to the producer span across async, retry-prone delivery.
Architecture Overview
Signal Forge's topology, service communication, trace propagation, and per-signal pipeline flow across local and Grafana Cloud deployment modes.