SignalForge Documentation

End-to-end technical and production-grade documentation for the SignalForge OTel Microservices Validation Lab.

Architecture & services

| Section | Description | | ------------------------------------------------ | --------------------------- | ------------------------------------------------------------------------- | | Architecture | System design, service topology, signal flow diagrams | | Architecture Decisions | 10 ADRs for every non-obvious design choice — browse architecture/adrs/ | | Service: gateway-api | .NET 8 API Gateway — endpoints, OTel, failure modes | | Service: order-api | .NET 8 gRPC Order Service — streaming, RabbitMQ publishing | | Service: notification-svc | Python FastAPI — async consumer, DLQ, Redis | | Service: frontend | Angular 17 SPA + Grafana Faro RUM |

Observability

| Section | Description | | ---------------- | -------------------------- | -------------------------------------------------------------- | | Pipeline | Grafana Alloy River config — processors and exporters | | OTel Signal Contracts | Per-service span names, metric instruments, log schemas | | Tail-Based Sampling | Sampling policies, validation, production tuning | | Log-to-Trace Correlation | Loki structured metadata, .NET vs Python field names | | Exemplars | End-to-end exemplar pipeline from SDK to Grafana | | SLOs & burn-rate alerts | Multi-window SLO math, PrometheusRule manifest, alert policy |

Infrastructure

| Section | Description | | ---------------------------------------------------- | ------------------------ | --------------------------------------------------------------------------- | | Datastores | MySQL, PostgreSQL, Redis, RabbitMQ — topology and config | | Datastore HA migration | Operator-backed HA (CNPG, Percona, RabbitMQ Operator) — prod path | | Kubernetes | Namespace layout, directory tree, secrets, RBAC, health probes | | Container hardening | securityContext, non-root UIDs, digest pins, Pod Security Standards | | Kustomize layout | k8s/base + overlays/{dev,staging,prod}, how deploy-local.sh consumes it |

Deployment

| Section | Description | | ------------------------------------------- | -------------------------- | ----------------------------------------------- | | Local Deployment | k3d cluster setup, local backends, step-by-step | | Grafana Cloud Deployment | Credentials, AKV integration, endpoint formats | | Helm Monitoring Stack | grafana/k8s-monitoring chart, Alloy roles |

Operations

| Section | Description | | ---------------------------------------------------------- | ----------------------- | ----------------------------------------------------------------- | | Runbooks | Troubleshooting playbooks for every failure mode | | Security | Secrets lifecycle, credential rotation, threat model | | Networking & TLS | NetworkPolicies, Ingress TLS via cert-manager, flannel caveat | | Reliability | PodDisruptionBudgets, pod anti-affinity, graceful shutdown | | Resilience Patterns | App-level retry/circuit-breaker/backoff/DLQ patterns, per service | | Supply-chain security | CI Trivy/Syft/cosign pipeline, digest pinning, SBOM verification | | Known Issues | Open limitations and accepted trade-offs, consolidated |

API reference

| Section | Description | | ------------------------------------ | ----------- | --------------------------------------------------- | | REST API | Endpoints, request/response schemas | | gRPC API | Proto definitions, error codes, streaming behaviour |

Replication guides

Ordered, copy-paste guides for standing up this project’s OTel instrumentation pattern in a different repository — for handing the pattern off to another team, not for working in this repo.

| Section | Description | | ----------------------------------------- | ------------------------------ | ------------------------------------------------------------- | | Guides index | Scope, assumptions, recommended order | | Collector & Pipeline Setup | Alloy + grafana/k8s-monitoring Helm chart — do this first | | .NET Instrumentation | SDK wiring, custom spans/metrics, RabbitMQ outbox propagation | | Python Instrumentation | SDK wiring, RabbitMQ consumer SpanLink propagation | | Frontend RUM Instrumentation | Grafana Faro setup, runtime config injection |

Quick orientation

flowchart LR
    Browser --> Gateway["gateway-api (.NET)"]
    Gateway --> OrderAPI["order-api (.NET, gRPC)"]
    OrderAPI --> RabbitMQ[RabbitMQ]
    RabbitMQ --> Notification["notification-svc (Python)"]
    Gateway --> MySQL[MySQL]
    OrderAPI --> PostgreSQL[PostgreSQL]
    Notification --> Redis[Redis]

All services export OTLP → alloy-receiver (Helm DaemonSet in the monitoring namespace). Logs are tailed at the node by alloy-logs, not shipped via OTLP.

Two deployment modes (set in conf.yml monitoring.mode):

local — bespoke Alloy DaemonSet exports to in-cluster Jaeger / Prometheus / Loki / Grafana
cloud — the Helm release’s Alloy agents export to Grafana Cloud Tempo / Mimir / Loki (credentials via Azure Key Vault)

Starting points by role

| You are… | Start here | | ------------------------------------ | ------------------------------------------------------------------- | ------------------------------------------------------------------------------- | -------------------------------------- | -------------- | | New to the lab | Architecture Overview | | Setting up for the first time | Local Deployment | | Debugging missing traces | Runbooks → No traces in Jaeger | | Adding a new service | Architecture Decisions + Observability Pipeline | | Enabling Grafana Cloud export | Grafana Cloud Deployment | | Understanding sampling behaviour | Tail-Based Sampling | | Reviewing signal contracts | OTel Signal Contracts | | Reviewing security posture | Security + Container hardening + Supply-chain | | Promoting to staging / prod | Datastore HA migration + Kustomize overlays + Reliability | | Writing SLO alerts | SLOs & burn-rate alerts | | Handing this pattern to another team | Replication guides |

Source

Full source code lives at github.com/shipsolid/app-signal-forge.

SignalForge Documentation MOC