# Projects
All Projects notes →Replication Guides: Instrumenting Your Own Project
Step-by-step, copy-paste guides for replicating SignalForge's OpenTelemetry instrumentation pattern in a new .NET/Python/Angular/RabbitMQ/K8s project.
Guide: Collector & Pipeline Setup
Step-by-step: stand up a Grafana Alloy + grafana/k8s-monitoring Helm chart pipeline that receives OTLP traces/metrics/logs from your services and exports to Grafana Cloud or a self-hosted backend.
Guide: .NET Instrumentation
Step-by-step: instrument an ASP.NET Core / gRPC .NET 8 service with OpenTelemetry — SDK wiring, custom spans and metrics, and RabbitMQ producer-side async trace propagation via the outbox pattern.
Guide: Frontend RUM Instrumentation
Step-by-step: instrument an Angular frontend with Grafana Faro for browser RUM — SDK setup, runtime config injection, source-map upload, and browser-to-backend trace linkage.
Guide: Python Instrumentation
Step-by-step: instrument a Python FastAPI service with OpenTelemetry — SDK wiring, custom metrics, and RabbitMQ consumer-side async trace propagation via manual context extraction and SpanLink.
SignalForge Documentation
Documentation hub for the SignalForge OTel Microservices Validation Lab — architecture, services, API, deployment, observability, and operations.
gRPC API Reference
Reference for the OrderService gRPC contract between gateway-api and order-api, covering RPCs, error codes, and trace propagation.
REST API Reference
Reference for gateway-api's REST endpoints covering projects, orders, notifications, and OTel trace context propagation.
Signal Forge ADR-008: Dead Letter Queue for poison message handling
Routes messages that exceed retry limits or are NACKed without requeue to a RabbitMQ DLQ, preventing infinite retry loops from starving the consumer.
Signal Forge ADR-006: Fail-fast on missing secrets
Services throw at startup when required connection strings are absent, instead of silently falling back to defaults that mask misconfiguration.
Signal Forge ADR-010: gRPC server-streaming via AsAsyncEnumerable (not ToListAsync)
Streams order rows to the gRPC client directly from the PostgreSQL cursor via AsAsyncEnumerable, keeping memory usage O(1) regardless of result set size.
Signal Forge ADR-004: Helm-managed Alloy stack (grafana/k8s-monitoring)
Standardizes on the grafana/k8s-monitoring Helm chart's five-role Alloy topology, keeping the hand-rolled DaemonSet only as a non-deployed reference.
Signal Forge ADR-009: K8s attribute enrichment at collector (not in SDK)
Adds Kubernetes pod/namespace/deployment attributes via Alloy's k8sattributes processor at the collector, keeping application SDKs Kubernetes-agnostic.
Signal Forge ADR-001: Log tailing instead of OTLP log export
Ships logs via node-level Alloy tailing of stdout instead of OTLP SDK log export, to keep log delivery decoupled from application health.
Signal Forge ADR-007: secretKeyRef for all credentials (no plaintext env vars)
Stores all database, RabbitMQ, and API credentials in Kubernetes Secrets referenced via secretKeyRef so manifests stay safe to commit.
Signal Forge ADR-005: Separate collector configmaps per deployment mode
Keeps cloud and local Alloy collector configs in separate files rather than one conditional configmap, so each mode's exporters stay explicit and uncoupled.
Signal Forge ADR-003: Span metrics generated before tail sampling
Places the spanmetrics connector ahead of tail_sampling so RED metrics reflect all traffic instead of only the ~25% of traces that survive sampling.
Signal Forge ADR-002: SpanLink for async RabbitMQ propagation (not parent-child)
Uses a SpanLink, not a parent-child span relationship, to connect RabbitMQ consumer spans back to the producer span across async, retry-prone delivery.
Architecture Overview
Signal Forge's topology, service communication, trace propagation, and per-signal pipeline flow across local and Grafana Cloud deployment modes.
Grafana Cloud Deployment
How Alloy exports traces, metrics, and logs to Grafana Cloud, including credential architecture, Azure Key Vault setup, and troubleshooting.
Helm Monitoring Stack
How the grafana/k8s-monitoring Helm chart deploys Alloy roles for collecting and exporting telemetry, and why it replaced the legacy Makefile flow.
Local Deployment
Step-by-step guide to deploying the full lab locally on k3d via deploy-local.sh, including cluster setup and mode switching.
Datastore HA — production migration notes
Migration paths from signal-forge's single-replica lab datastores to production HA via CloudNativePG, MySQL/Percona, RabbitMQ, and Redis operators.
Datastores
Reference for signal-forge's four datastores — MySQL, PostgreSQL, Redis, RabbitMQ — covering schemas, credentials, OTel instrumentation, and deploy order.
Container & Pod Hardening
Reference for signal-forge's Kubernetes Pod Security Standards hardening — per-image UIDs, Dockerfile conventions, and the security controls on every workload.
Kubernetes Infrastructure
Reference for signal-forge's Kubernetes infrastructure — namespaces, secrets, deployments, RBAC, ingress, health probes, and deploy order.
Kustomize layout
How signal-forge's Kustomize base and per-environment overlays are laid out, rendered, and consumed by deploy-local.sh.
Log-to-Trace Correlation
How SignalForge correlates logs to traces via node-level tailing and Loki structured metadata, across both local and cloud monitoring modes.
Exemplars
How exemplars link histogram metric observations to sampled traces end-to-end, from SDK emission through Prometheus/Mimir to Grafana.
OTel Signal Contracts
The OpenTelemetry signal contracts—spans, metrics, and log fields—for every SignalForge service and the frontend RUM app.
Observability Pipeline
How the Grafana Alloy collector pipeline differs between SignalForge's local (hand-authored River) and cloud (Helm chart) monitoring modes.
Tail-Based Sampling
How SignalForge's tail-based sampling policies decide which traces to keep, and why span metrics are generated before sampling runs.
SLOs & burn-rate alerts
SignalForge's published SLOs, how their SLIs are computed from span metrics, and how multi-window burn-rate alerts are structured.
Known issues
Recurring limitations and accepted trade-offs across Signal Forge, consolidated in one place to check before assuming a gap is new.
Networking & TLS
Network-plane security for Signal Forge: NetworkPolicy default-deny model, Ingress TLS via cert-manager, and the k3d flannel enforcement caveat.
Reliability controls
Workload-level Kubernetes controls protecting Signal Forge availability during disruption: PodDisruptionBudgets, anti-affinity, and graceful shutdown.
Resilience patterns
Application-level failure handling in Signal Forge: retries, circuit breakers, backoff, and delivery-safety patterns for downstream dependency failures.
Runbooks
Troubleshooting playbooks for every known Signal Forge failure mode, from missing traces to Grafana Cloud export errors.
Security
Threat model and secrets lifecycle reference for Signal Forge, covering credential management, input validation, CORS, RBAC, and rotation procedures.
Supply-chain security
What CI verifies before a Signal Forge image ships: vulnerability scanning, SBOM generation, and cosign keyless signing.
SignalForge Instrumentation Reference
Reference explaining every OpenTelemetry instrumentation decision in the signal-forge lab — what's configured, why, and what correct behavior looks like.
Service: frontend (Angular SPA)
The Angular SPA's role, Faro browser instrumentation, nginx configuration, and browser-to-backend trace linkage.
Service: gateway-api
gateway-api's endpoints, domain model, configuration, OTel instrumentation, resilience patterns, and failure modes.
Service: notification-svc
notification-svc's RabbitMQ consumer architecture, dead-letter routing, idempotency handling, and OTel instrumentation.
Service: order-api
order-api's gRPC service definition, outbox-pattern RabbitMQ publishing, streaming pattern, and OTel instrumentation.
SignalForge: OTel Microservices Validation Lab
The architecture spec for SignalForge, a multi-service .NET/Python/Angular lab validating every OpenTelemetry instrumentation pattern end-to-end.
Testing
Reference for signal-forge's 140 automated tests across all four services, including setup commands, per-suite coverage, and known gaps.
10 — Templates
Reusable authoring templates for ADRs, RFCs, runbooks, post-mortems, and Confluence pages across the ShipSolid platform.
RFC: Editor & Agentic IDE Selection — VS Code vs. Google Antigravity
Engagement Model
How teams engage the platform team — intake, support tiers, and SLAs.
Glossary
Shared vocabulary for the platform.
Quickstart: Onboard in 30 Minutes
Get a service emitting metrics, logs, and traces to Grafana Cloud in **30 minutes**.
00 — Start Here
Orientation for anyone joining or consuming the ShipSolid observability platform.
Team Charter & Ownership Model
Who owns the platform, how decisions get made, and what teams can expect from us.
Vision & Mission
Why the ShipSolid observability platform exists and where it is going.
ADR-008: Adopt Grafana Cloud + OpenTelemetry + Alloy as the ShipSolid Observability Standard
- **Status**: Accepted - **Date**: 2026-06-09
ADR-006: Pin OpenTelemetry Semantic Conventions to v1.26 as Platform Baseline
- **Status**: Proposed - **Date**: 2026-05-07
ADR-007: Adopt Two-Tier Grafana Alloy Collector Topology
- **Status**: Proposed - **Date**: 2026-05-07
Dependency Map
Map of platform dependencies — what the platform relies on, and what relies on it.
Control Plane Architecture
How the platform is configured, governed, and changed — the control plane.
Data Plane Architecture
How telemetry flows from source to storage — the data plane.
Technical Design Documents
Index of **ShipSolid platform** Technical Design Documents.
01 — Platform Architecture
How the platform is built — views, planes, signals, and the decision record indexes.
Platform Overview (C4 L1-L2)
The platform at a glance — C4 Level 1 (System Context) and Level 2 (Containers).
Signal Catalog
Canonical catalog of the signals the platform supports and the golden signals every service should
Logs Instrumentation Guide
How to instrument a service for **logs** on the ShipSolid observability platform.
Metrics Instrumentation Guide
How to instrument a service for **metrics** on the ShipSolid observability platform.
Naming & Label Schema
The canonical label and resource-attribute schema every signal must follow.
Onboarded Teams Registry
Registry of teams onboarded to the platform and their onboarding state.
Onboarding Checklist
The definitive checklist for taking a service from zero to fully observable.
Service Catalog
Registry of all services emitting to the platform, with owner and on-call.
02 — Service Onboarding
Paved-road guides to take a service from zero to fully observable.
Traces Instrumentation Guide
How to instrument a service for **distributed traces** on the ShipSolid observability platform (OTel
Error Budget Policy
How error budgets are calculated, consumed, and enforced.
Production Readiness Review (PRR)
The Production Readiness Review gate — what a service must satisfy before prod.
03 — Reliability Engineering
The SRE practice: charter, SLOs, error budgets, and the prod-readiness gate.
Reliability Review Archive
Archive of completed reliability / PRR reviews.
SLO Registry
Registry of all SLOs across ShipSolid platform services.
SRE Charter
The SRE practice charter — scope, principles, and how reliability work is prioritized.
Alert Runbooks
Runbooks invoked directly from paging alerts.
Runbook — ShipSolidApiGateway5xxHigh
- **Service:** api-gateway - **Owner Team:** Platform SRE
Communication Templates
Copy-paste communication templates for incidents.
Incident Response Playbook
The step-by-step incident response flow for platform-impacting incidents.
Incident Trends & Themes
Cross-incident analysis — recurring themes, top contributors, and where to invest.
On-Call Handbook
Everything an on-call engineer needs for a shift on the observability platform.
04 — Operations & Incident Response
Running the platform: on-call, incident response, runbooks, and post-mortems.
2026-05-21 — billing-service elevated latency
- **Incident Commander:** On-call Engineer (Platform SRE)
Post-Mortems
Blameless post-mortems for resolved incidents.
Rotation Schedule
Current on-call rotation and schedule.
Runbook Index
Index of all operational runbooks.
Severity Definitions
Severity definitions so everyone agrees on what SEV-n means.
Alert Rules Catalog
Catalog of alert rules managed by the platform (Terraform/config-driven).
Collector Config Templates
Reusable Alloy / OTel collector configuration templates by workload class.
Dashboard Catalog
Catalog of shared dashboards and the golden-signal starter pack.
Feature Flags & Config Management
How platform feature flags and configuration are managed and rolled out.
05 — Platform Configuration
The knobs: collector configs, alert/dashboard catalogs, sampling and retention.
Retention Policy
Retention windows per signal type and tier.
Sampling Policy
Trace (and log) sampling policy — what we keep, what we drop, and why.
06 — Build & Release
From local dev to production: CI/CD, releases, deploys, and rollbacks.
CI/CD Pipeline Guide
How platform changes flow through CI/CD.
Deployment Runbooks
Step-by-step deployment runbooks per component.
Development Environment Setup
Set up a local dev environment to work on platform config.
Release Notes
Per-release notes.
Release Process
The platform release process — cadence, versioning, and approvals.
Rollback Procedures
How to roll back each component safely.
Cardinality Governance
How the platform governs cardinality — the primary cost and stability risk.
07 — Cost & Governance
Keeping the platform affordable and well-governed: budgets, cardinality, vendors.
Ingest Budget by Team
Ingest budget allocation per team and current consumption.
Integration Specifications
Specifications for integrations between the platform and external systems (IRM, BigPanda, SNOW,
Monthly Cost Reports
Monthly observability cost reports.
Toil Budget Policy
Cap and track operational toil so it doesn't crowd out engineering.
Vendor Registry
Registry of vendors and external services the platform depends on.
Portfolio Initiatives
Active and proposed initiatives in the platform portfolio.
Quarterly OKRs
Quarterly objectives and key results for the platform.
Roadmap
Where the platform is going — themes and milestones.
08 — Strategy & Planning
Direction: roadmap, OKRs, and the initiative portfolio.
09 — Archive
Frozen history.
Breaking Changes Register
Register of breaking changes to platform contracts (label schema, APIs, config).
Deprecated Designs
Deprecated architecture/designs, kept for context.
Historical Releases
Archive of historical releases.
Retired Services
Services retired from the platform, with off-boarding date.
Communication Templates
Copy-paste communication templates for incidents.
ShipSolid Observability Platform
Documentation hub for the **ShipSolid observability platform** — Grafana Cloud (Mimir / Loki /
MDIxAI Alerts — Standards (extracted from current state)
Rule group fields are identical across the 4 env files except for the env suffix.
Grafana Cloud Usage Guide
1.
Grafana Learning Path: From Beginner to Expert in Observability
Understand Grafana, observability concepts, and basic usage.
Telemetry Schema Design — Azure Container Apps on Grafana Cloud
This schema is optimised for: - 50+ Azure Container Apps
ADR-002: Adopt Grafana Cloud for Unified Observability
Accepted 2024-02-01 The Architect Learning Lab runs heterogeneous services across multiple technology stacks (.
ADR-005: Adopt AIOps Pillar for Intelligent Operations
Accepted 2026-03-26 The Architect Learning Lab has mature observability (f-observability), tooling (i-tooling/srekit),
ADR-001: Adopt Eight-Pillar Monorepo Structure
Accepted — pillar table amended (see [Amendments](#amendments) below).
Architecture Overview
The ShipSolid Platform Engineering Lab demonstrates enterprise-grade cloud-native patterns through a
Architectural Design
This document outlines the architectural design of a scalable, modular observability framework
Data Source Strategy
To ensure comprehensive observability across infrastructure, applications, and business workflows,
Collector Deployment Strategy
A key pillar of the observability framework is the standardized and automated deployment of
Faro / RUM Technical Design
**Applies to:** Angular 17+ SPAs **SDK versions:** `@grafana/faro-web-sdk ^2.
Observability Overview
The observability pillar combines Alloy collector examples with a Terraform-managed Grafana Cloud
The Pillar Model
The monorepo is organized into nine alphabetically-prefixed pillars, each representing a distinct
Platform Overview
The platform pillar is currently centered on Terraform workload roots under
RFC-001: Adopt Grafana Cloud for Centralized Observability
- **RFC ID**: rfc-001-adopt-grafana-cloud-for-centralized-observability
Logging Implementation Guidelines
**Applies to:** All teams deploying services to the ShipSolid SRE Observability platform (AKS, Azure
Logging Contract
**Applies to:** All application teams deploying to AKS clusters monitored by the ShipSolid SRE
Metrics Contract
**Applies to:** Application teams exposing custom business or application metrics on the ShipSolid SRE
Onboarding a New Service to the Observability Platform
**Owner:** SRE Team **Last Updated:** 2025-05-01 **Applies to:** Any new service being deployed to
Resource Label & Tag Naming Convention
**Status:** Active **Last Updated:** 2026-03-24 **Applies To:** Grafana Cloud (cost attribution),
Tracing Contract
**Applies to:** Application teams implementing distributed tracing on the ShipSolid SRE Observability
KPIs, SLIs, SLOs & SLAs
Defines the **metrics hierarchy** used to align technical observability signals with business
Incident Notification & Response
**Owner:** SRE Team **Last Updated:** 2025-05-01 **Applies to:** All production alerts routed
Notification & Alerting Strategy
A unified approach to delivering actionable alerts, minimizing noise, and ensuring timely
SRE Toolkit
The `srekit` CLI is a Python-based toolkit for SRE operations, located in
Visualization, Alerting & SLOs
An effective observability system translates raw telemetry into actionable insights through
AKS Helm Implementation Guidelines
**Applies to:** Services deploying to Azure Kubernetes Service (AKS) via Helm charts on the ShipSolid
Alerting Contract
**Applies to:** All teams authoring or requesting Grafana alert rules on the ShipSolid SRE
grafana_tf — How-To Guides
Dashboards are auto-discovered from the `grafana_tf/dashboards/` directory — no changes to any `.
grafana_tf — Operations
All tokens default to a 90-day TTL (`token_ttl_seconds = 7776000`).
grafana_tf — CI/CD Workflows
The platform runs on **HCP Terraform (TFC) with VCS-driven workspaces**.
Azure Key Vault Secrets — Reference
All secrets live in `mf-cc-dt-azrsrp-prd-kv`.
Label-Based Access Control (LBAC)
LBAC restricts what time-series data each Grafana team can query from Mimir and Loki.
grafana_tf — Module Reference
All shared logic lives in `modules/`.
grafana_tf — Products Registry
[`products.
Technical Design — grafana_tf Platform
`grafana_tf/` is the Terraform-based control plane for ShipSolid Foods' Grafana Cloud observability
Grafana Terraform Provider — Capability Audit
Audit of [grafana/grafana](https://registry.
Security, Access & Compliance
Observability data often contains sensitive operational, business, or user-level insights.
Adopting the ShipSolid Central Terraform CI/CD Pipeline
- Org consistency across all ShipSolid Terraform repos (Grafana Cloud IaC, SDLC Portal infra,
CI/CD Overview
The repo currently uses three CI/CD layers:
Code Standards
This document defines the mandatory code standards for all services and infrastructure in this
Migrating to Terraform Cloud (TFC / HCP Terraform)
Migrating to Terraform Cloud (TFC / HCP Terraform) is mostly mechanical, but there's one
Naming Conventions
This document defines the naming patterns for all resources, files, and artifacts in this monorepo.
Pre-commit Hooks
This repo uses [pre-commit](https://pre-commit.
Terraform Driver Styles
The repo runs Terraform two different ways.
Business Alignment & Cost Transparency
The observability framework is not just a technical initiative—it is designed to **deliver
Metric Label Standards for Cost Attribution
**Goal:** Attribute Grafana Cloud metrics ingestion cost to business unit, product, and environment.
AIOps Overview
The **h-aiops** pillar is an experimental sandbox for the in-house SRE Agent and related AIOps
Future-Readiness & Extensibility
The observability framework is architected with scalability, flexibility, and longevity in mind.
Platform & Cloud Maturity Model (L1 → L5)
This model defines maturity progression across core platform and cloud architecture pillars.
Production Readiness Review Template
- **Service**: [FILL] - **Service Team TL**: [FILL]
SLO Document Template
- **Service**: [Full service name] - **Owner**: [Team name]
Confluence Content Templates — SRE / Observability / Platform
**Space:** `OBS` or `PLT` **Parent page:** `Decision Log (ADRs)`
Confluence Data Lifecycle & Governance — SRE / Observability / Platform
The initiative passes through four phases.
Confluence Space Structure — SRE / Observability / Platform Initiative
Use **three dedicated Confluence spaces**, not one.
ADR Template
- **Status**: Proposed | Accepted | Rejected | Superseded
Alert Runbook Template
- **Alert Name**: [FILL: exact name from alerting system — must match 1:1]
Confluence Content Templates — SRE / Observability / Platform
**Space:** `OBS` or `PLT` **Parent page:** `Decision Log (ADRs)`
Service Onboarding Checklist Template
- **Team**: [FILL] - **Service(s)**: [FILL: comma-separated if multiple]
Incident Post-Mortem Template
- **Incident Commander**: [FILL: name] - **Severity**: SEV1 | SEV2 | SEV3
RFC Template
- **RFC ID**: rfc-YYYY-MM-<slug> - **Authors**: [Name(s), Role(s)]