Known issues
Recurring limitations and accepted trade-offs, consolidated from notes scattered across reliability.md, supply-chain.md, testing.md, and resilience-patterns.md “what this doesn’t cover” sections, plus code-level findings from writing those pages. This is not an incident log — this is a personal lab with no on-call and no incident process (see README.md § Ownership Boundary) — it is a single place to check before assuming a gap is new.
Planned/deferred work that’s already tracked with a target is in README.md § Roadmap (SLO rules default-off, Pyroscope profiling, Faro source-map upload, datastore HA operator migration) and is not duplicated here.
| ID | Symptom | Workaround | Root cause | Reference | Status |
| ------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------ | ----------------------------------------------------------------------------- |
| KI-001 | Sustained Redis outage causes notification-svc to tight-loop redelivery on affected messages (no backoff) | /readyz pulls the pod from Service rotation for new HTTP traffic, but the consumer thread keeps retrying at full speed | Intentional trade-off: requeue=True on redis.RedisError avoids permanently dead-lettering live traffic on a short blip, at the cost of no backoff on a long one | consumer.py:271-281, resilience-patterns.md | Open — accepted trade-off, not yet backed by a max-attempt or backoff policy |
| KI-002 | OutboxRelayWorker retries every 5s indefinitely on a sustained RabbitMQ outage, no exponential backoff | None — acceptable at lab volume | Simplicity over production-grade backoff; documented directly in the worker’s own header comment | OutboxRelayWorker.cs, resilience-patterns.md | Open — would need backoff before this pattern goes to a real production queue |
| KI-003 | None of the app-level resilience patterns (retry, circuit breaker, backoff, DLQ) are exercised by automated fault injection | Manual validation only (kill a pod, watch Jaeger/logs) | No chaos-engineering tooling in the repo; the k6 job in k8s/loadtest/ generates volume, not faults | resilience-patterns.md § What is not implemented | Open |
| KI-004 | docs/README.md’s “Reviews & audits” table links to reviews/2026-07-08-principal-staff-review.md, and a docstring in consumer.py references the same review — the file was deleted in the most recent commit (9953b611) and neither reference was updated | None yet | The review’s finding (dedup TTL mismatch, its own §2.2) was fixed in code before the review file was deleted, but the two backlinks and this doc’s own code snippet were left stale — the snippet has since been corrected as part of this pass, the two links have not | git log --diff-filter=D -- "docs/reviews/*" | Open — needs an owner decision: restore the review file, or remove the two dangling references (docs/README.md row + the “see docs/reviews/…” comment/reference) |
| KI-005 | SLO-burn-rate-driven HPA not wired — the prod overlay’s HPA scales on raw CPU utilization, not the burn-rate recording rules in slo-rules.yaml | Illustrative CPU-based HPA in the meantime | Needs a custom-metrics adapter (e.g. prometheus-adapter) not installed by deploy-local.sh | reliability.md § What this doesn’t cover | Open |
| KI-006 | No VerticalPodAutoscaler — resource requests/limits are fixed per environment, no right-sizing automation | Manual tuning | Not needed at lab scale | reliability.md | Open — add if right-sizing becomes a concern |
| KI-007 | No topology spread constraints — pod anti-affinity handles same-node avoidance only, no AZ-level spread | N/A — single-node k3d has no AZ concept | Would need topologySpreadConstraints with topology.kubernetes.io/zone for a real multi-AZ prod cluster | reliability.md | Open |
| KI-008 | Admission-time image-signature verification not wired — CI signs and verifies at build time, but nothing in the cluster refuses to schedule an unsigned image at deploy time | None | Needs a cluster-scoped admission controller (sigstore/policy-controller, connaisseur, or kyverno) not installed | supply-chain.md § Admission enforcement | Open |
| KI-009 | Dependency-vulnerability remediation is ad-hoc — CI gates PRs on dotnet list package --vulnerable / pip-audit / npm audit, but nothing proactively opens upgrade PRs | Manual, as-needed bumps | No Dependabot/Renovate configured | supply-chain.md § What this doesn’t cover | Open |
| KI-010 | No SLSA provenance attestation — cosign signing proves “built by this workflow,” not build hermeticity | None | Would need slsa-framework/slsa-github-generator alongside the existing SBOM/signing steps | supply-chain.md | Open |
| KI-011 | No proto schema contract test between gateway-api’s client stub and order-api’s server implementation | None — drift would surface at runtime | No schema registry in the lab | testing.md § What is not covered | Open |
| KI-012 | RabbitMQ consumer backpressure is untested | None | Requires a real broker under sustained load; out of scope for the current unit-test suite | testing.md | Open |
| KI-013 | Concurrent (not sequential) idempotency-key submissions to CreateOrder are untested | DB unique index is the backstop, but it’s not exercised by a concurrency test | Sequential-retry case is covered; a true concurrent-race test wasn’t written | testing.md | Open |
Local graph
Linked from 5 notes
Reliability controls
Workload-level Kubernetes controls protecting Signal Forge availability during disruption: PodDisruptionBudgets, anti-affinity, and graceful shutdown.
Resilience patterns
Application-level failure handling in Signal Forge: retries, circuit breakers, backoff, and delivery-safety patterns for downstream dependency failures.
Runbooks
Troubleshooting playbooks for every known Signal Forge failure mode, from missing traces to Grafana Cloud export errors.
Supply-chain security
What CI verifies before a Signal Forge image ships: vulnerability scanning, SBOM generation, and cosign keyless signing.
SignalForge Documentation
Documentation hub for the SignalForge OTel Microservices Validation Lab — architecture, services, API, deployment, observability, and operations.
Related notes
Networking & TLS
Network-plane security for Signal Forge: NetworkPolicy default-deny model, Ingress TLS via cert-manager, and the k3d flannel enforcement caveat.
Reliability controls
Workload-level Kubernetes controls protecting Signal Forge availability during disruption: PodDisruptionBudgets, anti-affinity, and graceful shutdown.
Resilience patterns
Application-level failure handling in Signal Forge: retries, circuit breakers, backoff, and delivery-safety patterns for downstream dependency failures.
Runbooks
Troubleshooting playbooks for every known Signal Forge failure mode, from missing traces to Grafana Cloud export errors.