Label-Based Access Control (LBAC)

Label-Based Access Control (LBAC)

LBAC restricts what time-series data each Grafana team can query from Mimir and Loki. A product team member can only see metrics and logs tagged with their product’s deployment_environment label — they cannot see data from other products even if they share the same data source.

---

How it works

Access is enforced in two layers, evaluated in order:

User query
    │
    ▼
Layer 1 — Data source permissions (grafana_data_source_permission)
    • Is this team listed? → No  → query denied
    │                      → Yes → continue
    ▼
Layer 2 — LBAC rules (grafana_data_source_config_lbac_rules)
    • Does the query match the team's selector? → No  → no data returned
    │                                           → Yes → data returned
    ▼
Results

Layer 1 — data source permissions sets an explicit allowlist on the Mimir and Loki data sources. Any user not in a listed team is blocked before LBAC is even evaluated.

Gap 1 — permissions and LBAC rules ignored on provisioned sources. Grafana Cloud silently ignores both grafana_data_source_permission and grafana_data_source_config_lbac_rules on Terraform-provisioned data sources (confirmed by Grafana support, 2026-04). For this reason, Mimir and Loki are no longer provisioned by Terraform — operators create them manually in the Grafana UI at the stable UIDs mimir-<uid_prefix> and loki-<uid_prefix> so dashboards and alerts continue to resolve without edits. This also closes Gap 3 (provisioned-source bypass via Explore): without a provisioned source, there is nothing for non-Admin users to query outside the LBAC-enforced manual sources.

Layer 2 — LBAC rules filter the data each team can see. The selector is matched against the deployment_environment label stamped by the Alloy pipeline on every metric and log line — the same value declared per-environment in products.yml. A team only receives time series where the label matches.

Both layers apply identically to the manual Mimir (metrics) and Loki (logs) data sources.

---

SRE access

SRE receives Admin permission on both data sources and a wildcard LBAC rule:

{deployment_environment=~".+"}

This matches every value — unrestricted access. It is explicit rather than a fallthrough so the intent is visible in code and in the Grafana UI.

---

Selector syntax

Environments	Generated selector
Single	{deployment_environment="daia-dev"}
Multiple	{deployment_environment=~"daia-dev|daia-qa|daia-prod"}
SRE wildcard	{deployment_environment=~".+"}

The selector is automatically generated by the lbac module from the deployment_environments list in module "lbac". You never write selectors by hand.

---

Prerequisites

Before LBAC takes effect:

1. Loki — GA on all Grafana Cloud plans. No additional steps.

2. Mimir — Requires:

   • A Grafana Cloud plan that includes GEM (Grafana Enterprise Metrics).
   • The teamHttpHeadersMimir feature toggle enabled on the Grafana instance (set by Grafana Cloud support, not via Terraform).

3. Manual LBAC data sources — per Gap 1 above, LBAC rules cannot bind to Terraform-provisioned data sources. Each stack needs two data sources created by hand in Connections → Data sources → Add data source. The checklist (name, UID, type, default flag, URL, basic-auth user, KV password secret name) is available as a Terraform output after the first apply, driven by meta.sre.lbac_datasources in products.yml:

   terraform -chdir=environments/<stack> output -json lbac_datasources_setup | jq
   

   The UIDs must match mimir-<uid_prefix> / loki-<uid_prefix> exactly — dashboards and alerts reference them by these strings. After saving, the UID is immutable, so type carefully.

   Fetch each basic-auth password from Key Vault using the password_kv field from the output:

   az keyvault secret show --vault-name mf-cc-dt-azrsrp-prd-kv \
     --name "$(terraform -chdir=environments/<stack> output -raw lbac_datasources_setup | jq -r .mimir.password_kv)" \
     --query value -o tsv
   

   Once both DS are saved in the UI, set the matching UIDs in meta.sre.lbac_datasources.<stack>.{mimir,loki}.uid in products.yml:

   meta:
     sre:
       lbac_datasources:
         shipsoliddev:
           mimir: { name: "mimir-shipsoliddev", type: prometheus, default: true,  uid: "mimir-shipsoliddev" }
           loki:  { name: "loki-shipsoliddev",  type: loki,       default: false, uid: "loki-shipsoliddev" }
   

   Until a stack’s Mimir and Loki UIDs are non-empty, module.lbac no-ops (the permission and config_lbac_rules resources are gated by count = … != "" ? 1 : 0). No tfvars change is needed — products.yml is the single source of truth.

   Mimir — additional config on the data source’s Settings page (needed to match prior provisioned behaviour; dashboards expect Mimir-specific features):

   • HTTP method: POST (PromQL payloads can exceed GET URL limits).
   • Prometheus type: Mimir, version 2.9.1.
   • Exemplars → add one entry: traceID → Internal link → data source Tempo → label View Trace in Tempo.

   Loki — additional config:

   • Derived fields (two entries, both Internal link → data source Tempo):
     • Name TraceID, regex traceID=(\w+), URL ${__value.raw}, label View Trace in Tempo.
     • Name TraceID_json, regex "traceId":"(\w+)", URL ${__value.raw}, label View Trace in Tempo.
   • Max lines: 5000.

   Stack activation status:

   • shipsoliddev: LBAC active.
   • shipsolid: pending manual DS creation.

---

Grafana Cloud auto-provisioned data sources (Gap 3 caveat)

Every Grafana Cloud stack ships with ten auto-provisioned data sources that Terraform does not create and cannot delete:

• grafanacloud-<uid_prefix>-prom (Prometheus; marked default by Grafana Cloud)
• grafanacloud-<uid_prefix>-logs (Loki)
• grafanacloud-<uid_prefix>-traces (Tempo)
• grafanacloud-<uid_prefix>-profiles (Pyroscope)
• grafanacloud-<uid_prefix>-ngalertmanager (Alertmanager)
• grafanacloud-<uid_prefix>-cardinality-management (Grafana cardinality)
• grafanacloud-<uid_prefix>-graphite (Graphite)
• grafanacloud-<uid_prefix>-usage-insights (Loki)
• grafanacloud-<uid_prefix>-alert-state-history (Loki)
• grafanacloud-k6 (k6 proxy, stack-independent)

Their expected UID strings are exposed as module.datasources.cloud_datasource_uids so downstream modules can reference them. Terraform does not data "grafana_data_source" them (an initial attempt failed when a subset returned 404 against the SA token) and does not manage them as resources (the integration sync silently reverts writes). The actual subset present on a given stack varies with the active integrations; consumers that reference a UID which isn’t provisioned will get “Data source not found” on the affected panel only.

LBAC impact (Gap 1 carry-over). grafana_data_source_config_lbac_rules and grafana_data_source_permission cannot bind to these auto-provisioned sources. A non-SRE user who opens Explore on grafanacloud-<uid_prefix>-prom or -logs will get unrestricted results — the LBAC scoping does not apply. This is the same bypass class that motivated removing the TF-provisioned Mimir/Loki in favour of manual LBAC DS; it is not closed for the auto DS and there is no supported way to close it short of Grafana Cloud itself changing how auto-provisioning interacts with LBAC.

Operational guidance.

• Dashboards and alerts should reference module.datasources.mimir_uid / .loki_uid (LBAC-enforced manual sources). The module.datasources.cloud_datasource_uids.* references are for Admin-only dashboards, platform integrations, or tooling that explicitly wants unscoped access.
• When creating the manual Mimir LBAC DS with “Default data source: yes” (per the Prerequisites table above), first open grafanacloud-<uid_prefix>-prom in the UI and uncheck its default flag — Grafana allows only one default and the manual LBAC DS should win.
• Do not attempt to delete the auto DS via API; Grafana Cloud re-provisions them on stack events.

---

How-to: add a new product team to LBAC

LBAC teams are fully auto-derived from products.yml — no .tf files need to change.

Step 1 — Add the product block to grafana_tf/products.yml with at least one active environment:

my-product:
  platform: aks
  team:
    name: "My Product Team"
    email: "myteam@shipsolid.com"
    dashboardRole: Editor
    members: []
  terraform_envs: [shipsoliddev]
  grafana:
    folder_key: golden-aks-my-product
    folder_title: Golden-AKS-MyProduct
  environments:
    dev:
      status: active
      deployment_environment: my-product-dev
      use_contact_points:
        - teams-my-product-nonprod

Step 2 — Run the generator and commit:

python grafana_tf/tools/generate.py
git add grafana_tf/alert_rules/generated/ grafana_tf/dashboards/generated/

Step 3 — Apply in shipsoliddev first, validate, then apply in shipsolid:

cd grafana_tf/environments/shipsoliddev
terraform plan   # should show new folder, team, LBAC rules, dashboards, alert groups
terraform apply

The module.rbac and module.lbac calls in main.tf both iterate over module.computed.products, so adding the product to products.yml is all that’s required. No manual wiring needed.

---

How-to: add a new environment to an existing product

Step 1 — Add the new environment block to products.yml with status: active:

daia:
  environments:
    staging:           # new
      status: active
      deployment_environment: daia-staging
      use_contact_points:
        - teams-aks-daia-nonprod

Step 2 — Run the generator and commit:

python grafana_tf/tools/generate.py
git add grafana_tf/alert_rules/generated/ grafana_tf/dashboards/generated/

Step 3 — Apply. The lbac module derives the deployment_environments list from active environments in products.yml — no manual edits to main.tf are needed. Rules are replaced atomically.

---

How-to: add a member to a product team

Edit grafana_tf/teams.yml and add the user’s Grafana login email:

daia:
  name: "DIA - AMS DAIA"
  members:
    - john.smith@shipsolid.com

Then apply. The user must have logged into Grafana via SSO at least once before Terraform can resolve their email to a Grafana user ID.

---

How-to: remove a product from LBAC

1. Remove the team entry from module "lbac" lbac_teams in both main.tf files.
2. Remove the team entry from module "rbac" additional_teams in both main.tf files.
3. Remove the team from grafana_tf/teams.yml.
4. Apply.

Terraform will destroy the grafana_data_source_permission entries and the LBAC rules for that team. Remaining teams are unaffected — the lbac module replaces all rules atomically on each apply.

---

Troubleshooting

User sees no data after being added to a team

• Confirm the environment has status: active in products.yml — planned environments are excluded from LBAC rules.
• Confirm the deployment_environment label is actually present on the metrics/logs being queried. Check the Alloy pipeline config for that product.
• Confirm the deployment_environment value in products.yml exactly matches the label stamped by Alloy (case-sensitive, no trailing whitespace).

terraform apply errors with “user not found” on grafana_team_member

The user has not logged into Grafana yet. They must complete at least one SSO login before Terraform can resolve their email. Ask them to log in, then re-apply.

User is in a team but still sees all data (no LBAC filtering)

• For Mimir: verify the teamHttpHeadersMimir feature toggle is enabled. Contact Grafana Cloud support if unsure — this cannot be set via Terraform.
• Check that the Grafana Cloud plan includes GEM. Without GEM, Mimir LBAC is silently ignored; Loki LBAC still applies.

A new environment’s data is visible to other teams

The Alloy pipeline for the new environment must stamp deployment_environment with a value that is not in any other team’s selector. Verify the label value in the Alloy config matches the deployment_environment set in products.yml.

---

Key files

File	Purpose
grafana_tf/modules/lbac/main.tf	LBAC module — permissions and rules resources
grafana_tf/modules/lbac/variables.tf	Module inputs
grafana_tf/products.yml	Source of truth — team, environments, and deployment_environment
grafana_tf/modules/observability-stack/main.tf	Derives lbac_teams from products.yml active environments
environments/*/main.tf	Wires module.computed, module.rbac, and module.lbac together