Grafana Cloud Deployment

Grafana Cloud Deployment

Alloy exports all signals to Grafana Cloud (Tempo, Mimir, Loki) when credentials are configured. When credentials are absent, cloud exporters are no-ops — local backends are unaffected.

---

Credential architecture

Grafana Cloud uses per-signal instance IDs as Basic Auth usernames — one for Tempo (traces), one for Mimir (metrics), one for Loki (logs). A single shared API key is the password for all three signals.

flowchart TD
    Token["Shared Access Policy token (glc_...)<br/>grafana-example-org-alloy-writer-example-org-token<br/>Scopes: metrics:write logs:write traces:write"]

    subgraph GC["Grafana Cloud Stack"]
        Tempo["Tempo (traces)<br/>instance ID: 1541184<br/>tempo-prod-xx.grafana.net"]
        Mimir["Mimir (metrics)<br/>instance ID: 3102416<br/>prometheus-us-central2.grafana.net"]
        Loki["Loki (logs)<br/>instance ID: 1546883<br/>logs-prod-037.grafana.net"]
    end

    Token -->|authenticates| Tempo
    Token -->|authenticates| Mimir
    Token -->|authenticates| Loki

Important: The Grafana Cloud API key (grafana-example-org-cloud-api-key, prefix glsa_) is a Grafana organisation service account token — it authenticates with the Grafana frontend (example-org.grafana.net) but is rejected by Mimir/Loki/Tempo data ingestion endpoints (HTTP 401). Always use grafana-example-org-alloy-writer-example-org-token (prefix glc_) for data plane writes.

---

Endpoint format requirements

The raw Grafana Cloud endpoint URLs require path/format adjustments before they work with Alloy:

Signal	Raw Grafana Cloud URL	Required format
Traces (OTLP gRPC)	https://tempo-prod-29-....grafana.net	tempo-prod-29-....grafana.net:443 — strip https://, append :443
Metrics (Prometheus remote_write)	https://prometheus-us-central2.grafana.net/api/prom	https://prometheus-us-central2.grafana.net/api/prom/push — append /push
Logs (Loki push)	https://logs-prod-037.grafana.net	https://logs-prod-037.grafana.net/loki/api/v1/push

Why Prometheus remote_write, not OTLP HTTP, for metrics? The Helm chart’s Alloy destinations use type: prometheus, which speaks Prometheus remote_write. App OTLP metrics arriving at alloy-receiver are converted OTLP → Prometheus inside Alloy before shipping. This gives a single ingestion path for scraped infra metrics and converted app metrics, and matches Grafana Cloud’s dashboard/query UX.

Why the difference for Tempo? gRPC uses HTTP/2 transport. Alloy’s otelcol.exporter.otlp expects a host:port endpoint without a URL scheme.

scripts/fetch-grafana-cloud-conf-from-akv.sh applies these adjustments automatically — append /push to the Mimir URL if missing, append /loki/api/v1/push to Loki, append :443 to Tempo.

---

Azure Key Vault integration

Credentials are stored in Azure Key Vault (example-org-prd-kv) under the grafana-example-org-* prefix. The AKV coordinates (tenant/subscription/RG/vault name) live in the env file named by conf.yml’s monitoring.grafana_cloud.use_env — as ARM_TENANT_ID / ARM_SUBSCRIPTION_ID / Resource_Group / Azure_KeyVault, the same keys the legacy Makefile flow (make secrets-fetch-akv) already reads from there. Safe to track in git — these are IDs/names, not credentials.

AKV secret name	env file key (== K8s Secret key)	Notes
grafana-example-org-alloy-writer-example-org-token	GRAFANA_CLOUD_API_KEY	glc_ access-policy token — required for data-plane writes
grafana-example-org-cloud-tempo-endpoint	GRAFANA_CLOUD_TEMPO_ENDPOINT	fetch script appends :443
grafana-example-org-cloud-tempo-username	GRAFANA_CLOUD_TEMPO_USER
grafana-example-org-cloud-mimir-endpoint	GRAFANA_CLOUD_MIMIR_ENDPOINT	fetch script appends /push if missing
grafana-example-org-cloud-mimir-username	GRAFANA_CLOUD_MIMIR_USER
grafana-example-org-cloud-loki-endpoint	GRAFANA_CLOUD_LOKI_ENDPOINT	fetch script appends /loki/api/v1/push if missing
grafana-example-org-cloud-loki-username	GRAFANA_CLOUD_LOKI_USER
grafana-example-org-faro-api-endpoint	FARO_COLLECTOR_URL	frontend runtime env
grafana-example-org-faro-sourcemap-token	FARO_API_KEY	webpack build arg

---

Setup

1. Azure auth

The fetch script first resolves the target env file from conf.yml’s monitoring.grafana_cloud.use_env, then sources it to load ARM_TENANT_ID / ARM_SUBSCRIPTION_ID / Resource_Group / Azure_KeyVault (vault coordinates) and ARM_CLIENT_ID / ARM_CLIENT_SECRET (service-principal credentials, if set).

# Interactive login (your user credentials) — used if ARM_CLIENT_ID/ARM_CLIENT_SECRET
# are empty in the env file:
az login

Or fill in ARM_CLIENT_ID / ARM_CLIENT_SECRET in the env file itself for service-principal auth — the script logs in as that principal automatically.

2. Fetch into the env file (in place)

# Preview the changes:
./scripts/fetch-grafana-cloud-conf-from-akv.sh --dry-run

# Apply in place (creates <env file>.bak):
./scripts/fetch-grafana-cloud-conf-from-akv.sh

The script updates only the nine GRAFANA_CLOUD_*/FARO_* keys in the env file named by use_env (appending any that don’t exist yet). Comments, ordering, and every other line (including the ARM_*/Resource_Group/Azure_KeyVault coordinates) are preserved — see smoke-test-conf-updater.sh for the regression test that enforces this.

3. Deploy (re-materialises the Secret)

./deploy-local.sh --skip-cluster --skip-build

deploy-local.sh (see Local Deployment) sources the same env file named by monitoring.grafana_cloud.use_env (required — it fails fast if empty or missing) and writes the grafana-cloud-secrets K8s Secret into both otel-lab (apps/FARO consumers) and monitoring (Helm chart‘s Alloy). Before helm upgrade, a contract validator asserts every key referenced by the rendered values file is present in the Secret — rename on either side fails fast.

---

Verifying cloud export is working

# Check Alloy receiver logs for successful exports
kubectl -n monitoring logs daemonset/grafana-k8s-alloy-receiver --tail=100 \
  | grep -E "grafana_cloud|export|error" | head -30

# Generate a trace
curl -s http://localhost:8080/api/projects

# Then check Grafana Cloud:
# → Explore → Tempo → search by service.name=gateway-api
# → Explore → Mimir → query: traces_spanmetrics_calls_total
# → Explore → Loki → query: {namespace="otel-lab"}

Expected log output when working:

level=debug component=otelcol.exporter.otlp.grafana_cloud_traces msg="successfully exported"
level=debug component=otelcol.exporter.otlphttp.grafana_cloud_metrics msg="successfully exported"

---

Troubleshooting

”endpoint is empty” in Alloy logs

The Secret was not applied or Alloy was not restarted after applying it.

# Verify the secret exists
kubectl -n otel-lab get secret grafana-cloud-secrets -o json | jq '.data | keys'

# Verify Alloy reads the env var
kubectl -n monitoring exec daemonset/grafana-k8s-alloy-receiver -- env | grep GRAFANA

# If env var is missing, Alloy needs a restart to pick up the new secret
kubectl -n monitoring rollout restart daemonset/grafana-k8s-alloy-receiver

”401 Unauthorized”

Wrong API key or wrong instance ID for that signal type.

make secrets-show   # verify all 7 values are non-empty and correct

Each signal type has its own instance ID. Using the Tempo ID for Mimir (or vice versa) causes 401 errors on that signal only.

”connection refused” for Tempo

Tempo endpoint must be host:443 without https://. If it includes https:// the gRPC transport fails.

make secrets-show
# GRAFANA_CLOUD_TEMPO_ENDPOINT should be:  tempo-prod-xx....grafana.net:443
# NOT: https://tempo-prod-xx....grafana.net

Re-run make secrets-fetch-akv to re-apply the adjusted format.

AKV authentication failing

source .env
az login --service-principal \
  --username  "$ARM_CLIENT_ID" \
  --password  "$ARM_CLIENT_SECRET" \
  --tenant    "$ARM_TENANT_ID"

# Verify SP has Key Vault Secrets User role
az keyvault show --name example-org-prd-kv \
  --query "properties.accessPolicies[?objectId=='<SP_OBJECT_ID>']"

# List available secrets
az keyvault secret list --vault-name example-org-prd-kv \
  --query "[?starts_with(name,'grafana-example-org')].name" -o tsv

---

Graceful degradation

When cloud credentials are absent:

• optional: true on every secretKeyRef means Alloy pods start normally
• Cloud exporters log: level=error msg="failed to export" err="endpoint is empty"
• Local backends (Jaeger, Prometheus, Loki if deployed) receive all signals normally
• No reconfiguration needed to switch modes — just apply or remove the Secret

---

Credential rotation

When the Grafana Cloud API key is rotated:

1. Update the secret in AKV:

   az keyvault secret set --vault-name example-org-prd-kv \
     --name grafana-example-org-cloud-api-key --value "glsa_newtoken..."
   

2. Re-fetch and apply:

   make secrets-fetch-akv
   

3. Alloy picks up the new env vars automatically on the next pod restart (or force it):

   kubectl -n monitoring rollout restart daemonset/grafana-k8s-alloy-receiver
   

The old key remains valid until explicitly revoked in the Grafana Cloud Access Policies UI.