Grafana as Code: Surviving Destroy-Recreate With Provisioning Files

· 9 min read · Jo4 Team
grafana devops monitoring automation

If you set up Grafana the way most people set up Grafana — log in, click around, add a datasource, build a dashboard, wire up Slack — every piece of that state lives in one sqlite file inside the container. Rebuild the droplet and you've started over: no dashboards, no alert rules, no contact points, no notification policies.

We learned this the way you'd guess: we destroyed our observability droplet on purpose to test the rebuild. Everything came back except the entire Grafana installation we'd spent an afternoon configuring. Here's the provisioning layout we now ship, and the one non-obvious gotcha (the pinned datasource UID) that keeps dashboards and alerts working across rebuilds.

The Lesson From a Real Rebuild

The original setup: a jo4-impress droplet running Prometheus + Grafana via docker-compose. Datasource via the Grafana HTTP API. Dashboards built in the UI. Three alert rules clicked through the alert builder. Slack contact point and notification policy the same way. All of it persisted in grafana_data — a Docker named volume backing Grafana's sqlite.

Then we ran a planned destroy-recreate to confirm setup-impress.sh could rebuild from scratch. The script worked. The droplet came back. Compose came up. We SSH-tunneled into Grafana — and got an empty install. No datasource, no dashboards, no alerts, no Slack.

The volume was new because the droplet was new. Anything not in the repo was gone. So we moved everything to provisioning files.

What Provisioning Covers

Grafana reads YAML/JSON from /etc/grafana/provisioning/ on container start, and again periodically for anything that supports live-reload. The pieces we needed:

  • Datasource (datasources/*.yaml) — points Grafana at Prometheus.
  • Dashboards (dashboards/*.yaml provider + dashboards/jo4/*.json for the actual JSON).
  • Alert rules (alerting/rules.yaml) — the rules themselves, with data, expressions, thresholds.
  • Contact points (alerting/contact-points.yaml) — where alerts go (Slack for us).
  • Notification policies (alerting/policies.yaml) — routing rules from rule → contact point.

The compose mount makes that whole tree read-only inside the container:

volumes:
  - grafana_data:/var/lib/grafana
  - ./grafana/provisioning:/etc/grafana/provisioning:ro

The named volume stays for transient sqlite state (sessions, ad-hoc dashboards an operator might create in the UI during incident response, etc). The repo tree is the source of truth for everything that matters.

Pin the Datasource UID

The single most important line in this whole setup is the datasource UID. Grafana generates one automatically if you don't supply one — something like PBFA97CFB590B2093. Dashboards and alert rules don't reference datasources by name; they reference them by UID. Auto-generated UIDs differ per install. After a destroy-recreate, every dashboard and every alert references a UID that doesn't exist on the new instance, and they all break with "datasource not found."

Pin it:

apiVersion: 1

datasources:
  - name: Prometheus
    uid: jo4-prometheus
    type: prometheus
    access: proxy
    url: http://prometheus:9090
    isDefault: true
    editable: false
    jsonData:
      timeInterval: 30s
      httpMethod: POST

uid: jo4-prometheus. Every dashboard JSON references "datasource": {"type": "prometheus", "uid": "jo4-prometheus"}. Every alert rule references datasourceUid: jo4-prometheus. Destroy the droplet a hundred times — the UID is the same on every rebuild, every reference keeps working.

editable: false is the other guardrail. An operator can't accidentally tweak this in the UI and have the change vanish on the next reload (which is exactly what would happen, because the file always wins).

Dashboards as JSON

The dashboard provider is one file that says "look in this directory":

apiVersion: 1

providers:
  - name: jo4
    folder: jo4
    type: file
    disableDeletion: false
    updateIntervalSeconds: 30
    allowUiUpdates: true
    options:
      path: /etc/grafana/provisioning/dashboards/jo4
      foldersFromFilesStructure: false

Notice what's not here: folderUid. Pinning a folder UID conflicts with the folder that alerting provisioning creates by name a few milliseconds earlier in the boot sequence — Grafana tries to make a second folder with the same name, errors, and the container exits. Leave folderUid unset and the dashboard provider reuses the folder alerting just created. (That's its own gotcha worth a separate post; short version: provisioning order matters, alerting runs first, and the folder you reference in folder: here had better match what alerting will create.)

The actual dashboard JSON lives at dashboards/jo4/*.json inside the provisioning tree. The relevant property in each panel:

"datasource": { "type": "prometheus", "uid": "jo4-prometheus" }

That's the binding that survives rebuilds, because the UID was pinned upstream.

allowUiUpdates: true lets an operator iterate on a dashboard in the UI during incident response. The discipline is to copy the JSON back into the repo before the next deploy, because at deploy time the file wins again.

Alert Rules in YAML

Each alert rule has three parts that have to line up: a data block (one entry per query/expression), a condition field pointing to the refId that fires the alert, and the for duration that gates flapping. Here's the warn-level executor saturation rule, copied verbatim:

apiVersion: 1
groups:
- orgId: 1
  name: jo4-async-executor
  folder: jo4
  interval: 1m
  rules:
  - uid: jo4-saturation-warn
    title: Executor saturation high
    condition: B
    data:
    - refId: A
      queryType: ''
      relativeTimeRange:
        from: 600
        to: 0
      datasourceUid: jo4-prometheus
      model:
        expr: mvc_async_executor_saturation
        instant: true
        intervalMs: 1000
        maxDataPoints: 43200
        refId: A
    - refId: B
      queryType: ''
      relativeTimeRange:
        from: 0
        to: 0
      datasourceUid: __expr__
      model:
        expression: $A > 0.8
        intervalMs: 1000
        maxDataPoints: 43200
        refId: B
        type: math
    noDataState: NoData
    execErrState: Error
    for: 5m
    annotations:
      description: Saturation is {{ $values.A }}. Pool is filling up; at 1.0 we reject new connections.
      summary: MVC async executor saturation > 0.8 for 5m
    labels:
      severity: warn
      team: jo4

The shape that matters:

  • refId: A is the PromQL query against the real datasource (datasourceUid: jo4-prometheus — the pinned UID again).
  • refId: B is a math expression against the synthetic __expr__ datasource that ships with Grafana. expression: $A > 0.8 is what turns a number into a boolean firing condition.
  • condition: B picks B as the rule's firing input.
  • uid: jo4-saturation-warn is the rule's own stable identifier — pin it for the same reason we pin the datasource UID. If you ever export, re-import, or reference a rule from elsewhere, the UID has to survive rebuilds.
  • for: 5m is the anti-flap — the condition must hold for 5 minutes before firing.

One thing worth flagging: noDataState matters per-rule. For ratio-based rules like 5xx-error-rate, NoData is the wrong default because zero-traffic windows are 0/0, which evaluates to NoData, which fires a synthetic DatasourceNoData alert every repeat_interval until traffic returns. Set those to noDataState: OK and use a separate up{} rule for "is the app even running."

Contact Point With Env-Var Interpolation

Grafana expands $VAR and ${VAR} references in provisioning files at startup, which means the Slack webhook URL never has to touch the repo:

apiVersion: 1

contactPoints:
  - orgId: 1
    name: slack-alerts
    receivers:
      - uid: slack-alerts-1
        type: slack
        settings:
          url: $SLACK_IMPRESS_WEBHOOK_URL
          title: '{{ .CommonLabels.alertname }}'
          text: |
            {{ range .Alerts }}*{{ .Labels.severity | toUpper }}* — {{ .Annotations.summary }}
            {{ .Annotations.description }}
            {{ end }}
        disableResolveMessage: false

The compose side passes the value through from a GitHub Actions secret to the container env:

- SLACK_IMPRESS_WEBHOOK_URL=${SLACK_IMPRESS_WEBHOOK_URL:?SLACK_IMPRESS_WEBHOOK_URL must be exported}

Two things to call out:

The :?required form. If the env var is unset, compose hard-fails on up. We do this on the admin password too. Silent defaults here would mean "deploy succeeded; Grafana came up with a blank-password admin and a contact point that posts to nowhere" — the worst kind of broken.

The webhook URL is bearer-equivalent. Anyone with it can post to that Slack channel. It never goes in the file or the repo. Flow: GitHub Secret → workflow env → compose env → container env → Grafana env-var interpolation at start.

One more compose env worth showing: GF_SERVER_ROOT_URL. Grafana uses this when constructing links inside Slack messages. Without it, alert links default to http://localhost:3000 — useless on a phone:

- GF_SERVER_ROOT_URL=https://graf.jo4.io
- GF_SERVER_DOMAIN=graf.jo4.io

You only notice this the first time an alert fires for real.

Notification Policy

The routing rule from "rule fires" → "Slack message" is one file:

apiVersion: 1

policies:
  - orgId: 1
    receiver: slack-alerts
    group_by:
      - grafana_folder
      - alertname
    group_wait: 30s
    group_interval: 5m
    repeat_interval: 4h

Root policy — every alert lands in slack-alerts unless a child route catches it first. One Slack channel for everything is fine at our scale. Grouping on (grafana_folder, alertname) coalesces two pods flapping the same rule into one message. group_wait: 30s gives related alerts time to batch. repeat_interval: 4h is "still on fire, here's another nudge."

The tuning matches what we'd set up clicking through the UI. The difference is that now it lives in the repo, gets code-reviewed, and survives rebuilds.

Testing the Rebuild

The whole point is destroy-recreate. So we tested it for real:

  1. doctl droplet delete jo4-impress -f
  2. Run the bootstrap workflow — provisions a fresh droplet, installs Docker, clones the repo, sources the env file, docker compose up -d.
  3. SSH-tunnel to Grafana, log in.

What we saw on first paint, without touching a single setting:

  • Datasource present, UID jo4-prometheus, marked Default. Test → green.
  • Folder jo4 exists (created by alerting, picked up by dashboards).
  • Dashboards loaded, panels rendering Prometheus data.
  • Three alert rules in the jo4-async-executor group, all evaluating.
  • Contact point slack-alerts configured, webhook URL interpolated from env.
  • Notification policy routing to slack-alerts.

Operator action between "droplet doesn't exist" and "fully wired Grafana receiving Slack alerts": SSH tunnel, login, look around.

Lessons Learned

  • Anything you set up via the Grafana UI lives in sqlite and dies with the volume. If you care about reproducibility, it has to be a provisioning file in the repo.
  • Pin the datasource UID. Auto-generated UIDs differ per install; every dashboard and alert reference breaks across rebuilds without a pinned UID. One line, enormous payoff.
  • Don't pin folderUid on the dashboard provider if alerting provisioning creates the same folder — they'll race and the container will exit. Let alerting create the folder; reference it by name from dashboards.
  • Grafana expands $VAR references in provisioning files at start. Use it for secrets like Slack webhook URLs — never commit the URL itself.
  • :?required env-var defaults at the compose layer turn "silent broken deploy" into "loud failed deploy." Use it for admin passwords and webhook URLs.
  • Set GF_SERVER_ROOT_URL. Alert links default to localhost otherwise, which makes Slack notifications useless on a phone.
  • Test the destroy-recreate before you need it. The whole reason we found these gotchas was running the rebuild deliberately, in a window where downtime didn't matter.

Have your own provisioning patterns? What survived your last destroy-recreate? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.