Why Your Grafana 5xx-Rate Alert Spams DatasourceNoData Every 4 Hours

· 8 min read · Jo4 Team
grafana monitoring debugging devops

For about a week our on-call Slack channel got the same alert, like clockwork, every four hours. Always the same body:

DatasourceNoData
WARN — 5xx error rate > 5% for 5m
Server-error ratio is n/a. Check recent deploys + downstream deps.

3am. 7am. 11am. 3pm. 7pm. 11pm. No actual 5xx errors. No actual outage. Just a metronome of phantom pages that everyone learned to swipe away — which is, of course, the worst possible state for an alerting system to be in.

The bug isn't one thing. It's three things layered on top of each other, and unpicking them taught us something about how Grafana's alerting pipeline actually behaves when a PromQL query has nothing to chew on.

The Alert That Wasn't

First gotcha: DatasourceNoData is not your rule firing. It's a meta-alert Grafana generates when it tried to evaluate your rule and got back an empty result set. The body you see — the summary, description, severity label — that all comes from your original rule's annotations and labels, because Grafana helpfully copies them onto the synthetic alert. So it looks exactly like a real firing of the "5xx error rate > 5% for 5m" rule. It is not.

That distinction matters because it sends you debugging the wrong thing. We spent a couple of hours staring at Spring Boot logs looking for 5xxs that weren't there before we noticed: every alert body said "ratio is n/a", not "ratio is 7.3%". The rule wasn't evaluating to firing. It was failing to evaluate, and the failure path was loud.

Three Things Are Combining

Once we stopped chasing nonexistent 5xxs, the actual chain unfolded.

1. PromQL 0/0 returns empty

The rule's query is the textbook server-error ratio:

sum(rate(http_server_requests_seconds_count{status=~"5.."}[5m]))
/
sum(rate(http_server_requests_seconds_count[5m]))

When traffic is quiet — 3am on a weekday for a developer-tool SaaS — both rate(...) calls evaluate over a 5-minute window with zero samples. Numerator 0, denominator 0. PromQL doesn't return NaN; it returns an empty result set. From Grafana's perspective, the query "returned no data". That's our trigger.

2. noDataState: NoData routes to Slack

Our rule had this:

noDataState: NoData
execErrState: Error

NoData means: when the query returns nothing, raise a DatasourceNoData alert and let it flow through the notification policy tree. Our root policy routes anything matching severity=warn,team=jo4 to Slack with repeat_interval: 4h. So the synthetic alert hits Slack, stays "active" because the query keeps coming back empty, and re-fires on the 4-hour repeat cadence forever.

3. humanizePercentage on nil blows up the template

This was the subtle one. Our annotation template was:

description: Server-error ratio is {{ $values.A | humanizePercentage }}. Check recent deploys + downstream deps.

When the query has data, $values.A is a float and humanizePercentage formats it as "7.3%". When the query returns empty, $values.A is nil. The Go template pipeline tries to coerce nil to float, and the renderer fails with:

template: __alert_jo4-5xx-rate-warn:1: can't convert <nil> to float

Grafana's behavior on a template error during a NoData event is to substitute a placeholder. In our case, the description came out as "Server-error ratio is n/a" — which read, at 3am, like a legitimate rendered alert. It wasn't. It was the renderer giving up.

So: empty query → synthetic NoData alert → template chokes on nil → "n/a" → Slack → 4-hour repeat → indistinguishable from a real 5xx storm.

The Asymmetric Fix

The first instinct is "flip every rule to noDataState: OK and call it done". Don't. The right setting depends on what "no data" means for the specific signal.

Our other two rules in the same file — jo4-saturation-warn and jo4-saturation-crit — both watch mvc_async_executor_saturation. If that metric stops reporting, it's not because the pool is idle; it's because jo4-api is down. We want a page for that. Those keep noDataState: NoData.

The 5xx ratio is different. "No data" there is the legitimate steady-state of any normal-traffic app during a quiet window. There's no error condition; there's just no traffic. Pretending otherwise creates the phantom-page pattern we lived through.

So our 5xx rule alone flips to noDataState: OK. If we ever need a true "is the API responding" alert, that's a separate dedicated rule on up{job="jo4-api"} == 0 — not overloaded onto a ratio metric.

We documented the asymmetry inline so the next person doesn't "fix the inconsistency":

# noDataState=OK (asymmetric vs the two saturation rules above which
# use NoData). The 5xx ratio is 0/0 when there are no HTTP requests in
# the 5m window — common at night / quiet periods. With NoData,
# Grafana fires a `DatasourceNoData` alert via the root policy every
# `repeat_interval` (4h), producing noise that doesn't reflect a real
# condition.
#
# If Prometheus or jo4-api goes fully dark we'd still want to know;
# that signal belongs on a dedicated `up{job="jo4-api"} == 0` rule,
# not overloaded onto this one.
noDataState: OK

That comment is load-bearing. Without it, "make the alerting config consistent" is a tempting-looking PR that re-introduces the bug.

The Guarded Template

Flipping noDataState to OK stops the Slack spam, but it doesn't fix the underlying template-render error. If the query ever does return nil for any other reason (a scrape gap, a recording-rule lag), the description will still throw a render error in Grafana's logs. So we guarded the template too.

Before:

description: Server-error ratio is {{ $values.A | humanizePercentage }}. Check recent deploys + downstream deps.

After:

description: |-
  Server-error ratio is {{ if $values.A }}{{ $values.A | humanizePercentage }}{{ else }}n/a{{ end }}. Check recent deploys + downstream deps.

The {{ if $values.A }} guard renders nothing-aware. When the query has data, you get "Server-error ratio is 7.3%". When it's nil, you get "Server-error ratio is n/a" — intentionally, this time, with no render error in the logs. The "n/a" reads identically to what we were seeing before, but its provenance is completely different: explicit fallback vs. silent renderer failure.

This is the rule we now apply to every annotation template that touches $values.X: if it goes through humanizePercentage, humanizeDuration, printf, or any pipeline that expects a typed scalar, wrap it in {{ if }}. Templates evaluate even in NoData and Error states — they're not skipped.

Verifying

We rolled the change through our sandbox first. Two things to confirm:

  1. The rule's health field in /api/v1/rules should be ok, not error. Pre-fix it was flapping between nodata and error depending on whether template-render or empty-query happened first in the eval cycle. Post-fix: solid ok.
  2. Grafana's grafana.log should be free of can't convert <nil> to float. Pre-fix that line showed up every 60s (our eval interval) during quiet hours. Post-fix: gone.

Then we waited 16 hours on a deliberately quiet sandbox. Pre-fix, that window would have generated four DatasourceNoData Slack messages. Post-fix: zero. Real 5xx synthetics injected into the sandbox still fired the rule correctly with "Server-error ratio is 14.2%" rendering exactly as you'd want.

Lessons Learned

  • DatasourceNoData is not your rule firing — it's Grafana saying it couldn't evaluate. When the alert body has placeholder values where you expect numbers, suspect the meta-alert path before suspecting the underlying metric.
  • noDataState is per-rule for a reason. Saturation metrics going dark = the app is down = alert. Ratio metrics going dark = no traffic = silent. Don't unify them.
  • PromQL 0/0 is empty, not NaN. Any ratio query on bursty or low-traffic services will hit empty windows. Treat it as the default case, not the edge case.
  • Annotation templates evaluate during NoData and Error states. If the template assumes a typed scalar, it will throw. Always wrap humanizePercentage / humanizeDuration in {{ if $values.X }}...{{ end }}.
  • Document asymmetric config inline. A future engineer "normalizing the config" can re-introduce a bug in a one-line PR. A comment block is the cheapest possible defense.
  • A page that always fires teaches the team to ignore pages. Even one phantom alert per day erodes trust in the whole notification pipeline. Fix it the day you notice it.

What other Grafana template gotchas have you hit? Drop the alert body + RCA in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.