Most alerting is threshold-based: page me when errors exceed N per minute. That catches the outage. It completely misses the bug that hits one user, once, and never trips any count — which is often the most important signal, because it's the brand-new failure mode you've never seen before.
TL;DR: A "novelty" detector alerts the first time a given kind of error is seen within a rolling window, regardless of volume. Back it with Redis SET key 1 NX EX ttl for atomic first-seen detection, normalize signatures so variants of one bug collapse together (and attackers can't inflate the keyspace), and pre-register the metric at zero so Grafana can actually see the 0 → 1 step.
The Problem
You want to know the moment a new class of error appears — a fresh exception type, a page that just started throwing — even if it happens exactly once. A rate(errors) > threshold alert can't do this; a single occurrence never crosses a meaningful threshold. And you can't label the metric with the raw error message to distinguish new from old, because attacker-influenced, unbounded strings as Prometheus labels is a cardinality bomb (a metric-DoS).
The Fix: Atomic First-Seen in Redis
SET key value NX EX ttl is the perfect primitive: it sets the key only if absent and returns whether it did. That "did I just create it?" boolean is your first-seen signal, atomically, with a rolling TTL window baked in:
public boolean isFirstSeen(NoveltyKind kind, Map<String, String> fields) {
String signature = kind.signatureFields().stream()
.map(f -> normalize(fields.get(f)))
.collect(Collectors.joining(FIELD_SEP));
String key = "novelty:" + kind.key() + ":" + sha256(signature);
try {
Boolean firstSeen = redisTemplate.opsForValue().setIfAbsent(key, "1", kind.ttl());
if (Boolean.TRUE.equals(firstSeen)) {
meterRegistry.counter("jo4.novel.errors", "kind", kind.key()).increment();
log.info("NEW {} kind first-seen", kind.key());
return true;
}
return false;
} catch (Exception e) {
// fail-closed: Redis down → treat as not-novel, never a false alert, never break the caller
return false;
}
}
The kind tag is a closed enum (low cardinality — safe as a Prometheus label). The unbounded, attacker-influenced content goes into the hashed signature, never into a label. Grafana alerts on increase(jo4_novel_errors_total{kind}[window]) and fires the instant a new signature appears.
Normalize Signatures So Variants Collapse
NullPointerException at line 42 and ... at line 87 are the same bug. Hash them raw and you get two "novel" alerts — and an attacker can mint unlimited unique messages to spam you. Normalize away the dynamic bits first:
static String normalize(String value) {
if (value == null || value.isBlank()) return "";
String s = value.trim().toLowerCase();
s = s.replaceAll("https?://\\S+", "<url>");
s = s.replaceAll("\\b[0-9a-f]{8}-[0-9a-f-]{20,}\\b", "<uuid>");
s = s.replaceAll("0x[0-9a-f]+", "<hex>");
s = s.replaceAll("\\d+", "#"); // line numbers, ids, counts
s = s.replaceAll("\\s+", " ");
return s;
}
Now all variants of one issue share a signature (one alert, not a storm), and the keyspace is bounded so nobody can inflate it with junk.
The Gotcha: Pre-Register the Counter at Zero
This one cost real debugging time. A lazily-created Prometheus counter first appears already at 1 — and increase() needs a prior sample as a baseline. With no 0 before the 1, the query can't detect the first (often only) occurrence. So we eagerly register every kind's counter at zero on boot:
for (NoveltyKind kind : NoveltyKind.values()) {
meterRegistry.counter("jo4.novel.errors", "kind", kind.key()); // seed at 0
}
Now the first real event is a visible 0 → 1 step the alert can catch. The kind set is a closed enum, so this stays cardinality-safe.
Fail-Closed on the Store
If Redis is unavailable, isFirstSeen returns false — never a false novelty alert, never a broken caller. That direction matters: a novelty detector that fails open would page you on every event during an outage, training you to ignore it. Fail-closed keeps the signal trustworthy, and you lose only the (rare) genuinely-novel event that happened to coincide with the datastore being down — an acceptable trade for an alerting heuristic.
One thing to be honest about: this is a best-effort signal, not a ledger. If your cache is configured to evict under memory pressure, a "seen" key can occasionally disappear before its TTL, and you'll get a repeat alert for a kind you've seen before. For an alerting heuristic that's fine — a duplicate "new error" ping is cheap; a missed brand-new failure is not. If you ever needed exactly-once semantics you'd reach for a persistent set, but for "tell me when something new shows up" the atomic, TTL'd SET NX EX is the right amount of machinery.
Pick the Signature Granularity Deliberately
The signatureFields you hash decide what "a new kind" means. Too coarse (hash only the exception class) and a genuinely new failure on a new page hides inside an existing kind — no alert. Too fine (hash the full message with all its dynamic bits) and normalization can't collapse variants, so every occurrence looks novel and you drown. Choose the fields that match the granularity you actually want to be paged at — often (error_type, location) — and make sure your alert query groups by the same dimensions, so one new combination produces exactly one alert.
Lessons Learned
- Thresholds miss the rare bug. The single-occurrence, never-before-seen failure is often the one you most want paged on. Detect novelty, not just volume.
- Hash the unbounded stuff; label the bounded stuff. Raw messages as labels are a cardinality-DoS. Put them in a normalized hash; keep labels a closed enum.
- Seed counters at zero.
increase()needs a baseline. A lazily-created counter that debuts at 1 is invisible to your first-seen alert.
How do you catch the never-before-seen error? Tell me what works for you.
Building jo4.io - a URL shortener with analytics and real observability for developers.