"Did this page change since I last looked at it?" sounds trivial until you try to answer it well. Refetch the URL, compare the bytes, done? Not even close — a timestamp, a rotating ad slot, or a random nonce means the raw HTML is different on every single fetch while the content is identical. You need to fingerprint what the page means, not what bytes it happened to serve.
This comes up everywhere: cache invalidation, regenerating a link-preview or Open Graph image only when the source moved, deciding whether to re-run an expensive downstream job, or notifying a user that a page they follow was updated. In every case you want a cheap, stable identifier that changes when the meaningful content changes and stays put when it doesn't.
TL;DR: Compute a deterministic fingerprint over the normalized, extracted content of a page — not the raw HTML. Same fingerprint means "unchanged, reuse whatever you computed last time." Different fingerprint means "it moved, do the work again." And embed a format version in the hash input, so the day you improve your normalization you don't silently invalidate every fingerprint you've ever stored.
The Problem
Two obvious approaches both fail:
- Hash the raw response. Timestamps, nonces, ad rotations, and CSRF tokens change every fetch. Your fingerprint is different every time, so "did it change?" is always "yes." Useless.
- Never fingerprint; just re-do the work. Correct, but you pay full price on every check — a headless render, an image regeneration, an API call — even when nothing moved. Expensive and slow at any real volume.
You want the cheap path when the content is stable and the expensive path only when it isn't. That requires a way to ask, precisely, "is this the same page I already processed?"
The Fix: Fingerprint the Extracted, Normalized Content
Extract the things a human would actually look at — the page's registrable domain, its title, its visible text — normalize them, and hash the result:
public static String compute(String finalUrl, String title, String visibleText) {
String domain = registrableDomain(finalUrl == null ? "" : finalUrl);
String normTitle = normalize(title);
String normText = normalize(visibleText);
if (normTitle.isEmpty() && normText.isEmpty()) {
return null; // nothing meaningful to fingerprint
}
return sha256(VERSION + "|" + domain + "|" + normTitle + "|" + normText);
}
/** Lowercase, collapse whitespace runs to a single space, trim. */
static String normalize(String s) {
if (s == null) return "";
return s.toLowerCase().replaceAll("\\s+", " ").trim();
}
Normalization is what makes this robust. Lowercasing and collapsing whitespace means a trivial reflow — an extra blank line, a capitalization tweak in a template — doesn't produce a false "content changed" signal. You're hashing the substance, not the formatting.
Returning null when there's nothing meaningful to fingerprint matters too: an empty or failed extraction should not masquerade as a stable, known page. Downstream, null means "I can't vouch for this," which is the honest answer.
The Gotcha: Version the Hash Input
Here's the mistake that surfaces months later. You improve your extraction — say you start stripping boilerplate navigation from the visible text. Now the same page hashes differently than it did before, and you have no way to distinguish "the page changed" from "my hashing changed." Every fingerprint you've stored silently becomes wrong, and downstream jobs either re-run needlessly or, worse, trust stale results.
Embed a version into the hashed input:
static final String VERSION = "v1";
When the normalization logic changes, bump to v2. Old v1 and new v2 fingerprints now live in different namespaces by construction — a format change forces one clean pass of recomputation instead of masquerading as a wave of content changes. It costs one string constant and saves you a genuinely confusing incident.
Why "Extracted" Beats "Rendered Bytes"
The instinct to hash the full rendered DOM is understandable — it feels more complete. But completeness is the problem. The DOM includes every dynamic, per-request artifact, so it's maximally unstable. By fingerprinting the extracted title and text (ideally the same extraction you already feed to whatever consumes the content), the fingerprint tracks the thing you care about and ignores the churn you don't. It also composes cleanly: any two callers that extract the same way will agree on the fingerprint, so it doubles as a content-identity key across services.
Pick Your Hash and Your Inputs Deliberately
SHA-256 is a fine default here — you want a stable, well-distributed digest, not a cryptographic commitment, but a strong hash costs nothing and sidesteps collision worries entirely. The inputs matter more than the algorithm: choose the smallest set of fields that captures "meaningfully different content" for your use case. Too many inputs (every attribute, every timestamp) and the fingerprint is noisy; too few and genuinely different pages collide. Domain + title + visible text is a good starting point for "is this the same page," and you can add fields as your definition of "changed" sharpens.
When This Pattern Pays Off
- Regenerate-on-change. Only rebuild a preview image, OG card, or summary when the fingerprint moves.
- Cache keys with meaning. A fingerprint is a natural cache key that survives cosmetic redeploys of the source page.
- Cross-service identity. "Is this the same content we saw elsewhere?" without shipping the whole body around.
Lessons Learned
- Fingerprint meaning, not markup. Raw HTML changes every fetch. Normalize and hash extracted content so cosmetic churn doesn't read as change.
- Normalize before you hash. Lowercase, collapse whitespace, trim. Small, deterministic, and it kills the bulk of false positives.
- Version your hash inputs. The day you improve normalization, an unversioned fingerprint can't tell "content changed" from "we changed." A version prefix makes that upgrade safe and boring.
How do you decide when a page has really changed? Share your approach in the comments.
Building jo4.io - a modern URL shortener with analytics, bio pages, and team workspaces.