OAuth client_credentials in Spring Boot for Prometheus Scrapes

· 13 min read · Jo4 Team
oauth springboot security prometheus

We stood up Prometheus on a self-hosted DigitalOcean droplet and pointed it at our Spring Boot API's /actuator/prometheus endpoint. The obvious question came next: how does Prometheus authenticate?

Until that moment, our OAuth server had spoken exactly two grants — authorization_code (with PKCE) and refresh_token. Both assume a user behind the request. Zapier, Make, Pipedream, the Chrome extension, the MCP clients — every existing integration represents a person who clicked "Connect". Prometheus represents nobody. It's a process on a host scraping a metrics endpoint every 30 seconds. There is no consenting user.

The RFC has a grant for exactly this — client_credentials, §4.4. The client authenticates itself to the token endpoint with its credentials, gets back a bearer, and presents that bearer on subsequent calls. No user, no consent screen, no refresh token. Here's how we slotted it into an existing Spring Security stack without regressing a single user-bound integration.

Why client_credentials Fits

Three properties of the Prometheus scrape are worth being explicit about, because each maps directly to a §4.4 design choice:

There is no user. Prometheus is a daemon. It has no Auth0 identity, no email, no role assignments. Forcing it through authorization_code would mean creating a synthetic "Prometheus user" — a pattern that's been a security footgun every time we've seen it (service accounts that drift into having too many permissions because they're shaped like real users).

The credentials live on the server, rotatably. Prometheus reads its OAuth client_secret from a file on disk that the deployment script writes. Rotating means redeploying the secret file and restarting Prometheus — same operational model as any other server-side credential. No browser, no consent.

The blast radius is one endpoint. Prometheus has exactly one thing it's allowed to do: GET /actuator/prometheus. Not list URLs, not read analytics, not touch user data. The credential should be incapable of doing anything else even if it leaks.

client_credentials is the RFC-defined shape for all three. The wire flow is two HTTP calls: POST /oauth/token with HTTP Basic-Auth-encoded client credentials returns an access_token; every subsequent scrape carries Authorization: Bearer <access_token> until the token expires. Per §4.4.3 there is no refresh token — when the access token expires, the client re-authenticates by hitting /oauth/token again. Prometheus's built-in oauth2: block does this transparently.

Three Schema Relaxations

Before any of the Java could change, the database had to admit the shape of a userless token and the shape of a redirect-URI-less client. Liquibase changeset 356 made three modifications:

- changeSet:
    id: 356-add-oauth-client-credentials-grant
    author: anand
    changes:
      - addColumn:
          tableName: oauth_clients
          columns:
            - column:
                name: grant_types
                type: VARCHAR(100)
                defaultValue: "authorization_code,refresh_token"
                constraints:
                  nullable: false
                remarks: "Comma-separated RFC 6749 grant_type values the client is permitted to use."

      - dropNotNullConstraint:
          tableName: oauth_clients
          columnName: redirect_uris
          columnDataType: TEXT

      - dropNotNullConstraint:
          tableName: oauth_access_tokens
          columnName: user_id
          columnDataType: BIGINT

Each change has a specific reason:

oauth_clients.grant_types (NEW column). Previously, the grant type set was implicit — every client got AC + RT. Now each client carries the exact list of grants it's allowed to request. Existing rows get backfilled to "authorization_code,refresh_token" via the column default during the addColumn step, which is the property that keeps every existing integration green. Zapier's row, Make's row, both Pipedream rows, the Chrome extension's row — all get the same string they were implicitly assuming. The new CC-only Prometheus client gets "client_credentials" and nothing else.

oauth_clients.redirect_uris nullable. A CC-only client legitimately has no browser callback. Without this relaxation, every CC client registration would have to invent a fake URL just to satisfy a NOT NULL constraint that was modeling the wrong invariant. Enforcement now moves to the service layer — it's still required for authorization_code clients, but the rule is "required when grant_types includes AC", not "required always".

oauth_access_tokens.user_id nullable. A CC token represents the client, not a user. The foreign key to users(id) ON DELETE CASCADE stays — NULL becomes the signal for "service-principal token, no user lookup needed". Token validation branches on this: if userId is null, skip the user-load step.

The changeset uses addColumn and dropNotNullConstraint rather than raw SQL, which sidesteps the Liquibase PL/pgSQL footgun (folded-block YAML corrupting dollar quotes). All three changes are reversible and the rollback is provided explicitly.

Token Issuance

The token endpoint is a switch on grant_type. We added a third branch:

@Transactional
public TokenResponse issueClientCredentialsToken(String clientId, String clientSecret,
                                                 String requestedScope, String requestedResource) {
    OAuthClientEntity client = validateTokenEndpointClient(clientId, clientSecret);

    if (client.isPublicClient()) {
        throw new UrlService.AppException(ErrorCode.OAUTH_INVALID_CLIENT_SECRET,
                "client_credentials grant requires a confidential client");
    }

    if (!client.supportsGrant("client_credentials")) {
        throw new UrlService.AppException(ErrorCode.OAUTH_CLIENT_DISABLED,
                "Client is not authorized to use client_credentials grant");
    }

    // Default to the client's full allowed scope set if none requested; otherwise
    // intersect requested scopes with allowed (RFC 6749 §3.3 narrowing).
    Set<String> allowed = client.getScopesSet();
    Set<String> granted;
    if (requestedScope == null || requestedScope.isBlank()) {
        granted = allowed;
    } else {
        Set<String> requested = parseScopes(requestedScope);
        if (!allowed.containsAll(requested)) {
            throw new UrlService.AppException(ErrorCode.OAUTH_INVALID_SCOPE,
                    "One or more requested scopes are not granted to this client");
        }
        granted = requested;
    }
    String grantedScopeString = String.join(",", granted);

    // Mint access token only — no refresh token per RFC 6749 §4.4.3.
    String accessToken = generateToken(TOKEN_LENGTH);
    String accessTokenHash = hashToken(accessToken);
    long accessExpiresAt = System.currentTimeMillis() + (client.getAccessTokenLifetimeSeconds() * 1000L);

    OAuthAccessTokenEntity accessTokenEntity = OAuthAccessTokenEntity.builder()
            .tokenHash(accessTokenHash)
            .clientId(client.getId())
            .userId(null)  // CC tokens have no user
            .scopes(grantedScopeString)
            .expiresAt(accessExpiresAt)
            .tenantId(client.getTenantId())
            .audience(requestedResource)
            .slug(UUID.randomUUID().toString().replace("-", "").substring(0, 16))
            .build();

    accessTokenRepository.save(accessTokenEntity);
    // ...returns TokenResponse with no refresh_token
}

Three things this is doing right:

Public-client rejection. A public client (token_endpoint_auth_method = "none") is one that authenticates via PKCE only. CC has no PKCE story — its security model is "the client knows the secret". Public clients can never use CC, full stop. We reject before issuing.

Per-client grant gating. supportsGrant("client_credentials") checks the row's grant_types column. A client registered for AC + RT cannot suddenly switch grants on the wire. The grant set is part of the client identity, not a request-time choice.

Scope narrowing. RFC 6749 §3.3 says the authorization server MAY issue a narrower scope than requested, MUST NOT issue a wider one. We default to "narrowest of allowed and requested", and reject if the request asks for anything outside allowed.

The token gets persisted with userId = null — the column relaxation from migration 356 is what allows that row to exist at all. No refresh token row is created.

Authentication Subclass

The hardest design decision wasn't on the token endpoint. It was on the validation side. When a bearer comes in on /actuator/prometheus, the existing filter chain inspects the token and constructs a Spring Security Authentication object to attach to the request. The existing code expects every authentication to have a user:

// OAuthTokenAuthentication.java (existing — unchanged)
public OAuthTokenAuthentication(UserEntity user, OAuthAccessTokenEntity token, OAuthClientEntity client) {
    super(buildAuthorities(user, token));
    this.user = user;
    this.userId = user.getId();  // would NPE for CC tokens
    // ...
}

The naive fix is to make user nullable on OAuthTokenAuthentication and sprinkle null-checks across every consumer. We considered that and rejected it — the codebase has dozens of instanceof OAuthTokenAuthentication checks across AuthContext, OAuthScopeEnforcementFilter, Jo4McpTools, and elsewhere. Making user nullable on the existing type would mean auditing every single one of them and adding null guards, with the constant risk of missing one and getting an NPE in production six weeks later.

We introduced a separate authentication type instead:

@Getter
public class OAuthClientCredentialsAuthentication extends AbstractAuthenticationToken {

    private final OAuthAccessTokenEntity accessToken;
    private final OAuthClientEntity client;
    private final Long clientId;
    private final Set<String> scopes;

    public OAuthClientCredentialsAuthentication(OAuthAccessTokenEntity accessToken, OAuthClientEntity client) {
        super(buildAuthorities(accessToken));
        this.accessToken = accessToken;
        this.client = client;
        this.clientId = client.getId();
        this.scopes = accessToken.getScopesSet();
        setAuthenticated(true);
    }

    /**
     * The "principal" of a client_credentials token is the OAuth client itself.
     */
    @Override
    public Object getPrincipal() {
        return client;
    }
    // ...
}

What this buys us is type-driven correctness. Every existing instanceof OAuthTokenAuthentication check naturally evaluates false for a CC token. Code that asks "who is the user behind this request?" gets Optional.empty() from AuthContext.getActualUserOptional(). Controllers that call authContext.getCurrentUser() throw UNAUTHORIZED automatically. The MCP tools that filter by userId never even see CC requests. We didn't have to audit dozens of sites; the type system did it for us.

The validation filter branches once, at the boundary:

// inside OAuthTokenAuthenticationFilter, paraphrased
ValidatedToken validated = oauthService.validateAccessToken(bearer);
Authentication auth = validated.getUser() == null
    ? new OAuthClientCredentialsAuthentication(validated.getToken(), validated.getClient())
    : new OAuthTokenAuthentication(validated.getUser(), validated.getToken(), validated.getClient());
SecurityContextHolder.getContext().setAuthentication(auth);

validateAccessToken itself has exactly one change — the user lookup becomes conditional on userId != null. That's the entire blast radius on the validation side.

The Path Allowlist

Scope-based authorization gets us most of the way. The Prometheus client has metrics:read scope only — not read, not write. Endpoints requiring read/write reject it because the bearer lacks those scopes.

But "lacks the right scope" is one layer. We wanted defense-in-depth, because scope sets get edited in admin UIs by humans and a future "let's give the metrics client read to test something" five-second mistake should not silently grant CC tokens access to user data. So we added a second filter that runs immediately after the token filter:

@Slf4j
@Component
@Profile("auth0")
public class ClientCredentialsTokenGateFilter extends OncePerRequestFilter {

    private final AntPathMatcher pathMatcher = new AntPathMatcher();

    @Value("${app.security.cc-token-paths:/actuator/prometheus}")
    private String ccTokenPaths;

    @Override
    protected void doFilterInternal(HttpServletRequest request, HttpServletResponse response, FilterChain filterChain)
            throws ServletException, IOException {

        Authentication auth = SecurityContextHolder.getContext().getAuthentication();
        // Only act on CC tokens. Everything else (JWT, API key, user-bound OAuth, anonymous) passes through.
        if (!(auth instanceof OAuthClientCredentialsAuthentication ccAuth)) {
            filterChain.doFilter(request, response);
            return;
        }

        String requestPath = request.getRequestURI();
        List<String> allowed = Arrays.stream(ccTokenPaths.split(","))
                .map(String::trim)
                .filter(s -> !s.isEmpty())
                .toList();

        boolean pathAllowed = allowed.stream().anyMatch(p -> pathMatcher.match(p, requestPath));
        if (pathAllowed) {
            filterChain.doFilter(request, response);
            return;
        }

        log.warn("client_credentials token attempted to access non-allowlisted path: clientId={}, path={}, scopes={}",
                ccAuth.getClientId(), requestPath, ccAuth.getScopes());

        // RFC 6750 §3.1: insufficient_scope on a Bearer-protected resource.
        response.setStatus(HttpStatus.FORBIDDEN.value());
        response.setHeader("WWW-Authenticate",
                "Bearer error=\"insufficient_scope\", error_description=\"client_credentials tokens are not valid for this endpoint\"");
        response.getWriter().write(
                "{\"error\":\"insufficient_scope\",\"error_description\":\"client_credentials tokens are not valid for this endpoint\"}");
    }
}

A few details that matter:

The instanceof guard is the no-op short-circuit. JWT requests, API-key requests, user-bound OAuth requests, anonymous requests — all four pass through the filter with one type check and zero allocations. The cost is paid only on the small set of CC requests.

Config-driven allowlist. app.security.cc-token-paths is comma-separated and AntPathMatcher-evaluated. Adding a future endpoint (say, an alertmanager receiver) is a config edit, not a code change. The default — /actuator/prometheus — is the only thing CC tokens can reach out of the box:

# application-security.yaml
app:
  security:
    # Paths where client_credentials OAuth tokens are accepted. CSV; AntPathMatcher syntax.
    cc-token-paths: "/actuator/prometheus"

WWW-Authenticate: Bearer error="insufficient_scope" is the RFC 6750 §3.1 shape. Prometheus and other OAuth-aware clients can read this and surface a meaningful error, rather than getting a generic 403 with no clue what went wrong.

Never throws. Filters that throw can mask the underlying error or tear down the chain in surprising ways. This one always writes a response or delegates.

Wiring the Scrape

On the Prometheus side, the entire setup is one block in prometheus.yml:

scrape_configs:
  - job_name: jo4-api
    metrics_path: /actuator/prometheus
    scheme: http
    scrape_interval: 30s
    oauth2:
      client_id: __PROMETHEUS_OAUTH_CLIENT_ID__
      client_secret_file: /etc/prometheus/oauth-client-secret
      token_url: http://10.108.0.3:8080/oauth/token
      scopes:
        - metrics:read
    static_configs:
      - targets: ['10.108.0.3:8080']
        labels:
          service: alertstage
          env: prod

Notes from setting this up:

client_id inline, client_secret_file from disk. Per OAuth 2.0, the client ID is public — fine to bake into the config and sed-substitute at deploy time. The secret stays in a separate file with restrictive permissions, written by the bootstrap script from a GitHub Secret. Never co-located, never logged.

Private VPC URLs. token_url and the scrape target both use the DigitalOcean private IP 10.108.0.3. Traffic never leaves the VPC, which is the primary security boundary; the OAuth check is defense-in-depth on top of that. The token endpoint accepts plain HTTP only because it's on a private subnet; this would be HTTPS for any path that crosses the public internet.

scopes: ["metrics:read"]. Prometheus's oauth2: block converts this into the scope form field of the /oauth/token request. Our issueClientCredentialsToken narrows it against the client's allowed set — which happens to be just metrics:read — and stamps the result on the issued access token.

No refresh_token config. Prometheus knows §4.4.3 — when it sees a token response with no refresh_token, it just re-runs the token flow when the current access token nears expiry. No refresh logic on either side.

Testing It End-To-End

A curl sequence to verify the whole loop, against a freshly registered Prometheus client (client ID jo4_Xxxx, client secret captured at registration):

# 1. Token exchange — HTTP Basic-Auth-encoded client credentials.
ACCESS_TOKEN=$(curl -s -X POST https://jo4-api.jo4.io/oauth/token \
  -u "jo4_Xxxx:secret_XXXXXXXX" \
  -d "grant_type=client_credentials" \
  -d "scope=metrics:read" | jq -r .access_token)

# 2. Scrape — bearer on the request.
curl -i https://jo4-api.jo4.io/actuator/prometheus \
  -H "Authorization: Bearer $ACCESS_TOKEN"
# → HTTP/2 200
# → # HELP http_server_requests_seconds ...

# 3. Negative test — same token against a user endpoint.
curl -i https://jo4-api.jo4.io/api/v1/urls/me \
  -H "Authorization: Bearer $ACCESS_TOKEN"
# → HTTP/2 403
# → WWW-Authenticate: Bearer error="insufficient_scope", ...
# → {"error":"insufficient_scope","error_description":"client_credentials tokens are not valid for this endpoint"}

That third call is the critical one to verify. The same bearer, presented at a non-allowlisted path, must be rejected — even though the bearer is technically valid. If you see anything other than 403 + insufficient_scope there, the gate filter isn't wired correctly.

Lessons Learned

  • A new grant type is mostly a schema problem. The Java for client_credentials is small. The work was three column relaxations and an audit of every codepath that assumed userId was non-null.
  • A separate Authentication subclass beats nullable fields on the existing one. instanceof checks across the codebase become the type-system's audit for you. Sprinkling null-checks would have been an ongoing tax forever.
  • Always backfill new NOT NULL columns via a default. Adding grant_types NOT NULL would have failed on every existing row without defaultValue: "authorization_code,refresh_token". The default is what makes the migration zero-downtime and the rollout silent for every existing client.
  • Defense-in-depth means two independent layers, not one layer twice. Scope-based authorization handles "the client wasn't granted this scope". The path-allowlist filter handles "even if someone misconfigures the scope set tomorrow, this token still can't hit user data". Both layers must agree before access is granted.
  • WWW-Authenticate on 401/403 from a bearer-protected resource is RFC 6750 §3.1. Clients can react to error="insufficient_scope" programmatically. A naked 403 with no header forces humans into the loop for every failure.
  • No refresh token for CC, period. RFC 6749 §4.4.3. Prometheus and every other OAuth-aware client know how to re-run the token flow on expiry. Issuing a refresh token would be inventing capability the spec explicitly forbids.
  • Validate every existing integration before shipping. We scanned ten risk categories across the codebase before touching a line — Zapier, Make, Pipedream, Chrome extension, MCP clients, OIDC discovery, /oauth/userinfo, the scope filter, public endpoints, the previously-public /actuator/prometheus path. The migration changed default-value-backfilled columns; behavior changed for exactly one path, and it was the one we intended.

Adding CC grant to an existing AC flow? What schema migration bit you? Drop it in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.