Every schema migration is a small act of faith. You run it, the app boots, and usually it's fine. But the one time it isn't — a bad Liquibase changeset, a column drop that shouldn't have shipped — you want a restore point from seconds before the migration, not from last night's 3 AM backup.
We already had a nightly backup. What we didn't have was a backup tied to the deploy — and, more importantly, a deploy that refuses to proceed if the backup fails. Here's how we wired that up.
TL;DR
- Reuse your existing backup workflow via
workflow_call— don't duplicate the logic. - Gate the deploy on the backup job with
needs:. A failed backup skips the deploy. No migration without a restore point. - If you deploy via image-pull (Watchtower, Flux, etc.), the trick is to push an immutable tag first and promote the "watched" tag last — so the backup runs between build and go-live.
The deploy mechanism matters
Our backend deploys with Watchtower: CI pushes an image to the registry, and a Watchtower container on the droplet polls, pulls :latest, and restarts the app. Liquibase migrates on startup.
That detail is the whole game. The migration happens when :latest changes — roughly 60 seconds after we push it, not when CI finishes. So a "pre-deploy" backup has to complete before that tag moves.
The naive version — take the backup as the first job, then build+test+push — technically works, but it's wasteful and wrong-ordered:
- If the test suite fails, you took a backup for a deploy that never happened.
- The backup is now several minutes stale by the time the migration actually runs.
What we actually want is: build & test → backup → go live.
Step 1: Make the backup reusable
Our nightly backup already did the hard, sensitive parts — pg_dump of a DigitalOcean Managed Postgres schema, upload to Spaces, and a firewall dance that temporarily trusts the runner's IP (with a preflight that refuses to run if it would lock production out). Duplicating that into the deploy pipeline would be asking for trouble.
Instead, we added a workflow_call trigger to the existing workflow so it can be called as well as scheduled:
on:
schedule:
- cron: '0 3 * * *'
workflow_dispatch:
workflow_call:
inputs:
prefix:
type: string
default: db-daily-backup
lifecycle_days:
type: number
default: 30
label:
type: string
default: ''
A tiny "resolve parameters" step maps those inputs (or their defaults, for the nightly run) into env vars, so the rest of the workflow is unchanged. Nightly behaves exactly as before; a caller can point it at a different prefix, retention, and filename label.
There's a bonus safety property here that duplication would have thrown away: the workflow keeps its concurrency group. A pre-deploy backup and the nightly backup can now never run at the same time and race on the database's trusted-source firewall rules — one run's cleanup could otherwise remove the other's still-in-use IP. Reuse isn't just DRY; it's safer.
Step 2: Split build from go-live
The key restructure in the deploy workflow: the build job pushes only immutable tags — never the tag Watchtower watches.
jobs:
build:
name: Build, Test & Push (:sha)
runs-on: ubuntu-latest
steps:
# ... checkout, JDK, gradle test, bootJar, docker build ...
- name: Build and Push Docker Image (immutable tags only)
uses: docker/build-push-action@v5
with:
push: true
tags: |
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.run_number }}
# NOTE: deliberately no :latest here.
Pushing :sha puts the image in the registry but triggers nothing — Watchtower only watches :latest.
Step 3: Backup between build and go-live
predeploy-backup:
name: Pre-deploy DB backup
needs: build
uses: ./.github/workflows/DigitalOcean-460-backup.yml
secrets: inherit
with:
prefix: db-predeploy-backup
lifecycle_days: 14
label: predeploy-${{ github.sha }}
needs: build means it only runs after a green build+test. secrets: inherit passes the DO tokens through. The result lands at s3://.../db-predeploy-backup/jo4db-<ts>-predeploy-<sha>.sql.gz — labelled with the exact commit, so finding the right restore point later is trivial.
Step 4: Promote :latest last — that's the deploy trigger
deploy:
name: Promote :latest & Verify
needs: predeploy-backup
runs-on: ubuntu-latest
steps:
- name: Promote image to :latest (triggers Watchtower)
run: |
docker buildx imagetools create \
--tag ${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:latest \
${{ env.REGISTRY }}/${{ env.IMAGE_NAME }}:${{ github.sha }}
# ... wait for Watchtower, health-check the deployed SHA ...
docker buildx imagetools create is the quiet hero: it adds the :latest tag to the already-built manifest, server-side. No rebuild, no pull. Tagging it is the single, deliberate act that makes the app go live.
The job graph reads exactly like the invariant we wanted:
build ──▶ predeploy-backup ──▶ deploy(:latest)
(test) (restore point) (migration)
Why the gate is unskippable
Walk the failure modes:
- Tests fail →
buildfails → backup and deploy are skipped. No backup wasted, nothing ships. - Backup fails →
deployis skipped →:latestnever moves → no migration. Your change simply doesn't go live until you fix the backup. It cannot half-deploy. - Backup succeeds, deploy proceeds → the migration runs with a restore point that's seconds old.
That last one is the whole point. The only time you reach for this backup is a bad migration — and now there's always one waiting, tagged with the commit that caused it.
Lessons learned
- Reuse the sensitive workflow, don't fork it. The shared concurrency group prevented a firewall race we didn't even set out to fix.
- Understand what triggers your deploy. For pull-based deploys, controlling which tag moves and when is how you sequence work around the go-live moment.
- A gate is only a gate if failure blocks.
needs:turns "we take a backup" into "we cannot deploy without one." - Label backups with the commit SHA. Future-you, mid-incident, will not want to guess which
.sql.gzis the right one.
How do you sequence backups around migrations? Pre-deploy snapshot, blue/green, or just trust the nightly? Curious what's working for others.
Building jo4.io — a URL shortener with analytics for developers who like their deploys boring.