Watchtower Tried to Upgrade Itself. It Didn't Come Back.

· 8 min read · Jo4 Team
docker devops debugging automation

On May 19 we pushed a fix to main. GitHub Actions built the image, tagged it :latest, shoved it into GHCR, and went home. Our droplet was supposed to pick that up within sixty seconds. It didn't. Not in sixty minutes, not in two days.

When we finally SSH'd in to ask why, the answer was in docker ps: our auto-update tool wasn't running. It had tried to update itself, succeeded at the "stop" half, and never made it to the "start" half. Nobody noticed for forty-eight hours because the API was up; only the deploy pipeline was silently dead.

This is the story of that incident and the two-line fix that makes it structurally impossible to repeat.

What Watchtower Is For

If you've never used Watchtower, it solves a small, useful problem: you push a new image to a registry, and Watchtower running on your host detects the new digest, pulls it, gracefully stops the old container, and starts a fresh one. No SSH, no docker compose pull && up -d, no human in the loop. For a tiny team on a single droplet, it's a five-line replacement for an entire CD pipeline. You point it at GHCR, give it the docker socket, and forget it exists.

Forgetting it exists is exactly the problem.

The Incident

Here's what we pieced together after the fact.

Our watchtower service was running :latest — the version everyone copy-pastes from blog posts. On May 19, upstream pushed a new :latest. Watchtower, doing its job, polled the registry, saw the new digest, and decided to update itself.

To update a container, Watchtower stops it, then starts the replacement. The replacement is launched by Watchtower. Which is the container being stopped.

You can see the deadlock from orbit. Watchtower issued docker stop against its own container. The Docker socket connection it was using to issue the "now start the new one" command got severed the instant the container died. The new container never launched. The host was left with no Watchtower at all — just the old API container, happily serving stale code, and no process polling the registry for new builds.

The comment we eventually added to docker-compose.yml captures it best:

image is PINNED, not :latest. On 2026-05-19 Watchtower self-detected an updated :latest, tried to stop itself to upgrade, cut its own Docker-socket connection mid-stop, and never restarted — leaving this droplet on stale code for 2 days until manual intervention. Bump this tag explicitly to upgrade Watchtower itself; do NOT track :latest.

Two days. We only caught it because someone noticed a bug fix that was definitely in main was definitely not in production.

Why It Couldn't Recover

Natural question: why didn't restart: unless-stopped save us? It's right there in the compose file.

restart policies fire when a container crashes — exits with a non-zero code, or the daemon restarts. They do not fire when a container is intentionally stopped via the API. From Docker's perspective, somebody asked Watchtower to stop. It stopped. Nothing to restart. The fact that the "somebody" was Watchtower itself — and that the next command in the sequence never ran because the issuer evaporated — is invisible to the daemon.

The deeper problem is that nothing was watching the watcher. Every other container on the host had Watchtower as its safety net. Watchtower had nothing. It was the bottom turtle, and when the bottom turtle quietly walks off, the whole stack just sits there.

The Two-Layer Fix

We fixed it twice, on purpose. The full Watchtower service now looks like this:

  # Watchtower - Auto-updates containers when new images are pushed to GHCR
  # Eliminates need for SSH-based deployments
  # NOTE: Using nickfedor/watchtower fork (containrrr/watchtower is archived
  # and incompatible with Docker 24+)
  #
  # IMPORTANT: image is PINNED, not :latest. On 2026-05-19 Watchtower
  # self-detected an updated :latest, tried to stop itself to upgrade, cut
  # its own Docker-socket connection mid-stop, and never restarted —
  # leaving this droplet on stale code for 2 days until manual intervention.
  # Bump this tag explicitly to upgrade Watchtower itself; do NOT track :latest.
  watchtower:
    image: nickfedor/watchtower:1.17.0
    container_name: jo4-watchtower
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock
      - /root/.docker/config.json:/config.json:ro
    environment:
      - WATCHTOWER_CLEANUP=true
      - WATCHTOWER_POLL_INTERVAL=60
      - WATCHTOWER_INCLUDE_STOPPED=false
      - WATCHTOWER_REVIVE_STOPPED=false
      - WATCHTOWER_ROLLING_RESTART=false
      # Opt-in mode: only watch containers labeled
      # com.centurylinklabs.watchtower.enable=true (i.e. jo4-api).
      - WATCHTOWER_LABEL_ENABLE=true
    restart: unless-stopped

And on the app service we want it to actually watch:

  app:
    image: ghcr.io/${GITHUB_REPOSITORY:-rathnasorg/jo4}/alertstage-api:${APP_VERSION:-latest}
    labels:
      - "com.centurylinklabs.watchtower.enable=true"

Two layers.

Layer 1: Pin Watchtower's own image

image: nickfedor/watchtower:1.17.0. Not :latest. Not :stable. A specific tag we wrote down on purpose.

This single change makes the May 19 sequence impossible. When Watchtower asks "is there a newer image for nickfedor/watchtower:1.17.0?" the answer is, by definition, no. The only way Watchtower gets a new image is when we edit this file and bump the version — a human in the loop who can verify the new container actually came up.

(We also moved off containrrr/watchtower while here — that repo is archived and broke on Docker 24+. nickfedor/watchtower is the maintained fork.)

Layer 2: Opt-in label mode

WATCHTOWER_LABEL_ENABLE=true flips Watchtower from "watch every container on the host" to "watch only containers carrying the label com.centurylinklabs.watchtower.enable=true." We add that label to jo4-api. We do not add it to jo4-watchtower. The Watchtower container is now structurally invisible to itself.

Why Both Layers Matter

You could argue either layer alone is enough. Pin the image, no more updates, no more deadlock. Or use label-mode, Watchtower can't pick itself up regardless of tag. Pick one and move on.

We didn't, because each layer protects against a different human error six months from now.

Layer 1 fails the day somebody — maybe one of us, on autopilot — edits the compose file and changes 1.17.0 back to :latest. It's the kind of "cleanup" that looks reasonable in a PR diff. If layer 2 isn't there, that edit reintroduces the exact bug.

Layer 2 fails the day somebody adds the watchtower.enable=true label to the Watchtower container itself — maybe they're labelling everything for an inventory script, maybe a tutorial says "label all your containers." If layer 1 isn't there, Watchtower starts watching itself again.

Either failure alone is a foot. Both simultaneously, on the same PR, by the same engineer, is a foot wrapped in a kevlar boot. Defense in depth is cheap when the depth is two YAML lines.

Lessons For Any Auto-Update Tool

The Watchtower-specific story generalizes. Anything that auto-updates itself — your dependency bot, your CI runner image, your sidecar agent, your secrets-rotator — has the same shape of bug latent in it. Some patterns we now apply across the board:

  • Pin the version of anything that can mutate itself. Auto-update tools should not auto-update. Their version belongs in source control where a human has to type the new digit. :latest is for things being watched, not for things doing the watching.
  • Prefer opt-in over opt-out for blast radius. Default-watch-everything tools sweep up the watcher itself, your debug containers, your one-off docker run experiments. Opt-in lists are slightly more verbose and dramatically harder to misfire.
  • restart: unless-stopped is not a safety net for orchestration bugs. It catches crashes. It does not catch "I stopped myself on purpose and then died." If something in your stack has the authority to stop itself, give it a watchdog or accept it will walk off.
  • Silent CD failure is the worst kind. The API was up the whole time. Health checks green. Monitoring happy. The only symptom was "fixes don't reach production," which nobody catches until they go looking. Add an alert on "no successful deploy in N hours" — it would have paged us at hour 13 instead of hour 48.
  • Write the comment you'd want six months from now. The block we added to docker-compose.yml is the entire postmortem in five sentences. The next operator who looks at that file knows exactly why the version is pinned and what happens if they unpin it. More durable than any wiki page.

The footgun isn't gone — we still trust an external tool to mutate our running containers. But the specific path where the trust loops back on itself is closed off twice. That's enough.

Have you been bitten by a self-updating tool? What recovered you? Drop the war story in the comments.

Building jo4.io — a URL shortener with analytics for developers who ship.