Blinkin Legacy Docs Docs
Porting · porting/2026-05-05-hetzner-backup-restore.md Docs Home

Hetzner Backup And Restore

Date: 2026-05-05

Objective

Add the first backup and restore runbook for the Hetzner port.

This does not make backups production-ready by itself. The first production-ready milestone is a successful restore rehearsal on a Hetzner staging or recovery host.

What Was Added

  • configs/hetzner/backup.env.example Placeholder-only backup configuration.
  • scripts/hetzner-backup.mjs Helper that checks support files, prints backup commands, runs guarded backups on Hetzner, writes redacted backup evidence, and prints restore plans.
  • package.json scripts:
    • hetzner:backup:plan
    • hetzner:backup:check
    • hetzner:backup:commands
    • hetzner:backup:run
    • hetzner:restore:plan
    • hetzner:restore:commands
    • hetzner:restore:rehearse
  • .gitignore and configs/hetzner/rsync-filter.rules now exclude local backups/.

Data That Must Be Protected

The first Hetzner backup scope covers:

  • Postgres logical dump through pg_dumpall.
  • MinIO object-storage volume.
  • Caddy data and config volumes for TLS/certificate state.
  • Runtime config archive for configs/hetzner/*.env and configs/hetzner/Caddyfile.

The runtime config archive contains secrets. It must never be committed and should only live in restricted, encrypted backup storage.

Compose Env Boundary

Backup and restore commands that call Docker Compose now use the same Compose env file as the stack, Python, and database gates:

docker compose --env-file configs/hetzner/frontend-build.env -f docker-compose.hetzner.yml ...

This matters because docker-compose.hetzner.yml interpolates Compose-level values such as POSTGRES_PASSWORD, MINIO_ROOT_USER, and MINIO_ROOT_PASSWORD. Restore must not restart the stack with placeholder defaults.

configs/hetzner/backup.env.example therefore includes:

COMPOSE_ENV_FILE=configs/hetzner/frontend-build.env

hetzner:backup:run and hetzner:restore:rehearse refuse to run if the real Compose env file is missing.

Commands

Check backup support files:

npm run hetzner:backup:check

Print the backup command block:

npm run hetzner:backup:plan

Print the exact guarded backup command list:

npm run hetzner:backup:commands

Run the backup on Hetzner only after reviewing the commands:

HETZNER_BACKUP_CONFIRM=1 npm run hetzner:backup:run

Print the restore rehearsal plan:

npm run hetzner:restore:plan

Print the destructive restore rehearsal command list:

npm run hetzner:restore:commands

Run the restore rehearsal only on a prepared recovery target:

HETZNER_RESTORE_CONFIRM=1 HETZNER_RESTORE_TARGET=recovery HETZNER_RESTORE_STAMP=YYYYMMDDTHHMMSSZ npm run hetzner:restore:rehearse

Backup Execution Guard

hetzner:backup:run is guarded:

  • It runs only on native Linux x64/amd64 unless HETZNER_BACKUP_ALLOW_NON_HETZNER=1 is set for intentional diagnostics.
  • It requires HETZNER_BACKUP_CONFIRM=1.
  • It requires configs/hetzner/backup.env to exist.
  • It requires the configured Compose env file to exist.
  • It writes redacted command-status evidence to:
docs/evidence/<timestamp>-hetzner-backup.md

Backup evidence proves backup command execution only. It does not replace restore rehearsal evidence.

Restore Rehearsal Guard

hetzner:restore:rehearse is more guarded because it is destructive:

  • It runs only on native Linux x64/amd64 unless HETZNER_RESTORE_ALLOW_NON_HETZNER=1 is set for intentional diagnostics.
  • It requires HETZNER_RESTORE_CONFIRM=1.
  • It requires HETZNER_RESTORE_TARGET=recovery.
  • It requires HETZNER_RESTORE_STAMP=YYYYMMDDTHHMMSSZ.
  • It requires configs/hetzner/backup.env to exist.
  • It requires the configured Compose env file to exist.
  • It requires matching docs/evidence/*-hetzner-backup.md backup evidence for the same backup stamp.
  • It verifies backup files exist before stopping services or restoring volumes.
  • It writes restore rehearsal evidence to:
docs/evidence/<timestamp>-backup-restore-rehearsal.md

Server Setup

On Hetzner, create the runtime backup config:

npm run hetzner:config:init:dry
npm run hetzner:config:init

Keep backup output outside the repository, for example:

sudo mkdir -p /opt/legacy-blinkin-backups
sudo chown "$USER":"$USER" /opt/legacy-blinkin-backups
chmod 700 /opt/legacy-blinkin-backups

Restore Rehearsal Gate

Backups are not considered proven until this has happened:

  1. A backup is produced on Hetzner staging.
  2. The backup files are copied to restricted external storage.
  3. A separate staging or recovery target is prepared.
  4. Postgres and object storage are restored.
  5. The stack starts again.
  6. npm run hetzner:health:check passes.
  7. One published App flow is manually verified.
  8. Restore evidence is documented.

The completion gate expects restore rehearsal evidence in:

docs/evidence/<timestamp>-backup-restore-rehearsal.md

The evidence file must include:

# Backup And Restore Rehearsal
Status: complete
Target: recovery
Backup command exit code: 0
Restore command exit code: 0
Health check exit code: 0

Current Verification Evidence

  • npm run hetzner:backup:check passes locally for support-file presence.
  • npm run hetzner:backup:plan prints a backup command block without printing secret values.
  • npm run hetzner:backup:commands prints the guarded backup commands without running them, and the Postgres dump command includes --env-file configs/hetzner/frontend-build.env.
  • HETZNER_BACKUP_ALLOW_NON_HETZNER=1 npm run hetzner:backup:run refuses without HETZNER_BACKUP_CONFIRM=1.
  • HETZNER_BACKUP_CONFIRM=1 npm run hetzner:backup:run refuses locally because this Mac is not native Linux x64/amd64.
  • npm run hetzner:restore:plan prints a restore rehearsal plan.
  • npm run hetzner:restore:commands prints the destructive restore rehearsal commands without running them, and every Docker Compose restore/restart command includes --env-file configs/hetzner/frontend-build.env.
  • HETZNER_RESTORE_ALLOW_NON_HETZNER=1 npm run hetzner:restore:rehearse refuses without HETZNER_RESTORE_CONFIRM=1.
  • HETZNER_RESTORE_CONFIRM=1 HETZNER_RESTORE_TARGET=recovery HETZNER_RESTORE_STAMP=20260505T120000Z npm run hetzner:restore:rehearse refuses locally because this Mac is not native Linux x64/amd64.
  • With diagnostics override and confirmation, restore rehearsal still refuses locally because configs/hetzner/backup.env is missing.
  • Real backup files are not generated locally.

Known Blockers

  • No backup has run on Hetzner yet.
  • No docs/evidence/*-hetzner-backup.md file exists yet.
  • No restore rehearsal has run on a recovery host yet.
  • No docs/evidence/*-backup-restore-rehearsal.md file exists yet.
  • Runtime config backup contains secrets and needs encrypted off-server storage.
  • MinIO and Caddy volume restore commands are destructive and must only run against an intentional recovery target.