Hetzner Milestone Runbook

Date: 2026-05-05

Objective

Turn the growing set of Hetzner gates into a safer milestone runner.

The project now has separate helpers for source sync, host readiness, env readiness, Python builds, database migrations, stack startup, health checks, backup checks, and evidence collection. This runbook groups them into reviewable milestones so the server run can stop at the first failing boundary.

What Was Added

• scripts/hetzner-runbook.mjs Guarded milestone runner for server-side execution.
• package.json scripts:
  • hetzner:runbook:plan
  • hetzner:runbook:commands
  • hetzner:runbook:check
  • hetzner:runbook:readiness
  • hetzner:runbook:python
  • hetzner:runbook:build
  • hetzner:runbook:database
  • hetzner:runbook:stack
  • hetzner:runbook:proxy

Safety Rules

• Execution is guarded for native Linux x64/amd64.
• Every run milestone requires HETZNER_RUNBOOK_CONFIRM=1.
• The runner stops at the first failed command.
• Each milestone writes a redacted runbook evidence file to docs/evidence/.
• Each milestone evidence file includes an expected command manifest and command count.
• The status evidence collector still runs at the end of each milestone through npm run hetzner:evidence:collect.
• The runner does not create or edit env files.
• The runner does not perform source sync; source sync remains an explicit local action through hetzner:sync:*.
• The proxy milestone is separate and should only run after internal health is green and DNS/proxy config is real.

Milestones

Print the plan:

npm run hetzner:runbook:plan

Print the raw command list:

npm run hetzner:runbook:commands

Run readiness after source sync, npm install, and real env values:

HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:readiness

Run Python services:

HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:python

Build remaining services:

HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:build

Run database migration/check:

HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:database

Start internal stack:

HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:stack

Start public proxy only after internal health is green:

HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:proxy

Evidence Contract

Each completed milestone writes a file named like:

docs/evidence/<timestamp>-hetzner-runbook-<milestone>.md

Completion evidence must include:

Status: complete
Milestone: <milestone>
Expected command count: <count>
Milestone complete: <milestone>
Platform: linux x64

It must also include every expected command twice:

• once in ## Expected Commands;
• once in ## Commands with the matching numbered command output section.

The final port:completion:* gate now checks this manifest. A thin evidence file that only says Status: complete is not enough.

Local Verification Evidence

• node --check scripts/hetzner-runbook.mjs passed.
• npm run hetzner:runbook:plan prints 6 milestones.
• npm run hetzner:runbook:commands prints 28 commands.
• Future milestone evidence now records Expected command count plus the expected command manifest.
• npm run hetzner:runbook:check fails locally as expected on darwin arm64.
• With HETZNER_RUNBOOK_ALLOW_NON_HETZNER=1, a run milestone still refuses to execute without HETZNER_RUNBOOK_CONFIRM=1.
• No local runbook evidence file was generated because no confirmed milestone was executed on this Mac.

Current Boundary

No milestone has been executed on a real Hetzner host yet.

The next real server step is:

HETZNER_SYNC_TARGET="USER@HOST:/opt/LegacyBlinkin-2-Hetzner/" npm run hetzner:sync:dry-run
HETZNER_SYNC_CONFIRM=1 HETZNER_SYNC_TARGET="USER@HOST:/opt/LegacyBlinkin-2-Hetzner/" npm run hetzner:sync:push

Then SSH to the server and start with:

cd /opt/LegacyBlinkin-2-Hetzner/
npm install
npm run hetzner:runbook:plan
HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:readiness