Blinkin Legacy Docs Docs
Porting · porting/2026-05-05-hetzner-stack-runbook.md Docs Home

Hetzner Stack Runbook

Date: 2026-05-05

Objective

Turn the first Hetzner Compose package into a repeatable server-side run sequence for config validation, image builds, stack startup, service inspection, and reverse-proxy startup.

This helper is intentionally server-guarded. Build and startup commands are meant for native Linux x64/amd64 on Hetzner, not the current low-disk Mac.

What Was Added

  • scripts/hetzner-stack.mjs Ordered Docker Compose runner for the Hetzner stack.
  • package.json scripts:
    • hetzner:stack:plan
    • hetzner:stack:commands
    • hetzner:stack:config
    • hetzner:stack:build:query
    • hetzner:stack:build
    • hetzner:stack:up
    • hetzner:stack:ps
    • hetzner:stack:proxy:up

Related milestone runner:

  • scripts/hetzner-runbook.mjs
  • npm run hetzner:runbook:plan
  • npm run hetzner:runbook:commands
  • HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:readiness
  • HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:python
  • HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:build
  • HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:database
  • HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:stack
  • HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:proxy

Related Python-first gate:

  • scripts/hetzner-python-gate.mjs
  • npm run hetzner:python:plan
  • npm run hetzner:python:build:query
  • npm run hetzner:python:build:ingestion
  • npm run hetzner:python:health

Related database gate:

  • scripts/database-gate.mjs
  • npm run db:local:check
  • npm run hetzner:db:plan
  • npm run hetzner:db:studio:migrate
  • npm run hetzner:db:check

Safety Rules

  • Heavy build and startup commands require native Linux x64/amd64.
  • The helper checks that real runtime config files exist before Compose commands run.
  • The proxy command additionally checks for configs/hetzner/reverse-proxy.env and configs/hetzner/Caddyfile.
  • No logs, database rows, or env values are printed by the helper.
  • The local Mac can still print plans and exact Docker Compose command lines without running builds.

Server Workflow

Run this after source sync and npm install on Hetzner:

npm run hetzner:stack:plan
npm run hetzner:stack:commands
npm run hetzner:runbook:plan
npm run hetzner:runbook:commands

Then run the real server gates:

npm run hetzner:bootstrap:check
npm run hetzner:host:check
npm run hetzner:config:status
npm run hetzner:config:init:dry
npm run hetzner:config:init
npm run hetzner:env:check

After real values are filled in:

npm run hetzner:stack:config
npm run hetzner:python:preflight
npm run hetzner:python:torch
npm run hetzner:python:build:query
npm run hetzner:python:build:ingestion
npm run hetzner:python:imports
npm run hetzner:python:up
npm run hetzner:python:ps
npm run hetzner:python:health
npm run hetzner:stack:build:query
npm run hetzner:stack:build
npm run hetzner:db:plan
npm run hetzner:db:studio:migrate
npm run hetzner:db:check
npm run hetzner:stack:up
npm run hetzner:stack:ps
npm run hetzner:health:check

npm run hetzner:python:ps parses docker compose ps --format json and fails unless redis, query-engine, and ingestion-worker are all running. This catches worker exits before the broader stack build continues.

Only after the internal stack is green:

npm run hetzner:stack:proxy:up
npm run hetzner:evidence:collect

Why Query Engine Builds First

The Query Engine is still the riskiest server-side build because the local Mac cannot finish a full Python/Torch Docker build safely. Building it first on Hetzner gives the fastest signal on the largest remaining blocker.

Local Verification Evidence

  • node --check scripts/hetzner-stack.mjs passed.
  • node --check scripts/hetzner-runbook.mjs passed.
  • node --check scripts/hetzner-python-gate.mjs passed.
  • node --check scripts/database-gate.mjs passed.
  • npm run hetzner:stack:plan prints the ordered server workflow.
  • npm run hetzner:stack:commands prints the exact Docker Compose commands without requiring Docker Compose locally.
  • npm run hetzner:runbook:plan and npm run hetzner:runbook:commands print the milestone runner without running server commands locally.
  • npm run hetzner:python:plan and npm run hetzner:python:commands print the Python-first gate without running heavy builds locally.
  • npm run hetzner:db:plan and npm run hetzner:db:commands print the database gate without running server commands locally.
  • npm run legacy:verify includes scripts/hetzner-stack.mjs as a required foundation file.

Current Boundary

The actual build and startup commands still need to run on Hetzner after real staging values are entered. The helper makes that operation repeatable, but it does not replace the required server-side build evidence.