Hetzner Staging Compose

Date: 2026-05-05

Objective

Create the first Hetzner staging package for the preserved legacy platform without touching blinkin-2-platform.

This is not the final production setup yet. It is the first server-side staging shape that can build and run the already imported legacy systems on a native Linux/amd64 host.

What Was Added

• docker-compose.hetzner.yml Container map for the first Hetzner staging stack.
• configs/hetzner/*.env.example Placeholder-only environment templates for services, Compose-level credentials, and frontend build arguments.
• configs/hetzner/reverse-proxy.env.example Placeholder-only domain and ACME email template for the reverse proxy.
• configs/hetzner/backup.env.example Placeholder-only backup configuration.
• configs/hetzner/Caddyfile.example Example Caddy routing file for TLS and public domains.
• configs/hetzner/rsync-filter.rules Source sync filter that excludes generated folders, caches, virtual environments, Wrangler output, and real env files.
• scripts/hetzner-staging-probe.mjs Read-only check script for required staging files and Compose/YAML validation.
• package.json scripts:
  • hetzner:plan
  • hetzner:sync:plan
  • hetzner:sync:check
  • hetzner:sync:dry-run
  • hetzner:sync:push
  • hetzner:bootstrap:plan
  • hetzner:bootstrap
  • hetzner:bootstrap:check
  • hetzner:config:plan
  • hetzner:config:status
  • hetzner:config:init:dry
  • hetzner:config:init
  • hetzner:host
  • hetzner:host:check
  • hetzner:check
  • hetzner:env
  • hetzner:env:check
  • hetzner:health
  • hetzner:health:check
  • hetzner:stack:plan
  • hetzner:stack:commands
  • hetzner:stack:config
  • hetzner:stack:build:query
  • hetzner:stack:build
  • hetzner:stack:up
  • hetzner:stack:ps
  • hetzner:stack:proxy:up
  • hetzner:runbook:plan
  • hetzner:runbook:commands
  • hetzner:runbook:check
  • hetzner:runbook:readiness
  • hetzner:runbook:python
  • hetzner:runbook:build
  • hetzner:runbook:database
  • hetzner:runbook:stack
  • hetzner:runbook:proxy
  • db:local:check
  • hetzner:db:plan
  • hetzner:db:commands
  • hetzner:db:studio:migrate
  • hetzner:db:check
  • hetzner:backup:plan
  • hetzner:backup:check
  • hetzner:restore:plan
  • hetzner:evidence:plan
  • hetzner:evidence:check
  • hetzner:evidence:collect

The .gitignore excludes real configs/hetzner/*.env files, so staging secrets are not committed.

Services In The First Staging Stack

The Compose file currently maps these services:

• reverse-proxy: optional Caddy TLS reverse proxy behind the proxy Compose profile.
• postgres: shared Postgres 16 database host for Studio API and Zweistein.
• redis: shared queue/cache service.
• minio: S3-compatible object storage for staging media and files.
• weaviate: optional AI vector database behind the ai Compose profile.
• studio-api: legacy Studio API NestJS backend.
• zweistein-server: legacy Zweistein NestJS backend.
• query-engine: Zweistein Python Query Engine.
• ingestion-worker: Zweistein Python ingestion worker.
• zweistein-admin: Zweistein Admin frontend.
• picasso-editor: Picasso Studio editor frontend.
• picasso-houston: Picasso Houston app runtime frontend.
• picasso-widget: Picasso embeddable widget served as static Nginx assets.
• zweistein-embedding-widget: Zweistein chat/agent embedding widget served as static Nginx assets.

The first staging decision is to expose both widgets as their own static services. A later production reverse proxy can mount these under stable public paths or domains.

Staging Ports

The first staging ports are intentionally explicit:

Production should later put these behind a reverse proxy and TLS. The first staging goal is to prove containers, health checks, and service connectivity.

Compose-Level Runtime Values

configs/hetzner/frontend-build.env is passed into Docker Compose through --env-file. Despite the historical name, it now contains both frontend build values and the Compose-level infrastructure values needed before containers start:

• POSTGRES_PASSWORD
• MINIO_ROOT_USER
• MINIO_ROOT_PASSWORD

The strict env gate checks these keys because Postgres and MinIO read them before service-specific env_file values are loaded.

Public Reverse Proxy

The first public edge is Caddy, defined as the optional reverse-proxy service in docker-compose.hetzner.yml.

It is behind the proxy profile so the internal stack can be built and health-checked before opening ports 80 and 443.

Create the proxy runtime files on Hetzner through the runtime config helper:

npm run hetzner:config:init:dry
npm run hetzner:config:init

Then replace every domain placeholder in configs/hetzner/reverse-proxy.env.

The example routes:

After the internal health gate is green, start the proxy:

docker compose --env-file configs/hetzner/frontend-build.env -f docker-compose.hetzner.yml --profile proxy up -d reverse-proxy

Safe Hetzner Setup Steps

Run these on the Hetzner/Linux staging host, not on the current low-disk Mac:

npm run hetzner:config:plan
npm run hetzner:config:status
npm run hetzner:config:init:dry
npm run hetzner:config:init

Then replace every placeholder in configs/hetzner/*.env with real staging values.

Important: never commit the real .env files.

Source Sync

Print the safe source sync plan:

npm run hetzner:sync:plan

Run the local safety check:

npm run hetzner:sync:check

Set the server target, then dry-run and push:

export HETZNER_SYNC_TARGET="USER@HOST:/opt/LegacyBlinkin-2-Hetzner/"
npm run hetzner:sync:dry-run
HETZNER_SYNC_CONFIRM=1 npm run hetzner:sync:push

The sync helper uses configs/hetzner/rsync-filter.rules so the server receives source files, docs, scripts, Dockerfiles, and .env.example templates without local node_modules, build outputs, virtual environments, caches, Wrangler output, backups, or real .env files.

Server Bootstrap

The first server bootstrap checklist lives in porting/2026-05-05-hetzner-bootstrap.md.

Useful commands:

npm run hetzner:bootstrap:plan
npm run hetzner:bootstrap
npm run hetzner:bootstrap:check

Run the bootstrap gate before treating a Hetzner host as ready for source sync and image builds.

Validation Commands

Print the server-side plan:

npm run hetzner:plan

Check required staging files and Compose/YAML shape:

npm run hetzner:host
npm run hetzner:check

Run the strict host gate on Hetzner before building:

npm run hetzner:host:check

The host gate checks for native Linux x64/amd64, Node.js 20+, Docker, Docker Compose, available disk headroom, and free staging ports.

Show whether real staging env files exist and whether obvious placeholders remain:

npm run hetzner:config:status
npm run hetzner:env

Run the strict env gate before building on Hetzner:

npm run hetzner:env:check

The strict env gate fails if runtime env files are missing, if obvious placeholders such as replace-with-... or example.com remain, or if required keys are empty. It prints only file names, key names, and line numbers, not secret values.

On the current Mac, hetzner:check only validates file presence and YAML because Docker Compose is not available in the local Docker CLI and disk is too tight for heavy image builds. On Hetzner, after the real env files exist, the same command should run Docker Compose config validation.

Stack Runbook

The guarded server-side stack runbook lives in porting/2026-05-05-hetzner-stack-runbook.md.

The milestone runner lives in porting/2026-05-05-hetzner-milestone-runbook.md.

Useful commands:

npm run hetzner:stack:plan
npm run hetzner:stack:commands
npm run hetzner:runbook:plan
npm run hetzner:runbook:commands

Build Order

Build the Python Query Engine first on native Linux/amd64:

npm run hetzner:stack:build:query

Then build the remaining services:

npm run hetzner:stack:build

Run the database gate before treating application startup as ready:

npm run hetzner:db:plan
npm run hetzner:db:studio:migrate
npm run hetzner:db:check

Start the first staging stack:

npm run hetzner:stack:up
npm run hetzner:stack:ps

Health Checks

After startup, inspect the health endpoints:

npm run hetzner:health

Use the strict health gate when staging should be considered ready:

npm run hetzner:health:check

The first green server milestone is not "the UI opens". It is: all containers build, start, and report health on the staging host.

Backup And Restore

The first backup/restore runbook lives in porting/2026-05-05-hetzner-backup-restore.md.

Useful commands:

npm run hetzner:backup:check
npm run hetzner:backup:plan
npm run hetzner:restore:plan

The backup scope covers Postgres, MinIO, Caddy data/config volumes, and runtime config files. Runtime config backups contain secrets and must stay out of git and in restricted, encrypted storage.

Evidence Collection

The first server-side evidence workflow lives in porting/2026-05-05-hetzner-staging-evidence.md.

Useful commands:

npm run hetzner:evidence:plan
npm run hetzner:evidence:check
npm run hetzner:evidence:collect

Run the collector on Hetzner after each major milestone. It writes Markdown evidence into docs/evidence/ and intentionally avoids env values, heavy build output, rsync push output, migration execution output, stack startup output, logs, database rows, and secret files.

Current Verification Evidence

• npm run hetzner:plan passed locally.
• npm run hetzner:host reports the current host shape. Locally it correctly reports that this Mac is not the target native Linux/amd64 build host and has too little free disk for heavy image builds.
• npm run hetzner:check passed locally with YAML fallback.
• npm run hetzner:env reports the expected missing real .env files locally without printing secret values.
• npm run hetzner:health is available as a post-start staging probe. Locally it reports expected failures because the Hetzner stack is not running on this Mac.
• picasso-widget and zweistein-embedding-widget now have first static Nginx Docker services in docker-compose.hetzner.yml.
• .dockerignore files were added for the new widget build contexts so local node_modules, dist, and .env files are not sent to Docker.
• Existing Studio API, Zweistein Server/Admin, Query Engine, and Ingestion Worker Docker contexts now also ignore local generated folders, caches, virtual environments, and real env files.
• npm run hetzner:sync:plan now prints the guarded source sync flow.
• npm run hetzner:sync:check verifies the rsync filter and refuses to proceed if real local Hetzner env/Caddy files exist.
• npm run hetzner:sync:push refuses to run rsync --delete unless HETZNER_SYNC_CONFIRM=1 or --yes is present.
• reverse-proxy is now defined as an optional Caddy service with template-based domain routing and TLS.
• Backup/restore helper scripts now print first-run backup and restore rehearsal plans.
• Evidence helper scripts now collect server-side status into Markdown without reading secret values or service logs.
• Bootstrap helper scripts now check whether the Hetzner host has the required base tools before source sync and image builds.
• Runtime config helper scripts now create missing ignored Hetzner config files from templates without overwriting existing files.
• configs/hetzner/frontend-build.env.example now includes required Compose-level Postgres and MinIO credentials so the stack cannot silently rely on placeholder defaults.
• Stack helper scripts now provide guarded server-side commands for Compose config, build, up, ps, and proxy startup.
• The local check reports that real configs/hetzner/*.env files are intentionally missing.
• docker-compose.hetzner.yml uses real .env filenames for runtime services, not .env.example.
• configs/hetzner/*.env is ignored by git.

Known Blockers

• The Query Engine Docker image still needs a successful full build on native Linux/amd64 or Hetzner staging.
• The current Mac is too low on disk for more heavy image builds.
• The current Mac fails the strict host gate because it is darwin arm64, has very little free disk, and has local MinIO/tunnel ports in use.
• Docker Compose config was not fully validated locally because the local Docker CLI does not include the Compose plugin.
• The widget containers still need a successful server-side image build and health check on Hetzner.
• The reverse proxy still needs real DNS, real configs/hetzner/reverse-proxy.env, real configs/hetzner/Caddyfile, and a server-side Caddy config validation.
• Backup and restore still need a real Hetzner run and restore rehearsal.
• No real Hetzner evidence file exists yet.
• No real Hetzner bootstrap evidence exists yet.
• End-to-end Studio Editor to Houston App to Zweistein Agent/AI runtime is not verified yet.

Next Slice

The next safe slice is to run the Hetzner staging build on a native Linux/amd64 host and capture exact evidence:

• Query Engine image build result.
• Ingestion Worker image build result.
• Backend health checks.
• Frontend health checks.
• One end-to-end published App flow.