Hetzner Port Completion Audit

Date: 2026-05-05

Verdict

The goal is not complete yet.

The local folder, imported source, local rebuild slices, coordinated non-Python local runtime smoke, database gate, guarded source sync helper, local full-stack evidence contract, Hetzner Compose package, config helpers, Python gate, stack runbook, milestone runner, completion gate, backup runbook, and documentation site are in place. The missing proof is still the Python-backed full stack running either as one local assembly or on a native Hetzner Linux/amd64 host.

Objective Restated As Success Criteria

Original objective:

Port the full code base of Picasso, Zweistein, and Studio API to Hetzner, first rebuilding it locally in the new folder.

Concrete success criteria:

1. Work happens only in LegacyBlinkin-2-Hetzner, not in blinkin-2-platform.
2. The legacy Picasso FE, Zweistein, and Studio API source trees are present in the new project.
3. Real secrets and real runtime env files are not copied or committed.
4. Local rebuild slices prove each major service can install/build/start where the current machine can support it.
5. A local runtime map shows the whole platform assembly order and current gaps.
6. Hetzner deployment files exist for the full platform stack.
7. Hetzner runtime config is initialized safely from templates and passes strict placeholder checks on the server.
8. Hetzner host bootstrap, Docker Compose config, image builds, stack startup, health checks, proxy startup, backup, and evidence collection run on a native Linux/amd64 host.
9. A complete end-to-end platform flow is verified: Studio Editor to published Houston App to Zweistein Agent/AI runtime.

Prompt-To-Artifact Checklist

Commands Inspected In This Audit

npm run legacy:verify
npm run hetzner:sync:check
npm run hetzner:sync:target
npm run db:local:check
npm run legacy:runtime:smoke:non-python
npm run legacy:runtime:status
npm run legacy:runtime:missing
npm run hetzner:python:plan
npm run hetzner:python:commands
npm run hetzner:python:preflight
npm run hetzner:runbook:plan
npm run hetzner:runbook:commands
npm run hetzner:runbook:check
npm run port:completion:plan
npm run port:completion:status
npm run port:completion:check
npm run local:full-stack:plan
npm run local:full-stack:probe
npm run local:full-stack:template
npm run local:full-stack:collect
npm run local:full-stack:check
npm run platform:e2e:plan
npm run platform:e2e:template
npm run platform:e2e:check
npm run hetzner:evidence:check
npm run hetzner:backup:commands
npm run hetzner:restore:commands
npm run hetzner:check
npm run hetzner:config:status
npm run hetzner:config:checklist
npm run hetzner:stack:commands
find docs/evidence -maxdepth 1 -type f
find configs/hetzner -maxdepth 1 \( -name '*.env' -o -name 'Caddyfile' \) -type f

Observed current facts:

• legacy:verify passed.
• hetzner:sync:check passed; it reported 0 local Hetzner runtime env/Caddy files and warned that root .env is excluded by the rsync filter.
• hetzner:sync:target fails locally as expected without HETZNER_SYNC_TARGET and refuses non-standard remote paths before SSH.
• hetzner:sync:push refuses to run rsync --delete without HETZNER_SYNC_CONFIRM=1 or --yes.
• Successful source sync dry-run and push commands now write redacted docs/evidence/*-hetzner-source-sync-*.md files.
• db:local:check passed; Studio API source/applied migration counts both equal 84, Studio has 37 public tables, and Zweistein has 39 public tables.
• Zweistein has no imported src/dataSource.ts or src/migrations directory and currently relies on the primary TypeORM connection's synchronize:true for schema creation.
• legacy:runtime:smoke:non-python passed and left the seven app ports free after shutdown.
• legacy:runtime:missing reports no missing required runtime assembly files.
• legacy:runtime:status during the repeatable coordinated smoke showed all non-Python app services responding with HTTP 200.
• hetzner:python:plan and hetzner:python:commands print the new server-side Python gate.
• hetzner:python:commands now prints docker compose ... ps --format json redis query-engine ingestion-worker.
• hetzner:python:ps is guarded for native Linux x64/amd64 and will parse Docker Compose JSON on Hetzner to fail unless redis, query-engine, and ingestion-worker are all running.
• hetzner:python:preflight fails locally as expected because this machine is darwin arm64 with about 7.6 GiB free disk and no Docker Compose plugin, while the gate requires native Linux x64/amd64 and 40 GiB free disk.
• hetzner:runbook:plan and hetzner:runbook:commands print 6 milestones and 28 commands.
• hetzner:runbook:check fails locally as expected because this machine is darwin arm64.
• hetzner:runbook:readiness refuses to run with diagnostics override unless HETZNER_RUNBOOK_CONFIRM=1 is set.
• Confirmed runbook milestone execution now writes redacted docs/evidence/*-hetzner-runbook-<milestone>.md files, but no confirmed milestone was executed locally.
• Future runbook milestone evidence includes Expected command count and a ## Expected Commands manifest; port:completion:* requires this manifest before accepting a milestone as complete.
• port:completion:plan and port:completion:status run locally.
• port:completion:check fails locally as expected because local full-stack evidence, real server runtime config, runbook evidence, source-sync evidence, end-to-end evidence, and backup/restore evidence are missing.
• local:full-stack:plan and local:full-stack:template run locally.
• local:full-stack:probe is available and performs live HTTP checks for the seven non-Python local URLs plus http://127.0.0.1:5001/docs for the Query Engine.
• local:full-stack:collect is available and refuses to write complete evidence unless all live HTTP probes pass, LOCAL_FULL_STACK_VERIFIER is set, LOCAL_FULL_STACK_EVIDENCE_CONFIRM=1 is set, and both manual observation flags are set.
• local:full-stack:check fails locally as expected because no completed docs/evidence/*-local-full-stack.md file exists.
• platform:e2e:plan and platform:e2e:template run locally.
• platform:e2e:check fails locally as expected because no completed docs/evidence/*-platform-e2e.md file exists.
• hetzner:evidence:check passes locally and reports 18 configured read-only/status evidence commands.
• hetzner:backup:commands prints the backup command list.
• hetzner:backup:commands now prints the Postgres dump with --env-file configs/hetzner/frontend-build.env.
• hetzner:backup:run refuses locally without HETZNER_BACKUP_CONFIRM=1 and refuses on macOS even with confirmation unless diagnostics override is explicitly set.
• hetzner:restore:commands prints the destructive restore rehearsal command list without running it.
• hetzner:restore:commands now prints Docker Compose stop, Postgres restore, and restart commands with --env-file configs/hetzner/frontend-build.env.
• hetzner:restore:rehearse refuses locally without HETZNER_RESTORE_CONFIRM=1, refuses on macOS even with confirmation, and still refuses with diagnostics override when configs/hetzner/backup.env is missing.
• Python disk readiness is still below the local install/build guards; the latest Python gate preflight saw about 7.6 GiB free, below the 8 GiB install guard, 20 GiB Docker build guard, and 40 GiB Hetzner Python gate guard.
• Ingestion Worker install dry-run reports 307 installs; Query Engine install dry-run reports only torch still pending in the local venv.
• hetzner:check passes locally only for file presence and YAML fallback.
• hetzner:config:status reports 8 real runtime config files missing locally.
• hetzner:config:checklist prints the required server value checklist from checked-in templates only and does not read runtime env values.
• hetzner:stack:commands prints the exact server Compose commands.
• docs/evidence/ currently has 0 real Hetzner evidence files.
• There is no docs/evidence/*-hetzner-backup.md backup evidence yet.
• There is no docs/evidence/*-backup-restore-rehearsal.md restore rehearsal evidence yet.
• configs/hetzner/ currently has 0 real runtime env/Caddy files locally.

Completion Blockers

The goal cannot be marked complete until these are proven:

1. Real runtime config files are created on Hetzner and npm run hetzner:env:check passes.
2. Full local-stack evidence exists for Studio API, Picasso, Zweistein, Query Engine, Ingestion Worker, and Agent/AI runtime behavior.
3. Source is actually synced to the Hetzner host with npm run hetzner:sync:dry-run followed by confirmed npm run hetzner:sync:push.
4. npm run hetzner:bootstrap:check and npm run hetzner:host:check pass on native Linux/amd64.
5. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:readiness passes on Hetzner.
6. npm run hetzner:stack:config passes with real server config.
7. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:python passes on Hetzner.
8. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:build passes on Hetzner.
9. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:database passes on Hetzner.
10. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:stack passes on Hetzner.
11. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:proxy starts the public edge after DNS/config is real.
12. One Studio Editor to Houston published App to Zweistein Agent/AI flow is manually verified.
13. npm run hetzner:evidence:collect writes real evidence files under docs/evidence/.
14. Backup and restore rehearsal evidence exists.

Next Concrete Action

Move from local preparation to a real Hetzner staging run:

npm run hetzner:sync:plan
npm run hetzner:sync:check
npm run db:local:check
npm run hetzner:db:plan
npm run hetzner:runbook:plan
npm run port:completion:plan
npm run hetzner:python:plan
npm run hetzner:stack:plan
npm run hetzner:stack:commands

Then run the listed server gates on a native Linux/amd64 Hetzner host and capture evidence.