Hetzner Port Completion Audit

Date: 2026-05-05

Verdict

The goal is not complete yet.

The local folder, imported source, local rebuild slices, coordinated non-Python local runtime smoke, database gate, guarded source sync helper, local full-stack evidence contract, Hetzner Compose package, config helpers, Python gate, stack runbook, milestone runner, completion gate, backup runbook, and documentation site are in place. The missing proof is still the Python-backed full stack running either as one local assembly or on a native Hetzner Linux/amd64 host.

Objective Restated As Success Criteria

Original objective:

Port the full code base of Picasso, Zweistein, and Studio API to Hetzner, first rebuilding it locally in the new folder.

Concrete success criteria:

1. Work happens only in LegacyBlinkin-2-Hetzner, not in blinkin-2-platform.
2. The legacy Picasso FE, Zweistein, and Studio API source trees are present in the new project.
3. Real secrets and real runtime env files are not copied or committed.
4. Local rebuild slices prove each major service can install/build/start where the current machine can support it.
5. A local runtime map shows the whole platform assembly order and current gaps.
6. Hetzner deployment files exist for the full platform stack.
7. Hetzner runtime config is initialized safely from templates and passes strict placeholder checks on the server.
8. Hetzner host bootstrap, Docker Compose config, image builds, stack startup, health checks, proxy startup, backup, and evidence collection run on a native Linux/amd64 host.
9. A complete end-to-end platform flow is verified: Studio Editor to published Houston App to Zweistein Agent/AI runtime.

Prompt-To-Artifact Checklist

Commands Inspected In This Audit

npm run legacy:verify
npm run hetzner:sync:check
npm run hetzner:sync:target
npm run db:local:check
npm run legacy:runtime:smoke:non-python
npm run legacy:runtime:status
npm run legacy:runtime:missing
npm run hetzner:python:plan
npm run hetzner:python:commands
npm run hetzner:python:preflight
npm run hetzner:runbook:plan
npm run hetzner:runbook:commands
npm run hetzner:runbook:check
npm run port:completion:plan
npm run port:completion:status
npm run port:completion:check
npm run local:full-stack:plan
npm run local:full-stack:doctor
npm run local:full-stack:probe
npm run local:full-stack:template
npm run local:full-stack:collect
npm run local:full-stack:check
npm run platform:e2e:plan
npm run platform:e2e:template
npm run platform:e2e:check
npm run hetzner:evidence:check
npm run hetzner:backup:commands
npm run hetzner:restore:commands
npm run hetzner:check
npm run hetzner:config:status
npm run hetzner:config:checklist
npm run hetzner:stack:commands
find docs/evidence -maxdepth 1 -type f
find configs/hetzner -maxdepth 1 \( -name '*.env' -o -name 'Caddyfile' \) -type f

Observed current facts:

• legacy:verify passed.
• hetzner:sync:check passed; it reported 0 local Hetzner runtime env/Caddy files and warned that root .env is excluded by the rsync filter.
• hetzner:sync:target fails locally as expected without HETZNER_SYNC_TARGET and refuses non-standard remote paths before SSH.
• hetzner:sync:push refuses to run rsync --delete without HETZNER_SYNC_CONFIRM=1 or --yes.
• Successful source sync dry-run and push commands now write redacted docs/evidence/*-hetzner-source-sync-*.md files.
• db:local:check passed; Studio API source/applied migration counts both equal 84, Studio has 37 public tables, and Zweistein has 39 public tables.
• Zweistein has no imported src/dataSource.ts or src/migrations directory and currently relies on the primary TypeORM connection's synchronize:true for schema creation.
• legacy:runtime:smoke:non-python passed again after the Docker Compose and disk fixes. All seven non-Python app surfaces returned HTTP 200, Zweistein public tables: 39 was reported, Query Engine remained not responding, Ingestion Worker remained manual, and the smoke stopped its own app processes afterward.
• legacy:runtime:missing reports no missing required runtime assembly files.
• legacy:runtime:status during the repeatable coordinated smoke showed all non-Python app services responding with HTTP 200.
• hetzner:python:plan and hetzner:python:commands print the new server-side Python gate.
• hetzner:python:commands now prints docker compose ... ps --format json redis query-engine ingestion-worker.
• hetzner:python:ps is guarded for native Linux x64/amd64 and will parse Docker Compose JSON on Hetzner to fail unless redis, query-engine, and ingestion-worker are all running.
• hetzner:python:preflight fails locally as expected because this machine is darwin arm64 with about 7.0 GiB free disk, while the gate requires native Linux x64/amd64 and 40 GiB free disk. It now also prints host memory, Docker daemon memory, and the Docker-visible memory minimum needed for the spider-rs Rust source build. Docker Compose is available locally.
• HETZNER_PYTHON_GATE_ALLOW_NON_HETZNER=1 HETZNER_PYTHON_GATE_ALLOW_LOW_DISK=1 npm run hetzner:python:preflight fails locally at the new memory guard because Docker exposes about 1.9 GiB, below the 4 GiB minimum for the spider-rs/chromiumoxide_cdp source build.
• The local Docker Compose blocker was reduced: Homebrew docker-compose 5.1.3 was installed and the broken ~/.docker/cli-plugins/docker-compose symlink was repointed, so docker compose version now works locally.
• hetzner:runbook:plan and hetzner:runbook:commands print 6 milestones and 28 commands.
• hetzner:runbook:check fails locally as expected because this machine is darwin arm64.
• hetzner:runbook:readiness refuses to run with diagnostics override unless HETZNER_RUNBOOK_CONFIRM=1 is set.
• Confirmed runbook milestone execution now writes redacted docs/evidence/*-hetzner-runbook-<milestone>.md files, but no confirmed milestone was executed locally.
• Future runbook milestone evidence includes Expected command count and a ## Expected Commands manifest; port:completion:* requires this manifest before accepting a milestone as complete.
• port:completion:plan and port:completion:status run locally.
• port:completion:check fails locally as expected because local full-stack evidence, real server runtime config, runbook evidence, source-sync evidence, end-to-end evidence, and backup/restore evidence are missing.
• local:full-stack:plan and local:full-stack:template run locally.
• local:full-stack:doctor is available and combines disk readiness, Docker CLI, Docker Compose, rebuilt artifact checks, Python runtime directory checks, Query Engine torch import, Ingestion Worker redis import, HTTP probes, and manual observation flags.
• local:full-stack:probe is available and performs live HTTP checks for the seven non-Python local URLs plus http://127.0.0.1:5001/docs for the Query Engine.
• local:full-stack:collect is available and refuses to write complete evidence unless all live HTTP probes pass, LOCAL_FULL_STACK_VERIFIER is set, LOCAL_FULL_STACK_EVIDENCE_CONFIRM=1 is set, and both manual observation flags are set.
• local:full-stack:check fails locally as expected because no completed docs/evidence/*-local-full-stack.md file exists.
• platform:e2e:plan and platform:e2e:template run locally.
• platform:e2e:check fails locally as expected because no completed docs/evidence/*-platform-e2e.md file exists.
• hetzner:evidence:check passes locally and reports 18 configured read-only/status evidence commands.
• hetzner:backup:commands prints the backup command list.
• hetzner:backup:commands now prints the Postgres dump with --env-file configs/hetzner/frontend-build.env.
• hetzner:backup:run refuses locally without HETZNER_BACKUP_CONFIRM=1 and refuses on macOS even with confirmation unless diagnostics override is explicitly set.
• hetzner:restore:commands prints the destructive restore rehearsal command list without running it.
• hetzner:restore:commands now prints Docker Compose stop, Postgres restore, and restart commands with --env-file configs/hetzner/frontend-build.env.
• hetzner:restore:rehearse refuses locally without HETZNER_RESTORE_CONFIRM=1, refuses on macOS even with confirmation, and still refuses with diagnostics override when configs/hetzner/backup.env is missing.
• Python disk readiness improved after clearing rebuildable developer caches, which allowed the real Query Engine Docker build attempts to start. After those attempts and generated dependency cleanup, the local project path has about 7.0 GiB free, which is below the 8 GiB local Poetry install guard, the 20 GiB local Docker build guard, and the 40 GiB Hetzner Python gate guard.
• npm run zweistein-query:docker-build reached the Linux dependency install phase, including CPU Torch, then became effectively idle during native spider-rs wheel build under Docker Desktop amd64 emulation. The build container was stopped deliberately and Docker reported exit code 137.
• ZWEISTEIN_DOCKER_PLATFORM=linux/arm64 ZWEISTEIN_DOCKER_LOW_MEMORY_RUST=1 ZWEISTEIN_PYTHON_ALLOW_LOW_DISK=1 npm run zweistein-query:docker-build:low-memory reached spider-rs on the Mac-native Docker architecture and proved the low-memory Rust profile was active, but still failed compiling chromiumoxide_cdp with SIGKILL under Docker Desktop's roughly 1.9 GiB memory limit.
• The Query Engine and Ingestion Worker Dockerfiles now include pkg-config, libssl-dev, CARGO_BUILD_JOBS=1, and optional RUST_LOW_MEMORY_RELEASE=1 support for constrained local builders.
• npm run zweistein-python:wheel-audit passes and reports torch lock entries without Linux wheels: 0 plus spider-rs Linux source-build requirements: 4, proving the current Linux build risk is the crawler's Rust source build.
• npm run zweistein-python:lazy-import-check passes and confirms api/v1/endpoints/crawler.py plus ingestion_worker/message_processor.py no longer import spider_rs at module load time.
• Generated node_modules directories under legacy-src/ were removed to free disk. Source and built dist artifacts remain, but repeated non-Python smoke runs need dependency reinstalls.
• Ingestion Worker install dry-run reports 307 installs; Query Engine install dry-run reports only torch still pending in the local venv.
• npm run zweistein-query:install was attempted after the disk guard passed and still failed because Poetry skipped all 38 locked Torch wheels for this Mac/Python 3.12 arm64 ABI.
• npm run zweistein-query:docker-torch-probe passed again on Linux/amd64 and resolved torch-2.10.0+cpu-cp311-cp311-manylinux_2_28_x86_64.whl.
• ZWEISTEIN_DOCKER_PLATFORM=linux/arm64 npm run zweistein-query:docker-torch-probe passed and resolved torch-2.10.0+cpu-cp311-cp311-manylinux_2_28_aarch64.whl, proving the Torch blocker is not Linux/arm64 availability either.
• hetzner:check passes locally only for file presence and YAML fallback.
• hetzner:config:status reports 8 real runtime config files missing locally.
• hetzner:config:checklist prints the required server value checklist from checked-in templates only and does not read runtime env values.
• hetzner:stack:commands prints the exact server Compose commands.
• docs/evidence/ currently has 0 real Hetzner evidence files.
• There is no docs/evidence/*-hetzner-backup.md backup evidence yet.
• There is no docs/evidence/*-backup-restore-rehearsal.md restore rehearsal evidence yet.
• configs/hetzner/ currently has 0 real runtime env/Caddy files locally.

Completion Blockers

The goal cannot be marked complete until these are proven:

1. Real runtime config files are created on Hetzner and npm run hetzner:env:check passes.
2. Full local-stack evidence exists for Studio API, Picasso, Zweistein, Query Engine, Ingestion Worker, and Agent/AI runtime behavior.
3. Source is actually synced to the Hetzner host with npm run hetzner:sync:dry-run followed by confirmed npm run hetzner:sync:push.
4. npm run hetzner:bootstrap:check and npm run hetzner:host:check pass on native Linux/amd64.
5. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:readiness passes on Hetzner.
6. npm run hetzner:stack:config passes with real server config.
7. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:python passes on Hetzner.
8. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:build passes on Hetzner.
9. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:database passes on Hetzner.
10. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:stack passes on Hetzner.
11. HETZNER_RUNBOOK_CONFIRM=1 npm run hetzner:runbook:proxy starts the public edge after DNS/config is real.
12. One Studio Editor to Houston published App to Zweistein Agent/AI flow is manually verified.
13. npm run hetzner:evidence:collect writes real evidence files under docs/evidence/.
14. Backup and restore rehearsal evidence exists.

Next Concrete Action

Move from local preparation to a real Hetzner staging run:

npm run hetzner:sync:plan
npm run hetzner:sync:check
npm run db:local:check
npm run hetzner:db:plan
npm run hetzner:runbook:plan
npm run port:completion:plan
npm run hetzner:python:plan
npm run hetzner:stack:plan
npm run hetzner:stack:commands

Then run the listed server gates on a native Linux/amd64 Hetzner host and capture evidence.