Blinkin Legacy Docs Docs
Porting · porting/2026-05-05-hetzner-python-gate.md Docs Home

Hetzner Python Gate

Date: 2026-05-05

Objective

Turn the remaining Python blocker into a repeatable server-side gate for the Hetzner rebuild.

This gate focuses only on the two Python services:

  • Zweistein Query Engine
  • Zweistein Ingestion Worker

It does not replace the full stack runbook. It gives the riskiest Python/Torch part its own early proof before the remaining application images are built.

What Was Added

  • scripts/hetzner-python-gate.mjs Server-guarded helper for Python preflight, Torch wheel probe, Query Engine build, Ingestion Worker build, import probes, Python subset startup, Compose inspection, and Query Engine health.
  • package.json scripts:
    • hetzner:python:plan
    • hetzner:python:commands
    • hetzner:python:preflight
    • hetzner:python:torch
    • hetzner:python:build:query
    • hetzner:python:build:ingestion
    • hetzner:python:build
    • hetzner:python:imports
    • hetzner:python:up
    • hetzner:python:ps
    • hetzner:python:health

Safety Rules

  • Heavy commands require native Linux x64/amd64.
  • The helper requires at least 40 GiB free disk unless HETZNER_PYTHON_GATE_ALLOW_LOW_DISK=1 is set intentionally.
  • The helper now requires at least 4 GiB memory visible to Docker unless HETZNER_PYTHON_GATE_ALLOW_LOW_MEMORY=1 is set intentionally. This is specifically for the Linux spider-rs/chromiumoxide_cdp Rust source build, which failed locally with only about 1.9 GiB available to Docker.
  • 8 GiB Docker-visible memory is recommended for the full Python image builds.
  • Runtime commands require real ignored Hetzner env files:
    • configs/hetzner/frontend-build.env
    • configs/hetzner/zweistein-query-engine.env
    • configs/hetzner/zweistein-ingestion-worker.env
  • The helper does not print env values, logs, tokens, or database rows.
  • The local Mac can print the plan and command lines, but it is expected to fail hetzner:python:preflight.

Server Workflow

Run this on the Hetzner host after source sync, npm install, and real runtime config:

npm run hetzner:python:plan
npm run hetzner:python:commands

Then run the server gate:

npm run hetzner:python:preflight
npm run hetzner:env:check
npm run hetzner:python:torch
npm run hetzner:python:build:query
npm run hetzner:python:build:ingestion
npm run hetzner:python:imports
npm run hetzner:python:up
npm run hetzner:python:ps
npm run hetzner:python:health

If this passes, continue the full platform stack:

npm run hetzner:stack:build
npm run hetzner:stack:up
npm run hetzner:health:check
npm run hetzner:evidence:collect

Exact Command Shape

The helper prints these server commands through npm run hetzner:python:commands:

docker run --rm --platform linux/amd64 python:3.11-slim python -m pip install --dry-run --index-url https://download.pytorch.org/whl/cpu 'torch==2.10.0+cpu'
docker compose --env-file configs/hetzner/frontend-build.env -f docker-compose.hetzner.yml build query-engine
docker compose --env-file configs/hetzner/frontend-build.env -f docker-compose.hetzner.yml build ingestion-worker
docker compose --env-file configs/hetzner/frontend-build.env -f docker-compose.hetzner.yml run --rm --no-deps --entrypoint python query-engine -c "import main; print('query main import ok')"
docker compose --env-file configs/hetzner/frontend-build.env -f docker-compose.hetzner.yml run --rm --no-deps --entrypoint python ingestion-worker -c "import redis_worker; import message_processor; print('ingestion imports ok')"
docker compose --env-file configs/hetzner/frontend-build.env -f docker-compose.hetzner.yml up -d redis query-engine ingestion-worker
docker compose --env-file configs/hetzner/frontend-build.env -f docker-compose.hetzner.yml ps --format json redis query-engine ingestion-worker
curl -fsS http://127.0.0.1:5001/healthz

Running-State Gate

npm run hetzner:python:ps is a real gate, not just a display command.

It runs Docker Compose with JSON output, parses the returned service rows, and fails unless all three Python-subset services are present and running:

  • redis
  • query-engine
  • ingestion-worker

This is especially important for ingestion-worker, because it is a worker process rather than an HTTP service with its own health endpoint.

Local Verification Evidence

  • node --check scripts/hetzner-python-gate.mjs passed.
  • npm run hetzner:python:plan prints the ordered Python server gate.
  • npm run hetzner:python:commands prints exact Docker and Docker Compose commands.
  • npm run hetzner:python:commands now shows ps --format json redis query-engine ingestion-worker.
  • npm run hetzner:python:ps fails locally as expected before any server state is inspected, because the command is guarded for native Linux x64/amd64.
  • npm run hetzner:python:preflight fails locally as expected:
    • platform is darwin arm64;
    • Docker Compose is now available locally after installing Homebrew docker-compose 5.1.3 and repointing the broken Docker CLI plugin symlink;
    • free disk is about 7.0 GiB after the real Query Engine Docker build attempts, architecture probes, and generated dependency cleanup;
    • minimum Python gate disk is 40 GiB;
    • host memory is about 16.0 GiB, but Docker daemon memory is about 1.9 GiB;
    • minimum Docker memory for the spider-rs source build is 4 GiB;
    • the helper is guarded for native Linux x64/amd64.
  • With HETZNER_PYTHON_GATE_ALLOW_NON_HETZNER=1 HETZNER_PYTHON_GATE_ALLOW_LOW_DISK=1, local preflight now continues past the platform/disk guards and fails at the new memory guard: Docker exposes about 1.9 GiB, below the 4 GiB minimum.
  • The local Query Engine Docker build path reached the heavy Linux dependency set, including the CPU Torch wheel, then became effectively idle while building the native spider-rs wheel under Docker Desktop amd64 emulation. The build was stopped deliberately and Docker reported exit code 137.
  • ZWEISTEIN_DOCKER_PLATFORM=linux/arm64 npm run zweistein-query:docker-torch-probe passes on this Mac's native Docker architecture and resolves torch-2.10.0+cpu-cp311-cp311-manylinux_2_28_aarch64.whl. This is useful local-development evidence, but it does not replace the Hetzner gate, because the target server remains native Linux x64/amd64.
  • The Query Engine and Ingestion Worker Dockerfiles now include pkg-config and libssl-dev for spider-rs/openssl-sys, set CARGO_BUILD_JOBS=1, and expose an optional low-memory Rust build profile through RUST_LOW_MEMORY_RELEASE=1.
  • ZWEISTEIN_DOCKER_PLATFORM=linux/arm64 ZWEISTEIN_DOCKER_LOW_MEMORY_RUST=1 ZWEISTEIN_PYTHON_ALLOW_LOW_DISK=1 npm run zweistein-query:docker-build:low-memory reached the real spider-rs native wheel build and proved the low-memory profile was applied (opt-level=1, no embedded bitcode, high codegen unit split). It still failed at chromiumoxide_cdp with SIGKILL under Docker Desktop's roughly 1.9 GiB memory limit.
  • npm run zweistein-python:wheel-audit reads the committed Poetry locks without installing packages. It reports that all torch entries have Linux wheels, while all four spider-rs 0.0.53 lock entries have zero Linux wheels and therefore require source builds on Linux.
  • Generated node_modules folders under legacy-src/ were removed to recover local disk. This does not change source or dist artifacts, but non-Python smoke reruns require reinstalling dependencies.

Current Boundary

This is still not Python runtime proof.

The actual gate must pass on the Hetzner/Linux host before the full port can be considered complete. The local Docker experiments and lockfile wheel audit prove the next server-side risk is the native spider-rs/chromiumoxide_cdp source build, not Torch wheel availability.