Blinkin Legacy Docs Docs
Porting · porting/2026-05-05-zweistein-python-services-local-probe.md Docs Home

Zweistein Python Services Local Probe

Date: 2026-05-05

Objective

Start the local rebuild of the imported Zweistein Python services:

  • python_server/query_engine: FastAPI query engine for chat, agentic apps, tools, RAG, media routing, and model orchestration.
  • python_server/ingestion_worker: Redis worker for document/media ingestion and indexing.

This slice verifies metadata, local Python selection, placeholder env coverage, partial Query Engine install, and the current blocker.

What Was Added

  • legacy-src/zweistein/python_server/query_engine/README.md
  • legacy-src/zweistein/python_server/ingestion_worker/README.md

The Poetry metadata referenced README.md, but the files were missing in the imported snapshot.

  • configs/local/zweistein-query-engine.env.example Complete local placeholder values for Query Engine required settings.
  • configs/local/zweistein-ingestion-worker.env.example Complete local placeholder values for Ingestion Worker required settings.
  • scripts/zweistein-python-local.mjs Small helper for Poetry checks, Python 3.12 environment selection, dry-run install plans, disk readiness checks, guarded install commands, import probes, Docker Torch probes, Docker build attempts, and optional low-memory Rust Docker builds.
  • scripts/zweistein-python-wheel-audit.mjs Reads the committed Poetry lockfiles and reports whether critical native dependencies have Linux wheels or require a source build.
  • scripts/zweistein-python-lazy-import-check.mjs Verifies that spider_rs is not imported at module load time in the Query Engine crawler endpoint or the Ingestion Worker website processor.
  • scripts/hetzner-python-gate.mjs Server-side follow-up helper for the same Python blocker on native Linux/amd64.
  • legacy-src/zweistein/python_server/query_engine/devops/docker/Dockerfile Adds the native build headers needed by spider-rs and openssl-sys, constrains Cargo parallelism, and accepts RUST_LOW_MEMORY_RELEASE=1 for constrained local Docker Desktop experiments.
  • legacy-src/zweistein/python_server/ingestion_worker/devops/docker/Dockerfile Mirrors the same native Rust/OpenSSL build readiness for the worker image.
  • package.json scripts:
    • zweistein-python:check
    • zweistein-python:disk
    • zweistein-python:wheel-audit
    • zweistein-python:lazy-import-check
    • zweistein-query:env
    • zweistein-query:install:plan
    • zweistein-query:install
    • zweistein-query:import
    • zweistein-query:main-import
    • zweistein-query:docker-torch-probe
    • zweistein-query:docker-torch-probe:amd64
    • zweistein-query:docker-torch-probe:arm64
    • zweistein-query:docker-build
    • zweistein-query:docker-build:low-memory
    • zweistein-ingestion:env
    • zweistein-ingestion:install:plan
    • zweistein-ingestion:install
    • zweistein-ingestion:import
    • zweistein-ingestion:docker-build
    • zweistein-ingestion:docker-build:low-memory

Sensitive Data Cleanup

During Python inspection, API-key-like defaults were found in the imported local Query Engine settings. A JWT-like Studio API fixture was also found during the expanded scan.

The local copy now uses placeholders instead. The foundation verifier now fails if it sees common hardcoded secret patterns such as private keys, JWT-like tokens, NVIDIA API keys, Google API keys, AWS keys, GitHub tokens, Stripe webhook secrets, or Stripe live/test secrets.

Local Commands

Check Poetry metadata for both services:

npm run zweistein-python:check

Check whether the current machine has enough disk for local Python installs/builds:

npm run zweistein-python:disk

Select Python 3.12 for Query Engine:

npm run zweistein-query:env

Preview the Query Engine install without writing packages:

npm run zweistein-query:install:plan

Attempt Query Engine dependency install:

npm run zweistein-query:install

Run the lightweight Query Engine import probe:

npm run zweistein-query:import

Run the Query Engine FastAPI app import probe:

npm run zweistein-query:main-import

Check the Hetzner-like Linux/amd64 Torch wheel without doing a full image build:

npm run zweistein-query:docker-torch-probe

Check the Torch wheel explicitly for the Hetzner target architecture:

npm run zweistein-query:docker-torch-probe:amd64

Check the Torch wheel explicitly for this Mac's native Docker architecture:

npm run zweistein-query:docker-torch-probe:arm64

Attempt the legacy Query Engine Dockerfile path:

npm run zweistein-query:docker-build

Select Python 3.12 for Ingestion Worker:

npm run zweistein-ingestion:env

Preview the Ingestion Worker install without writing packages:

npm run zweistein-ingestion:install:plan

Verification Evidence

  • poetry check initially failed for both Python services because README.md was missing.
  • After adding local placeholder READMEs, npm run zweistein-python:check passed for both services with Poetry 2 metadata warnings only.
  • npm run zweistein-query:env selected /opt/homebrew/bin/python3.12.
  • Raw POETRY_VIRTUALENVS_IN_PROJECT=true poetry install --no-root --no-interaction --no-ansi for Query Engine created a local .venv and installed many dependencies.
  • Query Engine install stopped with torch (2.10.0+cpu) because Poetry could not find a compatible wheel for this local macOS/Python 3.12 arm64 environment from the configured pytorch-cpu source.
  • npm run zweistein-query:import succeeded for the partial environment and printed:
3.12.12
0.111.1
  • poetry run python -c "import torch" failed with ModuleNotFoundError, confirming the Torch dependency is not installed.
  • npm run zweistein-query:main-import reached the legacy FastAPI app import path and then failed at llama_index.postprocessor.colbert_rerank, which imports torch. This confirms the next local startup blocker is the missing Torch dependency, not missing env placeholders.
  • npm run zweistein-ingestion:env created the local Ingestion Worker Python 3.12 environment.
  • The broken local Docker Compose symlink was replaced with the Homebrew-installed plugin, and docker compose version now reports Docker Compose version 5.1.3.
  • Clearing rebuildable developer caches temporarily freed enough local disk for both local Python guards. Follow-up real Docker builds and generated dependency cleanup left the current Mac with about 7.0 GiB free at the project path. That is below the 8 GiB local Poetry install guard, the 20 GiB local Docker build guard, and the 40 GiB Hetzner Python gate guard.
  • To free local disk for the Python experiments, generated node_modules folders under legacy-src/ were removed. Source files and built dist artifacts were kept, but non-Python local smoke commands will need dependency reinstalls before they can be repeated.
  • npm run zweistein-query:install:plan reports only one remaining Query Engine install operation in the current local venv: torch (2.10.0+cpu).
  • npm run zweistein-query:install was attempted after the disk guard passed. It still failed because Poetry identified 38 torch wheels but skipped all of them as unsupported for the current local environment ABI tags.
  • npm run zweistein-ingestion:import currently fails at the first import because redis is not installed in the Ingestion Worker venv.
  • npm run zweistein-ingestion:install:plan reports 307 installs, including redis, opencv-python, spider-rs, torch (2.5.1), and openai-whisper.
  • npm run zweistein-ingestion:install is now guarded by the same disk helper. It is no longer blocked by the 8 GiB local install guard, but the dry-run still shows a very large install set and should run on Hetzner/native Linux unless we intentionally spend the local disk budget.
  • npm run legacy:verify now reports potential hardcoded secrets: 0.
  • docker run --rm --platform linux/amd64 python:3.11-slim python -m pip install --dry-run --index-url https://download.pytorch.org/whl/cpu 'torch==2.10.0+cpu' passed. The Linux container resolved torch-2.10.0+cpu-cp311-cp311-manylinux_2_28_x86_64.whl and ended with Would install ... torch-2.10.0+cpu ....
  • ZWEISTEIN_DOCKER_PLATFORM=linux/arm64 npm run zweistein-query:docker-torch-probe passed on this Mac's native Docker architecture. The Linux container resolved torch-2.10.0+cpu-cp311-cp311-manylinux_2_28_aarch64.whl and ended with Would install ... torch-2.10.0+cpu ....
  • docker build --platform linux/amd64 -f legacy-src/zweistein/python_server/query_engine/devops/docker/Dockerfile -t legacy-blinkin-query-engine:local legacy-src/zweistein/python_server/query_engine started the real legacy Query Engine Dockerfile path. It installed Debian build tooling, Rust, Poetry 1.8.3, the first locked dependency set, and reached torch (2.6.0+cpu), transformers, weaviate-client, crewai, presidio, spider-rs, and other service dependencies before becoming effectively idle while building the native spider-rs wheel under Docker Desktop amd64 emulation. The build container was stopped deliberately after CPU and IO went flat; Docker then reported build exit code 137 for the dependency install step. The generated intermediate container/image were removed.
  • The Dockerfiles were patched to include pkg-config and libssl-dev, because the first linux/arm64 build reached spider-rs and failed in openssl-sys native build detection.
  • The Dockerfiles now set CARGO_BUILD_JOBS=1, and the helper can pass RUST_LOW_MEMORY_RELEASE=1 through npm run zweistein-query:docker-build:low-memory.
  • ZWEISTEIN_DOCKER_PLATFORM=linux/arm64 ZWEISTEIN_DOCKER_LOW_MEMORY_RUST=1 ZWEISTEIN_PYTHON_ALLOW_LOW_DISK=1 npm run zweistein-query:docker-build:low-memory reached the real spider-rs native wheel build on this Mac's native Docker architecture. The low-memory profile took effect: the failing rustc command used -C opt-level=1, -C embed-bitcode=no, and -C codegen-units=256 instead of the original optimized/LTO profile.
  • The linux/arm64 low-memory build still failed compiling chromiumoxide_cdp with signal: 9, SIGKILL under Docker Desktop's roughly 1.9 GiB memory limit. This makes the current local blocker a Docker Desktop resource limit around spider-rs/chromiumoxide_cdp, not Torch wheel availability.
  • npm run zweistein-python:wheel-audit passed against the four committed Query Engine and Ingestion Worker lockfiles. It reports torch lock entries without Linux wheels: 0 and spider-rs Linux source-build requirements: 4.
  • The same audit shows spider-rs 0.0.53 has no Linux wheel in any committed lockfile; it has one macOS arm64 wheel plus a source archive. That means a Hetzner/Linux build must compile spider-rs from source unless the legacy crawler implementation is deliberately replaced.
  • The direct spider_rs imports in api/v1/endpoints/crawler.py and ingestion_worker/message_processor.py were moved behind small lazy helper functions. Crawler behavior still requires spider-rs when used, but module import no longer fails solely because the crawler dependency has not been built yet.
  • npm run zweistein-python:lazy-import-check passed and verifies those two files have no top-level spider_rs import.
  • npm run local:full-stack:doctor now reports 14 blocker(s): Python install disk, Python Docker build disk, Query Engine torch import, Ingestion Worker redis import, seven stopped HTTP probes, Query Engine HTTP probe, and the two manual observation flags.

Current Blocker

The Query Engine lock resolves torch to a CPU wheel from the explicit pytorch-cpu source. That source did not provide a compatible wheel for this Mac/Python 3.12 arm64 setup.

The Linux/amd64 Docker probe and the lockfile wheel audit confirm that the required CPU Torch wheels are available for Hetzner-like Linux targets. The Linux/arm64 Docker probe confirms that the same locked Torch version is also available for this Mac's native Docker architecture. The remaining local blocker is no longer Torch availability on Linux; it is that spider-rs has no locked Linux wheel and its native chromiumoxide_cdp source build is too heavy for this Mac's current Docker Desktop resource limits.

The crawler dependency is now isolated to crawler feature use instead of Python module import. This does not complete the full rebuild, but it reduces the blast radius of the native Rust dependency while preserving the legacy crawler feature boundary.

For the Hetzner path, the next decision should be one of:

  • verify the full Query Engine Docker build on a native Linux amd64 machine or Hetzner staging box with npm run hetzner:python:build:query;
  • optionally retry a local Mac-native linux/arm64 Docker image build only after increasing Docker Desktop memory well beyond 1.9 GiB and freeing disk, while keeping Hetzner proof on native linux/amd64;
  • verify the Ingestion Worker Docker build on the same target with npm run hetzner:python:build:ingestion;
  • run npm run hetzner:python:imports, npm run hetzner:python:up, and npm run hetzner:python:health before continuing the full stack;
  • pin a macOS-compatible local development Torch source/version separately from production;
  • split heavyweight ML dependencies behind optional groups so lightweight API/import checks can run without Torch;
  • regenerate the lockfile for the target Hetzner architecture after choosing the runtime image.

Not Yet Done

  • Query Engine does not fully install locally on macOS yet.
  • Query Engine Docker build did not complete on this Mac under amd64 emulation; the real attempt was stopped after becoming idle in spider-rs native wheel build and Docker returned exit code 137.
  • Query Engine Docker build also did not complete on this Mac's native linux/arm64 Docker platform. After the OpenSSL header fix and the low-memory Rust profile, it still failed at chromiumoxide_cdp with SIGKILL under the current Docker Desktop memory limit.
  • Query Engine FastAPI app was not started.
  • Ingestion Worker dependencies were not installed. The dry-run install plan requires 307 packages and should run on Hetzner/native Linux unless we intentionally spend the local disk budget.
  • Ingestion Worker Docker build was not attempted yet.
  • Redis worker processing was not exercised.
  • Weaviate/vector storage was not exercised.
  • Document/media ingestion, OCR, transcription, RAG, and agentic tool routes were not tested.
  • The final Docker/Hetzner Python runtime shape is not decided yet.