Hetzner Staging Evidence

Date: 2026-05-05

Objective

Create a repeatable evidence workflow for the first real Hetzner staging run.

The goal is to avoid vague "it worked on the server" status. Every server milestone should leave a small, reviewable evidence file.

What Was Added

• scripts/hetzner-evidence.mjs Read-only evidence collector for server status.
• scripts/hetzner-runbook.mjs Guarded milestone runner that writes redacted per-milestone evidence files.
• package.json scripts:
  • hetzner:evidence:plan
  • hetzner:evidence:check
  • hetzner:evidence:collect

Evidence Command

After syncing source and creating runtime config files on Hetzner, run:

npm run hetzner:evidence:collect

The command writes a Markdown file into:

docs/evidence/

Confirmed runbook milestones also write files there automatically:

docs/evidence/<timestamp>-hetzner-runbook-<milestone>.md

What It Collects

The collector captures:

• source sync safety status;
• host preflight output;
• staging file readiness;
• env readiness status without secret values;
• Python gate preflight, service status, and Query Engine health;
• database gate status without database rows;
• backup support status;
• Docker and Docker Compose versions;
• Docker daemon status;
• disk space;
• Compose service status through npm run hetzner:stack:ps;
• Compose image status;
• health probe output;
• completion-gate status;
• platform E2E evidence status.

What It Does Not Collect

The collector intentionally does not collect:

• .env values;
• Caddyfile content;
• build output from heavy image builds;
• rsync push output;
• migration execution output unless separately run by the operator;
• stack startup output;
• service logs;
• database rows;
• API payloads;
• private keys;
• user content.

How To Use Evidence

Create evidence after each milestone:

1. Source tree synced to Hetzner.
2. Runtime env files created.
3. Host and env gates pass.
4. Query Engine image builds.
5. All images build.
6. Stack starts.
7. Health gate passes.
8. Reverse proxy/TLS starts.
9. Backup runs.
10. Restore rehearsal passes.

Each evidence file should be reviewed before the next milestone is treated as done.

Current Verification Evidence

• npm run hetzner:evidence:check passes locally.
• npm run hetzner:evidence:plan prints the server-side collection workflow.
• The collector now has 18 read-only/status commands configured, including source sync, Python gate, database gate, completion gate, and platform E2E evidence status.
• Confirmed milestone runs now write redacted hetzner-runbook-<milestone> evidence files.
• No local evidence file was generated in this slice because the real target is the Hetzner host.

Known Blockers

• No Hetzner evidence file exists yet.
• Server-side evidence still needs real Linux/amd64 host output.
• Health output will remain partially failing until the full stack is built and running.