Hetzner migration runbook

Ordered steps to move Oter off GCP onto a single Hetzner CPX11 x86 VPS with Cloudflare R2 (storage), Cloudflare Access (docs auth), and Cloudflare Pages (static WASM + public docs).

Historical note (kept as an ops reference): the actual migration ran on 2026-07-08 and is complete. GCP teardown was performed the same day. The attachments-to-R2 phase from the original plan was skipped — there was no attachment data to move, and R2 is the greenfield store. Phases 6 and 10 have been dropped from this doc; if you need the GCP-era commands, see git history at 0b26bf0 or earlier.

Motivation and alternatives-considered: gcp-migration-analysis.md.

Phase 0 — prerequisites

Accounts you need active:

• Hetzner Cloud (project created, payment method attached)
• Cloudflare (oterapp.com and negodex.co transferred or DNS pointed to CF nameservers)
• GitHub with repo admin on esteban505r/LifeCommanderKMP and the Negodex repo
• Access to the existing GCP project oter-490318 with roles/owner (needed for one-time data export and later teardown)

Local tools:

brew install hcloud gcloud cloudflared jq postgresql@16 awscli
# or apt: hetzner-cli / hcloud, google-cloud-sdk, cloudflared, jq, postgresql-client, awscli

Phase 1 — provision the VPS

# Generate a deploy keypair (private stays on your workstation; public goes
# to Hetzner + the VM's authorized_keys; private also goes to GitHub as SSH_KEY).
[ -f ~/.ssh/oter_ed25519 ] || ssh-keygen -t ed25519 -f ~/.ssh/oter_ed25519 -C "oter-deploy" -N ""
 
export HCLOUD_TOKEN=…             # Hetzner Cloud API token, project scope
hcloud ssh-key create --name oter-deploy --public-key-from-file ~/.ssh/oter_ed25519.pub
 
# Server type. Hetzner's ARM Ampere line (CAX11 — 4 GB / €3.79/mo) is EU-only
# (fsn1/nbg1/hel1). US locations (ash/hil) only ship x86:
#   cpx11  — 2 vCPU AMD dedicated / 2 GB / 40 GB / 20 TB / ~$5/mo  (tight on RAM
#            for the full stack; workable with swap)
#   cpx21  — 3 vCPU AMD / 4 GB / 80 GB / 20 TB / ~$9/mo            (recommended
#            headroom for Ktor + Postgres + Nest + docs + Caddy)
# Pick based on LATAM-latency preference:
#   Ashburn US East (ash/hil) → cpx11 or cpx21
#   Falkenstein / Helsinki (fsn1/hel1) → cax11 (cheapest, +30-60ms LATAM)
hcloud server create \
  --name oter-prod \
  --type cpx11 \
  --location ash \
  --image ubuntu-24.04 \
  --ssh-key oter-deploy
 
VM_IP=$(hcloud server ip oter-prod)
echo "VM public IP: $VM_IP"

Persist VM_IP and an SSH alias so every later step in this runbook works with a plain ssh oter-prod (and matches what CI will do). Skip either block if you already have equivalents:

# Save VM_IP so a new shell can pick it up
grep -q 'VM_IP=' ~/.oter-migration.env 2>/dev/null || cat >> ~/.oter-migration.env <<ENV
export VM_IP=${VM_IP}
ENV
# In any new shell before continuing:
#   source ~/.oter-migration.env
 
# SSH config alias — makes the deploy key + accept-new host key the default
if ! grep -q '^Host oter-prod' ~/.ssh/config 2>/dev/null; then
  cat >> ~/.ssh/config <<CFG
 
Host oter-prod
  HostName ${VM_IP}
  User root
  IdentityFile ~/.ssh/oter_ed25519
  IdentitiesOnly yes
  StrictHostKeyChecking accept-new
CFG
  chmod 0600 ~/.ssh/config
fi

Quick sanity check before running the heredoc (this catches the two common failure modes — wrong/missing key and unresolved $VM_IP):

: "${VM_IP:?VM_IP is empty — run 'source ~/.oter-migration.env' or re-export it}"
ssh -i ~/.ssh/oter_ed25519 -o IdentitiesOnly=yes -o StrictHostKeyChecking=accept-new \
    root@"$VM_IP" 'echo ok: $(hostname)'
# Expect: "ok: oter-prod". If you see "Permission denied (publickey)" the key
# wasn't uploaded to Hetzner — re-run `hcloud ssh-key create` and recreate the
# VM (Hetzner only injects keys at first boot).

SSH in and harden. Use the alias so the identity + accept-new are always applied — running this without -i ~/.ssh/oter_ed25519 is the #1 cause of “the command just doesn’t do anything” (SSH silently falls back to your default key, auth fails, heredoc is discarded):

`ssh oter-prod bash -s <<'REMOTE'`
set -euo pipefail
timedatectl set-timezone UTC
export DEBIAN_FRONTEND=noninteractive
apt-get update -qq
apt-get install -y ca-certificates curl gnupg ufw fail2ban unattended-upgrades
install -m 0755 -d /etc/apt/keyrings
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | gpg --dearmor -o /etc/apt/keyrings/docker.gpg
chmod a+r /etc/apt/keyrings/docker.gpg
echo "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/docker.gpg] https://download.docker.com/linux/ubuntu $(lsb_release -cs) stable" > /etc/apt/sources.list.d/docker.list
apt-get update -qq
apt-get install -y docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin
 
# Create the deploy user + directory (idempotent — safe to re-run)
id -u deploy >/dev/null 2>&1 || useradd -m -s /bin/bash deploy
usermod -aG docker deploy
install -d -o deploy -g deploy -m 0755 /opt/oter /opt/oter/secrets
 
# Copy the SSH key so CI can log in as `deploy`
install -d -o deploy -g deploy -m 0700 /home/deploy/.ssh
cp /root/.ssh/authorized_keys /home/deploy/.ssh/authorized_keys
chown deploy:deploy /home/deploy/.ssh/authorized_keys
chmod 0600 /home/deploy/.ssh/authorized_keys
 
# Firewall — only 22/80/443 from anywhere
ufw allow OpenSSH
ufw allow 80
ufw allow 443
ufw --force enable
 
# Unattended upgrades
dpkg-reconfigure -f noninteractive unattended-upgrades
 
echo "== harden done =="
REMOTE

If the heredoc still returns immediately with no output, run it interactively first to see the real error:

ssh oter-prod   # should land you at a root@oter-prod prompt; then paste blocks

Phase 2 — Cloudflare (R2 + Pages + Access)

R2 buckets (from Cloudflare dashboard → R2):

• oter-attachments-prod — main attachments bucket
• oter-db-backups — nightly pg dumps

Create an API token with Object Read & Write on both buckets. Record the Access Key ID + Secret. Grab your account ID from the R2 dashboard sidebar; the endpoint is https://<account_id>.r2.cloudflarestorage.com.

Cloudflare Pages (dashboard → Workers & Pages → Pages → Direct Upload):

• Project oter-web — will host the WASM app (custom domain: oterapp.com)
• Project oter-docs — will host the public docs (custom domain: oterapp.com/docs via a Worker route or a subdomain)

Cloudflare Access (dashboard → Zero Trust → Access → Applications):

• App: Oter internal docs, domain docs-internal.oterapp.com
• Policy: allow only esteban505r@gmail.com (+ any other allowlist)
• Under the app → Advanced → Enable origin request header with name Cf-Access-Origin-Token and a random string. Save that string — it becomes CF_ACCESS_ORIGIN_TOKEN in /opt/oter/.env.

API token for GitHub Actions (dashboard → My Profile → API Tokens):

• Template: Custom Token
• Permissions: Account | Cloudflare Pages | Edit, Account | Account Settings | Read
• Save as CLOUDFLARE_API_TOKEN in the repo.

Phase 3 — GitHub repo config

Add repo secrets (Settings → Secrets and variables → Actions):

Add repo variables:

Remove the now-unused GCP variables/secrets: GCP_WIF_PROVIDER, GCP_WIF_SERVICE_ACCOUNT, GCP_PROJECT_ID, GCP_REGION, GCP_AR_REPOSITORY, GCP_WEB_BUCKET, GCP_DOCS_INTERNAL_BUCKET, CLOUD_RUN_SERVICE, CLOUD_SQL_INSTANCE, GCP_RUNTIME_SERVICE_ACCOUNT, GCS_BUCKET_NAME, GCP_SM_*.

Phase 4 — first-deploy the stack (empty DB)

Add a second alias for the deploy user (once):

if ! grep -q '^Host oter-deploy' ~/.ssh/config 2>/dev/null; then
  cat >> ~/.ssh/config <<CFG
 
Host oter-deploy
  HostName ${VM_IP}
  User deploy
  IdentityFile ~/.ssh/oter_ed25519
  IdentitiesOnly yes
  StrictHostKeyChecking accept-new
CFG
fi

Copy the deploy files to the VM and fill in /opt/oter/.env:

scp -r deploy/* oter-deploy:/opt/oter/
scp deploy/server.env.example oter-deploy:/opt/oter/.env  # then edit on the VM
scp path/to/firebase-oter-8c4e4.json oter-deploy:/opt/oter/secrets/firebase.json
 
ssh oter-deploy bash -s <<'REMOTE'
set -euo pipefail
chmod 0600 /opt/oter/.env
chmod 0400 /opt/oter/secrets/firebase.json
cd /opt/oter
docker compose --env-file .env pull
docker compose --env-file .env up -d postgres caddy
docker compose --env-file .env ps
REMOTE

DNS point api.oterapp.com and docs-internal.oterapp.com at the VM IP (Cloudflare → DNS → add A records, proxy status DNS only for now so LE HTTP-01 can succeed). Wait for Caddy to obtain certs:

ssh oter-deploy docker compose --env-file /opt/oter/.env logs caddy | grep -i "certificate obtained"

Phase 4 gotchas we hit — read before starting

• postgres:18-alpine refuses first-boot when the volume is mounted at /var/lib/postgresql/data (the pg18 image now expects /var/lib/postgresql and a version-named subdir). Compose is pinned to postgres:17-alpine for this reason; do not bump without also changing the mount.
• postgres-init.sh runs inside the postgres container, so the DB env vars (OTER_DB_*, NEGODEX_DB_*) must be declared on the postgres service in docker-compose.yml, not just on oter-server. If they’re missing, the init script bails with OTER_DB_USER: unbound variable and no app DBs get created.
• OTER_API_HOST_ALT (secondary hostname for the same server, e.g. api.estebanruano.com) must be set — even if empty — because Caddy’s site block references it and a bare comma is a parse error. Compose defaults OTER_API_HOST_ALT to OTER_API_HOST if unset; Caddy dedupes.
• The first Publish Server Release run creates the GHCR package as private. Either mark it public in GitHub’s package UI after the first push, or docker login ghcr.io on the VM with a read:packages PAT. Otherwise the deploy step 500s with denied.
• logback.xml used to unconditionally reference an appender that was defined inside a conditional block; if the conditional was false, logback silently detached all appenders and every logger went dark. Fixed in commit 5ebdf7c. If you see zero app logs after startup, this class of bug is the first thing to suspect — enable LOGBACK_DEBUG=true in .env to get logback’s own init status on stdout.

Phase 5 — restore Postgres data (Oter only)

Negodex isn’t on this VM yet; postgres-init.sh created its empty DB on first boot, and Negodex will bring its own migrations when it moves here.

The dump format we actually have is a plain-SQL file produced by pg_dump 18 against Cloud SQL. That produces three classes of content that don’t restore cleanly into a fresh empty DB and need to be stripped:

1. \restrict / \unrestrict psql meta-commands (pg18 security markers — they refuse subsequent backslash commands, including \. COPY terminators).
2. CREATE ROLE / ALTER ROLE for GCP roles (cloudsqladmin, cloudsqlsuperuser, pg_*) that don’t exist and aren’t wanted here.
3. CREATE DATABASE / ALTER DATABASE oter_prod OWNER TO cloudsqlsuperuser / \connect — the dump was made with --create semantics; we already have the DB and a local owner.

Restoring as postgres also leaves every table owned by postgres — the app’s connection user (oter_app) then hits permission denied on every SELECT. So the last step of a clean restore is to reassign ownership of everything in public to oter_app.

5.1 — expose postgres on VM loopback for a workstation tunnel

Add to docker-compose.yml (already there in current repo — this is a note about why):

postgres:
  ports:
    - "127.0.0.1:5432:5432"    # workstation SSH tunnel only; NOT public

Restart postgres to pick it up:

ssh oter-deploy 'cd /opt/oter && docker compose --env-file .env up -d postgres'

Open the tunnel in a terminal you’ll keep open:

ssh -N -L 5432:localhost:5432 oter-deploy

5.2 — restore

We ship deploy/restore-oter.sh (see below for the source of that script). It stops the app so Flyway can’t reconnect mid-restore, drops+recreates oter_prod, strips the three problem classes, restores via psql, reassigns object ownership to oter_app, and restarts the app.

# On your workstation, with the tunnel from 5.1 open in another terminal:
export PGPASSWORD='<POSTGRES_SUPER_PASSWORD from /opt/oter/.env>'
bash deploy/restore-oter.sh ~/oter-backups/dump-oter_prod-<STAMP>.sql

Expected end of output:

== 6. Sanity check ==
 users
-------
   N
(1 row)
== 7. Start oter-server ==
== done ==

5.3 — verify

curl -i https://api.oterapp.com/api/v1/version

HTTP/2 200 with a JSON payload = server is up on restored data.

5.4 — close the postgres port when done

Once the restore is verified, either drop the ports: mapping from the postgres service and docker compose up -d postgres, or leave it (loopback-only means it’s only reachable over SSH tunnel anyway, so exposure is low). The runbook keeps it — DBeaver access for future admin work is worth the tradeoff.

5.5 — anti-patterns we hit and shouldn’t repeat

• DBeaver’s Restore tool shells out to pg_restore --format=c unconditionally. Plain-SQL dumps (.sql) will always fail with did not find magic string in file header. Use psql from the terminal instead — DBeaver stays useful for browsing.
• Restore + oter-server running simultaneously: Flyway reconnects, notices the schema-history table is empty during restore, and tries to re-migrate on top of the incoming schema. Always docker compose stop oter-server before restore.
• Restore with two -c statements in one psql call (DROP DATABASE ...; CREATE DATABASE ...;): psql wraps -c in a single implicit transaction and DROP DATABASE refuses to run inside one. Use two separate psql -c calls (or a .sql file).

Phase 7 — DNS cutover

Trigger the publish workflows once so GHCR has the current image tags:

• Actions → Publish Server Release → Run workflow
• Actions → Publish WASM Web Release → Run workflow
• Actions → Publish Docs Release → Run workflow (both)

DNS updates (Cloudflare → DNS):

Enable Cloudflare proxy (Proxied orange cloud) for api.oterapp.com after Caddy has a fresh cert.

Phase 8 — smoke test

curl -fsS https://api.oterapp.com/health              # 200 with JSON
curl -fsS https://api.oterapp.com/api/v1/version      # 200 with JSON

The other two hostnames from the original plan aren’t wired yet — oterapp.com (Cloudflare Pages oter-web) still needs a WASM release, and docs-internal.oterapp.com will 502 until oter-docs-internal is published and Cloudflare Access is configured. Both are follow-up work, not blockers for cutover.

App-level checks:

• Login on the Android/Desktop client — new access token issued.
• Create a task and a habit — round-trip through Postgres.
• Upload an attachment — round-trip through R2 (aws s3 ls s3://oter-attachments-prod --endpoint-url ...).
• Push notification — send a test FCM from Firebase Console → device receives it.

Enable pg-backup cron on the VM:

ssh oter-deploy bash -s <<'REMOTE'
crontab -l 2>/dev/null | grep -v '^# pg-backup' > /tmp/ct || true
cat >> /tmp/ct <<'CRON'
# pg-backup — nightly pg_dumpall to R2 at 03:00 UTC
0 3 * * * cd /opt/oter && docker compose --env-file .env run --rm pg-backup >> /var/log/oter-pg-backup.log 2>&1
CRON
crontab /tmp/ct
rm -f /tmp/ct
REMOTE

Phase 9 — Negodex coordination

The Negodex repo needs its own PR mirroring this one:

• Drop Cloud SQL socket factory from its Nest ORM config; switch to standard DATABASE_URL=postgres://negodex_api:…@<vm-ip>:5432/myecommercedb (or, better, add a negodex-api service to this docker-compose.yml so both apps share the internal network without exposing Postgres to the internet).
• Rewrite its deploy.yml to push to GHCR + SSH deploy against the same VM.
• Change any Cloud IAP-protected routes to use Cloudflare Access.
• Move any Negodex GCS buckets to R2.

Recommended: add negodex-api to this stack’s docker-compose.yml (already scaffolded as a commented block in deploy/Caddyfile). One VM, one Postgres, two apps.

Rollback (historical)

Rollback to Cloud Run + Cloud SQL was possible during the first 48h — DNS could be flipped back and the GCP resources would still answer. After the 2026-07-08 teardown that option is gone; the only rollback path now is another VPS + a fresh restore from the R2 pg-backup bucket. Keep at least 7 days of nightly R2 backups before making any structural change to the Hetzner stack.

Related

• gcp-migration-analysis.md — the decision doc
• gcp-billing.md — historical baseline the migration was measured against
• ../server/README.md — updated env vars + deploy pointer