🗺️ Platform Deployment Map

last verified 2026-06-29
PROD CPU req 60% · mem ~86% STAGING 16 GB · 914 GB disk
📡 Open Live Status → 🧭 Product Roadmap → Real-time service health & recent deploys at status.dloizides.com; the product roadmap (flowchart · gantt · kanban) lives here too. This page is the static topology companion.

What runs where live topology

graph TB
    subgraph internet["🌐 Public Internet — *.dloizides.com (real LE TLS)"]
        users["End users / testers"]
    end

    subgraph prod["🟢 PROD — Hetzner prod (4 vCPU / 8 GB · k3s) — CPU req 60% / mem ~86% · weekly API-restart cron"]
        ingress["Traefik ingress + LE certs
ALL public URLs terminate here"] subgraph prodapis["Product APIs (+ BFF) → shared saas-db"] kefiapi["kefi-api / bff-kefi"] tenant["tenant-api"] quest["questioner-api"] onlinemenu["onlinemenu-api"] content["content-api"] payment["payment-api"] notif["notification-api (MT consumer)"] end subgraph prodweb["Web / SPA / static"] kefiweb["kefi-web SPA · kefi-marketing"] kefiland["kefi-landings (vestigial/crashloop)"] erevna["erevna-web · katalogos-web"] statics["~35 static apps/games · HQ"] end kc["keycloak + keycloak-db (heap capped)"] subgraph prodstate["Stateful infra (hibernate-able → 0/0)"] saasdb[("saas-db (shared; *-db aliases)")] rmq["rabbitmq"] sw["seaweedfs (S3)"] maddy["maddy (mail)"] pouenidb[("poueni-postgres")] end amlmkt["aml-marketing (static landing — stays)"] promagent["prometheus AGENT (remote-write)"] pubjob["kefi publish Job: build-site→kaniko→rollout
CPU req 500m→200m (now schedules)"] gproxy["selector-less Service + EndpointSlice
grafana · analytics · aml-* · dynalux ·
theme-studio(+mock-api) · webmail · status → staging"] end subgraph staging["🟡 STAGING — staging (WireGuard-only · 8 vCPU / 16 GB · 914 GB disk)"] registry["Docker registry (staging) + weekly GC cron"] graf["Grafana + Prometheus(store) + Loki + Alertmanager"] umami["Umami + umami-db"] amlstg["AML: aml-screening+aml-postgres ·
aml-identity+identity-postgres · aml-ner"] dynaluxstg["dynalux-api + dynalux-db (demo)"] miscstg["theme-studio · mock-api · webmail"] statushub["Katastasi status hub + feeders"] swstg["seaweedfs (E2E report store)"] janitor["pod-janitor cron (clears evicted pods /6h)"] subgraph e2e["E2E runners (CronJobs) — run HERE, TARGET prod"] e2ekefi["kefi-lifecycle 03:00 (deadline 1800)"] e2epoueni["poueni-reset 03:40"] e2efull["full suite 04:00"] end end users -->|HTTPS| ingress ingress --> prodapis ingress --> prodweb ingress --> kc ingress -->|analytics · grafana · aml-* · dynalux · theme-studio · webmail · status| gproxy gproxy -.->|WireGuard ~100ms| umami gproxy -.->|WireGuard ~100ms| graf gproxy -.->|WireGuard ~100ms| amlstg gproxy -.->|WireGuard ~100ms| dynaluxstg gproxy -.->|WireGuard ~100ms| miscstg gproxy -.->|WireGuard ~100ms| statushub promagent -.->|remote-write WG| graf kefiapi -->|create Job| pubjob pubjob -->|push image| registry e2e -.->|test over public DNS| ingress e2e -->|reports| swstg prod -.->|pull images| registry classDef off fill:#3d1414,stroke:#f85149,color:#ffd7d5; classDef fix fill:#13331b,stroke:#3fb950,color:#aff5b4; class kefiland off; class pubjob fix; class amlstg,dynaluxstg,miscstg fix;
live (prod) runs on staging, public URL proxied from prod hibernate-able / scheduled sleep off by design

Deployment inventory status by host

🟢 PROD — Hetzner prod (public)

Hetzner prod · 4 vCPU / 8 GB · real LE TLS
  • tenant · kefi · questioner · onlinemenu · content · payment · notification — product APIs + BFFs → shared saas-db
  • keycloak (+ keycloak-db) — auth; all realms
  • kefi-web · erevna-web · katalogos-web — product SPAs
  • ~35 static apps / games · hq.dloizides.com
  • saas-db · rabbitmq · seaweedfs · maddy · poueni-postgres — stateful; scaled to 0/0 when hibernated → APIs go "Running but not Ready"
  • aml-marketing — static AML landing (stays on prod)
  • grafana · analytics · aml-screening/identity · dynalux-api · theme-studio(+mock-api) · webmail · status — all MIGRATED to staging; only the public URL + LE cert + ingress stay on prod, proxied over WG
  • kefi-landings — vestigial; SPA replaced it (empty nginx root → crashloop)
  • kefi publish build Jobs — kaniko on Publish click; CPU req 200m + empty-webroot guard
  • prometheus AGENT — scrapes prod, remote-writes to staging
  • api-weekly-restart cron — Sun 05:00 UTC, restarts the 7 fat APIs one-at-a-time (heap hygiene)

🟡 STAGING — staging (WireGuard-only)

WireGuard-only · 8 vCPU / 16 GB · 914 GB disk (LV extended from 100 GB 2026-06-29) · no public TLS
  • Docker registry (staging) — prod pulls images from here; weekly GC CronJob keeps blobs from filling the disk
  • Grafana · Prometheus (store/query) · Loki · Alertmanager — prod agent remote-writes here; grafana.dloizides.com proxied from prod
  • Umami + umami-db — migrated 2026-06-27; analytics.dloizides.com proxied
  • AML stack — migrated 2026-06-28: aml-screening+aml-postgres, aml-identity+identity-postgres, aml-ner; proxied from prod
  • dynalux-api + dynalux-db — demo, migrated 2026-06-28; dynalux-api.dloizides.com proxied
  • theme-studio · mock-api · webmail — migrated 2026-06-29; proxied from prod (webmail → maddy on prod)
  • Katastasi status hub + feeders status.dloizides.com (env health + tests + 13 task boards); collector on a PVC
  • seaweedfs — E2E canary report store
  • E2E CronJobs (target prod) — kefi-lifecycle 03:00 · poueni-reset 03:40 · full suite 04:00 · share one canary lock
  • pod-janitor cron — clears evicted/failed pods every 6 h (prevents ephemeral-storage pile-up)

Status & recent changes 2026-06-29

✅ Offloads complete — prod slimmed

Umami, AML, dynalux, grafana + theme-studio/mock-api/webmail all moved to staging via the proxy pattern (public URL + cert + ingress stay on prod). Plus the .NET memory limits + keycloak heap were tightened. Prod CPU-req 94%→60%.

✅ Staging disk — fixed at the root

Staging kept hitting "full" and evicting pods (took grafana + the hub down). Cause: the Ubuntu installer capped the root volume at 100 GB on a ~928 GB disk. Extended online → 914 GB, 5% used. Plus a weekly registry-GC + a 6-hourly pod-janitor CronJob as hygiene.

✅ Status hub live (this map's live twin)

status.dloizides.com — env health + unit/E2E test status + 13 task boards (one row per repo). Collector on a PVC so it survives restarts. kefi self-serve publish fixed + guarded.

⚠️ Prod still tight (~86% mem) — +RAM is the real fix

The 7 product APIs + keycloak are the load and can't leave prod (latency/auth). A weekly api-restart cron resets heap accumulation, but the structural fix is +RAM. Also: the hibernation gotcha — products "Running but not Ready" = a stateful workload at 0/0 (kubectl get statefulset, scale DB-first).