# 031 - svc-net-tor: Full Tor Onion Node ## User Story As a self-hoster running Infinito.Nexus behind NAT/CGNAT/dynamic IP (Raspberry Pi, edge node, home server), I want a `svc-net-tor` role that makes the node's primary domain a Tor v3 `.onion` address so that the entire stack (web, SSO, LDAP, CA) is reachable over Tor without public IP, port forwarding, VPS, or DynDNS — while individual apps can still attach extra public domains with their own TLS. ## Model **`DOMAIN_PRIMARY` = the node's `.onion` address.** Everything derives from `DOMAIN_PRIMARY`, so the whole node becomes Tor-native with **zero domain-transform code**: - Vhosts: `next.cloud.{{ DOMAIN_PRIMARY }}` → `next.cloud..onion` - OIDC issuer, redirect URIs, web-origins: onion — one realm, one issuer, consistent - `LDAP_DN_BASE`, user/admin emails, oauth2-proxy `email_domains`, CA identity: all onion, mutually consistent (users `@.onion` pass the oauth2-proxy email filter) The `.onion` is a **deploy-time input**: a CLI helper mints the v3 key offline during inventory build, vaults it, and sets `DOMAIN_PRIMARY`. It must NOT be a runtime fact — `DOMAIN_PRIMARY` is consumed at build time (LDAP base, CA, applications cache). The role restores the minted key into the hidden-service volume before Tor starts. **All-or-nothing:** a partial shift (web onion, identity clearnet) breaks oauth2-proxy email matching ([oauth2-proxy-keycloak.cfg.j2:32](../../roles/web-app-keycloak/templates/sso_proxy/oauth2-proxy-keycloak.cfg.j2) sets `email_domains = "{{ DOMAIN_PRIMARY }}"`). Fresh onion nodes only; migrating an existing clearnet node is out of scope (base DN + all identities would shift). ## Codebase facts (verified; build on these) | Fact | Where | |---|---| | OpenResty runs `network_mode: host`, binds 80/443, `proxy_pass http://127.0.0.1:` | [`roles/svc-prx-openresty/templates/compose.yml.j2`](../../roles/svc-prx-openresty/templates/compose.yml.j2), [`roles/sys-svc-proxy/templates/location/html.conf.j2`](../../roles/sys-svc-proxy/templates/location/html.conf.j2) | | TLS-disabled vhosts serve plain HTTP on `:80`; the http→https 301 exists only in the TLS layer | [`roles/sys-front-tls/templates/http.conf.j2`](../../roles/sys-front-tls/templates/http.conf.j2) | | TLS is resolved per `(domain\|application_id)` with `TLS_MODE`/`TLS_ENABLED` defaults, flavors in `AVAILABLE_FLAVORS` | [`plugins/lookup/tls.py`](../../plugins/lookup/tls.py), `utils/tls_common` | | `DOMAIN_PRIMARY` defined once; `LDAP_DN_BASE`, CA identity, contact email derive from it | [`group_vars/all/00_general.yml:50`](../../group_vars/all/00_general.yml), [`group_vars/all/12_ldap.yml`](../../group_vars/all/12_ldap.yml), [`group_vars/all/02_tls.yml`](../../group_vars/all/02_tls.yml), [`group_vars/all/14_about.yml`](../../group_vars/all/14_about.yml) | | Keycloak redirect URIs/web-origins are computed from `lookup('domains')` at deploy | [`plugins/lookup/redirect_uris.py`](../../plugins/lookup/redirect_uris.py), [`roles/web-app-keycloak/vars/main.yml`](../../roles/web-app-keycloak/vars/main.yml) | | Apps register one vhost per domain via `include_role: sys-stk-front-proxy` with `domain` + `http_port`; multi-domain is established practice | [`roles/sys-stk-front-proxy/tasks/main.yml`](../../roles/sys-stk-front-proxy/tasks/main.yml), [`roles/web-app-matrix/tasks/flavor/compose/03_webserver.yml`](../../roles/web-app-matrix/) | | Extra app domains live in `applications..server.domains.{canonical,aliases}` (str/list/dict) | [`utils/domains/application_domain_index.py`](../../utils/domains/application_domain_index.py), [`plugins/filter/canonical_domains_map.py`](../../plugins/filter/canonical_domains_map.py) | | Inventory build/secrets tooling (mint-helper integration point) | [`cli/administration/inventory/credentials/`](../../cli/administration/inventory/) | | Container-service role archetype (compose sidecar, `meta/services.yml` entity, `run_once_` guard) | [`roles/svc-ai-ollama/`](../../roles/svc-ai-ollama/) | | Role file/entry policy enforced by lint (mandatory files, galaxy_info canon, README sections + exact Credits block, lifecycle values, service bond) | `utils/roles/mapping.py`, `tests/lint/ansible/roles/`, `tests/integration/roles/meta/test_mapping.py` | | No prior server-side Tor art in the repo (`desk-torbrowser` is desktop client only) | repo-wide grep | ## Tor facts (verified upstream) - **Subdomains:** `sub..onion` routes to the **same** hidden service; the client sends the full `Host:` header → vhost routing works exactly like clearnet. ([address-spec](https://spec.torproject.org/address-spec), [RFC 7686 §2](https://www.rfc-editor.org/rfc/rfc7686.html)) - **v3 address algorithm** (needed for offline mint): `address = base32(PUBKEY ‖ CHECKSUM ‖ VERSION) + ".onion"`, `CHECKSUM = SHA3-256(".onion checksum" ‖ PUBKEY ‖ VERSION)[:2]`, `VERSION = 0x03`, PUBKEY = 32-byte ed25519 master pubkey. - **Key-file formats** in `HiddenServiceDir`: `hostname` (address + `\n`), `hs_ed25519_public_key` (`== ed25519v1-public: type0 ==` header + 32-byte key), `hs_ed25519_secret_key` (`== ed25519v1-secret: type0 ==` header + **64-byte expanded** secret key = SHA-512(seed) clamped). - **`http://*.onion` is a Secure Context** → `Secure` cookies + secure-context APIs work over plain-HTTP onion; hence forward `X-Forwarded-Proto: https`. ([W3C Secure Contexts](https://www.w3.org/TR/secure-contexts/)) - **`.onion` is in the Public Suffix List** → `user@x.onion` passes most validators. - **No Let's Encrypt for `.onion`** (ACME needs public DNS). Onion transport is already encrypted + server-authenticated (address = pubkey). ## Decisions (operator-confirmed; do not re-litigate) | # | Decision | |---|---| | 1 | Role `svc-net-tor`, container-service archetype ([`svc-ai-ollama`](../../roles/svc-ai-ollama/) pattern), `application_id: svc-net-tor`. Category `svc-net-*` per [`meta/categories.yml`](../../meta/categories.yml). | | 2 | **Full onion node**: `DOMAIN_PRIMARY` = node `.onion`; entire stack shifts together. No web/identity split, no domain-transform lookup. | | 3 | **One node key + per-app subdomains** (`..onion`), Host-header-routed. No per-app onions in MVP. | | 4 | **Tor = compose sidecar**, `network_mode: host`, keys on a named volume, restored from the vaulted mint **before** Tor starts. Backup via `svc-bkp-container-2-local`. | | 5 | **HTTP on the onion side** (TLS forced off for `.onion`), vhosts on `:80`, upstream gets `X-Forwarded-Proto: https`. | | 6 | **Offline Python mint** at inventory build: ed25519 → address → exact Tor key-file formats → vault → `DOMAIN_PRIMARY`. Idempotent (existing key is reused, never regenerated). No Docker/Tor needed at build time. | | 7 | SSO needs no special handling: Keycloak domain is onion ⇒ one onion issuer/realm; onion emails match `email_domains`. | | 8 | **Tor image built via `files/Dockerfile`** from a pinned Debian `tor` package (`deb.debian.org`; no sandbox-allowlist change). `deb.torproject.org` = documented alternative requiring an allowlist entry. | | 9 | **Per-domain TLS via app-level override**: extra public domains under `applications..server.domains` deploy as additional HTTPS vhosts; flavor from `applications..tls.mode` (default `TLS_MODE`). `.onion` → always HTTP. No per-domain TLS map. | | 10 | **`TOR_EGRESS_ENABLED: false` default.** Flag + plumbing in MVP (SOCKS `127.0.0.1:9050` reachable for containers when enabled, documented); transparent egress torification out of scope. Inbound is always onion. | | 11 | **CI runs a real Tor E2E** (`torsocks curl` against the minted onion) with retry logic; unit tests remain the merge-blocking base. | ## Target Schema ### Role layout ``` roles/svc-net-tor/ ├── README.md # lint-conform (sections, exact Credits block, no heading emojis) ├── files/Dockerfile # Debian base + pinned `tor` ├── meta/ │ ├── main.yml # canonical galaxy_info (author/license/company/platforms exact canon) │ ├── services.yml # `tor` entity (below) │ └── volumes.yml # tor_data -> HiddenServiceDir ├── tasks/ │ ├── main.yml # run_once_svc_net_tor guard │ ├── 01_restore_key.yml # vaulted hostname + hs_ed25519_* -> volume, strict perms, never overwrite │ └── 02_core.yml # network routine + compose up + render torrc (handler-driven reload) ├── templates/ │ ├── compose.yml.j2 # sidecar: network_mode host, tor_data volume, base/container includes │ └── torrc.j2 └── vars/main.yml # application_id + lookup('config', ...) resolved vars ``` No onion vhost template, no domain lookup changes: onion domains fall out of `DOMAIN_PRIMARY`; the standard vhost serves them on `:80` because TLS resolves disabled. ### `meta/services.yml` ```yaml tor: bond: 1 enabled: "{{ 'svc-net-tor' in group_names }}" shared: "{{ 'svc-net-tor' in group_names }}" build: true image: svc-net-tor version: "" name: tor backup: no_stop_required: false # stop for consistent key-volume copy lifecycle: alpha # no public ports; SocksPort/ControlPort bind 127.0.0.1 only ``` ### `torrc.j2` ``` SocksPort 127.0.0.1:9050 ControlPort 127.0.0.1:9051 DataDirectory /var/lib/tor HiddenServiceDir /var/lib/tor/infinito/node/ HiddenServiceVersion 3 HiddenServicePort 80 127.0.0.1:80 ``` One `HiddenServicePort` covers all apps: every `..onion` request arrives on `127.0.0.1:80` where OpenResty routes by `Host`. ### CLI mint helper Location: under [`cli/administration/inventory/`](../../cli/administration/inventory/), integrated with `credentials/{emit,vault}.py`; runs automatically during inventory build when `svc-net-tor` is in the target groups. 1. If a vaulted key exists → reuse (idempotent, address stability is the contract). 2. Else: generate ed25519 seed → compute expanded secret key (SHA-512(seed), clamp) → derive address per the algorithm above → write the three Tor key files byte-exact. 3. Vault `hostname` + both key files; emit `DOMAIN_PRIMARY:
.onion` into the generated inventory. 4. Unit tests: known seed → known address test vector; idempotent reuse; file-format byte-exactness (headers + lengths). `tasks/01_restore_key.yml` writes the vaulted files into the volume-backed `HiddenServiceDir` (owner = in-container tor UID, `0700` dir, `0600` secret) before the Tor container starts. Existing on-disk keys are never overwritten. ### Per-domain TLS (extend `tls.py` / `utils/tls_common`) - Domain ends `.onion` → `enabled=false`, `protocols.web=http`, `ports.web=80`, no ACME. Overrides `TLS_MODE` unconditionally. - Non-onion domain → flavor from `applications..tls.mode`, else `TLS_MODE`. - Foreign domains under `applications..server.domains` register as separate HTTPS vhosts (existing per-domain `sys-stk-front-proxy` calls), same upstream as the onion vhost. ### Config ```yaml DOMAIN_PRIMARY: ".onion" # written by the mint helper TOR_EGRESS_ENABLED: false # Decision #10 applications: : server: domains: canonical: [ "app.example.org" ] # optional extra public domain tls: mode: letsencrypt # flavor for that app's non-onion domains ``` ## Acceptance Criteria ### Scaffolding & lint - [ ] `roles/svc-net-tor/` matches the layout; `application_id: svc-net-tor`; `run_once_svc_net_tor` guard; all meta lint green (mapping, galaxy canon, layout, bond, README). - [ ] `files/Dockerfile` pins `tor` (no `:latest`); image builds in the stack. ### Mint helper & key persistence - [ ] Helper mints offline, vaults `hostname` + `hs_ed25519_{public,secret}_key`, emits `DOMAIN_PRIMARY`; re-running reuses the key (address unchanged). - [ ] Unit tests: known-vector address determinism, idempotent reuse, byte-exact file formats. - [ ] `01_restore_key.yml` restores keys with strict perms before Tor starts; never overwrites existing keys; Tor serves exactly the minted address. - [ ] Address survives redeploy, update, full-cycle reinstall; restoring the vault on a fresh box reproduces the same address; key volume covered by `svc-bkp-container-2-local`. ### Serving - [ ] Tor sidecar bootstraps (`network_mode: host`); 9050/9051 bind `127.0.0.1` only; zero published ports. - [ ] `lookup('tls', '.onion')` → http/disabled/80, no ACME attempted; app vhosts serve onion `server_name`s on `:80`; every app reachable at `..onion` through `HiddenServicePort 80 → 127.0.0.1:80`. - [ ] Onion requests reach upstreams with `X-Forwarded-Proto: https`; no `.onion` request is 301'd to https; alias/www canonicalization stays onion-internal; CSP origins (CDN, Matomo, Keycloak, logout) resolve onion — no clearnet leak. ### SSO / identity consistency - [ ] OIDC login completes end-to-end over `.onion`: issuer, redirect URIs, web-origins all onion; no redirect-URI/issuer mismatch. - [ ] oauth2-proxy `email_domains` matches provisioned `@.onion` users; `LDAP_DN_BASE` + CA identity resolve consistently. ### Per-domain TLS - [ ] Extra domain under `applications..server.domains` + `tls.mode` deploys as a separate HTTPS vhost (correct flavor) alongside the onion vhost, same upstream. - [ ] `tls.py` unit tests: onion→forced off; foreign→app-level flavor; unset→`TLS_MODE`. ### Egress - [ ] `TOR_EGRESS_ENABLED` defaults `false` (outbound untouched); `true` exposes SOCKS to containers, documented. ### CI / E2E - [ ] CI job: Tor bootstraps, `torsocks curl -I http://..onion` returns the app response; retried against transient Tor-network failures; unit tests independent of Tor-network availability. ### Quality - [ ] Whole flow idempotent (second run: no changes); reloads handler-driven; clear failure if Tor cannot bootstrap. - [ ] `make test` green tree-wide. - [ ] README documents: model, mint helper usage, key backup/restore, limitations — no LE for `.onion`; outbound public mail undeliverable from `@onion` senders; fresh-node only (no clearnet migration); extra public domains = per-app opt-in. ## Onion service extensions (phase 2, layered on a green onion core) These build on the working onion node and are implemented after the web core is verified. ### Mailu listening on the onion (operator: "if possible") - The node hidden service exposes Mailu's mail ports in addition to `80`, via extra `HiddenServicePort` entries: `HiddenServicePort 25`, `587`, `465`, `143`, `993` → Mailu's local front binds on `127.0.0.1`. Mailu is then reachable at `.onion` for SMTP/submission/IMAP. - Mechanism: `torrc.j2` renders extra ports from `TOR_ONION_EXTRA_PORTS` (list of `{onion_port, target}`), populated when `web-app-mailu` is deployed. Mailu's front is bound to `127.0.0.1` in the onion node. - [ ] With Mailu deployed on an onion node, `.onion:993` / `:587` accept connections over Tor; the extra `HiddenServicePort` lines are rendered only when Mailu is present. ### Mail egress over Tor (operator: "must") - Outbound mail to `.onion` recipient domains is routed through Tor's SOCKS proxy (`127.0.0.1:9050`) instead of clearnet DNS/MX — enabling onion-to-onion / federated mail without public MX/rDNS. - Mechanism: a Postfix transport for `.onion` next-hops via a SOCKS-aware delivery path (e.g. a `torsocks`-wrapped transport or a SOCKS relay), gated on `svc-net-tor` being active. Public-Internet mail to non-onion domains stays on the normal path. - [ ] On an onion node, mail addressed to a `*.onion` domain is delivered via the Tor SOCKS proxy; delivery to non-onion domains is unaffected. ### Playwright over Tor (operator: "probably") - When the target under test is an `.onion` domain, the Playwright browser routes through Tor's SOCKS proxy so specs exercise the real onion path (aligns with Decision #11's real-Tor E2E). - Mechanism: the `test-e2e-playwright` runner sets the browser proxy to `socks5://127.0.0.1:9050` (with `socks5` remote DNS for `.onion` resolution) when the resolved domain ends in `.onion`. - [ ] A Playwright spec against a `..onion` surface passes through the Tor SOCKS proxy on an onion node; clearnet targets are unaffected. ## Out of scope Dual clearnet+onion with `Onion-Location` bridge and mirrored SSO realms; per-app dedicated onions / Tor Client Authorization; distributed backups over onion; transparent full-egress torification (only `.onion`-destination mail egress is in phase 2); migration of existing clearnet nodes. ## Validation ```bash # mint runs during inventory build (sets DOMAIN_PRIMARY), then: INFINITO_APPS="svc-net-tor,web-app-keycloak,web-app-nextcloud" make deploy-fresh-purged-apps INFINITO_FULL_CYCLE=true torsocks curl -I http://next.cloud..onion # app over Tor, no public ports # complete an SSO login over .onion; redeploy; confirm address unchanged ``` ## Prerequisites Read [AGENTS.md](../../AGENTS.md), then [Role Loop](../agents/action/iteration/README.md) and the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md). Verify key-file byte formats and vhost/TLS behavior **in the DiD container first**, then redeploy. ## Implementation order 1. Mint helper + unit tests (address vector, formats, idempotency). 2. Role scaffold: Dockerfile, compose, torrc, key restore, volume + backup wiring. 3. `tls.py`/`tls_common`: onion branch + app-level flavor override + unit tests. 4. E2E CI job (torsocks, retries). 5. README + cross-link the implementing PR here. ## Commit Policy - No `git commit` until every Acceptance Criterion is checked. - When green, operator runs `git-sign-push` outside the sandbox; the agent MUST NOT push. ## Context - — v3 address + key formats - — `.onion` special-use domain - — subdomain hidden-service reference - — `.onion` secure context - [`roles/svc-ai-ollama/`](../../roles/svc-ai-ollama/) — archetype - [`plugins/lookup/tls.py`](../../plugins/lookup/tls.py) — TLS resolution - [`cli/administration/inventory/`](../../cli/administration/inventory/) — mint integration point