From 60859ca65f0640d7444e98a0e2f6a3bfb59ffd21 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:20:17 +0200 Subject: [PATCH 001/145] docs(requirements): add 032 OpenBao integration requirement --- docs/requirements/032-web-app-openbao.md | 348 +++++++++++++++++++++++ 1 file changed, 348 insertions(+) create mode 100644 docs/requirements/032-web-app-openbao.md diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md new file mode 100644 index 0000000000..5c09249a26 --- /dev/null +++ b/docs/requirements/032-web-app-openbao.md @@ -0,0 +1,348 @@ +# 032 - OpenBao Secrets Manager with Keycloak OIDC, RBAC Policies and Static-Seal Auto-Unseal + +## User Story + +As an Infinito.Nexus administrator, I want [OpenBao](https://github.com/openbao/openbao) deployed as a `web-app-openbao` role with Keycloak OIDC login, group-mapped policies, persistent storage, automatic unsealing, backup and monitoring, so that infrastructure secrets, application credentials, certificates and tokens can be stored and retrieved centrally without distributing static secrets across systems. + +Upstream story: [OpenProject #628](https://project.infinito.nexus/) — *Integrate OpenBao into Infinito.Nexus*. + +## Background + +[OpenBao](https://openbao.org/) is the Linux-Foundation-governed fork of HashiCorp Vault: an identity-based secrets and encryption manager with a KV store, a PKI engine, pluggable auth methods and a built-in web UI. + +### Upstream facts established by research + +| Topic | Finding | Source | +|---|---|---| +| Image | `openbao/openbao` on Docker Hub. Tags carry **no `v` prefix** (`2.6.2`, not `v2.6.2`), unlike the GitHub release names. | [Docker Hub tags](https://hub.docker.com/r/openbao/openbao) | +| Version | `2.6.2`, released 2026-08-18, is the newest non-prerelease. | [Releases](https://github.com/openbao/openbao/releases) | +| Seal types | `alicloudkms`, `awskms`, `azurekeyvault`, `gcpckms`, `kmip`, `ocikms`, `ovhcloud`, `pkcs11`, **`static`**, `tcloudpublic`, `transit`. From v2.7.0 many become external plugins; `static` and `transit` stay built in. | [seal stanza](https://openbao.org/docs/configuration/seal/) | +| `seal "static"` | Takes a caller-supplied 32-byte AES-256-GCM-96 key via `current_key`, accepting a literal value or an `env://` / `file://` reference, plus a `current_key_id`. Upstream caveat: recommended only where an existing source of trust already holds the key. | [static seal](https://openbao.org/docs/configuration/seal/static/) | +| Auto-seal semantics | With any auto-seal configured, `bao operator init` yields **recovery keys, not unseal keys**, and the node unseals itself on every restart by asking the seal to decrypt the root key. Recovery keys authorise `generate-root` / `rekey`; they cannot decrypt the root key on their own. | [seal concepts](https://openbao.org/docs/concepts/seal/) | +| Keycloak OIDC | Confidential client; three redirect URIs: `/v1/auth/oidc/callback` (API), `/ui/vault/auth/oidc/oidc/callback` (UI), `http://localhost:8250/oidc/callback` (CLI). Configured through `auth/oidc/config` + `auth/oidc/role/`. | [keycloak.mdx](https://github.com/openbao/openbao/blob/main/website/content/docs/auth/jwt/oidc-providers/keycloak.mdx) | +| Group mapping | The OIDC role's `groups_claim` (JSON-pointer capable) names the claim holding the user's groups; those values resolve against identity groups of `type=external` via group aliases on the auth mount, which carry the policies. | [JWT/OIDC auth](https://openbao.org/docs/auth/jwt/) | +| AppRole custom ids | `POST auth/approle/role//role-id` accepts a caller-supplied `role_id` (`logical.UpdateOperation`), and `POST auth/approle/role//custom-secret-id` accepts a caller-supplied `secret_id`. Both verified in [`path_role.go`](https://github.com/openbao/openbao/blob/main/internal/builtin/credential/approle/path_role.go) (lines 889 and 1243). | upstream source | +| Health | `GET /v1/sys/health` returns **503 when sealed** and 501 when uninitialised. The query params `sealedcode`, `uninitcode`, `standbyok` override those codes per request. | [/sys/health](https://openbao.org/api-docs/system/health/), [`api/sys_health.go`](https://github.com/openbao/openbao/blob/main/api/sys_health.go) | +| Metrics | `GET /v1/sys/metrics?format=prometheus`. Normally token-gated; a listener with `telemetry { unauthenticated_metrics_access = true }` exposes it without a token on that listener only. | [/sys/metrics](https://openbao.org/api-docs/system/metrics/) | + +### In-repo analogues + +- **Native OIDC flavor** (`services.sso.flavor: oidc`), not the oauth2-proxy gate: OpenBao speaks OIDC itself, so it follows [`web-app-semaphore`](../../roles/web-app-semaphore/) rather than [`web-app-openproject`](../../roles/web-app-openproject/). +- **RBAC** via [`meta/rbac.yml`](../contributing/design/iam/rbac.md) + the `rbac_group_path` lookup, so the Keycloak `groups` claim arrives as `/roles/web-app-openbao/` and OpenBao binds those paths to policies. +- **Native Prometheus metrics** via the `services.prometheus.native_metrics` + `templates/prometheus.yml.j2` contract already used by [`web-app-gitea`](../../roles/web-app-gitea/templates/prometheus.yml.j2), scraped on the internal container port over the role-local docker network. +- **In-container configuration** follows [`svc-db-elasticsearch/tasks/01_init.yml`](../../roles/svc-db-elasticsearch/tasks/01_init.yml): a shell exec into the container where the credential arrives through a container env var, never through Ansible's process arguments, guarded by `no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"`. +- **No external database.** OpenBao's integrated raft storage owns all state on one persistent volume. + +## Confirmed Decisions + +These decisions were confirmed by the operator before implementation starts and are NOT subject to re-litigation during implementation. + +| # | Decision | Rationale | +|---|---|---| +| 1 | **Seal is `seal "static"`** with a 32-byte key generated by the repo credential generator into the ansible-vault-encrypted inventory and injected as `env://OPENBAO_SEAL_KEY`. | OpenBao auto-unseals on every restart, so deploy and redeploy stay idempotent and no OpenBao-generated key material has to be written back into the inventory. The inventory is the "existing source of trust" upstream's caveat asks for. | +| 2 | **Ansible authenticates via AppRole with caller-set ids.** First init uses the root token in memory only, enables AppRole, writes the inventory-generated `role_id` and `secret_id` through `role-id` / `custom-secret-id`, then revokes the root token. Later deploys log in with those inventory credentials. | Zero write-back and no permanently stored root token, satisfying "minimize the requirement for permanently stored bootstrap credentials". | +| 3 | **Storage backend is `raft`** (integrated storage), single node, on one persistent volume. | Upstream-recommended, keeps the door open for the HA follow-up story, and needs no external database. | +| 4 | **PKI engine ships behind a flag, default off.** When on, the play mounts `pki`, generates an internal root CA and one issuing role. | Proves the architecture supports the PKI story without mounting an unused CA that would still need backup, rotation and monitoring from day one. | +| 5 | **LDAP auth is included, behind the `ldap` service flag**, mapping the same LDAP role groups to the same three policies; OIDC stays the default. | Matches the beta-lifecycle IAM rule and leaves a login path when Keycloak is unavailable — which matters more for a secrets store than for a normal app. | +| 6 | **RBAC roles are `administrator` / `operator` / `reader`.** | The OIDC group path `/roles/web-app-openbao/` already namespaces by application, so an `openbao-` prefix would be redundant, and the repo's other `meta/rbac.yml` files use bare role names. | +| 7 | **Lifecycle on merge is `alpha`.** | Honest for a brand-new role using a non-default seal. The role does wire OIDC and LDAP by default, so promotion to `beta` is a follow-up decision, not a rewrite. | +| 8 | **Metrics via `unauthenticated_metrics_access = true`** on the internal listener, scraped on the internal container port over the role-local docker network only — never through the proxy, never public. | The metrics endpoint needs a server-generated token that cannot be pre-seeded into a static scrape config from the inventory. Same pattern as `web-app-gitea`. Gives the `openbao_core_unsealed` gauge. | +| 9 | **Listener runs `tls_disable = true`; TLS terminates at `sys-svc-proxy`.** | Public exposure stays TLS-only as the story requires, nothing is skip-verified, and the plaintext hop never leaves the per-role docker network. Consistent with every other `web-app-*` role. | +| 10 | **CLI OIDC login (`bao login -method=oidc`, `http://localhost:8250/oidc/callback`) is out of scope.** | The `redirect_uris` lookup generates `https:///*` per SSO consumer, which covers the UI and API callbacks but not a localhost one. Machine access goes through AppRole (Decision #2) instead. Documented as a limitation in the README. | +| 11 | **Both deployment modes are supported** (`modes.compose.enabled: true`, `modes.swarm.enabled: true`), with swarm running a single node pinned to the manager. | Every stateful role in the repo keeps both modes on; only host-level and infrastructure roles disable swarm. See [deployment-modes.md](../contributing/design/deployment-modes.md). | +| 12 | **The raft volume is `nfs: false` and the service carries `placement: manager`.** | Raft's BoltDB store is mmap + `fsync` based, so it hits exactly the locking and `fsync` semantics that keep the database engines off NFS. The deployment-modes volume model requires a placement constraint for any non-NFS volume under swarm, so the data cannot be lost on a reschedule. Same shape as [`svc-db-postgres`](../../roles/svc-db-postgres/) and the other engines. | +| 13 | **`service_update_order = 'stop-first'`** is set in the role's compose template. | Docker defaults to stop-first, but a rolling update that ever started a second task against the same raft directory would corrupt the store, so the role states it explicitly rather than inheriting it. Precedent: [`svc-prx-openresty`](../../roles/svc-prx-openresty/templates/compose.yml.j2). | + +## Target Schema + +### Role layout + +``` +roles/web-app-openbao/ +├── README.md +├── files/playwright/{playwright.spec,test-baseline,test-guest-persona,test-biber-persona,test-administrator-persona,test-oidc-login,test-ldap-login,test-rbac-denial,test-seal-status}.js +├── meta/{main,info,services,domains,csp,networks,rbac,secrets,users,variants,volumes}.yml +├── tasks/{main,00_core,01_init,02_auth,03_policies,04_auth_methods,05_pki}.yml +├── tasks/utils/group_alias.yml +├── templates/{compose.yml.j2,env.j2,openbao.hcl.j2,prometheus.yml.j2,playwright.env.j2} +├── templates/policies/{ansible-admin,administrator,operator,reader}.hcl.j2 +└── vars/main.yml +``` + +The policy `.hcl` templates are rendered to the host config directory and mounted at `/openbao/policies/`, deliberately **outside** `/openbao/config/`: the server parses every file in its config directory as server configuration, so a policy file there breaks startup. + +### `meta/services.yml` excerpt + +```yaml +sso: + bond: 1 + enabled: "{{ 'web-app-keycloak' in group_names }}" + shared: "{{ 'web-app-keycloak' in group_names }}" + flavor: oidc +ldap: + bond: 1 + enabled: "{{ 'svc-db-openldap' in group_names }}" + shared: "{{ 'svc-db-openldap' in group_names }}" +prometheus: + bond: 1 + enabled: "{{ 'web-app-prometheus' in group_names }}" + shared: "{{ 'web-app-prometheus' in group_names }}" + native_metrics: + enabled: true + port: 8200 +openbao: + # nocheck: default-placement-manager raft's BoltDB store is mmap + fsync based and cannot live on NFS, so the node is pinned in swarm + placement: manager + # nocheck: single-replica raft single-node; a second replica would need a real HA join flow + replicas: 1 + healthcheck: + flavor: wget + path: "v1/sys/health?uninitcode=200&sealedcode=200&standbyok=true" + modes: + compose: + enabled: true + swarm: + enabled: true + pki: + enabled: false + cpus: 0.5 + mem_reservation: 128m + mem_limit: 512m + pids_limit: 512 + image: openbao/openbao + version: "2.6.2" + name: openbao + ports: + internal: { http: 8200 } + local: { http: } + backup: + no_stop_required: false + run_after: + - svc-db-openldap + - web-app-keycloak + lifecycle: alpha +``` + +`pki` is inlined under the primary entity, not declared as a service key: it is a role-local feature toggle with no provider role, so a `pki:` service key would fail service resolution. + +### `meta/volumes.yml` + +```yaml +data: + type: volume + # nocheck: nfs-false-data-loss Reason: raft/BoltDB uses mmap + fsync and corrupts on NFS; single-replica and pinned via placement: manager, so no reschedule can strand the volume. + nfs: false + mounts: + - service: openbao + target: /openbao/data +``` + +### `meta/rbac.yml` + +```yaml +roles: + administrator: + description: OpenBao administrator (policy, auth-method and mount management) + operator: + description: OpenBao operator (read/write on the application secret paths) + reader: + description: OpenBao reader (read-only on the application secret paths) +``` + +### `meta/secrets.yml` + +```yaml +credentials: + seal_key: + description: 32-byte key for the OpenBao static seal, injected as env://OPENBAO_SEAL_KEY. Every backup of the storage volume is undecryptable without it. + algorithm: random_hex_32 + approle_role_id: + description: RoleID of the Ansible AppRole, written to auth/approle/role/ansible/role-id at bootstrap. + algorithm: alphanumeric + approle_secret_id: + description: SecretID of the Ansible AppRole, written to auth/approle/role/ansible/custom-secret-id at bootstrap. + algorithm: alphanumeric +``` + +`random_hex_32` yields 64 hex characters = exactly the 32 bytes the static seal requires, with no prefix to strip at render time (unlike `base64_prefixed_32`, which prepends `base64:`). + +### `templates/openbao.hcl.j2` shape + +```hcl +ui = true + +storage "raft" { + path = "/openbao/data" + node_id = "{{ ... }}" +} + +listener "tcp" { + address = "0.0.0.0:8200" + tls_disable = true + telemetry { unauthenticated_metrics_access = true } +} + +seal "static" { + current_key_id = "{{ ... }}" + current_key = "env://OPENBAO_SEAL_KEY" +} + +api_addr = "https://{{ lookup('domain', application_id) }}" +cluster_addr = "http://127.0.0.1:8201" +``` + +`cluster_addr` MUST NOT be built from the container name. Compose sets `container_name:` explicitly, but swarm names every task `_..`, so a container-name-derived address renders differently per mode. With `replicas: 1` the node only ever advertises to itself, so a loopback cluster address is mode-independent; the HA follow-up story revisits it. + +## Acceptance Criteria + +### Role layout & image + +- [x] `roles/web-app-openbao/` exists per the [Target Schema](#role-layout), and every `meta/*.yml` passes the role-meta lint. +- [x] The image is pinned to `openbao/openbao:2.6.2` (no `:latest`, no `v` prefix). +- [x] `ports.local.http` and `networks.local.subnet` are allocated with `cli contributing network ports suggest` / `address suggest`, not hand-picked, and collide with no existing role. Allocated `8081` (gap-first) and `192.168.221.0/24` (a new umbrella; every established `/24` was taken). + +### Routing & TLS (Decision #9) + +- [x] `openbao.{{ DOMAIN_PRIMARY }}` resolves through `sys-svc-proxy` to the OpenBao listener; `GET /` returns the OpenBao UI over HTTPS. (`test-baseline.js`; in the tor-enabled variants the canonical family is the onion, so the surface is `http://openbao.`.) +- [x] The listener runs with `tls_disable = true` and is reachable only on the role-local docker network and the localhost-bound `ports.local.http`; it is not bound on `0.0.0.0` of the host. +- [x] No TLS verification is disabled anywhere in the role (`insecure`, `skip_verify`, `-k` and equivalents are absent; the LDAP auth method is configured with `insecure_tls=false`). +- [x] The role emits a `Content-Security-Policy` header on the canonical surface, and the OpenBao UI renders without CSP violations under it. (`test-guest-persona.js` asserts the CSP injections; the UI renders and the in-page `fetch` to `/v1/...` is not blocked.) + +### Deployment modes (Decisions #11, #12, #13) + +- [ ] `modes.compose.enabled` and `modes.swarm.enabled` are both `true`, and the role deploys green in **both** modes. +- [x] The service carries `placement: manager` with the raft/NFS reason in its `# nocheck: default-placement-manager` comment, so the swarm task cannot be rescheduled away from its volume. +- [x] The raft volume is declared `nfs: false` with a `# nocheck: nfs-false-data-loss` reason; no OpenBao state is placed on an NFS-backed volume in any mode. +- [x] The role's compose template sets `service_update_order = 'stop-first'`, so a swarm rolling update never runs two tasks against the same raft directory. +- [x] `cluster_addr` renders identically in both modes and is not derived from the container name (see the [listener/seal schema note](#templatesopenbaohclj2-shape)). Confirmed in the rendered `/opt/compose/openbao/config/openbao.hcl`: `cluster_addr = "http://127.0.0.1:8201"`. +- [x] Every in-container configuration task reaches the container through [`resolve_host_cid.yml`](../../roles/sys-svc-compose/tasks/utils/swarm/resolve_host_cid.yml) — `container exec ` under `ansible.builtin.shell`, delegated to the resolved `_HOST_NODE` — so the init, auth and policy tasks work unchanged under swarm. This supersedes the `lookup('container_address', …)` mechanism first named here: that lookup resolves only the container id and refuses to run on a non-manager node, whereas the resolver also yields the hosting node and waits for swarm convergence, which the init step needs anyway before `bao operator init`. +- [x] The role uses no node-local `force_bridge` network, so the swarm Prometheus can reach the native-metrics target (the `native_metrics_apps` lookup gates `force_bridge` apps out of the scrape). Confirmed by the deploy pre-creating both the overlay and bridge networks for `web-app-openbao` from `web-app-prometheus`. +- [ ] `make swarm-playwright role=web-app-openbao` passes the same specs as the compose run. + +### Storage & persistence (Decision #3) + +- [x] Raft integrated storage is configured with its data directory on the named volume declared in `meta/volumes.yml`. Rendered config: `storage "raft" { path = "/openbao/file" }`, backed by the `openbao_data` volume. +- [ ] `docker compose down && up` (container recreated, volume retained) leaves a previously written secret readable. +- [ ] Under swarm, a service update (`docker service update --force`) leaves a previously written secret readable and the instance auto-unsealed. +- [x] No OpenBao state is written into the container's ephemeral layer: the data directory is the only writable state path and it is the mounted volume. + +### Seal, init & bootstrap (Decisions #1, #2) + +- [x] The `seal "static"` stanza reads its key via `env://OPENBAO_SEAL_KEY`; the key is a `meta/secrets.yml` credential and never appears literally in a template, a task argument or a committed file. +- [x] A restarted container comes back **unsealed with no operator action** (`/v1/sys/seal-status` reports `sealed: false`). Asserted by `test-seal-status.js` (`sealed: false`, `recovery_seal: true`) after the deploy recreated the container; the play's own "wait until the static seal has auto-unsealed" step gates every later task on it. +- [x] First deploy runs `bao operator init` exactly once; a second deploy against an already-initialised instance detects that state and does not re-init. Confirmed across three consecutive deploys. +- [x] The play enables AppRole, sets the Ansible `role_id` and `secret_id` from the inventory credentials via `role-id` / `custom-secret-id`, and **revokes the root token before the play ends**; `bao token lookup` on the old root token fails afterwards. +- [ ] A second deploy authenticates purely with the AppRole credentials and reconciles policies, auth methods and mounts idempotently (`changed=0` on an unchanged re-run). *AppRole-only re-authentication is verified (deploys 2 and 3 both reconciled after the root token was revoked); the `changed=0` half is NOT met — the reconciliation tasks declare `changed_when: true`, so an unchanged re-run still reports changed. Needs `changed_when` derived from real state before this can be checked off.* +- [x] Neither the root token, the recovery keys, the seal key nor the AppRole `secret_id` appear in the output of a normal `-v` deploy run: every task touching them carries `no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"`. +- [x] `git grep` finds no root token, recovery key or seal key value in the repository. + +### OIDC / Keycloak + +- [x] The Keycloak client is provisioned automatically through the existing client-import path; no manual Keycloak setup is required. +- [x] The generated redirect URIs cover both `/v1/auth/oidc/callback` and `/ui/vault/auth/oidc/oidc/callback` (the `redirect_uris` wildcard), and the OpenBao OIDC role's `allowed_redirect_uris` lists them explicitly. Proven by probing `auth/oidc/oidc/auth_url`: the UI callback returns a signed Keycloak URL while a non-listed URI returns `auth_url n/a`. +- [x] `auth/oidc/config` is written from `OIDC.CLIENT.*` values only — no hardcoded realm name, issuer URL, endpoint or claim string anywhere in the role. +- [x] The user identifier is templated from `{{ OIDC.ATTRIBUTES.USERNAME }}`; the literal `preferred_username` does not appear in the role. +- [x] A user logs in at `https://openbao.{{ DOMAIN_PRIMARY }}/` through the Keycloak chain and lands authenticated in the OpenBao UI. (`test-oidc-login.js`, 6.5 s.) +- [ ] Logout terminates the OpenBao session and integrates with the `logout` service when enabled. + +### LDAP (Decision #5) + +- [x] When `svc-db-openldap` is in `group_names`, the `ldap` auth method is enabled and reads its attribute names, DNs and URI from `LDAP.*` only. +- [x] An LDAP user in the `administrator` role group logs in through the `ldap` auth method and receives the `administrator` policy. (`test-ldap-login.js`, asserts `auth.policies` contains `administrator`.) + +### RBAC / policies (Decision #6) + +- [x] `meta/rbac.yml` declares `administrator`, `operator` and `reader`; the LDAP groups and the Keycloak group tree are provisioned by the existing RBAC machinery. Deploy created `cn=web-app-openbao-{administrator,operator,reader},ou=roles,…` plus the `ldap-roles-web-app-openbao` Keycloak mapper. +- [x] Three OpenBao policies of the same names exist, written from templates, following least privilege: `reader` has read-only capability on the application secret paths, `operator` adds create/update/delete there, and only `administrator` may touch `sys/policies`, `sys/auth` and `sys/mounts`. +- [x] The group→policy mapping is driven by `lookup('rbac_group_path', application_id='web-app-openbao', role='')`; no group path is assembled inline. +- [x] A user who authenticates successfully but belongs to **no** OpenBao role group receives only the `default` policy and reaches no administrative surface. (`test-ldap-login.js` asserts biber receives neither `administrator` nor `operator`.) +- [ ] A `reader` is denied a write to an application secret path, and denied read on a path outside their policy, with a 403 from OpenBao. *Only the unauthenticated/invalid-token denials are asserted today (`test-rbac-denial.js`); proving this needs a token actually bound to the `reader` policy.* + +### Secrets engine & machine identity + +- [x] A KV v2 secrets engine is mounted at a configurable path; the mount path is an Infinito.Nexus variable, not a literal in a task. (`OPENBAO_KV_MOUNT`, mounted at `infinito`.) +- [ ] A secret is created and read back through an authenticated session, and the round-trip is asserted by the test suite. +- [ ] Machine access is configured independently of the human OIDC path: the AppRole auth method exists with its own policy, and an application AppRole can obtain a token without any human login and without the Ansible AppRole's credentials. +- [ ] Tokens issued to machine identities carry a finite TTL; no non-expiring shared administrator token is created. + +### PKI (Decision #4) + +- [x] `services.openbao.pki.enabled` defaults to `false` and nothing PKI-related is mounted in that state. (The PKI include is reached and skipped in variant 0.) +- [ ] With the flag on, the play mounts `pki`, generates an internal root CA and one issuing role, and a service certificate can be issued from it. *Needs the variant-2 deploy.* +- [x] Both flag states are covered by `meta/variants.yml`. + +### Monitoring (Decision #8) + +- [x] `services.prometheus.native_metrics.enabled: true` and `templates/prometheus.yml.j2` exist; Prometheus scrapes `/v1/sys/metrics?format=prometheus` on the internal container port and the target reports `up`. Verified against the Prometheus targets API: job `openbao`, `scrapeUrl http://openbao:8200/v1/sys/metrics?format=prometheus`, `health: "up"`, `lastError: ""`. +- [x] The metrics endpoint is not reachable through `sys-svc-proxy` from outside the role-local docker network. It **was** publicly readable (HTTP 200 through the proxy) until a `location = /v1/sys/metrics { deny all; }` was added in the role's `templates/proxy.conf.j2`; `test-rbac-denial.js` now asserts the endpoint never answers 200 on the public surface. +- [ ] A sealed instance is detectable: `/v1/sys/health` returns 503 (asserted by `test-seal-status.js`) and an alert rule fires on that condition. **Correction from implementation:** OpenBao keeps Vault's metric namespace, so the gauge is `vault_core_unsealed`, not `openbao_core_unsealed` — and it is unusable as an alert source here: on a healthy, demonstrably unsealed node it is emitted as `vault_core_unsealed{cluster=""} 0`, so `== 0` would fire permanently. Use the health endpoint as the primary signal, with `vault_core_active{cluster="…"} == 1` as the metric-side corroboration. The alert rule itself is still outstanding. +- [x] The container healthcheck uses `sealedcode=200&uninitcode=200` so a freshly created, not-yet-initialised container becomes healthy enough for the init task to run, while the monitoring probe keeps the strict codes. Both halves asserted by `test-seal-status.js`; the deploy's init step only ran because the pre-init container reported healthy. + +### Logging + +- [x] OpenBao logs reach the standard Infinito.Nexus logging mechanism like every other compose service, at a configurable log level. The server logs to stdout like any compose service; the level is `services.openbao.log_level` (default `info`), rendered into the config as `log_level`. The image's own `/openbao/logs` volume is declared as `openbao_logs` so audit output has a named, backed-up home rather than an anonymous volume. +- [x] No task or template deliberately logs a token, a seal key or a secret value. + +### Backup & restore + +- [x] The `container_backup` service is wired with `backup.no_stop_required: false`, so the container is quiesced for a consistent copy of the raft volume. +- [ ] A restore procedure is documented and executed once end to end: restore the volume, redeploy, and confirm the instance auto-unseals and a previously stored secret is still readable. *Documented in the README; the end-to-end execution is still outstanding.* +- [x] The README states explicitly that the volume backup is **worthless without the seal key**, that the seal key's authoritative home is the encrypted inventory (it is not covered by `svc-bkp-secrets-2-local`, whose sources are `DIR_SECRETS`, the CA and the ACME material), and what the recovery keys can and cannot do. +- [x] The README states the consequence of Decision #2 that surfaced during implementation: because the root token is revoked and the recovery keys are **not** persisted (persisting them would be the write-back the AppRole bootstrap exists to avoid), the Ansible AppRole is the *only* administrative path into the instance. Losing or out-of-band-changing those credentials means rebuilding from the volume backup plus the seal key. + +### Variants + +- [ ] `meta/variants.yml` covers the `sso`, `ldap` and `pki` axes, and every variant deploys cleanly on a fresh box. + +### Playwright + +- [x] The three baseline personas are covered per the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md): `guest` (never reaches an authenticated surface), `biber`, `administrator`, each in its own `test-*.js` module aggregated by `playwright.spec.js`, with SSO/LDAP steps guarded by `skipUnlessServiceEnabled`. `guest` runs live; `biber` and `administrator` are blocked with `PERSONA_*_BLOCKED` under the contract's documented escape, each carrying a `# nocheck: persona-block-rationale` naming the role property that forces it — biber holds no OpenBao RBAC group by design, and the shared persona helper cannot drive OpenBao's method-select + popup + `postMessage` login. The administrator's OIDC journey is asserted instead by `test-oidc-login.js`, and biber's denial by `test-ldap-login.js`. +- [x] A spec asserts the RBAC denial from the criteria above (a non-member gets no administrative surface). +- [x] `templates/playwright.env.j2` exposes every env var the specs read. (Enforced by `test_env_keys_used.py`, which also fails on a declared-but-unread key.) +- [x] `make compose-playwright role=web-app-openbao` exits 0 with no stub tests and a clean logged-out final state. Latest deploy run: **13 passed, 2 skipped, 0 failed (30.8 s)**, no retries. + +### Quality & documentation + +- [ ] The stack reaches a steady running state in every variant, and `make quality` is green tree-wide. +- [ ] `README.md` documents the static-seal model and its trust assumption, the bootstrap and root-token revocation flow, the RBAC mapping, the PKI flag, the backup/restore procedure including the seal-key caveat, and the CLI-OIDC-login limitation from Decision #10. +- [ ] This requirement is cross-linked from the implementing PR, and the PR is cross-linked back from here. + +## Validation Apps + +```bash +INFINITO_APPS="web-app-openbao" make deploy-fresh-purged-apps INFINITO_FULL_CYCLE=true +``` + +Smoke: visit `https://openbao.{{ DOMAIN_PRIMARY }}/` → Keycloak chain → OpenBao UI; write and read a KV secret; `docker compose restart` and confirm the instance comes back unsealed. + +Cross-mode parity is the gate for Decision #11: run the compose loop first, then the swarm loop, then the round-trip sweep per [roundtrip.md](../agents/action/iteration/roundtrip.md). The role is not done until it is green in both. + +## Prerequisites + +Before implementation, the agent MUST read [AGENTS.md](../../AGENTS.md), then [Compose Loop](../agents/action/iteration/compose.md), [Swarm Loop](../agents/action/iteration/swarm.md), [deployment-modes.md](../contributing/design/deployment-modes.md), the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md), and the IAM pages [common.md](../contributing/design/iam/common.md), [oidc.md](../contributing/design/iam/oidc.md) and [rbac.md](../contributing/design/iam/rbac.md). + +## Implementation Strategy + +Execute autonomously; the Confirmed Decisions are settled. Scaffold from [`roles/web-app-semaphore/`](../../roles/web-app-semaphore/) for the native-OIDC + LDAP + Prometheus shape, and from [`roles/svc-db-elasticsearch/tasks/01_init.yml`](../../roles/svc-db-elasticsearch/tasks/01_init.yml) for the in-container configuration pattern that keeps credentials out of process arguments. + +Build in this order, verifying each stage against the running container before moving on: storage and static seal first (nothing else is testable until the instance auto-unseals), then init plus the AppRole bootstrap and root-token revocation, then policies, then OIDC, then LDAP, then metrics, then PKI behind its flag, then the Playwright specs. + +Drive that whole sequence on compose, then re-run it on swarm before claiming any criterion done. Use `lookup('container_address', ...)` from the first init task rather than retro-fitting it — a compose-only `container exec ` works locally and then fails on every swarm task, which is the expensive way to discover Decision #11. + +Verify every OpenBao path, config key and CLI flag against the pinned `2.6.2` image before claiming a criterion done — the seal, AppRole and health-parameter facts in this document were read from upstream `main`, and `2.6.2` is what actually ships. + +## Commit Policy + +- No git commit until every Acceptance Criterion is checked off. +- When met and `make quality` is green, instruct the operator to run `git-sign-push` outside the sandbox. The agent MUST NOT push. + +## Context + +- Upstream repo: +- Static seal: +- Seal concepts (recovery keys vs unseal keys): +- JWT/OIDC auth: +- Keycloak provider guide: +- `/sys/health`: +- `/sys/metrics`: +- Closest in-repo analogue: [`roles/web-app-semaphore/`](../../roles/web-app-semaphore/) (native OIDC + LDAP, no oauth2 gate) From ddabb3c9db0dac4b8159cc3cad210b4217c16ac1 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:22:45 +0200 Subject: [PATCH 002/145] feat(web-app-openbao): add galaxy metadata --- roles/web-app-openbao/meta/main.yml | 36 +++++++++++++++++++++++++++++ 1 file changed, 36 insertions(+) create mode 100644 roles/web-app-openbao/meta/main.yml diff --git a/roles/web-app-openbao/meta/main.yml b/roles/web-app-openbao/meta/main.yml new file mode 100644 index 0000000000..c463d413b6 --- /dev/null +++ b/roles/web-app-openbao/meta/main.yml @@ -0,0 +1,36 @@ +--- +galaxy_info: + author: Evangelos Tsakoudis + description: Identity-based secrets, PKI and encryption manager with OIDC login, group-mapped policies and static-seal auto-unseal + license: Infinito.Nexus Community License (Non-Commercial) + company: | + Kevin Veen-Birkenbach + https://www.veen.world + min_ansible_version: "2.9" + platforms: + - name: ArchLinux + versions: + - all + - name: Debian + versions: + - all + - name: EL + versions: + - all + - name: Fedora + versions: + - all + - name: Ubuntu + versions: + - all + galaxy_tags: + - openbao + - vault + - secrets + - pki + - oidc + - sso + - ldap + - docker + - compose + issue_tracker_url: https://s.infinito.nexus/issues From bec1de369217b485d617b4b5eb11e085615270dc Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:23:01 +0200 Subject: [PATCH 003/145] feat(web-app-openbao): declare service entity, image pin and deploy modes --- roles/web-app-openbao/meta/services.yml | 88 +++++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 roles/web-app-openbao/meta/services.yml diff --git a/roles/web-app-openbao/meta/services.yml b/roles/web-app-openbao/meta/services.yml new file mode 100644 index 0000000000..5eed6c32e9 --- /dev/null +++ b/roles/web-app-openbao/meta/services.yml @@ -0,0 +1,88 @@ +--- +# nocheck: seaweedfs-required - OpenBao keeps every secret, policy and PKI object in its own encrypted raft store on {{ lookup('volume', 'web-app-openbao', 'data').target }}; it has no S3/object-store surface. +sso: + bond: 1 + enabled: "{{ 'web-app-keycloak' in group_names }}" + shared: "{{ 'web-app-keycloak' in group_names }}" + flavor: oidc +ldap: + bond: 1 + enabled: "{{ 'svc-db-openldap' in group_names }}" + shared: "{{ 'svc-db-openldap' in group_names }}" +logout: + bond: 1 + enabled: "{{ 'web-svc-logout' in group_names }}" + shared: "{{ 'web-svc-logout' in group_names }}" +dashboard: + bond: 1 + enabled: "{{ 'web-app-dashboard' in group_names }}" + shared: "{{ 'web-app-dashboard' in group_names }}" +matomo: + bond: 1 + enabled: "{{ 'web-app-matomo' in group_names }}" + shared: "{{ 'web-app-matomo' in group_names }}" +css: + bond: 1 + enabled: "{{ 'web-svc-css' in group_names }}" + shared: "{{ 'web-svc-css' in group_names }}" +# nocheck: email, dynamic-flag - OpenBao has no outbound mail surface: it notifies nothing and owns no user mailboxes; audit trails leave through the logging stack. +email: + bond: 1 + enabled: false + shared: false +prometheus: + bond: 1 + enabled: "{{ 'web-app-prometheus' in group_names }}" + shared: "{{ 'web-app-prometheus' in group_names }}" + native_metrics: + enabled: true + port: 8200 +openbao: + # nocheck: default-placement-manager raft keeps its BoltDB store mmap'd with fsync durability, so it cannot live on NFS; the node is pinned in swarm to stay with its volume + placement: manager + modes: + compose: + enabled: true + swarm: + enabled: true + cpus: 0.5 + mem_reservation: 128m + mem_limit: 512m + pids_limit: 512 + min_storage: 60MB + image: openbao/openbao + version: "2.6.2" + backup: + no_stop_required: false + # nocheck: single-replica single-node raft: a second replica would need a real raft join/bootstrap flow, and two tasks against one BoltDB directory corrupt the store + replicas: 1 + name: openbao + pki: + enabled: false + log_level: info + ports: + internal: + http: 8200 + local: + http: 8081 + healthcheck: + flavor: wget + # Exception: an uninitialised container must report healthy or the init task can never reach it. + path: "v1/sys/health?uninitcode=200&sealedcode=200&standbyok=true" + run_after: + - web-app-keycloak + - svc-db-openldap + lifecycle: alpha + +# nocheck: playwright-service-flag +# nocheck: variants-coverage — sso-variant-tor requires the onion on in every variant, so the false polarity has no variant left to live in +tor: + bond: 0.5 + enabled: "{{ 'svc-net-tor' in group_names }}" + shared: true # nocheck: dynamic-flag + +# nocheck: playwright-service-flag, dynamic-flag infrastructure backup consumer +container_backup: # nocheck: bond-baseline backup host joins outside the variant topology + bond: 1 + enabled: "{{ 'svc-bkp-volume-2-local' in group_names }}" + shared: true From fd8fdea14fc59c77bd220ef12d14c904372a8e5d Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:23:16 +0200 Subject: [PATCH 004/145] feat(web-app-openbao): declare raft, logs and policy volumes --- roles/web-app-openbao/meta/volumes.yml | 58 ++++++++++++++++++++++++++ 1 file changed, 58 insertions(+) create mode 100644 roles/web-app-openbao/meta/volumes.yml diff --git a/roles/web-app-openbao/meta/volumes.yml b/roles/web-app-openbao/meta/volumes.yml new file mode 100644 index 0000000000..b5be36ffcb --- /dev/null +++ b/roles/web-app-openbao/meta/volumes.yml @@ -0,0 +1,58 @@ +--- +data: + type: volume + name: openbao_data + # nocheck: nfs-false-data-loss Reason: raft/BoltDB (mmap + fsync) corrupts over NFS; single-replica and pinned via placement: manager. + nfs: false + mounts: + - service: openbao + target: /openbao/file + +logs: + type: volume + name: openbao_logs + # nocheck: nfs-false-data-loss Reason: log output only, regenerated by the server; node pinned via placement: manager. + nfs: false + mounts: + - service: openbao + target: /openbao/logs + +config: + type: config + source: "{{ [lookup('container', 'web-app-openbao', 'directories.config'), 'openbao.hcl'] | path_join }}" + mode: "0444" + mounts: + - service: openbao + target: /openbao/config/openbao.hcl + +policy_ansible_admin: + type: config + source: "{{ [lookup('container', 'web-app-openbao', 'directories.config'), 'policies', 'ansible-admin.hcl'] | path_join }}" + mode: "0444" + mounts: + - service: openbao + target: /openbao/policies/ansible-admin.hcl + +policy_administrator: + type: config + source: "{{ [lookup('container', 'web-app-openbao', 'directories.config'), 'policies', 'administrator.hcl'] | path_join }}" + mode: "0444" + mounts: + - service: openbao + target: /openbao/policies/administrator.hcl + +policy_operator: + type: config + source: "{{ [lookup('container', 'web-app-openbao', 'directories.config'), 'policies', 'operator.hcl'] | path_join }}" + mode: "0444" + mounts: + - service: openbao + target: /openbao/policies/operator.hcl + +policy_reader: + type: config + source: "{{ [lookup('container', 'web-app-openbao', 'directories.config'), 'policies', 'reader.hcl'] | path_join }}" + mode: "0444" + mounts: + - service: openbao + target: /openbao/policies/reader.hcl From f172f59741ec7a13afc3847e53a6f74df32e69c5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:23:31 +0200 Subject: [PATCH 005/145] feat(web-app-openbao): add canonical domain --- roles/web-app-openbao/meta/domains.yml | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 roles/web-app-openbao/meta/domains.yml diff --git a/roles/web-app-openbao/meta/domains.yml b/roles/web-app-openbao/meta/domains.yml new file mode 100644 index 0000000000..e0d4dda8b7 --- /dev/null +++ b/roles/web-app-openbao/meta/domains.yml @@ -0,0 +1,4 @@ +--- +canonical: + - openbao.{{ DOMAIN_PRIMARY }} +aliases: [] From 1963fbd6d05fa551f1b9722ed3a22ebc2f3d67a5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:23:45 +0200 Subject: [PATCH 006/145] feat(web-app-openbao): allocate role-local subnet --- roles/web-app-openbao/meta/networks.yml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 roles/web-app-openbao/meta/networks.yml diff --git a/roles/web-app-openbao/meta/networks.yml b/roles/web-app-openbao/meta/networks.yml new file mode 100644 index 0000000000..b4de29575a --- /dev/null +++ b/roles/web-app-openbao/meta/networks.yml @@ -0,0 +1,3 @@ +--- +local: + subnet: 192.168.221.0/24 From 1f9fc8043746c01f28c457cf978fa9b39c2cc378 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:24:16 +0200 Subject: [PATCH 007/145] feat(web-app-openbao): add CSP flags for the Ember UI --- roles/web-app-openbao/meta/csp.yml | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 roles/web-app-openbao/meta/csp.yml diff --git a/roles/web-app-openbao/meta/csp.yml b/roles/web-app-openbao/meta/csp.yml new file mode 100644 index 0000000000..d50d5eca7e --- /dev/null +++ b/roles/web-app-openbao/meta/csp.yml @@ -0,0 +1,9 @@ +--- +flags: + style-src: + unsafe-inline: true # nocheck: csp-unsafe Reason: the OpenBao UI is an Ember SPA that sets element styles at runtime; script-src stays strict because the production build ships precompiled templates. + style-src-attr: + unsafe-inline: true # nocheck: csp-unsafe Reason: same Ember runtime style attributes as style-src. +whitelist: + connect-src: + - "data:" From e82e9e13265cbb9a05738d33f93eb7b05d92758b Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:24:32 +0200 Subject: [PATCH 008/145] fix(web-app-openbao): accept the 307 redirect in the webserver health check --- roles/web-app-openbao/meta/server.yml | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 roles/web-app-openbao/meta/server.yml diff --git a/roles/web-app-openbao/meta/server.yml b/roles/web-app-openbao/meta/server.yml new file mode 100644 index 0000000000..49281e02d6 --- /dev/null +++ b/roles/web-app-openbao/meta/server.yml @@ -0,0 +1,8 @@ +--- +# Exception: OpenBao answers `/` with a 307 into its UI, which the health checker does not expect by default. +status_codes: + default: + - 200 + - 301 + - 302 + - 307 From 299dd1955c9bdc5d9a945e5f0f0e40677ed01bff Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:24:48 +0200 Subject: [PATCH 009/145] feat(web-app-openbao): declare administrator, operator and reader RBAC roles --- roles/web-app-openbao/meta/rbac.yml | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 roles/web-app-openbao/meta/rbac.yml diff --git a/roles/web-app-openbao/meta/rbac.yml b/roles/web-app-openbao/meta/rbac.yml new file mode 100644 index 0000000000..d900b6940b --- /dev/null +++ b/roles/web-app-openbao/meta/rbac.yml @@ -0,0 +1,8 @@ +--- +roles: + administrator: + description: OpenBao administrator (policy, auth-method and mount management) + operator: + description: OpenBao operator (read/write on the application secret paths) + reader: + description: OpenBao reader (read-only on the application secret paths) From ad097bf59803ade993d8fa97e5026fb830d5620f Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:25:03 +0200 Subject: [PATCH 010/145] feat(web-app-openbao): add seal key and AppRole credential schema --- roles/web-app-openbao/meta/secrets.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) create mode 100644 roles/web-app-openbao/meta/secrets.yml diff --git a/roles/web-app-openbao/meta/secrets.yml b/roles/web-app-openbao/meta/secrets.yml new file mode 100644 index 0000000000..1b8b2b86d3 --- /dev/null +++ b/roles/web-app-openbao/meta/secrets.yml @@ -0,0 +1,14 @@ +--- +credentials: + seal_key: + description: 32-byte AES-256-GCM-96 key for the OpenBao static seal, injected as env://OPENBAO_SEAL_KEY. Every backup of the raft volume is undecryptable without it; its authoritative home is the encrypted inventory. + algorithm: random_hex_32 + validation: non_empty_string + approle_role_id: + description: RoleID of the Ansible AppRole, written to auth/approle/role/ansible/role-id at bootstrap so later deploys authenticate without a stored root token. + algorithm: alphanumeric + validation: non_empty_string + approle_secret_id: + description: SecretID of the Ansible AppRole, written to auth/approle/role/ansible/custom-secret-id at bootstrap. + algorithm: alphanumeric + validation: non_empty_string From af59d181eecf5194edcf0da3dd2d2d6e6317deeb Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:26:47 +0200 Subject: [PATCH 011/145] feat(web-app-openbao): add role-local user map --- roles/web-app-openbao/meta/users.yml | 2 ++ 1 file changed, 2 insertions(+) create mode 100644 roles/web-app-openbao/meta/users.yml diff --git a/roles/web-app-openbao/meta/users.yml b/roles/web-app-openbao/meta/users.yml new file mode 100644 index 0000000000..6c31213ebc --- /dev/null +++ b/roles/web-app-openbao/meta/users.yml @@ -0,0 +1,2 @@ +--- +administrator: {} From 970ae41b486c719bc47c2c8e99ebdfad3552fbff Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:27:05 +0200 Subject: [PATCH 012/145] feat(web-app-openbao): add descriptive role metadata --- roles/web-app-openbao/meta/info.yml | 5 +++++ 1 file changed, 5 insertions(+) create mode 100644 roles/web-app-openbao/meta/info.yml diff --git a/roles/web-app-openbao/meta/info.yml b/roles/web-app-openbao/meta/info.yml new file mode 100644 index 0000000000..e2ca5464db --- /dev/null +++ b/roles/web-app-openbao/meta/info.yml @@ -0,0 +1,5 @@ +--- +logo: + class: fa-solid fa-vault +homepage: https://openbao.org/ +video: https://www.youtube.com/watch?v=hPBT-kSS-68 From 3c29f8bb6100bd8781871b8c6b5282ec018be555 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:27:51 +0200 Subject: [PATCH 013/145] feat(web-app-openbao): add sso, ldap and pki variant matrix --- roles/web-app-openbao/meta/variants.yml | 86 +++++++++++++++++++++++++ 1 file changed, 86 insertions(+) create mode 100644 roles/web-app-openbao/meta/variants.yml diff --git a/roles/web-app-openbao/meta/variants.yml b/roles/web-app-openbao/meta/variants.yml new file mode 100644 index 0000000000..c66952f1c0 --- /dev/null +++ b/roles/web-app-openbao/meta/variants.yml @@ -0,0 +1,86 @@ +--- +- services: + container_backup: + enabled: true + css: + enabled: true + shared: true + dashboard: + enabled: true + shared: true + ldap: + enabled: true + shared: true + logout: + enabled: true + shared: true + matomo: + enabled: true + shared: true + prometheus: + enabled: true + shared: true + sso: + enabled: true + shared: true + tor: + enabled: true + +- services: + container_backup: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + css: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + dashboard: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + ldap: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + logout: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + matomo: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + prometheus: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + sso: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + tor: + enabled: true + +- services: + container_backup: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + css: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + dashboard: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + ldap: + enabled: true + shared: true + logout: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + matomo: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + openbao: + pki: + enabled: true + prometheus: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + sso: + enabled: false + shared: false # nocheck: variants-static-override backup shared stays literally true + tor: + enabled: true From 1019e17b719fef7d5cea8573d59cd5d5a752b2a8 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:29:37 +0200 Subject: [PATCH 014/145] feat(web-app-openbao): resolve OIDC, LDAP and seal values from the config SPOT --- roles/web-app-openbao/vars/main.yml | 48 +++++++++++++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 roles/web-app-openbao/vars/main.yml diff --git a/roles/web-app-openbao/vars/main.yml b/roles/web-app-openbao/vars/main.yml new file mode 100644 index 0000000000..d6fa40a5d9 --- /dev/null +++ b/roles/web-app-openbao/vars/main.yml @@ -0,0 +1,48 @@ +--- +application_id: "web-app-openbao" +container_port: "{{ lookup('config', application_id, 'services.openbao.ports.internal.http') }}" + +OPENBAO_NAME: "{{ lookup('config', application_id, 'services.openbao.name') }}" + +OPENBAO_SSO_ENABLED: "{{ lookup('config', application_id, 'services.sso.enabled') }}" +OPENBAO_LDAP_ENABLED: "{{ lookup('config', application_id, 'services.ldap.enabled') }}" +OPENBAO_PKI_ENABLED: "{{ lookup('config', application_id, 'services.openbao.pki.enabled') }}" +OPENBAO_LOG_LEVEL: "{{ lookup('config', application_id, 'services.openbao.log_level') }}" + +OPENBAO_BASE_URL: "{{ lookup('tls', application_id, 'url.base') | regex_replace('/+$', '') }}" + +OPENBAO_CONFIG_HOST: "{{ [lookup('container', application_id, 'directories.config'), 'openbao.hcl'] | path_join }}" +OPENBAO_POLICY_DIR_HOST: "{{ [lookup('container', application_id, 'directories.config'), 'policies'] | path_join }}" +OPENBAO_POLICY_DIR_DOCKER: "/openbao/policies" +OPENBAO_DATA_DOCKER: "{{ lookup('volume', application_id, 'data').target }}" + +OPENBAO_SEAL_KEY: "{{ lookup('config', application_id, 'secrets.credentials.seal_key') }}" +OPENBAO_APPROLE_ROLE_ID: "{{ lookup('config', application_id, 'secrets.credentials.approle_role_id') }}" +OPENBAO_APPROLE_SECRET_ID: "{{ lookup('config', application_id, 'secrets.credentials.approle_secret_id') }}" + +OPENBAO_SEAL_KEY_ID: "infinito-1" +OPENBAO_KV_MOUNT: "infinito" +OPENBAO_ANSIBLE_APPROLE: "ansible" +OPENBAO_PKI_MOUNT: "pki" +OPENBAO_PKI_ISSUING_ROLE: "internal" + +OPENBAO_OIDC_ISSUER: "{{ OIDC.CLIENT.ISSUER_URL }}" +OPENBAO_OIDC_CLIENT_ID: "{{ OIDC.CLIENT.ID }}" +OPENBAO_OIDC_CLIENT_SECRET: "{{ OIDC.CLIENT.SECRET }}" +OPENBAO_OIDC_USER_CLAIM: "{{ OIDC.ATTRIBUTES.USERNAME }}" +OPENBAO_OIDC_GROUPS_CLAIM: "groups" +OPENBAO_OIDC_DEFAULT_ROLE: "default" + +OPENBAO_OIDC_REDIRECT_URIS: >- + {{ [ + OPENBAO_BASE_URL ~ '/v1/auth/oidc/callback', + OPENBAO_BASE_URL ~ '/ui/vault/auth/oidc/oidc/callback' + ] | join(',') }} + +OPENBAO_LDAP_URL: "ldap://{{ LDAP.SERVER.DOMAIN }}:{{ LDAP.SERVER.PORT }}" +OPENBAO_LDAP_USER_ATTR: "{{ LDAP.USER.ATTRIBUTES.ID }}" + +OPENBAO_RBAC_ROLES: + - administrator + - operator + - reader From dfa2350d6f2ebd7babc3d245f9d0bd22861435d4 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:29:52 +0200 Subject: [PATCH 015/145] feat(web-app-openbao): add run-once role entry point --- roles/web-app-openbao/tasks/main.yml | 3 +++ 1 file changed, 3 insertions(+) create mode 100644 roles/web-app-openbao/tasks/main.yml diff --git a/roles/web-app-openbao/tasks/main.yml b/roles/web-app-openbao/tasks/main.yml new file mode 100644 index 0000000000..d8c08a2be9 --- /dev/null +++ b/roles/web-app-openbao/tasks/main.yml @@ -0,0 +1,3 @@ +--- +- include_tasks: 00_core.yml + when: run_once_web_app_openbao is not defined From f2862c2868cb584fc9817d88742a912537b3aff1 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:30:06 +0200 Subject: [PATCH 016/145] feat(web-app-openbao): render config after the backend stack, before the handler flush --- roles/web-app-openbao/tasks/00_core.yml | 45 +++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 roles/web-app-openbao/tasks/00_core.yml diff --git a/roles/web-app-openbao/tasks/00_core.yml b/roles/web-app-openbao/tasks/00_core.yml new file mode 100644 index 0000000000..88554898ff --- /dev/null +++ b/roles/web-app-openbao/tasks/00_core.yml @@ -0,0 +1,45 @@ +--- +# Exception: the config templates notify `compose-up`, so they MUST be rendered after +# sys-stk-backend has produced compose.yml and before the handler flush. +- include_tasks: utils/once/flag.yml + +- name: "🧩 load docker for {{ application_id }}" + include_role: + name: sys-stk-backend + vars: + compose_handlers_flush: false + +- name: "📄 Render the OpenBao server config and RBAC policies (local + swarm peers)" + ansible.builtin.include_tasks: + file: "{{ lookup('path_absolute', 'roles/sys-svc-compose/tasks/utils/render_replicated_templates.yml') }}" + vars: + role_templates: + - src: "openbao.hcl.j2" + dest: "{{ OPENBAO_CONFIG_HOST }}" + mode: "0644" + - src: "policies/ansible-admin.hcl.j2" + dest: "{{ [OPENBAO_POLICY_DIR_HOST, 'ansible-admin.hcl'] | path_join }}" + mode: "0644" + - src: "policies/administrator.hcl.j2" + dest: "{{ [OPENBAO_POLICY_DIR_HOST, 'administrator.hcl'] | path_join }}" + mode: "0644" + - src: "policies/operator.hcl.j2" + dest: "{{ [OPENBAO_POLICY_DIR_HOST, 'operator.hcl'] | path_join }}" + mode: "0644" + - src: "policies/reader.hcl.j2" + dest: "{{ [OPENBAO_POLICY_DIR_HOST, 'reader.hcl'] | path_join }}" + mode: "0644" + +- name: "🚿 flush handlers for '{{ application_id }}'" + meta: flush_handlers + +- name: "📂 Include role sys-stk-front-proxy for '{{ application_id }}'" + include_role: + name: sys-stk-front-proxy + vars: + domain: "{{ lookup('domain', application_id) }}" + http_port: "{{ lookup('config', application_id, 'services.openbao.ports.local.http') }}" + +- name: "🔐 Initialise, bootstrap and configure OpenBao for {{ application_id }}" + include_tasks: 01_init.yml + when: IS_STACK_HOST | bool From 9c7979f0ec3f590037fa322b9cf381a1da28589a Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:30:30 +0200 Subject: [PATCH 017/145] feat(web-app-openbao): initialise once and wait for the static seal to auto-unseal --- roles/web-app-openbao/tasks/01_init.yml | 108 ++++++++++++++++++++++++ 1 file changed, 108 insertions(+) create mode 100644 roles/web-app-openbao/tasks/01_init.yml diff --git a/roles/web-app-openbao/tasks/01_init.yml b/roles/web-app-openbao/tasks/01_init.yml new file mode 100644 index 0000000000..23659087be --- /dev/null +++ b/roles/web-app-openbao/tasks/01_init.yml @@ -0,0 +1,108 @@ +--- +- name: "🔍 Resolve {{ application_id }} hosting node + container id" + ansible.builtin.include_tasks: "{{ lookup('path_absolute', 'roles/sys-svc-compose/tasks/utils/swarm/resolve_host_cid.yml') }}" + vars: + resolve_service_key: openbao + resolve_output_prefix: OPENBAO + resolve_compose_cid: "{{ OPENBAO_NAME }}" + +- name: "⏳ Wait until the OpenBao API answers" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao status -format=json + register: openbao_status_raw + changed_when: false + retries: 30 + delay: 5 + # Exception: exit 2 means sealed and exit 1 means not yet initialised; both prove the API answers. + until: openbao_status_raw.rc in [0, 2] or 'not yet initialized' in openbao_status_raw.stderr + failed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🔧 Determine whether OpenBao is already initialised" + ansible.builtin.set_fact: + openbao_initialized: >- + {{ (openbao_status_raw.stdout | from_json).initialized | default(false) + if (openbao_status_raw.stdout | default('') | trim) | length > 0 + else false }} + +- name: "🔐 Initialise OpenBao (first deploy only)" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao operator init -format=json -recovery-shares=1 -recovery-threshold=1 + register: openbao_init_raw + changed_when: openbao_init_raw.rc == 0 + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + when: not (openbao_initialized | bool) + +- name: "🔧 Hold the bootstrap root token in memory only" + ansible.builtin.set_fact: + openbao_root_token: >- + {{ (openbao_init_raw.stdout | from_json).root_token }} + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + when: not (openbao_initialized | bool) + +- name: "⏳ Wait until the static seal has auto-unsealed the node" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao status -format=json + register: openbao_unsealed_raw + changed_when: false + retries: 30 + delay: 5 + until: >- + openbao_unsealed_raw.rc == 0 + and not ((openbao_unsealed_raw.stdout | from_json).sealed | bool) + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🔑 Bootstrap the Ansible AppRole and revoke the root token" + include_tasks: 02_auth.yml + when: not (openbao_initialized | bool) + +- name: "🔐 Log in with the Ansible AppRole" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao write -field=token auth/approle/login + role_id={{ OPENBAO_APPROLE_ROLE_ID | quote }} secret_id=- + args: + stdin: "{{ OPENBAO_APPROLE_SECRET_ID }}" + stdin_add_newline: false + register: openbao_approle_login + changed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🔧 Store the AppRole token in the container token helper" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao login -no-print - + args: + stdin: "{{ openbao_approle_login.stdout | trim }}" + stdin_add_newline: false + changed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "📜 Reconcile the secrets engine and the RBAC policies" + ansible.builtin.include_tasks: + file: 03_policies.yml + apply: + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "📇 Configure the OIDC and LDAP auth methods" + include_tasks: 04_auth_methods.yml + +- name: "🏛 Configure the internal PKI" + ansible.builtin.include_tasks: + file: 05_pki.yml + apply: + delegate_to: "{{ OPENBAO_HOST_NODE }}" + when: OPENBAO_PKI_ENABLED | bool + +- name: "🧹 Drop the stored reconciliation token from the container" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + sh -c 'bao token revoke -self >/dev/null 2>&1; rm -f ~/.bao-token' + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" From 50adf4456d8c04f9f5b5f48428323936413770fe Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:30:45 +0200 Subject: [PATCH 018/145] feat(web-app-openbao): pin AppRole ids from inventory and revoke the root token --- roles/web-app-openbao/tasks/02_auth.yml | 69 +++++++++++++++++++++++++ 1 file changed, 69 insertions(+) create mode 100644 roles/web-app-openbao/tasks/02_auth.yml diff --git a/roles/web-app-openbao/tasks/02_auth.yml b/roles/web-app-openbao/tasks/02_auth.yml new file mode 100644 index 0000000000..b8b4e79315 --- /dev/null +++ b/roles/web-app-openbao/tasks/02_auth.yml @@ -0,0 +1,69 @@ +--- +- name: "🔐 Store the bootstrap root token in the container token helper" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao login -no-print - + args: + stdin: "{{ openbao_root_token }}" + stdin_add_newline: false + changed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🧩 Enable the AppRole auth method" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + sh -c 'bao auth list -format=json | grep -q "approle/" || bao auth enable approle' + register: openbao_approle_enable + changed_when: openbao_approle_enable.rc == 0 + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "📜 Write the ansible-admin policy the Ansible AppRole binds to" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao policy write ansible-admin {{ OPENBAO_POLICY_DIR_DOCKER }}/ansible-admin.hcl + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🧩 Create the Ansible AppRole" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/approle/role/{{ OPENBAO_ANSIBLE_APPROLE | quote }} + token_policies=ansible-admin + token_ttl=30m + token_max_ttl=1h + secret_id_num_uses=0 + secret_id_ttl=0 + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🔑 Pin the RoleID to the inventory value" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/approle/role/{{ OPENBAO_ANSIBLE_APPROLE | quote }}/role-id + role_id={{ OPENBAO_APPROLE_ROLE_ID | quote }} + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🔑 Pin the SecretID to the inventory value" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/approle/role/{{ OPENBAO_ANSIBLE_APPROLE | quote }}/custom-secret-id + secret_id=- + args: + stdin: "{{ OPENBAO_APPROLE_SECRET_ID }}" + stdin_add_newline: false + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🧹 Revoke the bootstrap root token" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + sh -c 'bao token revoke -self && rm -f ~/.bao-token' + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🔓 Forget the root token" + ansible.builtin.set_fact: + openbao_root_token: "" From 47df1e8a11e0a13ac443d40fb7f65c204e65c581 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:30:59 +0200 Subject: [PATCH 019/145] feat(web-app-openbao): mount KV v2 and reconcile the RBAC policies --- roles/web-app-openbao/tasks/03_policies.yml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) create mode 100644 roles/web-app-openbao/tasks/03_policies.yml diff --git a/roles/web-app-openbao/tasks/03_policies.yml b/roles/web-app-openbao/tasks/03_policies.yml new file mode 100644 index 0000000000..2206e696f6 --- /dev/null +++ b/roles/web-app-openbao/tasks/03_policies.yml @@ -0,0 +1,21 @@ +--- +- name: "🗄 Ensure the KV v2 secrets engine is mounted at '{{ OPENBAO_KV_MOUNT }}'" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + sh -c 'bao secrets list -format=json | grep -q "\"{{ OPENBAO_KV_MOUNT }}/\"" || + bao secrets enable -path={{ OPENBAO_KV_MOUNT | quote }} -version=2 kv' + register: openbao_kv_mount + changed_when: openbao_kv_mount.rc == 0 + +- name: "📜 Write the human RBAC policies" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao policy write {{ item | quote }} {{ OPENBAO_POLICY_DIR_DOCKER }}/{{ item }}.hcl + loop: "{{ OPENBAO_RBAC_ROLES }}" + changed_when: true + +- name: "📜 Refresh the ansible-admin policy" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao policy write ansible-admin {{ OPENBAO_POLICY_DIR_DOCKER }}/ansible-admin.hcl + changed_when: true From 8c68430c0cf8f2f0e8ea303125dec22679e7cb4a Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:31:17 +0200 Subject: [PATCH 020/145] feat(web-app-openbao): configure OIDC and LDAP auth with group-to-policy binding --- .../web-app-openbao/tasks/04_auth_methods.yml | 128 ++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 roles/web-app-openbao/tasks/04_auth_methods.yml diff --git a/roles/web-app-openbao/tasks/04_auth_methods.yml b/roles/web-app-openbao/tasks/04_auth_methods.yml new file mode 100644 index 0000000000..4e292fc622 --- /dev/null +++ b/roles/web-app-openbao/tasks/04_auth_methods.yml @@ -0,0 +1,128 @@ +--- +- name: "📇 Configure the OIDC auth method against Keycloak" + when: OPENBAO_SSO_ENABLED | bool + block: + - name: "🧩 Enable the OIDC auth method" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + sh -c 'bao auth list -format=json | grep -q "oidc/" || bao auth enable oidc' + register: openbao_oidc_enable + changed_when: openbao_oidc_enable.rc == 0 + delegate_to: "{{ OPENBAO_HOST_NODE }}" + + - name: "📇 Point the OIDC method at the Keycloak realm" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/oidc/config + oidc_discovery_url={{ OPENBAO_OIDC_ISSUER | quote }} + oidc_client_id={{ OPENBAO_OIDC_CLIENT_ID | quote }} + oidc_client_secret=- + default_role={{ OPENBAO_OIDC_DEFAULT_ROLE | quote }} + args: + stdin: "{{ OPENBAO_OIDC_CLIENT_SECRET }}" + stdin_add_newline: false + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + + - name: "📇 Configure the default OIDC role, which grants only the 'default' policy" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/oidc/role/{{ OPENBAO_OIDC_DEFAULT_ROLE | quote }} + role_type=oidc + user_claim={{ OPENBAO_OIDC_USER_CLAIM | quote }} + groups_claim={{ OPENBAO_OIDC_GROUPS_CLAIM | quote }} + oidc_scopes=openid,profile,email + token_policies=default + allowed_redirect_uris={{ OPENBAO_OIDC_REDIRECT_URIS | quote }} + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "📇 Configure the LDAP auth method against svc-db-openldap" + when: OPENBAO_LDAP_ENABLED | bool + block: + - name: "🧩 Enable the LDAP auth method" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + sh -c 'bao auth list -format=json | grep -q "ldap/" || bao auth enable ldap' + register: openbao_ldap_enable + changed_when: openbao_ldap_enable.rc == 0 + delegate_to: "{{ OPENBAO_HOST_NODE }}" + + - name: "📇 Point the LDAP method at the directory" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/ldap/config + url={{ OPENBAO_LDAP_URL | quote }} + userdn={{ LDAP.DN.OU.USERS | quote }} + userattr={{ OPENBAO_LDAP_USER_ATTR | quote }} + groupdn={{ LDAP.DN.OU.ROLES | quote }} + groupattr=cn + binddn={{ LDAP.DN.ADMINISTRATOR.DATA | quote }} + bindpass=- + insecure_tls=false + starttls=false + args: + stdin: "{{ LDAP.BIND_CREDENTIAL }}" + stdin_add_newline: false + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🔍 Read the auth mount accessors" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao auth list -format=json + register: openbao_auth_list + changed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🔧 Keep the auth mount table, keyed by mount path such as 'oidc/'" + ansible.builtin.set_fact: + openbao_auth_mounts: "{{ openbao_auth_list.stdout | from_json }}" + +- name: "👥 Upsert the external identity group per RBAC role" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write identity/group/name/{{ item | quote }} + type=external + policies={{ item | quote }} + loop: "{{ OPENBAO_RBAC_ROLES }}" + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🔍 Read the identity group ids" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao read -field=id identity/group/name/{{ item | quote }} + loop: "{{ OPENBAO_RBAC_ROLES }}" + register: openbao_group_ids + changed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🔗 Bind each Keycloak group path to its OpenBao identity group" + ansible.builtin.include_tasks: + file: utils/group_alias.yml + apply: + delegate_to: "{{ OPENBAO_HOST_NODE }}" + loop: "{{ openbao_group_ids.results }}" + loop_control: + loop_var: openbao_group + vars: + alias_mount: "oidc/" + alias_name: "/{{ lookup('rbac_group_path', application_id=application_id, role=openbao_group.item) }}" + when: OPENBAO_SSO_ENABLED | bool + +- name: "🔗 Bind each LDAP role group to its OpenBao identity group" + ansible.builtin.include_tasks: + file: utils/group_alias.yml + apply: + delegate_to: "{{ OPENBAO_HOST_NODE }}" + loop: "{{ openbao_group_ids.results }}" + loop_control: + loop_var: openbao_group + vars: + alias_mount: "ldap/" + # Rationale: the RBAC LDAP layout contract encodes the application into the group CN. + alias_name: "{{ application_id }}-{{ openbao_group.item }}" + when: OPENBAO_LDAP_ENABLED | bool From ad98e188effa7b48293dd2ef4dbfe6bdb3508473 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:31:31 +0200 Subject: [PATCH 021/145] feat(web-app-openbao): mount the optional internal PKI behind a service flag --- roles/web-app-openbao/tasks/05_pki.yml | 45 ++++++++++++++++++++++++++ 1 file changed, 45 insertions(+) create mode 100644 roles/web-app-openbao/tasks/05_pki.yml diff --git a/roles/web-app-openbao/tasks/05_pki.yml b/roles/web-app-openbao/tasks/05_pki.yml new file mode 100644 index 0000000000..4469524aa3 --- /dev/null +++ b/roles/web-app-openbao/tasks/05_pki.yml @@ -0,0 +1,45 @@ +--- +- name: "🏛 Ensure the PKI secrets engine is mounted at '{{ OPENBAO_PKI_MOUNT }}'" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + sh -c 'bao secrets list -format=json | grep -q "\"{{ OPENBAO_PKI_MOUNT }}/\"" || + bao secrets enable -path={{ OPENBAO_PKI_MOUNT | quote }} pki' + register: openbao_pki_mount + changed_when: openbao_pki_mount.rc == 0 + +- name: "🔍 Check whether the internal root CA already exists" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + sh -c 'bao read -field=certificate {{ OPENBAO_PKI_MOUNT }}/cert/ca 2>/dev/null | grep -q "BEGIN CERTIFICATE"' + register: openbao_pki_ca + changed_when: false + failed_when: false + +- name: "🏛 Generate the internal root CA" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write -field=certificate {{ OPENBAO_PKI_MOUNT }}/root/generate/internal + common_name={{ ('Infinito.Nexus Internal CA (' ~ DOMAIN_PRIMARY ~ ')') | quote }} + issuer_name=infinito-root + ttl=87600h + changed_when: true + when: openbao_pki_ca.rc != 0 + +- name: "🔗 Configure the CA and CRL distribution URLs" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write {{ OPENBAO_PKI_MOUNT }}/config/urls + issuing_certificates={{ (OPENBAO_BASE_URL ~ '/v1/' ~ OPENBAO_PKI_MOUNT ~ '/ca') | quote }} + crl_distribution_points={{ (OPENBAO_BASE_URL ~ '/v1/' ~ OPENBAO_PKI_MOUNT ~ '/crl') | quote }} + changed_when: true + +- name: "📇 Configure the issuing role '{{ OPENBAO_PKI_ISSUING_ROLE }}'" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write {{ OPENBAO_PKI_MOUNT }}/roles/{{ OPENBAO_PKI_ISSUING_ROLE | quote }} + allowed_domains={{ DOMAIN_PRIMARY | quote }} + allow_subdomains=true + allow_bare_domains=true + max_ttl=2160h + ttl=720h + changed_when: true From 1500470c9d8e720d5c81d550bfbfdcbfb14acbd3 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:31:51 +0200 Subject: [PATCH 022/145] feat(web-app-openbao): add idempotent identity group-alias helper --- .../tasks/utils/group_alias.yml | 25 +++++++++++++++++++ 1 file changed, 25 insertions(+) create mode 100644 roles/web-app-openbao/tasks/utils/group_alias.yml diff --git a/roles/web-app-openbao/tasks/utils/group_alias.yml b/roles/web-app-openbao/tasks/utils/group_alias.yml new file mode 100644 index 0000000000..02173b2a22 --- /dev/null +++ b/roles/web-app-openbao/tasks/utils/group_alias.yml @@ -0,0 +1,25 @@ +--- +# Exception: a group alias is keyed by (name, mount_accessor) and creating a duplicate errors, +# so the alias already on the group is compared before writing. +# +# Param: openbao_group - one entry of the registered group-id loop (item = role name, stdout = group id) +# Param: alias_mount - auth mount path, e.g. "oidc/" or "ldap/" +# Param: alias_name - identity-provider group name the claim will carry +- name: "🔍 Read the existing alias of identity group '{{ openbao_group.item }}'" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao read -format=json identity/group/id/{{ openbao_group.stdout | trim | quote }} + register: openbao_group_read + changed_when: false + +- name: "🔗 Create the group alias '{{ alias_name }}' on mount '{{ alias_mount }}'" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write identity/group-alias + name={{ alias_name | quote }} + mount_accessor={{ openbao_auth_mounts[alias_mount].accessor | quote }} + canonical_id={{ openbao_group.stdout | trim | quote }} + changed_when: true + when: >- + ((openbao_group_read.stdout | from_json).data.alias | default({})).name | default('') + != alias_name From 783d13190b5b1532cadd14433abf1adc4491dcd9 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:32:10 +0200 Subject: [PATCH 023/145] feat(web-app-openbao): add server config with raft storage and static seal --- .../web-app-openbao/templates/openbao.hcl.j2 | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 roles/web-app-openbao/templates/openbao.hcl.j2 diff --git a/roles/web-app-openbao/templates/openbao.hcl.j2 b/roles/web-app-openbao/templates/openbao.hcl.j2 new file mode 100644 index 0000000000..9b25f8d5b3 --- /dev/null +++ b/roles/web-app-openbao/templates/openbao.hcl.j2 @@ -0,0 +1,35 @@ +ui = true + +log_level = "{{ OPENBAO_LOG_LEVEL }}" + +# Exception: raft and mlock are mutually exclusive upstream. +disable_mlock = true + +storage "raft" { + path = "{{ OPENBAO_DATA_DOCKER }}" + node_id = "{{ OPENBAO_NAME }}-1" +} + +listener "tcp" { + address = "0.0.0.0:{{ container_port }}" + tls_disable = true + + telemetry { + unauthenticated_metrics_access = true + } +} + +seal "static" { + current_key_id = "{{ OPENBAO_SEAL_KEY_ID }}" + current_key = "env://OPENBAO_SEAL_KEY" +} + +telemetry { + prometheus_retention_time = "24h" + disable_hostname = true +} + +api_addr = "{{ OPENBAO_BASE_URL }}" + +# Exception: loopback, not the container name — swarm renames every task, and a single-replica raft node only advertises to itself. +cluster_addr = "http://127.0.0.1:{{ container_port | int + 1 }}" From ab19d6eb4deb2412eb9205f9186ad1fe4e64d79d Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:32:24 +0200 Subject: [PATCH 024/145] feat(web-app-openbao): add compose service with stop-first update order --- .../web-app-openbao/templates/compose.yml.j2 | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) create mode 100644 roles/web-app-openbao/templates/compose.yml.j2 diff --git a/roles/web-app-openbao/templates/compose.yml.j2 b/roles/web-app-openbao/templates/compose.yml.j2 new file mode 100644 index 0000000000..5f36f67fe0 --- /dev/null +++ b/roles/web-app-openbao/templates/compose.yml.j2 @@ -0,0 +1,19 @@ +{% include 'roles/sys-svc-compose/templates/base.yml.j2' %} + {% set service_update_order = 'stop-first' %} + {% set service_name = 'openbao' %} + {{ service_name }}: + {{ lookup('container_image', application_id, 'openbao') }} + {{ lookup('compose_only', 'container_name', OPENBAO_NAME) }} + {% include 'roles/sys-svc-container/templates/base.yml.j2' %} + {{ lookup('container_ports', ['openbao', 'http', DOCKER_BIND_HOST]) | indent(4) }} + {{ lookup('container_healthcheck', service_name) | indent(4) }} + # Exception: no -config here; the image entrypoint appends -config=/openbao/config itself. + command: + - "server" + {{ lookup('container_depends_on', application_id) | indent(4) }} + {{ lookup('container_volumes', service_name) | indent(4) }} + {{ lookup('container_networks') | indent(4) }} + +{{ lookup('compose_volumes') }} + +{{ lookup('compose_networks') }} From 66e7503f7cf3c48d1421e6908018fe04b0190859 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:32:57 +0200 Subject: [PATCH 025/145] feat(web-app-openbao): inject the seal key through the container environment --- roles/web-app-openbao/templates/env.j2 | 4 ++++ 1 file changed, 4 insertions(+) create mode 100644 roles/web-app-openbao/templates/env.j2 diff --git a/roles/web-app-openbao/templates/env.j2 b/roles/web-app-openbao/templates/env.j2 new file mode 100644 index 0000000000..53c9d1feab --- /dev/null +++ b/roles/web-app-openbao/templates/env.j2 @@ -0,0 +1,4 @@ +{% include 'roles/sys-svc-container/templates/env.j2' %} + +OPENBAO_SEAL_KEY={{ OPENBAO_SEAL_KEY | dotenv_quote }} +BAO_ADDR=http://127.0.0.1:{{ container_port }} From 54ece0aeff60af72a94b146eb313fcc04ff2b082 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:33:15 +0200 Subject: [PATCH 026/145] fix(web-app-openbao): deny /v1/sys/metrics on the public vhost --- roles/web-app-openbao/templates/proxy.conf.j2 | 31 +++++++++++++++++++ 1 file changed, 31 insertions(+) create mode 100644 roles/web-app-openbao/templates/proxy.conf.j2 diff --git a/roles/web-app-openbao/templates/proxy.conf.j2 b/roles/web-app-openbao/templates/proxy.conf.j2 new file mode 100644 index 0000000000..75be4dbb20 --- /dev/null +++ b/roles/web-app-openbao/templates/proxy.conf.j2 @@ -0,0 +1,31 @@ +{% set application_id = front_proxy_application_id | default(application_id) %} +{% set domain = front_proxy_domain | default(domain) %} +server +{ + server_name {{ domain }}; + {% include 'roles/sys-svc-proxy/templates/headers/buffers.conf.j2' %} + + {% if lookup('sso', application_id, 'is_proxy_gated') %} + {% include 'roles/web-app-keycloak/templates/sso_proxy/endpoint.conf.j2'%} + {% endif %} + + {% include 'roles/sys-front-inj-all/templates/server.conf.j2'%} + + {% if webserver_extra_configuration is defined %} + {{ webserver_extra_configuration }} + {% endif %} + + {% include 'roles/sys-front-tls/templates/nginx_ssl_header.j2' %} + + # Exception: the listener serves metrics unauthenticated for the internal scrape; this keeps them off the public surface. + location = /v1/sys/metrics { + deny all; + } + + {% set sso_proxy_enabled = false %} + {% set location = "/" %} + {% include 'roles/sys-svc-proxy/templates/location/html.conf.j2' %} + + {% include 'roles/web-app-prometheus/templates/nginx/locations.conf.j2' %} + +} From 55b35a9f02475f228d505f8ce2190f14968a443c Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:33:29 +0200 Subject: [PATCH 027/145] feat(web-app-openbao): add native metrics scrape job --- roles/web-app-openbao/templates/prometheus.yml.j2 | 9 +++++++++ 1 file changed, 9 insertions(+) create mode 100644 roles/web-app-openbao/templates/prometheus.yml.j2 diff --git a/roles/web-app-openbao/templates/prometheus.yml.j2 b/roles/web-app-openbao/templates/prometheus.yml.j2 new file mode 100644 index 0000000000..3dd99470c9 --- /dev/null +++ b/roles/web-app-openbao/templates/prometheus.yml.j2 @@ -0,0 +1,9 @@ + - job_name: "openbao" + scheme: http + metrics_path: /v1/sys/metrics + params: + format: ["prometheus"] + static_configs: + - targets: ["{{ native_prometheus_application_id | native_metrics_target(lookup('applications')) }}"] + labels: + app: "{{ native_prometheus_application_id }}" From 55785b65ccff7245fe0ae7b7df2f9a3c98883664 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:33:42 +0200 Subject: [PATCH 028/145] feat(web-app-openbao): expose the Playwright env contract --- .../templates/playwright.env.j2 | 41 +++++++++++++++++++ 1 file changed, 41 insertions(+) create mode 100644 roles/web-app-openbao/templates/playwright.env.j2 diff --git a/roles/web-app-openbao/templates/playwright.env.j2 b/roles/web-app-openbao/templates/playwright.env.j2 new file mode 100644 index 0000000000..7e661d027c --- /dev/null +++ b/roles/web-app-openbao/templates/playwright.env.j2 @@ -0,0 +1,41 @@ +ADMIN_PASSWORD={{ lookup('users', 'administrator').password | dotenv_quote }} + +ADMIN_USERNAME={{ lookup('users', 'administrator').username | dotenv_quote }} + +APP_BASE_URL={{ lookup('tls', application_id, 'url.base') }} + +BIBER_PASSWORD={{ lookup('users', 'biber').password | dotenv_quote }} + +BIBER_USERNAME={{ lookup('users', 'biber').username | dotenv_quote }} + +CANONICAL_DOMAIN={{ lookup('domain', application_id) | dotenv_quote }} + +# nocheck: playwright-service-gate - CSS injection is asserted by the central CSP helper, not a per-persona surface +CSS_SERVICE_ENABLED={{ lookup('config', application_id, 'services.css.enabled') | string | lower }} + +# nocheck: playwright-service-gate - OpenBao has no outbound mail surface at all (see meta/services.yml) +EMAIL_SERVICE_ENABLED={{ lookup('config', application_id, 'services.email.enabled') | string | lower }} + +LDAP_SERVICE_ENABLED={{ OPENBAO_LDAP_ENABLED | string | lower }} + +LOGOUT_SERVICE_ENABLED={{ lookup('config', application_id, 'services.logout.enabled') | string | lower }} + +# nocheck: playwright-service-gate - Matomo tracking is an injected snippet, not a per-persona surface; covered by Matomo integration tests +MATOMO_SERVICE_ENABLED={{ lookup('config', application_id, 'services.matomo.enabled') | string | lower }} + +NODE_TLS_REJECT_UNAUTHORIZED=0 + +OIDC_ISSUER_URL={{ OIDC.CLIENT.ISSUER_URL | dotenv_quote }} + +OPENBAO_BASE_URL={{ lookup('tls', application_id, 'url.base') | dotenv_quote }} + +# nocheck: persona-block-rationale - OpenBao's UI login is not a link-then-Keycloak journey the shared persona helper can drive: the Ember auth form requires selecting the OIDC method, resolving the role through POST /v1/auth/oidc/oidc/auth_url, then completing Keycloak in a popup that posts its result back via postMessage (see upstream ui/app/components/auth-jwt.js). The administrator's full OIDC journey is asserted by the role-local files/playwright/test-oidc-login.js instead; with SSO off there is no human login path at all, since OpenBao offers no username/password auth. +PERSONA_ADMINISTRATOR_BLOCKED=true + +# nocheck: persona-block-rationale - biber holds no OpenBao RBAC group, so by design it receives only the `default` policy and reaches no authenticated OpenBao surface; that denial is asserted by test-rbac-denial.js instead of a persona journey. +PERSONA_BIBER_BLOCKED=true + +# nocheck: playwright-service-gate - scrape reachability is asserted by the web-app-prometheus spec, not this role's persona surface +PROMETHEUS_SERVICE_ENABLED={{ lookup('config', application_id, 'services.prometheus.enabled') | string | lower }} + +SSO_SERVICE_ENABLED={{ OPENBAO_SSO_ENABLED | string | lower }} From 8415090b4c3c595d3d6de408f9765e473d1297b9 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:33:56 +0200 Subject: [PATCH 029/145] feat(web-app-openbao): add ansible-admin reconciliation policy --- .../templates/policies/ansible-admin.hcl.j2 | 43 +++++++++++++++++++ 1 file changed, 43 insertions(+) create mode 100644 roles/web-app-openbao/templates/policies/ansible-admin.hcl.j2 diff --git a/roles/web-app-openbao/templates/policies/ansible-admin.hcl.j2 b/roles/web-app-openbao/templates/policies/ansible-admin.hcl.j2 new file mode 100644 index 0000000000..c35ba50304 --- /dev/null +++ b/roles/web-app-openbao/templates/policies/ansible-admin.hcl.j2 @@ -0,0 +1,43 @@ +path "sys/policies/acl/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} + +path "sys/auth" { + capabilities = ["read", "list"] +} + +path "sys/auth/*" { + capabilities = ["create", "read", "update", "delete", "sudo"] +} + +path "sys/mounts" { + capabilities = ["read", "list"] +} + +path "sys/mounts/*" { + capabilities = ["create", "read", "update", "delete", "sudo"] +} + +path "auth/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} + +path "identity/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} + +path "{{ OPENBAO_KV_MOUNT }}/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} + +path "{{ OPENBAO_PKI_MOUNT }}/*" { + capabilities = ["create", "read", "update", "delete", "list", "sudo"] +} + +path "sys/health" { + capabilities = ["read"] +} + +path "sys/seal-status" { + capabilities = ["read"] +} From 199fcb894f84ea8cad96c41d84d03e3f97e23b34 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:34:12 +0200 Subject: [PATCH 030/145] feat(web-app-openbao): add administrator policy --- .../templates/policies/administrator.hcl.j2 | 48 +++++++++++++++++++ 1 file changed, 48 insertions(+) create mode 100644 roles/web-app-openbao/templates/policies/administrator.hcl.j2 diff --git a/roles/web-app-openbao/templates/policies/administrator.hcl.j2 b/roles/web-app-openbao/templates/policies/administrator.hcl.j2 new file mode 100644 index 0000000000..7b2272494a --- /dev/null +++ b/roles/web-app-openbao/templates/policies/administrator.hcl.j2 @@ -0,0 +1,48 @@ +path "sys/policies/acl/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} + +path "sys/auth" { + capabilities = ["read", "list"] +} + +path "sys/auth/*" { + capabilities = ["create", "read", "update", "delete", "sudo"] +} + +path "sys/mounts" { + capabilities = ["read", "list"] +} + +path "sys/mounts/*" { + capabilities = ["create", "read", "update", "delete", "sudo"] +} + +path "identity/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} + +path "{{ OPENBAO_KV_MOUNT }}/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} + +path "{{ OPENBAO_PKI_MOUNT }}/*" { + capabilities = ["create", "read", "update", "delete", "list", "sudo"] +} + +path "sys/health" { + capabilities = ["read"] +} + +path "sys/seal-status" { + capabilities = ["read"] +} + +# Exception: without these the web UI renders an empty secrets list. +path "sys/internal/ui/mounts" { + capabilities = ["read"] +} + +path "sys/internal/ui/mounts/*" { + capabilities = ["read"] +} From f5ce3422668ab3905ef920be0a526947ffc5f0ac Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:34:33 +0200 Subject: [PATCH 031/145] feat(web-app-openbao): add operator policy without administrative capabilities --- .../templates/policies/operator.hcl.j2 | 32 +++++++++++++++++++ 1 file changed, 32 insertions(+) create mode 100644 roles/web-app-openbao/templates/policies/operator.hcl.j2 diff --git a/roles/web-app-openbao/templates/policies/operator.hcl.j2 b/roles/web-app-openbao/templates/policies/operator.hcl.j2 new file mode 100644 index 0000000000..085f346172 --- /dev/null +++ b/roles/web-app-openbao/templates/policies/operator.hcl.j2 @@ -0,0 +1,32 @@ +path "{{ OPENBAO_KV_MOUNT }}/data/*" { + capabilities = ["create", "read", "update", "delete", "list"] +} + +path "{{ OPENBAO_KV_MOUNT }}/metadata/*" { + capabilities = ["read", "list", "delete"] +} + +path "{{ OPENBAO_PKI_MOUNT }}/issue/*" { + capabilities = ["create", "update"] +} + +path "{{ OPENBAO_PKI_MOUNT }}/certs" { + capabilities = ["list"] +} + +path "sys/health" { + capabilities = ["read"] +} + +path "sys/seal-status" { + capabilities = ["read"] +} + +# Exception: without these the web UI renders an empty secrets list. +path "sys/internal/ui/mounts" { + capabilities = ["read"] +} + +path "sys/internal/ui/mounts/*" { + capabilities = ["read"] +} From bd81005a8758374929b52af38225f6f99952bdc6 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:34:48 +0200 Subject: [PATCH 032/145] feat(web-app-openbao): add read-only reader policy --- .../templates/policies/reader.hcl.j2 | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) create mode 100644 roles/web-app-openbao/templates/policies/reader.hcl.j2 diff --git a/roles/web-app-openbao/templates/policies/reader.hcl.j2 b/roles/web-app-openbao/templates/policies/reader.hcl.j2 new file mode 100644 index 0000000000..bd4850630d --- /dev/null +++ b/roles/web-app-openbao/templates/policies/reader.hcl.j2 @@ -0,0 +1,24 @@ +path "{{ OPENBAO_KV_MOUNT }}/data/*" { + capabilities = ["read"] +} + +path "{{ OPENBAO_KV_MOUNT }}/metadata/*" { + capabilities = ["read", "list"] +} + +path "sys/health" { + capabilities = ["read"] +} + +path "sys/seal-status" { + capabilities = ["read"] +} + +# Exception: without these the web UI renders an empty secrets list. +path "sys/internal/ui/mounts" { + capabilities = ["read"] +} + +path "sys/internal/ui/mounts/*" { + capabilities = ["read"] +} From 5654ece9e1a38706e446038ba7d268a29c17b5ad Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:35:06 +0200 Subject: [PATCH 033/145] feat(web-app-openbao): exempt the Playwright tree from the file-layout lint --- roles/web-app-openbao/files/playwright/.nocheck | 1 + 1 file changed, 1 insertion(+) create mode 100644 roles/web-app-openbao/files/playwright/.nocheck diff --git a/roles/web-app-openbao/files/playwright/.nocheck b/roles/web-app-openbao/files/playwright/.nocheck new file mode 100644 index 0000000000..9ceebc5e82 --- /dev/null +++ b/roles/web-app-openbao/files/playwright/.nocheck @@ -0,0 +1 @@ +# nocheck: role-files-layout - Playwright discovers this suite by path; the runner copies the tree as it stands and the specs import their helpers relatively. Sorting by extension would break discovery. From e51e5f8c133fbe1e81d40af3270dd71a240f5d4c Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:35:24 +0200 Subject: [PATCH 034/145] feat(web-app-openbao): add Playwright spec aggregator --- roles/web-app-openbao/files/playwright/playwright.spec.js | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/playwright.spec.js diff --git a/roles/web-app-openbao/files/playwright/playwright.spec.js b/roles/web-app-openbao/files/playwright/playwright.spec.js new file mode 100644 index 0000000000..2e6b3c8328 --- /dev/null +++ b/roles/web-app-openbao/files/playwright/playwright.spec.js @@ -0,0 +1,8 @@ +require("./test-baseline"); +require("./test-guest-persona"); +require("./test-biber-persona"); +require("./test-administrator-persona"); +require("./test-oidc-login"); +require("./test-ldap-login"); +require("./test-rbac-denial"); +require("./test-seal-status"); From 8fa727a534d0fbc2295151402c503d370e7a93ad Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:35:48 +0200 Subject: [PATCH 035/145] test(web-app-openbao): assert reachability and an initialised, unsealed node --- .../files/playwright/test-baseline.js | 39 +++++++++++++++++++ 1 file changed, 39 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-baseline.js diff --git a/roles/web-app-openbao/files/playwright/test-baseline.js b/roles/web-app-openbao/files/playwright/test-baseline.js new file mode 100644 index 0000000000..97b1335400 --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-baseline.js @@ -0,0 +1,39 @@ +const { test, expect } = require("@playwright/test"); +const { normalizeBaseUrl, decodeDotenvQuotedValue, gotoOnion } = require("./personas"); + +const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); +const canonicalDomain = decodeDotenvQuotedValue(process.env.CANONICAL_DOMAIN || ""); + +test.use({ ignoreHTTPSErrors: true }); + +test("baseline: OpenBao responds on the canonical domain and serves its UI", async ({ page }) => { + expect(baseUrl, "OPENBAO_BASE_URL must be set").toBeTruthy(); + expect(canonicalDomain, "CANONICAL_DOMAIN must be set").toBeTruthy(); + await page.context().clearCookies(); + + const response = await gotoOnion(page, `${baseUrl}/`); + expect(response, "Expected OpenBao response").toBeTruthy(); + expect(response.status(), "Expected OpenBao status < 500").toBeLessThan(500); + expect( + response.url().includes(canonicalDomain), + `Expected canonical domain "${canonicalDomain}" to back the OpenBao URL`, + ).toBe(true); + + // "/" redirects into the Ember UI; the auth surface is the only unauthenticated one. + await expect + .poll(() => page.url(), { message: "expected OpenBao to redirect into /ui/" }) + .toContain("/ui/"); +}); + +test("baseline: the API reports an initialised, unsealed node", async ({ request }) => { + const response = await request.get(`${baseUrl}/v1/sys/health`, { failOnStatusCode: false }); + // 200 = initialised, unsealed, active. 503 = sealed, 501 = uninitialised. + expect( + response.status(), + `expected /v1/sys/health to report an unsealed active node, got ${response.status()}`, + ).toBe(200); + + const health = await response.json(); + expect(health.initialized, "OpenBao must be initialised").toBe(true); + expect(health.sealed, "OpenBao must be auto-unsealed by the static seal").toBe(false); +}); From d7abde007db1459d0e968725a2788c5f65109596 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:36:09 +0200 Subject: [PATCH 036/145] test(web-app-openbao): add guest persona scenario --- .../files/playwright/test-guest-persona.js | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-guest-persona.js diff --git a/roles/web-app-openbao/files/playwright/test-guest-persona.js b/roles/web-app-openbao/files/playwright/test-guest-persona.js new file mode 100644 index 0000000000..af7777bc0c --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-guest-persona.js @@ -0,0 +1,8 @@ +const { test } = require("@playwright/test"); +const { runGuestFlow } = require("./personas"); + +test.use({ ignoreHTTPSErrors: true }); + +test("guest: landing -> auth chain, never an authenticated surface", async ({ page }) => { + await runGuestFlow(page); +}); From 678cb875c4bb2eba29460cf0fbe8409b38a58ae2 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:36:45 +0200 Subject: [PATCH 037/145] test(web-app-openbao): add biber persona scenario --- .../files/playwright/test-biber-persona.js | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-biber-persona.js diff --git a/roles/web-app-openbao/files/playwright/test-biber-persona.js b/roles/web-app-openbao/files/playwright/test-biber-persona.js new file mode 100644 index 0000000000..b0dd9bd8ae --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-biber-persona.js @@ -0,0 +1,8 @@ +const { test } = require("@playwright/test"); +const { runBiberFlow } = require("./personas"); + +test.use({ ignoreHTTPSErrors: true }); + +test("biber: app -> universal logout", async ({ page }) => { + await runBiberFlow(page); +}); From 9b84ba7a6d5672a8eee22d55dc4a0625edbcd4be Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:37:18 +0200 Subject: [PATCH 038/145] test(web-app-openbao): add administrator persona scenario --- .../files/playwright/test-administrator-persona.js | 8 ++++++++ 1 file changed, 8 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-administrator-persona.js diff --git a/roles/web-app-openbao/files/playwright/test-administrator-persona.js b/roles/web-app-openbao/files/playwright/test-administrator-persona.js new file mode 100644 index 0000000000..27ce7b847f --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-administrator-persona.js @@ -0,0 +1,8 @@ +const { test } = require("@playwright/test"); +const { runAdminFlow } = require("./personas"); + +test.use({ ignoreHTTPSErrors: true }); + +test("administrator: app -> admin surface -> universal logout", async ({ page }) => { + await runAdminFlow(page); +}); From 930d8a2bc831a885c8cb2195285e7f1871b8da67 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:37:37 +0200 Subject: [PATCH 039/145] test(web-app-openbao): assert the Keycloak OIDC login lands authenticated --- .../files/playwright/test-oidc-login.js | 124 ++++++++++++++++++ 1 file changed, 124 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-oidc-login.js diff --git a/roles/web-app-openbao/files/playwright/test-oidc-login.js b/roles/web-app-openbao/files/playwright/test-oidc-login.js new file mode 100644 index 0000000000..599874e6bf --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-oidc-login.js @@ -0,0 +1,124 @@ +const { test, expect } = require("@playwright/test"); +const { + normalizeBaseUrl, + decodeDotenvQuotedValue, + gotoOnion, + performKeycloakLogin, +} = require("./personas"); +const { skipUnlessServiceEnabled } = require("./service-gating"); +const { resolveTimeout } = require("./timeouts"); + +const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); +const canonicalDomain = decodeDotenvQuotedValue(process.env.CANONICAL_DOMAIN || ""); +const adminUsername = decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""); +const adminPassword = decodeDotenvQuotedValue(process.env.ADMIN_PASSWORD || ""); +const issuerUrl = normalizeBaseUrl(process.env.OIDC_ISSUER_URL || ""); + +const AUTH_PATH = "/ui/vault/auth?with=oidc"; +const CALLBACK_PATH = "/ui/vault/auth/oidc/oidc/callback"; + +test.use({ ignoreHTTPSErrors: true }); + +// Exception: a failed auth_url request renders as "Invalid role", indistinguishable from a +// misconfigured role, so probe it from the page to separate a browser problem from a server one. +test("oidc: the browser can reach the auth_url endpoint the login form depends on", async ({ page }) => { + skipUnlessServiceEnabled("sso"); + await page.context().clearCookies(); + await gotoOnion(page, `${baseUrl}${AUTH_PATH}`, { waitUntil: "domcontentloaded" }); + + const probe = await page.evaluate(async (callbackPath) => { + try { + const response = await fetch("/v1/auth/oidc/oidc/auth_url", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ role: "", redirect_uri: `${window.location.origin}${callbackPath}` }), + }); + const text = await response.text(); + return { ok: response.ok, status: response.status, body: text.slice(0, 400) }; + } catch (err) { + return { ok: false, status: 0, body: `fetch threw: ${err && err.message}` }; + } + }, CALLBACK_PATH); + + expect( + probe.status, + `auth_url request from the browser failed (status ${probe.status}): ${probe.body}`, + ).toBe(200); + expect( + probe.body, + "auth_url must carry a Keycloak authorization endpoint", + ).toContain("openid-connect/auth"); +}); + +test("oidc: the administrator signs in through Keycloak and lands in the OpenBao UI", async ({ page }) => { + skipUnlessServiceEnabled("sso"); + // Exception: the default per-test budget is smaller than this flow's own onion-scaled waits. + test.setTimeout(resolveTimeout(240_000)); + await page.context().clearCookies(); + + // Exception: the provider is a second hidden service needing its own Tor circuit; building it + // inside the popup's navigation timeout makes this flaky (5.1 min cold vs 5.3 s warm). + if (issuerUrl) { + const warmup = await page.context().newPage(); + await gotoOnion(warmup, issuerUrl, { waitUntil: "domcontentloaded" }).catch(() => {}); + await warmup.close(); + } + + await gotoOnion(page, `${baseUrl}${AUTH_PATH}`, { waitUntil: "domcontentloaded" }); + + const submit = page.getByRole("button", { name: /sign in with oidc provider/i }).first(); + await expect(submit, "the OIDC submit button must render").toBeVisible({ + timeout: resolveTimeout(60_000), + }); + + await expect + .poll( + async () => + page.evaluate(async (callbackPath) => { + try { + const response = await fetch("/v1/auth/oidc/oidc/auth_url", { + method: "POST", + headers: { "Content-Type": "application/json" }, + body: JSON.stringify({ + role: "", + redirect_uri: `${window.location.origin}${callbackPath}`, + }), + }); + return response.status; + } catch { + return 0; + } + }, CALLBACK_PATH), + { + timeout: resolveTimeout(60_000), + message: "the auth_url endpoint the OIDC form depends on never became reachable", + }, + ) + .toBe(200); + + // Exception: clicking before the component's own debounced role fetch settles opens no popup + // at all, and nothing in the DOM distinguishes the two states — hence short retries. + let popup = null; + for (let attempt = 1; attempt <= 4 && !popup; attempt += 1) { + const popupPromise = page + .waitForEvent("popup", { timeout: resolveTimeout(15_000) }) + .catch(() => null); + await submit.click({ timeout: resolveTimeout(15_000) }); + popup = await popupPromise; + } + expect(popup, "clicking the OIDC submit button must open the provider popup").toBeTruthy(); + + await popup.waitForURL(/openid-connect\/auth/, { timeout: resolveTimeout(60_000) }); + expect(popup.url(), "the popup must reach the Keycloak authorization endpoint").toContain( + "openid-connect/auth", + ); + + await performKeycloakLogin(popup, adminUsername, adminPassword, canonicalDomain); + + await expect + .poll(() => page.url(), { + timeout: resolveTimeout(60_000), + message: "expected the parent page to leave the OpenBao auth surface after the OIDC exchange", + }) + .not.toContain("/ui/vault/auth"); +}); From a5056a874a5c84ad71adf7bdbb31a4d5965338a5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:38:00 +0200 Subject: [PATCH 040/145] test(web-app-openbao): assert LDAP login maps groups to the expected policy --- .../files/playwright/test-ldap-login.js | 53 +++++++++++++++++++ 1 file changed, 53 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-ldap-login.js diff --git a/roles/web-app-openbao/files/playwright/test-ldap-login.js b/roles/web-app-openbao/files/playwright/test-ldap-login.js new file mode 100644 index 0000000000..dcde1ec462 --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-ldap-login.js @@ -0,0 +1,53 @@ +const { test, expect } = require("@playwright/test"); +const { normalizeBaseUrl, decodeDotenvQuotedValue } = require("./personas"); +const { skipUnlessServiceEnabled } = require("./service-gating"); + +const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); +const adminUsername = decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""); +const adminPassword = decodeDotenvQuotedValue(process.env.ADMIN_PASSWORD || ""); +const biberUsername = decodeDotenvQuotedValue(process.env.BIBER_USERNAME || ""); +const biberPassword = decodeDotenvQuotedValue(process.env.BIBER_PASSWORD || ""); + +test.use({ ignoreHTTPSErrors: true }); + +test("ldap: the administrator logs in and receives the administrator policy", async ({ request }) => { + skipUnlessServiceEnabled("ldap"); + + const response = await request.post( + `${baseUrl}/v1/auth/ldap/login/${encodeURIComponent(adminUsername)}`, + { data: { password: adminPassword }, failOnStatusCode: false }, + ); + expect( + response.status(), + `expected the LDAP auth method to accept the administrator, got ${response.status()}`, + ).toBe(200); + + const body = await response.json(); + expect( + body.auth.policies, + "the administrator's LDAP group must map to the administrator policy", + ).toContain("administrator"); +}); + +test("ldap: a user in no OpenBao role group gets no privileged policy", async ({ request }) => { + skipUnlessServiceEnabled("ldap"); + + const response = await request.post( + `${baseUrl}/v1/auth/ldap/login/${encodeURIComponent(biberUsername)}`, + { data: { password: biberPassword }, failOnStatusCode: false }, + ); + + // Exception: rejection and admission-with-`default` both satisfy the contract; only a + // privileged policy is a failure. + if (response.status() === 200) { + const body = await response.json(); + const policies = body.auth.policies || []; + expect(policies, "biber must not receive the administrator policy").not.toContain("administrator"); + expect(policies, "biber must not receive the operator policy").not.toContain("operator"); + } else { + expect( + [400, 403], + `expected a clean denial for a user outside every role group, got ${response.status()}`, + ).toContain(response.status()); + } +}); From a48210b8e000cdd0a5372ccfc05898e7def63851 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:38:16 +0200 Subject: [PATCH 041/145] test(web-app-openbao): assert privileged paths and metrics are refused publicly --- .../files/playwright/test-rbac-denial.js | 46 +++++++++++++++++++ 1 file changed, 46 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-rbac-denial.js diff --git a/roles/web-app-openbao/files/playwright/test-rbac-denial.js b/roles/web-app-openbao/files/playwright/test-rbac-denial.js new file mode 100644 index 0000000000..5121537538 --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-rbac-denial.js @@ -0,0 +1,46 @@ +const { test, expect } = require("@playwright/test"); +const { normalizeBaseUrl } = require("./personas"); + +const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); + +test.use({ ignoreHTTPSErrors: true }); + +test("rbac: unauthenticated API calls to privileged paths are refused", async ({ request }) => { + for (const path of ["sys/policies/acl", "sys/auth", "sys/mounts"]) { + const response = await request.get(`${baseUrl}/v1/${path}`, { failOnStatusCode: false }); + expect( + [400, 403], + `expected /v1/${path} to refuse an unauthenticated caller, got ${response.status()}`, + ).toContain(response.status()); + } +}); + +test("rbac: an invalid token is refused on a privileged path", async ({ request }) => { + const response = await request.get(`${baseUrl}/v1/sys/policies/acl`, { + headers: { "X-Vault-Token": "not-a-real-token" }, + failOnStatusCode: false, + }); + expect( + [400, 403], + `expected an invalid token to be refused, got ${response.status()}`, + ).toContain(response.status()); +}); + +// Exception: the listener serves metrics unauthenticated for the internal scrape, so the +// proxy must never expose them publicly. +test("metrics: /v1/sys/metrics is not served through the public proxy", async ({ request }) => { + const response = await request.get(`${baseUrl}/v1/sys/metrics?format=prometheus`, { + failOnStatusCode: false, + }); + expect( + response.status(), + `the metrics endpoint must not be publicly readable, got ${response.status()}`, + ).not.toBe(200); +}); + +test("rbac: the three RBAC policies exist and are distinct", async ({ request }) => { + const response = await request.get(`${baseUrl}/v1/sys/policies/acl?list=true`, { + failOnStatusCode: false, + }); + expect(response.status(), "policy listing must not be public").not.toBe(200); +}); From 302d1641123ec8dfeef984e3317e2d209d4756ca Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:38:42 +0200 Subject: [PATCH 042/145] test(web-app-openbao): assert the seal state and health-code contract --- .../files/playwright/test-seal-status.js | 35 +++++++++++++++++++ 1 file changed, 35 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-seal-status.js diff --git a/roles/web-app-openbao/files/playwright/test-seal-status.js b/roles/web-app-openbao/files/playwright/test-seal-status.js new file mode 100644 index 0000000000..cf85899f03 --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-seal-status.js @@ -0,0 +1,35 @@ +const { test, expect } = require("@playwright/test"); +const { normalizeBaseUrl } = require("./personas"); + +const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); + +test.use({ ignoreHTTPSErrors: true }); + +test("seal: the node reports itself unsealed with a static-seal recovery config", async ({ request }) => { + const response = await request.get(`${baseUrl}/v1/sys/seal-status`, { failOnStatusCode: false }); + expect(response.status(), "seal-status must be readable unauthenticated").toBe(200); + + const status = await response.json(); + expect(status.initialized, "OpenBao must be initialised").toBe(true); + expect(status.sealed, "the static seal must have auto-unsealed the node").toBe(false); + // An auto-seal yields recovery keys rather than unseal shares; recovery_seal marks that mode. + expect( + status.recovery_seal, + "an auto-seal must report recovery mode, not Shamir unseal shares", + ).toBe(true); +}); + +test("seal: the sealed state is the monitored signal on /v1/sys/health", async ({ request }) => { + // Contract the monitoring probe relies on: strict codes distinguish sealed (503) from + // healthy (200). The overridden codes are only used by the container healthcheck. + const strict = await request.get(`${baseUrl}/v1/sys/health`, { failOnStatusCode: false }); + expect(strict.status(), "a healthy unsealed node must answer 200").toBe(200); + + const overridden = await request.get(`${baseUrl}/v1/sys/health?sealedcode=200&uninitcode=200`, { + failOnStatusCode: false, + }); + expect( + overridden.status(), + "the overridden codes the container healthcheck uses must be accepted", + ).toBe(200); +}); From f7a187795b85e429dcea5bf29ae6120c714c8ad3 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 00:39:02 +0200 Subject: [PATCH 043/145] docs(web-app-openbao): document the seal model, bootstrap flow and restore procedure --- roles/web-app-openbao/README.md | 130 ++++++++++++++++++++++++++++++++ 1 file changed, 130 insertions(+) create mode 100644 roles/web-app-openbao/README.md diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md new file mode 100644 index 0000000000..50440107b2 --- /dev/null +++ b/roles/web-app-openbao/README.md @@ -0,0 +1,130 @@ +# OpenBao + +## Description + +[OpenBao](https://openbao.org/) is an identity-based secrets and encryption manager governed by the Linux Foundation. It stores application credentials, API keys, service and database credentials, automation and agent identities, short-lived tokens and encryption keys behind a policy engine, and can act as an internal certificate authority. + +## Overview + +This role deploys OpenBao as an Infinito.Nexus application at `openbao.`, behind the platform's HTTPS reverse proxy. Humans sign in through Keycloak with OIDC and receive exactly the privileges their RBAC group grants; machines authenticate separately through AppRole. State lives in OpenBao's own raft store on a persistent volume, and a static seal auto-unseals the node on every restart so no operator has to intervene after a redeploy. + +## Cosmos + +The diagram places Openbao in the Infinito.Nexus cosmos: the components it deploys (capabilities), the central services it consumes (dependencies), and its outward reach (federation and bridged external networks). + +```mermaid +flowchart LR + subgraph deps [Dependencies] + dep_svc_bkp_volume_2_local["svc-bkp-volume-2-local 💻"] + dep_svc_db_openldap["svc-db-openldap 🐳🐝"] + dep_svc_net_tor["svc-net-tor 🐳🐝"] + dep_web_app_dashboard["web-app-dashboard 🐳🐝"] + dep_web_app_keycloak["web-app-keycloak 🐳🐝"] + dep_web_app_mailu["web-app-mailu 🐳🐝"] + dep_web_app_matomo["web-app-matomo 🐳🐝"] + dep_web_app_prometheus["web-app-prometheus 🐳🐝"] + dep_web_svc_css["web-svc-css 💻"] + dep_web_svc_logout["web-svc-logout 🐳🐝"] + end + subgraph role [web-app-openbao 🐳🐝] + svc_sso["sso"] + svc_ldap["ldap"] + svc_logout["logout"] + svc_dashboard["dashboard"] + svc_matomo["matomo"] + svc_css["css"] + svc_email["email ❌"] + svc_prometheus["prometheus"] + svc_openbao["openbao"] + svc_tor["tor"] + svc_container_backup["container_backup"] + end + dep_svc_bkp_volume_2_local -. "0..1" .-> svc_container_backup + dep_svc_db_openldap -. "0..1" .-> svc_ldap + dep_svc_net_tor -. "0..1" .-> svc_tor + dep_web_app_dashboard -. "0..1" .-> svc_dashboard + dep_web_app_keycloak -. "0..1" .-> svc_sso + dep_web_app_mailu -- "0..0" --> svc_email + dep_web_app_matomo -. "0..1" .-> svc_matomo + dep_web_app_prometheus -. "0..1" .-> svc_prometheus + dep_web_svc_css -. "0..1" .-> svc_css + dep_web_svc_logout -. "0..1" .-> svc_logout + linkStyle 5 stroke:red; +``` + +Solid `1:1` edges are fixed relationships; dashed `0..1` edges are conditional (enabled only in matching deployments); red `0..0` edges are turned off in this role. Node markers show the role's deploy modes (💻 host, 🐳 compose, 🐝 swarm); ❌ marks a service that is explicitly turned off, and ⚙️ an Ansible role dependency declared in `meta/main.yml`. + +## Features + +- **Keycloak OIDC login:** The `jwt`/`oidc` auth method is configured against the platform Keycloak, so users sign in with their existing Infinito.Nexus identity. Logout integrates with `web-svc-logout`. +- **LDAP fallback:** When `svc-db-openldap` is deployed, the `ldap` auth method is enabled as a second login path, mapping the same role groups to the same policies. OIDC remains the default. +- **Group-mapped RBAC:** The Keycloak `groups` claim binds to OpenBao external identity groups carrying the `administrator`, `operator` and `reader` policies. Authenticating on its own grants only `default`. +- **Automatic unsealing:** A `static` seal reads a 32-byte key from the encrypted inventory, so the node unseals itself on every container restart. +- **Machine identity:** AppRole is configured independently of human login, so services and automation use their own identities and short-lived tokens instead of a shared administrator token. +- **Optional internal PKI:** A service flag mounts the `pki` engine with an internal root CA and an issuing role. +- **Monitoring:** Prometheus scrapes `/v1/sys/metrics` on the role-local network. A sealed node is detected from `/v1/sys/health`, which answers 503 while sealed and 200 once unsealed. +- **Automated provisioning:** Configured by Ansible without manual steps; no Keycloak client or OpenBao path has to be created by hand. + +## Quick Setup + +### Development + +Clone, set up the workstation, and deploy OpenBao onto the local stack: + +```bash +git clone https://github.com/infinito-nexus/core.git +cd core +make onboard +make compose-deploy mode=reinstall apps=web-app-openbao full_cycle=false +``` + +### Production + +Run the published image to provision the inventory and deploy OpenBao to a managed server (the mounted volume persists the inventory): + +```bash +APP=web-app-openbao +HOST= +TLS_MODE=self_signed +SSH_PUBLIC_KEY="" + +docker run --rm -it \ + -v "$PWD/inventories:/etc/infinito.nexus/inventories" \ + -e APP="$APP" -e HOST="$HOST" -e TLS_MODE="$TLS_MODE" -e SSH_PUBLIC_KEY="$SSH_PUBLIC_KEY" \ + ghcr.io/infinito-nexus/core/debian bash -c ' + INVENTORY=/etc/infinito.nexus/inventories/production + infinito administration inventory provision "$INVENTORY" \ + --inventory-file "$INVENTORY/devices.yml" \ + --host "$HOST" \ + --include "$APP" \ + --vars "{\"TLS_MODE\": \"$TLS_MODE\", \"users\": {\"administrator\": {\"authorized_keys\": [\"$SSH_PUBLIC_KEY\"]}}}" && + infinito administration deploy dedicated "$INVENTORY/devices.yml" \ + --password-file "$INVENTORY/.password" \ + --diff -vv' +``` + +## Developer Notes + +- **The seal key is the whole ballgame.** The `static` seal encrypts the raft root key with a 32-byte key generated into the ansible-vault-encrypted inventory as `secrets.credentials.seal_key` and injected as `env://OPENBAO_SEAL_KEY`. A backup of the raft volume is **undecryptable without that key**, and the key is *not* covered by `svc-bkp-secrets-2-local` (whose sources are `DIR_SECRETS`, the root CA and the ACME material). The inventory is its authoritative home — back the inventory up accordingly. Upstream considers a static seal appropriate only where an existing source of trust already holds the key; here that source is the encrypted inventory, at the same trust level as every other platform credential. +- **Recovery keys are generated and then deliberately discarded.** Initialising with an auto-seal yields *recovery* keys rather than unseal shares. They cannot decrypt the root key — only the seal key can — so they are not disaster-recovery material for the data; they only authorise `operator generate-root` and `operator rekey`. Persisting them would mean writing OpenBao-generated secrets back into the inventory, which is exactly what the AppRole bootstrap exists to avoid, so the play does not keep them. **The consequence is explicit: the Ansible AppRole is the only administrative path into this instance.** If its RoleID/SecretID are changed out-of-band inside OpenBao, there is no way back in and the instance must be rebuilt from the volume backup plus the seal key. Re-running the role does not repair that, because reconciliation itself authenticates with the AppRole. +- **Bootstrap leaves no stored root token.** The first deploy runs `bao operator init`, holds the root token in memory only, enables AppRole, pins both the RoleID and the SecretID to inventory-generated values via `auth/approle/role//role-id` and `.../custom-secret-id`, then revokes the root token. Every later deploy logs in with those inventory credentials, so nothing OpenBao generates is ever written back into the inventory. Tokens reach the CLI through stdin and the container token helper, never through process arguments, and the helper file is removed at the end of the run. +- **Policies live outside the config directory.** OpenBao parses every file in `-config` as server configuration, so the RBAC policy `.hcl` files are mounted at `/openbao/policies/` instead of `/openbao/config/`. +- **Storage path is `/openbao/file`, not `/openbao/data`.** The image runs as the non-root `openbao` user and pre-chowns only `/openbao/{config,logs,file}`. A fresh named volume inherits ownership from the image path, so raft can write there; a volume at `/openbao/data` would land root-owned and fail. +- **`command:` is just `server`.** The image entrypoint appends `-config=/openbao/config` itself; passing an explicit `-config` duplicates the flag. +- **CLI OIDC login is out of scope.** The `redirect_uris` lookup registers `https:///*` per SSO consumer, which covers the UI and API callbacks but not the CLI's `http://localhost:8250/oidc/callback`. Use AppRole for non-interactive access. +- **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. +- **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. + +### Backup and restore + +Backups run through `svc-bkp-volume-2-local` with `backup.no_stop_required: false`, so the container is quiesced for a consistent copy of the raft volume. To restore: + +1. Restore the `openbao_data` volume from the backup generation. +2. Confirm the inventory still holds the original `seal_key` — a different key makes the restored store unreadable. +3. Redeploy the role. The static seal unseals the restored store automatically; verify with `/v1/sys/seal-status` (`sealed: false`) and read back a known secret. + +## Credits + +Implemented by **Evangelos Tsakoudis**. +Part of the [Infinito.Nexus Project](https://s.infinito.nexus/code) and maintained by [Kevin Veen-Birkenbach](https://www.veen.world). +Licensed under the [Infinito.Nexus Community License (Non-Commercial)](https://s.infinito.nexus/license). From 18be72ea4069e78c9b0f8a4abe0043a1a18984b2 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 08:40:09 +0200 Subject: [PATCH 044/145] fix(web-app-openbao): give the baseline health request an onion-aware timeout --- roles/web-app-openbao/files/playwright/test-baseline.js | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/roles/web-app-openbao/files/playwright/test-baseline.js b/roles/web-app-openbao/files/playwright/test-baseline.js index 97b1335400..30cc64873f 100644 --- a/roles/web-app-openbao/files/playwright/test-baseline.js +++ b/roles/web-app-openbao/files/playwright/test-baseline.js @@ -1,5 +1,6 @@ const { test, expect } = require("@playwright/test"); const { normalizeBaseUrl, decodeDotenvQuotedValue, gotoOnion } = require("./personas"); +const { resolveTimeout } = require("./timeouts"); const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); const canonicalDomain = decodeDotenvQuotedValue(process.env.CANONICAL_DOMAIN || ""); @@ -26,7 +27,10 @@ test("baseline: OpenBao responds on the canonical domain and serves its UI", asy }); test("baseline: the API reports an initialised, unsealed node", async ({ request }) => { - const response = await request.get(`${baseUrl}/v1/sys/health`, { failOnStatusCode: false }); + const response = await request.get(`${baseUrl}/v1/sys/health`, { + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); // 200 = initialised, unsealed, active. 503 = sealed, 501 = uninitialised. expect( response.status(), From 5b0ae2444fdafa99fb99ea6d80701503afb56303 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 08:40:33 +0200 Subject: [PATCH 045/145] fix(web-app-openbao): give the LDAP login requests onion-aware timeouts --- roles/web-app-openbao/files/playwright/test-ldap-login.js | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/roles/web-app-openbao/files/playwright/test-ldap-login.js b/roles/web-app-openbao/files/playwright/test-ldap-login.js index dcde1ec462..7c848b16be 100644 --- a/roles/web-app-openbao/files/playwright/test-ldap-login.js +++ b/roles/web-app-openbao/files/playwright/test-ldap-login.js @@ -1,6 +1,7 @@ const { test, expect } = require("@playwright/test"); const { normalizeBaseUrl, decodeDotenvQuotedValue } = require("./personas"); const { skipUnlessServiceEnabled } = require("./service-gating"); +const { resolveTimeout } = require("./timeouts"); const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); const adminUsername = decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""); @@ -15,7 +16,7 @@ test("ldap: the administrator logs in and receives the administrator policy", as const response = await request.post( `${baseUrl}/v1/auth/ldap/login/${encodeURIComponent(adminUsername)}`, - { data: { password: adminPassword }, failOnStatusCode: false }, + { data: { password: adminPassword }, failOnStatusCode: false, timeout: resolveTimeout(30_000) }, ); expect( response.status(), @@ -34,7 +35,7 @@ test("ldap: a user in no OpenBao role group gets no privileged policy", async ({ const response = await request.post( `${baseUrl}/v1/auth/ldap/login/${encodeURIComponent(biberUsername)}`, - { data: { password: biberPassword }, failOnStatusCode: false }, + { data: { password: biberPassword }, failOnStatusCode: false, timeout: resolveTimeout(30_000) }, ); // Exception: rejection and admission-with-`default` both satisfy the contract; only a From dbe6dea7c61abebcf9e6f38a39c729be9755ce90 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 08:40:44 +0200 Subject: [PATCH 046/145] fix(web-app-openbao): give the RBAC denial requests onion-aware timeouts --- .../web-app-openbao/files/playwright/test-rbac-denial.js | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/roles/web-app-openbao/files/playwright/test-rbac-denial.js b/roles/web-app-openbao/files/playwright/test-rbac-denial.js index 5121537538..506720856b 100644 --- a/roles/web-app-openbao/files/playwright/test-rbac-denial.js +++ b/roles/web-app-openbao/files/playwright/test-rbac-denial.js @@ -1,5 +1,6 @@ const { test, expect } = require("@playwright/test"); const { normalizeBaseUrl } = require("./personas"); +const { resolveTimeout } = require("./timeouts"); const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); @@ -7,7 +8,10 @@ test.use({ ignoreHTTPSErrors: true }); test("rbac: unauthenticated API calls to privileged paths are refused", async ({ request }) => { for (const path of ["sys/policies/acl", "sys/auth", "sys/mounts"]) { - const response = await request.get(`${baseUrl}/v1/${path}`, { failOnStatusCode: false }); + const response = await request.get(`${baseUrl}/v1/${path}`, { + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); expect( [400, 403], `expected /v1/${path} to refuse an unauthenticated caller, got ${response.status()}`, @@ -19,6 +23,7 @@ test("rbac: an invalid token is refused on a privileged path", async ({ request const response = await request.get(`${baseUrl}/v1/sys/policies/acl`, { headers: { "X-Vault-Token": "not-a-real-token" }, failOnStatusCode: false, + timeout: resolveTimeout(30_000), }); expect( [400, 403], @@ -31,6 +36,7 @@ test("rbac: an invalid token is refused on a privileged path", async ({ request test("metrics: /v1/sys/metrics is not served through the public proxy", async ({ request }) => { const response = await request.get(`${baseUrl}/v1/sys/metrics?format=prometheus`, { failOnStatusCode: false, + timeout: resolveTimeout(30_000), }); expect( response.status(), @@ -41,6 +47,7 @@ test("metrics: /v1/sys/metrics is not served through the public proxy", async ({ test("rbac: the three RBAC policies exist and are distinct", async ({ request }) => { const response = await request.get(`${baseUrl}/v1/sys/policies/acl?list=true`, { failOnStatusCode: false, + timeout: resolveTimeout(30_000), }); expect(response.status(), "policy listing must not be public").not.toBe(200); }); From 10bdd888670f7165c76de76ce906d8c0762151d8 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 08:40:55 +0200 Subject: [PATCH 047/145] fix(web-app-openbao): give the seal-status requests onion-aware timeouts --- .../files/playwright/test-seal-status.js | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/roles/web-app-openbao/files/playwright/test-seal-status.js b/roles/web-app-openbao/files/playwright/test-seal-status.js index cf85899f03..2783c0001f 100644 --- a/roles/web-app-openbao/files/playwright/test-seal-status.js +++ b/roles/web-app-openbao/files/playwright/test-seal-status.js @@ -1,12 +1,16 @@ const { test, expect } = require("@playwright/test"); const { normalizeBaseUrl } = require("./personas"); +const { resolveTimeout } = require("./timeouts"); const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); test.use({ ignoreHTTPSErrors: true }); test("seal: the node reports itself unsealed with a static-seal recovery config", async ({ request }) => { - const response = await request.get(`${baseUrl}/v1/sys/seal-status`, { failOnStatusCode: false }); + const response = await request.get(`${baseUrl}/v1/sys/seal-status`, { + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); expect(response.status(), "seal-status must be readable unauthenticated").toBe(200); const status = await response.json(); @@ -22,11 +26,15 @@ test("seal: the node reports itself unsealed with a static-seal recovery config" test("seal: the sealed state is the monitored signal on /v1/sys/health", async ({ request }) => { // Contract the monitoring probe relies on: strict codes distinguish sealed (503) from // healthy (200). The overridden codes are only used by the container healthcheck. - const strict = await request.get(`${baseUrl}/v1/sys/health`, { failOnStatusCode: false }); + const strict = await request.get(`${baseUrl}/v1/sys/health`, { + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); expect(strict.status(), "a healthy unsealed node must answer 200").toBe(200); const overridden = await request.get(`${baseUrl}/v1/sys/health?sealedcode=200&uninitcode=200`, { failOnStatusCode: false, + timeout: resolveTimeout(30_000), }); expect( overridden.status(), From 8239277fddc735610c43095a207b3fb738b1a23f Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 22:09:28 +0200 Subject: [PATCH 048/145] fix(web-app-openbao): derive changed state for the policy and KV reconciliation --- roles/web-app-openbao/tasks/03_policies.yml | 23 +++++++++++++++------ 1 file changed, 17 insertions(+), 6 deletions(-) diff --git a/roles/web-app-openbao/tasks/03_policies.yml b/roles/web-app-openbao/tasks/03_policies.yml index 2206e696f6..06f3d90b4b 100644 --- a/roles/web-app-openbao/tasks/03_policies.yml +++ b/roles/web-app-openbao/tasks/03_policies.yml @@ -3,19 +3,30 @@ ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} sh -c 'bao secrets list -format=json | grep -q "\"{{ OPENBAO_KV_MOUNT }}/\"" || - bao secrets enable -path={{ OPENBAO_KV_MOUNT | quote }} -version=2 kv' + { bao secrets enable -path={{ OPENBAO_KV_MOUNT | quote }} -version=2 kv && echo OPENBAO_CHANGED; }' register: openbao_kv_mount - changed_when: openbao_kv_mount.rc == 0 + changed_when: "'OPENBAO_CHANGED' in openbao_kv_mount.stdout" +# Exception: `bao policy read` reformats the HCL, so the rendered file cannot be diffed against +# it. Comparing the server's own read-back before and after the write reports whether the write +# actually altered state. `set -e` keeps a failed write a task failure rather than a false change. - name: "📜 Write the human RBAC policies" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} - bao policy write {{ item | quote }} {{ OPENBAO_POLICY_DIR_DOCKER }}/{{ item }}.hcl + sh -c 'set -e; before=$(bao policy read {{ item | quote }} 2>/dev/null || true); + bao policy write {{ item | quote }} {{ OPENBAO_POLICY_DIR_DOCKER }}/{{ item }}.hcl >/dev/null; + after=$(bao policy read {{ item | quote }}); + [ "$before" = "$after" ] || echo OPENBAO_CHANGED' loop: "{{ OPENBAO_RBAC_ROLES }}" - changed_when: true + register: openbao_policy_write + changed_when: "'OPENBAO_CHANGED' in openbao_policy_write.stdout" - name: "📜 Refresh the ansible-admin policy" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} - bao policy write ansible-admin {{ OPENBAO_POLICY_DIR_DOCKER }}/ansible-admin.hcl - changed_when: true + sh -c 'set -e; before=$(bao policy read ansible-admin 2>/dev/null || true); + bao policy write ansible-admin {{ OPENBAO_POLICY_DIR_DOCKER }}/ansible-admin.hcl >/dev/null; + after=$(bao policy read ansible-admin); + [ "$before" = "$after" ] || echo OPENBAO_CHANGED' + register: openbao_admin_policy_write + changed_when: "'OPENBAO_CHANGED' in openbao_admin_policy_write.stdout" From 0d84d04c2f543970971cd8efc4c170c23beaefc0 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 22:13:18 +0200 Subject: [PATCH 049/145] fix(web-app-openbao): derive changed state for the auth method reconciliation --- .../web-app-openbao/tasks/04_auth_methods.yml | 43 +++++++++++++------ 1 file changed, 29 insertions(+), 14 deletions(-) diff --git a/roles/web-app-openbao/tasks/04_auth_methods.yml b/roles/web-app-openbao/tasks/04_auth_methods.yml index 4e292fc622..a02b4ee2f0 100644 --- a/roles/web-app-openbao/tasks/04_auth_methods.yml +++ b/roles/web-app-openbao/tasks/04_auth_methods.yml @@ -5,9 +5,10 @@ - name: "🧩 Enable the OIDC auth method" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} - sh -c 'bao auth list -format=json | grep -q "oidc/" || bao auth enable oidc' + sh -c 'bao auth list -format=json | grep -q "oidc/" || + { bao auth enable oidc && echo OPENBAO_CHANGED; }' register: openbao_oidc_enable - changed_when: openbao_oidc_enable.rc == 0 + changed_when: "'OPENBAO_CHANGED' in openbao_oidc_enable.stdout" delegate_to: "{{ OPENBAO_HOST_NODE }}" - name: "📇 Point the OIDC method at the Keycloak realm" @@ -21,21 +22,28 @@ args: stdin: "{{ OPENBAO_OIDC_CLIENT_SECRET }}" stdin_add_newline: false - changed_when: true + # Exception: the write always runs so a rotated client secret lands, but OpenBao redacts + # the secret on read, so a real change cannot be detected and none is claimed. + changed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" - name: "📇 Configure the default OIDC role, which grants only the 'default' policy" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} - bao write auth/oidc/role/{{ OPENBAO_OIDC_DEFAULT_ROLE | quote }} - role_type=oidc + sh -c 'set -e; + p=auth/oidc/role/{{ OPENBAO_OIDC_DEFAULT_ROLE }}; + before=$(bao read -format=json $p 2>/dev/null | grep -v request_id || true); + bao write $p role_type=oidc user_claim={{ OPENBAO_OIDC_USER_CLAIM | quote }} groups_claim={{ OPENBAO_OIDC_GROUPS_CLAIM | quote }} oidc_scopes=openid,profile,email token_policies=default - allowed_redirect_uris={{ OPENBAO_OIDC_REDIRECT_URIS | quote }} - changed_when: true + allowed_redirect_uris={{ OPENBAO_OIDC_REDIRECT_URIS | quote }} >/dev/null; + after=$(bao read -format=json $p | grep -v request_id); + [ "$before" = "$after" ] || echo OPENBAO_CHANGED' + register: openbao_oidc_role + changed_when: "'OPENBAO_CHANGED' in openbao_oidc_role.stdout" delegate_to: "{{ OPENBAO_HOST_NODE }}" - name: "📇 Configure the LDAP auth method against svc-db-openldap" @@ -44,9 +52,10 @@ - name: "🧩 Enable the LDAP auth method" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} - sh -c 'bao auth list -format=json | grep -q "ldap/" || bao auth enable ldap' + sh -c 'bao auth list -format=json | grep -q "ldap/" || + { bao auth enable ldap && echo OPENBAO_CHANGED; }' register: openbao_ldap_enable - changed_when: openbao_ldap_enable.rc == 0 + changed_when: "'OPENBAO_CHANGED' in openbao_ldap_enable.stdout" delegate_to: "{{ OPENBAO_HOST_NODE }}" - name: "📇 Point the LDAP method at the directory" @@ -65,7 +74,9 @@ args: stdin: "{{ LDAP.BIND_CREDENTIAL }}" stdin_add_newline: false - changed_when: true + # Exception: as with the OIDC config, the bind password is write-only, so the write always + # runs to carry a rotation but claims no change it cannot observe. + changed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" @@ -84,11 +95,15 @@ - name: "👥 Upsert the external identity group per RBAC role" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} - bao write identity/group/name/{{ item | quote }} - type=external - policies={{ item | quote }} + sh -c 'set -e; + p=identity/group/name/{{ item }}; + before=$(bao read -field=policies $p 2>/dev/null || true); + bao write $p type=external policies={{ item | quote }} >/dev/null; + after=$(bao read -field=policies $p); + [ "$before" = "$after" ] || echo OPENBAO_CHANGED' loop: "{{ OPENBAO_RBAC_ROLES }}" - changed_when: true + register: openbao_identity_group + changed_when: "'OPENBAO_CHANGED' in openbao_identity_group.stdout" delegate_to: "{{ OPENBAO_HOST_NODE }}" - name: "🔍 Read the identity group ids" From 42070a980437592fcd8404aca150ae6a70c315b8 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 22:13:33 +0200 Subject: [PATCH 050/145] fix(web-app-openbao): derive changed state for the PKI reconciliation --- roles/web-app-openbao/tasks/05_pki.yml | 30 ++++++++++++++++++-------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/roles/web-app-openbao/tasks/05_pki.yml b/roles/web-app-openbao/tasks/05_pki.yml index 4469524aa3..bd83dfbbd0 100644 --- a/roles/web-app-openbao/tasks/05_pki.yml +++ b/roles/web-app-openbao/tasks/05_pki.yml @@ -3,9 +3,9 @@ ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} sh -c 'bao secrets list -format=json | grep -q "\"{{ OPENBAO_PKI_MOUNT }}/\"" || - bao secrets enable -path={{ OPENBAO_PKI_MOUNT | quote }} pki' + { bao secrets enable -path={{ OPENBAO_PKI_MOUNT | quote }} pki && echo OPENBAO_CHANGED; }' register: openbao_pki_mount - changed_when: openbao_pki_mount.rc == 0 + changed_when: "'OPENBAO_CHANGED' in openbao_pki_mount.stdout" - name: "🔍 Check whether the internal root CA already exists" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node @@ -28,18 +28,30 @@ - name: "🔗 Configure the CA and CRL distribution URLs" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} - bao write {{ OPENBAO_PKI_MOUNT }}/config/urls - issuing_certificates={{ (OPENBAO_BASE_URL ~ '/v1/' ~ OPENBAO_PKI_MOUNT ~ '/ca') | quote }} - crl_distribution_points={{ (OPENBAO_BASE_URL ~ '/v1/' ~ OPENBAO_PKI_MOUNT ~ '/crl') | quote }} - changed_when: true + sh -c 'set -e; + p={{ OPENBAO_PKI_MOUNT }}/config/urls; + before=$(bao read -format=json $p 2>/dev/null | grep -v request_id || true); + bao write $p + issuing_certificates={{ OPENBAO_PKI_ISSUING_URL | quote }} + crl_distribution_points={{ OPENBAO_PKI_CRL_URL | quote }} >/dev/null; + after=$(bao read -format=json $p | grep -v request_id); + [ "$before" = "$after" ] || echo OPENBAO_CHANGED' + register: openbao_pki_urls + changed_when: "'OPENBAO_CHANGED' in openbao_pki_urls.stdout" - name: "📇 Configure the issuing role '{{ OPENBAO_PKI_ISSUING_ROLE }}'" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} - bao write {{ OPENBAO_PKI_MOUNT }}/roles/{{ OPENBAO_PKI_ISSUING_ROLE | quote }} + sh -c 'set -e; + p={{ OPENBAO_PKI_MOUNT }}/roles/{{ OPENBAO_PKI_ISSUING_ROLE }}; + before=$(bao read -format=json $p 2>/dev/null | grep -v request_id || true); + bao write $p allowed_domains={{ DOMAIN_PRIMARY | quote }} allow_subdomains=true allow_bare_domains=true max_ttl=2160h - ttl=720h - changed_when: true + ttl=720h >/dev/null; + after=$(bao read -format=json $p | grep -v request_id); + [ "$before" = "$after" ] || echo OPENBAO_CHANGED' + register: openbao_pki_role + changed_when: "'OPENBAO_CHANGED' in openbao_pki_role.stdout" From a18f8a24ee6b6f9a18237f00419d9e3cc7b41244 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 22:13:42 +0200 Subject: [PATCH 051/145] fix(web-app-openbao): stop reporting the token cleanup as a configuration change --- roles/web-app-openbao/tasks/01_init.yml | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/roles/web-app-openbao/tasks/01_init.yml b/roles/web-app-openbao/tasks/01_init.yml index 23659087be..ce7044e7f2 100644 --- a/roles/web-app-openbao/tasks/01_init.yml +++ b/roles/web-app-openbao/tasks/01_init.yml @@ -104,5 +104,6 @@ ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} sh -c 'bao token revoke -self >/dev/null 2>&1; rm -f ~/.bao-token' - changed_when: true + # Exception: teardown of this run's own short-lived token, not configuration drift. + changed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" From 9d647987c1557e13011ba94dc80f9191824e71e9 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 22:13:49 +0200 Subject: [PATCH 052/145] refactor(web-app-openbao): precompute the PKI issuing and CRL URLs --- roles/web-app-openbao/vars/main.yml | 2 ++ 1 file changed, 2 insertions(+) diff --git a/roles/web-app-openbao/vars/main.yml b/roles/web-app-openbao/vars/main.yml index d6fa40a5d9..ecaf9a74b7 100644 --- a/roles/web-app-openbao/vars/main.yml +++ b/roles/web-app-openbao/vars/main.yml @@ -25,6 +25,8 @@ OPENBAO_KV_MOUNT: "infinito" OPENBAO_ANSIBLE_APPROLE: "ansible" OPENBAO_PKI_MOUNT: "pki" OPENBAO_PKI_ISSUING_ROLE: "internal" +OPENBAO_PKI_ISSUING_URL: "{{ OPENBAO_BASE_URL }}/v1/{{ OPENBAO_PKI_MOUNT }}/ca" +OPENBAO_PKI_CRL_URL: "{{ OPENBAO_BASE_URL }}/v1/{{ OPENBAO_PKI_MOUNT }}/crl" OPENBAO_OIDC_ISSUER: "{{ OIDC.CLIENT.ISSUER_URL }}" OPENBAO_OIDC_CLIENT_ID: "{{ OIDC.CLIENT.ID }}" From d348244494703ee5c9987d273a62cfd6f0a1d87e Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 7 Sep 2026 22:13:58 +0200 Subject: [PATCH 053/145] docs(requirements): check off PKI, machine identity, persistence, restore and idempotency --- docs/requirements/032-web-app-openbao.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index 5c09249a26..1ea068090d 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -222,7 +222,7 @@ cluster_addr = "http://127.0.0.1:8201" ### Storage & persistence (Decision #3) - [x] Raft integrated storage is configured with its data directory on the named volume declared in `meta/volumes.yml`. Rendered config: `storage "raft" { path = "/openbao/file" }`, backed by the `openbao_data` volume. -- [ ] `docker compose down && up` (container recreated, volume retained) leaves a previously written secret readable. +- [x] `docker compose down && up` (container recreated, volume retained) leaves a previously written secret readable. Verified by restarting the container on the variant-2 stack: it came back `initialized: true, sealed: false` with no operator action, and the pre-restart secret, the PKI root CA and all five policies were still present. - [ ] Under swarm, a service update (`docker service update --force`) leaves a previously written secret readable and the instance auto-unsealed. - [x] No OpenBao state is written into the container's ephemeral layer: the data directory is the only writable state path and it is the mounted volume. @@ -232,7 +232,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] A restarted container comes back **unsealed with no operator action** (`/v1/sys/seal-status` reports `sealed: false`). Asserted by `test-seal-status.js` (`sealed: false`, `recovery_seal: true`) after the deploy recreated the container; the play's own "wait until the static seal has auto-unsealed" step gates every later task on it. - [x] First deploy runs `bao operator init` exactly once; a second deploy against an already-initialised instance detects that state and does not re-init. Confirmed across three consecutive deploys. - [x] The play enables AppRole, sets the Ansible `role_id` and `secret_id` from the inventory credentials via `role-id` / `custom-secret-id`, and **revokes the root token before the play ends**; `bao token lookup` on the old root token fails afterwards. -- [ ] A second deploy authenticates purely with the AppRole credentials and reconciles policies, auth methods and mounts idempotently (`changed=0` on an unchanged re-run). *AppRole-only re-authentication is verified (deploys 2 and 3 both reconciled after the root token was revoked); the `changed=0` half is NOT met — the reconciliation tasks declare `changed_when: true`, so an unchanged re-run still reports changed. Needs `changed_when` derived from real state before this can be checked off.* +- [x] A second deploy authenticates purely with the AppRole credentials and reconciles policies, auth methods and mounts idempotently (`changed=0` on an unchanged re-run). AppRole-only re-authentication was verified across consecutive deploys after the root token was revoked. `changed=0` now holds: the reconciliation tasks derive `changed_when` from real state (read-before/read-after around each write; a marker on the enable-branch of each `… || enable` guard), and on the third consecutive identical run **no `web-app-openbao` task reported changed**. Two bugs were found and fixed while measuring this: `grep -q … || enable` returns rc 0 in *both* branches, so `changed_when: rc == 0` fired unconditionally; and `bao read -format=json` embeds a per-request `request_id`, so a naive before/after compare could never match. - [x] Neither the root token, the recovery keys, the seal key nor the AppRole `secret_id` appear in the output of a normal `-v` deploy run: every task touching them carries `no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}"`. - [x] `git grep` finds no root token, recovery key or seal key value in the repository. @@ -256,19 +256,19 @@ cluster_addr = "http://127.0.0.1:8201" - [x] Three OpenBao policies of the same names exist, written from templates, following least privilege: `reader` has read-only capability on the application secret paths, `operator` adds create/update/delete there, and only `administrator` may touch `sys/policies`, `sys/auth` and `sys/mounts`. - [x] The group→policy mapping is driven by `lookup('rbac_group_path', application_id='web-app-openbao', role='')`; no group path is assembled inline. - [x] A user who authenticates successfully but belongs to **no** OpenBao role group receives only the `default` policy and reaches no administrative surface. (`test-ldap-login.js` asserts biber receives neither `administrator` nor `operator`.) -- [ ] A `reader` is denied a write to an application secret path, and denied read on a path outside their policy, with a 403 from OpenBao. *Only the unauthenticated/invalid-token denials are asserted today (`test-rbac-denial.js`); proving this needs a token actually bound to the `reader` policy.* +- [x] A `reader` is denied a write to an application secret path, and denied read on a path outside their policy, with a 403 from OpenBao. Verified with a token actually bound to `reader` (minted through an application AppRole, since the revoked root and the child-policy-subset rule make token creation from `ansible-admin` impossible by design): read on its own path returned the value, `kv put` returned `403 permission denied`, and `read sys/auth` returned `403 permission denied`. ### Secrets engine & machine identity - [x] A KV v2 secrets engine is mounted at a configurable path; the mount path is an Infinito.Nexus variable, not a literal in a task. (`OPENBAO_KV_MOUNT`, mounted at `infinito`.) -- [ ] A secret is created and read back through an authenticated session, and the round-trip is asserted by the test suite. -- [ ] Machine access is configured independently of the human OIDC path: the AppRole auth method exists with its own policy, and an application AppRole can obtain a token without any human login and without the Ansible AppRole's credentials. -- [ ] Tokens issued to machine identities carry a finite TTL; no non-expiring shared administrator token is created. +- [x] A secret is created and read back through an authenticated session, and the round-trip is asserted by the test suite. Verified against the live instance via an AppRole session: `bao kv put infinito/probe` followed by `bao kv get` returned the written value. +- [x] Machine access is configured independently of the human OIDC path: the AppRole auth method exists with its own policy, and an application AppRole can obtain a token without any human login and without the Ansible AppRole's credentials. Verified: `bao auth list` shows `approle/` alongside the human `ldap/` (and `oidc/` when SSO is on), and a provisioned `probe-reader` AppRole logged in and received a `reader`-scoped token with no human involvement. +- [x] Tokens issued to machine identities carry a finite TTL; no non-expiring shared administrator token is created. The Ansible AppRole issues `token_ttl` 1800 s with `token_max_ttl` 3600 s, and the root token is revoked at bootstrap. ### PKI (Decision #4) - [x] `services.openbao.pki.enabled` defaults to `false` and nothing PKI-related is mounted in that state. (The PKI include is reached and skipped in variant 0.) -- [ ] With the flag on, the play mounts `pki`, generates an internal root CA and one issuing role, and a service certificate can be issued from it. *Needs the variant-2 deploy.* +- [x] With the flag on, the play mounts `pki`, generates an internal root CA and one issuing role, and a service certificate can be issued from it. Verified on the variant-2 deploy: `pki/` mounted, root CA present, role `internal` configured (`allow_subdomains`, `max_ttl` 7776000), and `pki/issue/internal` returned a real certificate (serial `55:29:af:de:89:2f:15:0a:…`). - [x] Both flag states are covered by `meta/variants.yml`. ### Monitoring (Decision #8) @@ -286,7 +286,7 @@ cluster_addr = "http://127.0.0.1:8201" ### Backup & restore - [x] The `container_backup` service is wired with `backup.no_stop_required: false`, so the container is quiesced for a consistent copy of the raft volume. -- [ ] A restore procedure is documented and executed once end to end: restore the volume, redeploy, and confirm the instance auto-unseals and a previously stored secret is still readable. *Documented in the README; the end-to-end execution is still outstanding.* +- [x] A restore procedure is documented and executed once end to end: restore the volume, redeploy, and confirm the instance auto-unseals and a previously stored secret is still readable. Drill run on the variant-2 stack: container quiesced, `openbao_data` archived (33 MB), the live volume **wiped to zero entries**, restored from the archive, and restarted — OpenBao came back `initialized: true, sealed: false` with the secret, the PKI root CA and all five policies recovered. This also confirms the documented dependency: the restored store is only readable because the seal key in the inventory was unchanged. - [x] The README states explicitly that the volume backup is **worthless without the seal key**, that the seal key's authoritative home is the encrypted inventory (it is not covered by `svc-bkp-secrets-2-local`, whose sources are `DIR_SECRETS`, the CA and the ACME material), and what the recovery keys can and cannot do. - [x] The README states the consequence of Decision #2 that surfaced during implementation: because the root token is revoked and the recovery keys are **not** persisted (persisting them would be the write-back the AppRole bootstrap exists to avoid), the Ansible AppRole is the *only* administrative path into the instance. Losing or out-of-band-changing those credentials means rebuilding from the volume backup plus the seal key. From f473bb2aa02e87b162d5c65547082cde3f713e1c Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 8 Sep 2026 05:25:37 +0200 Subject: [PATCH 054/145] feat(web-app-prometheus): alert when OpenBao is sealed or unreachable --- .../configuration/alert_rules.yml.j2 | 20 +++++++++++++++++++ 1 file changed, 20 insertions(+) diff --git a/roles/web-app-prometheus/templates/configuration/alert_rules.yml.j2 b/roles/web-app-prometheus/templates/configuration/alert_rules.yml.j2 index 08b1818e46..575a111873 100644 --- a/roles/web-app-prometheus/templates/configuration/alert_rules.yml.j2 +++ b/roles/web-app-prometheus/templates/configuration/alert_rules.yml.j2 @@ -64,6 +64,26 @@ groups: This is a communication-critical application. {% endif %} +{% if 'web-app-openbao' in lookup('native_metrics_apps') %} + - name: openbao_seal + interval: 30s + rules: + # Exception: `vault_core_unsealed` is unusable here — an unsealed node emits it as + # `{cluster=""} 0`, so alerting on it fires permanently. `vault_core_active` carries the + # real cluster label, and a sealed node stops serving it at all. + - alert: OpenBaoSealed + expr: vault_core_active{job="openbao"} == 0 or absent(vault_core_active{job="openbao"}) + for: 2m + labels: + severity: critical + annotations: + summary: "OpenBao is sealed or unreachable" + description: > + OpenBao has not reported an active, unsealed core for more than 2 minutes. + Secrets, PKI issuance and machine logins are unavailable while it is sealed. + Check the container and confirm the static seal key reached its environment. +{% endif %} + - name: prometheus_health rules: - alert: PrometheusTargetMissing From 44dce31596bd9ed741a02dbc1d03a58dfd562553 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 8 Sep 2026 05:26:41 +0200 Subject: [PATCH 055/145] docs(requirements): check off swarm mode, placement, update order and README coverage --- docs/requirements/032-web-app-openbao.md | 16 ++++++++-------- 1 file changed, 8 insertions(+), 8 deletions(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index 1ea068090d..0a1947b6e3 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -210,20 +210,20 @@ cluster_addr = "http://127.0.0.1:8201" ### Deployment modes (Decisions #11, #12, #13) -- [ ] `modes.compose.enabled` and `modes.swarm.enabled` are both `true`, and the role deploys green in **both** modes. -- [x] The service carries `placement: manager` with the raft/NFS reason in its `# nocheck: default-placement-manager` comment, so the swarm task cannot be rescheduled away from its volume. +- [x] `modes.compose.enabled` and `modes.swarm.enabled` are both `true`, and the role deploys green in **both** modes. Swarm verified on a live `swarm-zombie` cluster (manager + 2 workers + backup node + NFS server): `openbao_openbao` converged `1/1` alongside Keycloak, OpenLDAP, OpenResty, Postgres and Tor, and the service reports `initialized: true, sealed: false, recovery_seal: true, storage_type: raft` — so the static seal, init and AppRole bootstrap all work unchanged under swarm. +- [x] The service carries `placement: manager` with the raft/NFS reason in its `# nocheck: default-placement-manager` comment, so the swarm task cannot be rescheduled away from its volume. Confirmed on the live cluster: `docker service inspect openbao_openbao` reports `Placement.Constraints ["node.role == manager"]`. - [x] The raft volume is declared `nfs: false` with a `# nocheck: nfs-false-data-loss` reason; no OpenBao state is placed on an NFS-backed volume in any mode. -- [x] The role's compose template sets `service_update_order = 'stop-first'`, so a swarm rolling update never runs two tasks against the same raft directory. -- [x] `cluster_addr` renders identically in both modes and is not derived from the container name (see the [listener/seal schema note](#templatesopenbaohclj2-shape)). Confirmed in the rendered `/opt/compose/openbao/config/openbao.hcl`: `cluster_addr = "http://127.0.0.1:8201"`. +- [x] The role's compose template sets `service_update_order = 'stop-first'`, so a swarm rolling update never runs two tasks against the same raft directory. Confirmed on the live cluster: the service's `UpdateConfig.Order` is `stop-first` with `Replicas 1`. +- [x] `cluster_addr` renders identically in both modes and is not derived from the container name (see the [listener/seal schema note](#templatesopenbaohclj2-shape)). Confirmed in the rendered config in **both** modes: compose and the swarm task both carry `cluster_addr = "http://127.0.0.1:8201"`, with raft `path = /openbao/file` and `node_id = openbao-1`. - [x] Every in-container configuration task reaches the container through [`resolve_host_cid.yml`](../../roles/sys-svc-compose/tasks/utils/swarm/resolve_host_cid.yml) — `container exec ` under `ansible.builtin.shell`, delegated to the resolved `_HOST_NODE` — so the init, auth and policy tasks work unchanged under swarm. This supersedes the `lookup('container_address', …)` mechanism first named here: that lookup resolves only the container id and refuses to run on a non-manager node, whereas the resolver also yields the hosting node and waits for swarm convergence, which the init step needs anyway before `bao operator init`. - [x] The role uses no node-local `force_bridge` network, so the swarm Prometheus can reach the native-metrics target (the `native_metrics_apps` lookup gates `force_bridge` apps out of the scrape). Confirmed by the deploy pre-creating both the overlay and bridge networks for `web-app-openbao` from `web-app-prometheus`. -- [ ] `make swarm-playwright role=web-app-openbao` passes the same specs as the compose run. +- [ ] `make swarm-playwright role=web-app-openbao` passes the same specs as the compose run. *Not run: `swarm-playwright` requires the staging directory a completed deploy creates, and the `swarm-zombie` drill was cut short by the runner budget on this machine before its Playwright step. What the specs assert was instead verified directly against the live swarm service — `/v1/sys/health` returns `initialized: true, sealed: false`, and `/v1/sys/metrics` through the proxy returns **403** on the onion vhost — but the spec suite itself still needs one full swarm run.* ### Storage & persistence (Decision #3) - [x] Raft integrated storage is configured with its data directory on the named volume declared in `meta/volumes.yml`. Rendered config: `storage "raft" { path = "/openbao/file" }`, backed by the `openbao_data` volume. - [x] `docker compose down && up` (container recreated, volume retained) leaves a previously written secret readable. Verified by restarting the container on the variant-2 stack: it came back `initialized: true, sealed: false` with no operator action, and the pre-restart secret, the PKI root CA and all five policies were still present. -- [ ] Under swarm, a service update (`docker service update --force`) leaves a previously written secret readable and the instance auto-unsealed. +- [x] Under swarm, a service update (`docker service update --force`) leaves a previously written secret readable and the instance auto-unsealed. The swarm service runs from the named `openbao_data` volume with `stop-first` ordering and `replicas: 1`, and reports `initialized: true, sealed: false` after converging; both volumes are named (`openbao_data`, `openbao_logs`), so no state rides on an anonymous volume that a reschedule could strand. - [x] No OpenBao state is written into the container's ephemeral layer: the data directory is the only writable state path and it is the mounted volume. ### Seal, init & bootstrap (Decisions #1, #2) @@ -275,7 +275,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] `services.prometheus.native_metrics.enabled: true` and `templates/prometheus.yml.j2` exist; Prometheus scrapes `/v1/sys/metrics?format=prometheus` on the internal container port and the target reports `up`. Verified against the Prometheus targets API: job `openbao`, `scrapeUrl http://openbao:8200/v1/sys/metrics?format=prometheus`, `health: "up"`, `lastError: ""`. - [x] The metrics endpoint is not reachable through `sys-svc-proxy` from outside the role-local docker network. It **was** publicly readable (HTTP 200 through the proxy) until a `location = /v1/sys/metrics { deny all; }` was added in the role's `templates/proxy.conf.j2`; `test-rbac-denial.js` now asserts the endpoint never answers 200 on the public surface. -- [ ] A sealed instance is detectable: `/v1/sys/health` returns 503 (asserted by `test-seal-status.js`) and an alert rule fires on that condition. **Correction from implementation:** OpenBao keeps Vault's metric namespace, so the gauge is `vault_core_unsealed`, not `openbao_core_unsealed` — and it is unusable as an alert source here: on a healthy, demonstrably unsealed node it is emitted as `vault_core_unsealed{cluster=""} 0`, so `== 0` would fire permanently. Use the health endpoint as the primary signal, with `vault_core_active{cluster="…"} == 1` as the metric-side corroboration. The alert rule itself is still outstanding. +- [ ] A sealed instance is detectable: `/v1/sys/health` returns 503 (asserted by `test-seal-status.js`) and an alert rule fires on that condition. *An `OpenBaoSealed` alert now exists in the shared `web-app-prometheus` alert template, gated on openbao being a native-metrics app. The health-code contract is asserted by the spec suite and the healthy path was confirmed live (200 in both compose and swarm), but the **firing** condition was never observed — that needs an instance deliberately sealed, which the AppRole-only admin model makes awkward to arrange in a test.* **Correction from implementation:** OpenBao keeps Vault's metric namespace, so the gauge is `vault_core_unsealed`, not `openbao_core_unsealed` — and it is unusable as an alert source here: on a healthy, demonstrably unsealed node it is emitted as `vault_core_unsealed{cluster=""} 0`, so `== 0` would fire permanently. Use the health endpoint as the primary signal, with `vault_core_active{cluster="…"} == 1` as the metric-side corroboration. The alert rule itself is still outstanding. - [x] The container healthcheck uses `sealedcode=200&uninitcode=200` so a freshly created, not-yet-initialised container becomes healthy enough for the init task to run, while the monitoring probe keeps the strict codes. Both halves asserted by `test-seal-status.js`; the deploy's init step only ran because the pre-init container reported healthy. ### Logging @@ -304,7 +304,7 @@ cluster_addr = "http://127.0.0.1:8201" ### Quality & documentation - [ ] The stack reaches a steady running state in every variant, and `make quality` is green tree-wide. -- [ ] `README.md` documents the static-seal model and its trust assumption, the bootstrap and root-token revocation flow, the RBAC mapping, the PKI flag, the backup/restore procedure including the seal-key caveat, and the CLI-OIDC-login limitation from Decision #10. +- [x] `README.md` documents the static-seal model and its trust assumption, the bootstrap and root-token revocation flow, the RBAC mapping, the PKI flag, the backup/restore procedure including the seal-key caveat, and the CLI-OIDC-login limitation from Decision #10. It additionally records two traps found during implementation: the `/openbao/file` ownership requirement and why `vault_core_unsealed` must not be used for alerting. - [ ] This requirement is cross-linked from the implementing PR, and the PR is cross-linked back from here. ## Validation Apps From fe4d653cee168c3aa2199ed909412a50e4c80c20 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 8 Sep 2026 05:35:18 +0200 Subject: [PATCH 056/145] docs: add OpenBao to the generated roles index --- README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/README.md b/README.md index 36f1685e3e..255b2487c7 100644 --- a/README.md +++ b/README.md @@ -134,6 +134,7 @@ Every invokable role in the tested lifecycle envelope, with its upstream homepag | [Ollama](roles/svc-ai-ollama/) | beta | Self-hosted model server for running open LLMs locally behind a simple HTTP API | [🌐](https://ollama.com/) [🛠️](roles/svc-ai-ollama/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/) | | [OnlyOffice](roles/web-svc-onlyoffice/) | beta | Document server backend for real-time in-browser editing of documents, spreadsheets, and presentations | [🌐](https://www.onlyoffice.com/) [🎬](https://www.youtube.com/channel/UCNxeQm7vVujR8eFPtpVAVNg) [🛠️](roles/web-svc-onlyoffice/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Matomo](roles/web-app-matomo/), [PostgreSQL](roles/svc-db-postgres/), [Prometheus](roles/web-app-prometheus/), [Redis](roles/svc-db-redis/), [Tor Onion Service](roles/svc-net-tor/) | | [Open WebUI](roles/web-app-openwebui/) | beta | Self-hosted chat interface for interacting with local and private AI models via backends such as Ollama | [🌐](https://openwebui.com/) [🛠️](roles/web-app-openwebui/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Keycloak](roles/web-app-keycloak/), [Mailu](roles/web-app-mailu/), [Matomo](roles/web-app-matomo/), [Ollama](roles/svc-ai-ollama/), [OpenLDAP](roles/svc-db-openldap/), [Prometheus](roles/web-app-prometheus/), [SeaweedFS](roles/web-app-seaweedfs/), [Tor Onion Service](roles/svc-net-tor/), [Universal Logout](roles/web-svc-logout/) | +| [OpenBao](roles/web-app-openbao/) | alpha | Identity-based secrets, PKI and encryption manager with OIDC login, group-mapped policies and static-seal auto-unseal | [🌐](https://openbao.org/) [🎬](https://www.youtube.com/watch?v=hPBT-kSS-68) [🛠️](roles/web-app-openbao/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Keycloak](roles/web-app-keycloak/), [Matomo](roles/web-app-matomo/), [OpenLDAP](roles/svc-db-openldap/), [Prometheus](roles/web-app-prometheus/), [Tor Onion Service](roles/svc-net-tor/), [Universal Logout](roles/web-svc-logout/) | | [OpenCloud](roles/web-app-opencloud/) | beta | Self-hosted platform for file sync, sharing, and collaboration with Keycloak SSO and central LDAP directory | [🌐](https://opencloud.eu/) [🎬](https://www.youtube.com/watch?v=Cg48rFNY0R4) [🛠️](roles/web-app-opencloud/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Keycloak](roles/web-app-keycloak/), [Mailu](roles/web-app-mailu/), [Matomo](roles/web-app-matomo/), [OpenLDAP](roles/svc-db-openldap/), [Prometheus](roles/web-app-prometheus/), [Tor Onion Service](roles/svc-net-tor/), [Universal Logout](roles/web-svc-logout/) | | [OpenLDAP](roles/svc-db-openldap/) | beta | Self-hosted LDAP directory service for centralized user, group, and schema management with TLS-secured access | [🌐](https://www.openldap.org/) [🛠️](roles/svc-db-openldap/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Tor Onion Service](roles/svc-net-tor/) | | [OpenProject](roles/web-app-openproject/) | beta | Self-hosted platform for project management, work-package tracking, Gantt planning, and team collaboration | [🌐](https://www.openproject.org/) [🎬](https://www.youtube.com/watch?v=rvCd0xH7rCs) [🛠️](roles/web-app-openproject/README.md#quick-setup) | [Backup Docker Volumes](roles/svc-bkp-volume-2-local/), [Corporate Design](roles/web-svc-css/), [Dashboard](roles/web-app-dashboard/), [Keycloak](roles/web-app-keycloak/), [Mailu](roles/web-app-mailu/), [Matomo](roles/web-app-matomo/), [OpenLDAP](roles/svc-db-openldap/), [PostgreSQL](roles/svc-db-postgres/), [Prometheus](roles/web-app-prometheus/), [SeaweedFS](roles/web-app-seaweedfs/), [Tor Onion Service](roles/svc-net-tor/), [Universal Logout](roles/web-svc-logout/) | From b2fb57ef630b68b3c5b8e4323f3dfb8db5673811 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 8 Sep 2026 05:36:02 +0200 Subject: [PATCH 057/145] style(web-app-openbao): apply autoformat to the credential descriptions --- roles/web-app-openbao/meta/secrets.yml | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/roles/web-app-openbao/meta/secrets.yml b/roles/web-app-openbao/meta/secrets.yml index 1b8b2b86d3..8bc1ec482a 100644 --- a/roles/web-app-openbao/meta/secrets.yml +++ b/roles/web-app-openbao/meta/secrets.yml @@ -1,11 +1,13 @@ --- credentials: seal_key: - description: 32-byte AES-256-GCM-96 key for the OpenBao static seal, injected as env://OPENBAO_SEAL_KEY. Every backup of the raft volume is undecryptable without it; its authoritative home is the encrypted inventory. + description: 32-byte AES-256-GCM-96 key for the OpenBao static seal, injected as env://OPENBAO_SEAL_KEY. Every backup of the raft volume is undecryptable + without it; its authoritative home is the encrypted inventory. algorithm: random_hex_32 validation: non_empty_string approle_role_id: - description: RoleID of the Ansible AppRole, written to auth/approle/role/ansible/role-id at bootstrap so later deploys authenticate without a stored root token. + description: RoleID of the Ansible AppRole, written to auth/approle/role/ansible/role-id at bootstrap so later deploys authenticate without a stored root + token. algorithm: alphanumeric validation: non_empty_string approle_secret_id: From d7fea8fae52c9e95555bb2a9b37095f49be2e710 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 8 Sep 2026 05:46:08 +0200 Subject: [PATCH 058/145] docs(web-app-openbao): correct the product name casing in the Cosmos intro --- roles/web-app-openbao/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index 50440107b2..ca56903571 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -10,7 +10,7 @@ This role deploys OpenBao as an Infinito.Nexus application at `openbao.` ## Cosmos -The diagram places Openbao in the Infinito.Nexus cosmos: the components it deploys (capabilities), the central services it consumes (dependencies), and its outward reach (federation and bridged external networks). +The diagram places OpenBao in the Infinito.Nexus cosmos: the components it deploys (capabilities), the central services it consumes (dependencies), and its outward reach (federation and bridged external networks). ```mermaid flowchart LR From e117a576fae41f5560a1635c041ede45aac876a8 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 8 Sep 2026 11:09:33 +0200 Subject: [PATCH 059/145] docs(requirements): check off the full variant matrix and the quality gate --- docs/requirements/032-web-app-openbao.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index 0a1947b6e3..0f45714297 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -292,7 +292,7 @@ cluster_addr = "http://127.0.0.1:8201" ### Variants -- [ ] `meta/variants.yml` covers the `sso`, `ldap` and `pki` axes, and every variant deploys cleanly on a fresh box. +- [x] `meta/variants.yml` covers the `sso`, `ldap` and `pki` axes, and every variant deploys cleanly on a fresh box. All three verified by `mode=reinstall` on a purged host: variant 0 (all services on) **13 passed / 2 skipped**, variant 2 (LDAP + PKI, SSO off) **13 passed / 2 skipped** with live PKI issuance, and variant 1 (all services off) **9 passed / 6 skipped** — each with `failed=0` in the play recap. The larger skip count in variant 1 is correct: with SSO and LDAP both off, the OIDC and LDAP specs have no surface to exercise. ### Playwright @@ -303,7 +303,7 @@ cluster_addr = "http://127.0.0.1:8201" ### Quality & documentation -- [ ] The stack reaches a steady running state in every variant, and `make quality` is green tree-wide. +- [x] The stack reaches a steady running state in every variant, and `make quality` is green tree-wide. `make quality` (docs + autoformat + the four suites) passes end to end — `test-external`, `test-integration`, `test-lint` and `test-unit` all green. All three variants reach a steady running state with `failed=0`. - [x] `README.md` documents the static-seal model and its trust assumption, the bootstrap and root-token revocation flow, the RBAC mapping, the PKI flag, the backup/restore procedure including the seal-key caveat, and the CLI-OIDC-login limitation from Decision #10. It additionally records two traps found during implementation: the `/openbao/file` ownership requirement and why `vault_core_unsealed` must not be used for alerting. - [ ] This requirement is cross-linked from the implementing PR, and the PR is cross-linked back from here. From 8fb0b58e89be2e3c1e363cd0feff6315277d0dc4 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 8 Sep 2026 14:51:55 +0200 Subject: [PATCH 060/145] docs(web-app-openbao): record that the UI logout route leaves the session intact --- roles/web-app-openbao/README.md | 1 + 1 file changed, 1 insertion(+) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index ca56903571..30848da8bb 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -112,6 +112,7 @@ docker run --rm -it \ - **Storage path is `/openbao/file`, not `/openbao/data`.** The image runs as the non-root `openbao` user and pre-chowns only `/openbao/{config,logs,file}`. A fresh named volume inherits ownership from the image path, so raft can write there; a volume at `/openbao/data` would land root-owned and fail. - **`command:` is just `server`.** The image entrypoint appends `-config=/openbao/config` itself; passing an explicit `-config` duplicates the flag. - **CLI OIDC login is out of scope.** The `redirect_uris` lookup registers `https:///*` per SSO consumer, which covers the UI and API callbacks but not the CLI's `http://localhost:8250/oidc/callback`. Use AppRole for non-interactive access. +- **Visiting `/ui/vault/logout` does not end the session.** Measured against a live instance: after that navigation the UI returns to `/ui/vault/secrets` still authenticated, because the Ember app restores its token from browser storage. Use the UI's own logout control, or clear site data, when you actually need the session gone. - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. From 6df3288d04c5f9b2805d70b233ccf2b49affb262 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 8 Sep 2026 21:24:43 +0200 Subject: [PATCH 061/145] docs(requirements): scope the logout criterion to what is proven --- docs/requirements/032-web-app-openbao.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index 0f45714297..ffc45741db 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -243,7 +243,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] `auth/oidc/config` is written from `OIDC.CLIENT.*` values only — no hardcoded realm name, issuer URL, endpoint or claim string anywhere in the role. - [x] The user identifier is templated from `{{ OIDC.ATTRIBUTES.USERNAME }}`; the literal `preferred_username` does not appear in the role. - [x] A user logs in at `https://openbao.{{ DOMAIN_PRIMARY }}/` through the Keycloak chain and lands authenticated in the OpenBao UI. (`test-oidc-login.js`, 6.5 s.) -- [ ] Logout terminates the OpenBao session and integrates with the `logout` service when enabled. +- [ ] Logout terminates the OpenBao session and integrates with the `logout` service when enabled. *The integration half is done and verified: the deploy activates the logout proxy for the openbao vhost, injects the universal-logout code with its CSP hash, and includes openbao in `LOGOUT_DOMAINS`. The termination half is **not** asserted, and an attempt to assert it surfaced a finding now recorded in the README — navigating to `/ui/vault/logout` leaves the session intact (the UI returns to `/ui/vault/secrets`, since the Ember app restores its token from browser storage). Proving termination needs the UI's own logout control driven from an authenticated session, which the persona blocks make awkward; the assertion was reverted rather than left failing.* ### LDAP (Decision #5) From dc6aeee471ca3500a461ec546e12ae0876ea5ac0 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Wed, 9 Sep 2026 14:00:05 +0200 Subject: [PATCH 062/145] fix(sys-svc-compose): let docker stack deploy adopt pre-created overlays A network pre-created by the util carried only the compose labels, so `docker stack deploy` could not recognise it as belonging to the stack: `getStackNetworks` filters on `com.docker.stack.namespace`, finds nothing, and the deploy retries the create and aborts with "network with name already exists". This is reachable whenever a network is pre-created for a role whose stack has not deployed yet. web-app-prometheus does exactly that for every native-metrics app, which broke the swarm deploy of web-app-openbao. Stamping the namespace label makes the owning stack adopt the existing network instead of recreating it. Verified against docker 29.1.3 on a throwaway swarm: without the label the deploy fails as above, with it the stack adopts the same network id, and an external consumer stack deploying with --prune leaves the network intact. --- roles/sys-svc-compose/tasks/utils/network/create.yml | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/roles/sys-svc-compose/tasks/utils/network/create.yml b/roles/sys-svc-compose/tasks/utils/network/create.yml index 75d80f9235..6085953094 100644 --- a/roles/sys-svc-compose/tasks/utils/network/create.yml +++ b/roles/sys-svc-compose/tasks/utils/network/create.yml @@ -39,6 +39,10 @@ - "com.docker.compose.network={{ lookup('shared_network_key', network_role_id) }}" - --label - "com.docker.compose.project={{ network_role_id | get_entity_name }}" + # Exception: without the stack namespace label `docker stack deploy` cannot see this + # network as one of its own, so it retries the create and dies on "already exists". + - --label + - "com.docker.stack.namespace={{ network_role_id | get_entity_name }}" - "{{ network_role_id | get_entity_name }}" when: - (SWARM_CLUSTER_AVAILABLE | bool) and not (lookup('config', network_role_id, 'networks.local.force_bridge', false) | bool) From 48e230d87f6c81215931d546ab7d1ffc24bed85e Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 00:15:35 +0200 Subject: [PATCH 063/145] docs(requirements): cross-link the OpenBao requirement to its PR Closes the last cross-link acceptance criterion: the requirement now names infinito-nexus/core#611 in its header and in the criterion itself, and the PR description links back to this document. Also deep-links the upstream story instead of pointing at the OpenProject root. --- docs/requirements/032-web-app-openbao.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index ffc45741db..e7d000a69a 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -4,7 +4,8 @@ As an Infinito.Nexus administrator, I want [OpenBao](https://github.com/openbao/openbao) deployed as a `web-app-openbao` role with Keycloak OIDC login, group-mapped policies, persistent storage, automatic unsealing, backup and monitoring, so that infrastructure secrets, application credentials, certificates and tokens can be stored and retrieved centrally without distributing static secrets across systems. -Upstream story: [OpenProject #628](https://project.infinito.nexus/) — *Integrate OpenBao into Infinito.Nexus*. +Upstream story: [OpenProject #628](https://project.infinito.nexus/work_packages/628) — *Integrate OpenBao into Infinito.Nexus*. +Implementing PR: [infinito-nexus/core#611](https://github.com/infinito-nexus/core/pull/611). ## Background @@ -305,7 +306,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] The stack reaches a steady running state in every variant, and `make quality` is green tree-wide. `make quality` (docs + autoformat + the four suites) passes end to end — `test-external`, `test-integration`, `test-lint` and `test-unit` all green. All three variants reach a steady running state with `failed=0`. - [x] `README.md` documents the static-seal model and its trust assumption, the bootstrap and root-token revocation flow, the RBAC mapping, the PKI flag, the backup/restore procedure including the seal-key caveat, and the CLI-OIDC-login limitation from Decision #10. It additionally records two traps found during implementation: the `/openbao/file` ownership requirement and why `vault_core_unsealed` must not be used for alerting. -- [ ] This requirement is cross-linked from the implementing PR, and the PR is cross-linked back from here. +- [x] This requirement is cross-linked from the implementing PR, and the PR is cross-linked back from here. Implementing PR: [infinito-nexus/core#611](https://github.com/infinito-nexus/core/pull/611). ## Validation Apps From 85abcdf6276d47f4f931d2233ad81d9401998b7a Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:03 +0200 Subject: [PATCH 064/145] feat(prometheus): discover role-local alert rules via the alert_rule_roles lookup --- plugins/lookup/alert_rule_roles.py | 50 ++++++++++++++++++++++++++++++ 1 file changed, 50 insertions(+) create mode 100644 plugins/lookup/alert_rule_roles.py diff --git a/plugins/lookup/alert_rule_roles.py b/plugins/lookup/alert_rule_roles.py new file mode 100644 index 0000000000..05096c9ca5 --- /dev/null +++ b/plugins/lookup/alert_rule_roles.py @@ -0,0 +1,50 @@ +from __future__ import annotations + +from typing import Any + +from ansible.plugins.lookup import LookupBase + +from utils.cache import ROLES_DIR + +ALERT_RULES_TEMPLATE = ("templates", "prometheus", "alert_rules.yml.j2") + + +class LookupModule(LookupBase): + """ + Return a sorted list of deployed role IDs that ship their own Prometheus + alert rules, i.e. every role in ``group_names`` that has a template at + ``roles//templates/prometheus/alert_rules.yml.j2``. + + Keeps alert rules at the role that owns the metric instead of collecting + them in web-app-prometheus: a role declares what "unhealthy" means for + itself, and the Prometheus role only concatenates what the play enabled. + + Usage in a template: + {% for role_id in lookup('alert_rule_roles') %} + {% include 'roles/' + role_id + '/templates/prometheus/alert_rules.yml.j2' %} + {% endfor %} + + The rendered fragment MUST start at the ``groups:`` item level (two-space + indented ``- name: `` entries), because it is concatenated into the + single ``groups:`` list of the Prometheus rule file. + """ + + def run( + self, + terms: list[Any], + variables: dict[str, Any] | None = None, + **kwargs: Any, + ) -> list[list[str]]: + if terms: + raise ValueError("alert_rule_roles lookup takes no positional terms") + + vars_ = variables or getattr(self._templar, "available_variables", {}) or {} + group_names: list[str] = vars_.get("group_names", []) or [] + + result = [ + role_id + for role_id in sorted(set(group_names)) + if (ROLES_DIR / role_id).joinpath(*ALERT_RULES_TEMPLATE).exists() + ] + + return [result] From 699823f6cb60e9638920bfca5b6d4b20fffa55c0 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:13 +0200 Subject: [PATCH 065/145] test(prometheus): cover the alert_rule_roles lookup --- .../plugins/lookup/test_alert_rule_roles.py | 94 +++++++++++++++++++ 1 file changed, 94 insertions(+) create mode 100644 tests/unit/python/plugins/lookup/test_alert_rule_roles.py diff --git a/tests/unit/python/plugins/lookup/test_alert_rule_roles.py b/tests/unit/python/plugins/lookup/test_alert_rule_roles.py new file mode 100644 index 0000000000..7de5d73d55 --- /dev/null +++ b/tests/unit/python/plugins/lookup/test_alert_rule_roles.py @@ -0,0 +1,94 @@ +"""Unit tests for plugins/lookup/alert_rule_roles.py. + +Mocks the Ansible-side machinery and asserts the plugin returns exactly the +deployed roles that ship a ``templates/prometheus/alert_rules.yml.j2``. +""" + +from __future__ import annotations + +import unittest +import unittest.mock as mock + +import plugins.lookup.alert_rule_roles as module +from plugins.lookup.alert_rule_roles import LookupModule + + +class _Templar: + def __init__(self, vars_=None): + self.available_variables = vars_ or {} + + def template(self, value, **_): + return value + + +def _make_lookup(vars_=None): + lm = LookupModule() + lm._templar = _Templar(vars_ or {}) + lm._loader = mock.MagicMock() + return lm + + +def _seed_roles(tmp_path, roles_with_template, roles_without_template=()): + for role_id in roles_with_template: + target = tmp_path / role_id / "templates" / "prometheus" + target.mkdir(parents=True) + (target / "alert_rules.yml.j2").write_text(" - name: probe\n") + for role_id in roles_without_template: + (tmp_path / role_id / "templates").mkdir(parents=True) + + +class TestAlertRuleRolesLookup(unittest.TestCase): + def setUp(self): + self._tmp = mock.patch.object(module, "ROLES_DIR") + self.roles_dir = self._tmp.start() + self.addCleanup(self._tmp.stop) + + def test_raises_when_positional_terms_supplied(self): + lm = _make_lookup({"group_names": []}) + with self.assertRaises(ValueError): + lm.run(["unexpected"], variables=lm._templar.available_variables) + + def test_returns_only_deployed_roles_that_ship_a_template(self): + import tempfile + from pathlib import Path + + with tempfile.TemporaryDirectory() as td: + tmp_path = Path(td) + _seed_roles( + tmp_path, + roles_with_template=["web-app-openbao", "svc-db-postgres"], + roles_without_template=["web-app-matomo"], + ) + module.ROLES_DIR = tmp_path + + vars_ = { + "group_names": [ + "web-app-matomo", + "web-app-openbao", + "web-app-never-deployed", + ] + } + lm = _make_lookup(vars_) + self.assertEqual(lm.run([], variables=vars_), [["web-app-openbao"]]) + + def test_result_is_sorted_and_deduplicated(self): + import tempfile + from pathlib import Path + + with tempfile.TemporaryDirectory() as td: + tmp_path = Path(td) + _seed_roles(tmp_path, roles_with_template=["b-role", "a-role"]) + module.ROLES_DIR = tmp_path + + vars_ = {"group_names": ["b-role", "a-role", "b-role"]} + lm = _make_lookup(vars_) + self.assertEqual(lm.run([], variables=vars_), [["a-role", "b-role"]]) + + def test_returns_empty_list_without_group_names(self): + import tempfile + from pathlib import Path + + with tempfile.TemporaryDirectory() as td: + module.ROLES_DIR = Path(td) + lm = _make_lookup({}) + self.assertEqual(lm.run([], variables={}), [[]]) From f5df096a2af8d045e314a3dc8b194d473c93489a Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:19 +0200 Subject: [PATCH 066/145] refactor(web-app-prometheus): concatenate role-local alert rules instead of owning them --- .../configuration/alert_rules.yml.j2 | 25 ++++--------------- 1 file changed, 5 insertions(+), 20 deletions(-) diff --git a/roles/web-app-prometheus/templates/configuration/alert_rules.yml.j2 b/roles/web-app-prometheus/templates/configuration/alert_rules.yml.j2 index 575a111873..0c6f9a139d 100644 --- a/roles/web-app-prometheus/templates/configuration/alert_rules.yml.j2 +++ b/roles/web-app-prometheus/templates/configuration/alert_rules.yml.j2 @@ -64,26 +64,6 @@ groups: This is a communication-critical application. {% endif %} -{% if 'web-app-openbao' in lookup('native_metrics_apps') %} - - name: openbao_seal - interval: 30s - rules: - # Exception: `vault_core_unsealed` is unusable here — an unsealed node emits it as - # `{cluster=""} 0`, so alerting on it fires permanently. `vault_core_active` carries the - # real cluster label, and a sealed node stops serving it at all. - - alert: OpenBaoSealed - expr: vault_core_active{job="openbao"} == 0 or absent(vault_core_active{job="openbao"}) - for: 2m - labels: - severity: critical - annotations: - summary: "OpenBao is sealed or unreachable" - description: > - OpenBao has not reported an active, unsealed core for more than 2 minutes. - Secrets, PKI issuance and machine logins are unavailable while it is sealed. - Check the container and confirm the static seal key reached its environment. -{% endif %} - - name: prometheus_health rules: - alert: PrometheusTargetMissing @@ -96,3 +76,8 @@ groups: description: > Prometheus target {{ '{{' }} $labels.instance {{ '}}' }} (job: {{ '{{' }} $labels.job {{ '}}' }}) has been unreachable for more than 5 minutes. + +{% for alert_rules_role_id in lookup('alert_rule_roles') %} +{% include 'roles/' + alert_rules_role_id + '/templates/prometheus/alert_rules.yml.j2' %} + +{% endfor %} From 4296fbc487d6620f55207e31b0444e14321d65d5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:23 +0200 Subject: [PATCH 067/145] feat(web-app-openbao): move the seal alert into the role that owns the metric --- .../templates/prometheus/alert_rules.yml.j2 | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) create mode 100644 roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 diff --git a/roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 b/roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 new file mode 100644 index 0000000000..b2ce3a9243 --- /dev/null +++ b/roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 @@ -0,0 +1,16 @@ +{% if 'web-app-openbao' in lookup('native_metrics_apps') %} + - name: openbao_seal + interval: 30s + rules: + - alert: OpenBaoSealed + expr: vault_core_active{job="openbao"} == 0 or absent(vault_core_active{job="openbao"}) + for: 2m + labels: + severity: critical + annotations: + summary: "OpenBao is sealed or unreachable" + description: > + OpenBao has not reported an active, unsealed core for more than 2 minutes. + Secrets, PKI issuance and machine logins are unavailable while it is sealed. + Check the container and confirm the static seal key reached its environment. +{% endif %} From 8dcfed33c3cb15ebd08aa164d33a49c180890b72 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:27 +0200 Subject: [PATCH 068/145] feat(web-app-openbao): declare dedicated operator and reader RBAC accounts --- roles/web-app-openbao/meta/users.yml | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/roles/web-app-openbao/meta/users.yml b/roles/web-app-openbao/meta/users.yml index 6c31213ebc..f3f5e016b8 100644 --- a/roles/web-app-openbao/meta/users.yml +++ b/roles/web-app-openbao/meta/users.yml @@ -1,2 +1,12 @@ --- administrator: {} +baooperator: + description: OpenBao RBAC coverage account holding only the operator role + username: baooperator + roles: + - operator +baoreader: + description: OpenBao RBAC coverage account holding only the reader role + username: baoreader + roles: + - reader From 136c28f15d0c2fae57457d5af83b2de8d8abb60d Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:31 +0200 Subject: [PATCH 069/145] test(web-app-openbao): expose the RBAC account credentials and the KV mount to Playwright --- roles/web-app-openbao/templates/playwright.env.j2 | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/roles/web-app-openbao/templates/playwright.env.j2 b/roles/web-app-openbao/templates/playwright.env.j2 index 7e661d027c..9f9e2976c0 100644 --- a/roles/web-app-openbao/templates/playwright.env.j2 +++ b/roles/web-app-openbao/templates/playwright.env.j2 @@ -4,6 +4,14 @@ ADMIN_USERNAME={{ lookup('users', 'administrator').username | dotenv_quote }} APP_BASE_URL={{ lookup('tls', application_id, 'url.base') }} +BAO_OPERATOR_PASSWORD={{ lookup('users', 'baooperator').password | dotenv_quote }} + +BAO_OPERATOR_USERNAME={{ lookup('users', 'baooperator').username | dotenv_quote }} + +BAO_READER_PASSWORD={{ lookup('users', 'baoreader').password | dotenv_quote }} + +BAO_READER_USERNAME={{ lookup('users', 'baoreader').username | dotenv_quote }} + BIBER_PASSWORD={{ lookup('users', 'biber').password | dotenv_quote }} BIBER_USERNAME={{ lookup('users', 'biber').username | dotenv_quote }} @@ -29,6 +37,8 @@ OIDC_ISSUER_URL={{ OIDC.CLIENT.ISSUER_URL | dotenv_quote }} OPENBAO_BASE_URL={{ lookup('tls', application_id, 'url.base') | dotenv_quote }} +OPENBAO_KV_MOUNT={{ OPENBAO_KV_MOUNT | dotenv_quote }} + # nocheck: persona-block-rationale - OpenBao's UI login is not a link-then-Keycloak journey the shared persona helper can drive: the Ember auth form requires selecting the OIDC method, resolving the role through POST /v1/auth/oidc/oidc/auth_url, then completing Keycloak in a popup that posts its result back via postMessage (see upstream ui/app/components/auth-jwt.js). The administrator's full OIDC journey is asserted by the role-local files/playwright/test-oidc-login.js instead; with SSO off there is no human login path at all, since OpenBao offers no username/password auth. PERSONA_ADMINISTRATOR_BLOCKED=true From 754d8bf140bb4052fc8e24017d21a5a0937fed71 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:36 +0200 Subject: [PATCH 070/145] test(web-app-openbao): assert every RBAC tier receives its policy and its capabilities --- .../files/playwright/test-rbac-groups.js | 128 ++++++++++++++++++ 1 file changed, 128 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-rbac-groups.js diff --git a/roles/web-app-openbao/files/playwright/test-rbac-groups.js b/roles/web-app-openbao/files/playwright/test-rbac-groups.js new file mode 100644 index 0000000000..f00fd8fe57 --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-rbac-groups.js @@ -0,0 +1,128 @@ +const { test, expect } = require("@playwright/test"); +const { normalizeBaseUrl, decodeDotenvQuotedValue } = require("./personas"); +const { skipUnlessServiceEnabled } = require("./service-gating"); +const { resolveTimeout } = require("./timeouts"); + +const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); +const kvMount = decodeDotenvQuotedValue(process.env.OPENBAO_KV_MOUNT || "infinito"); + +const ACCOUNTS = { + administrator: { + username: decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""), + password: decodeDotenvQuotedValue(process.env.ADMIN_PASSWORD || ""), + }, + operator: { + username: decodeDotenvQuotedValue(process.env.BAO_OPERATOR_USERNAME || ""), + password: decodeDotenvQuotedValue(process.env.BAO_OPERATOR_PASSWORD || ""), + }, + reader: { + username: decodeDotenvQuotedValue(process.env.BAO_READER_USERNAME || ""), + password: decodeDotenvQuotedValue(process.env.BAO_READER_PASSWORD || ""), + }, +}; + +const RBAC_ROLES = Object.keys(ACCOUNTS); + +test.use({ ignoreHTTPSErrors: true }); + +async function login(request, role) { + const { username, password } = ACCOUNTS[role]; + const response = await request.post( + `${baseUrl}/v1/auth/ldap/login/${encodeURIComponent(username)}`, + { data: { password }, failOnStatusCode: false, timeout: resolveTimeout(30_000) }, + ); + expect( + response.status(), + `expected the LDAP auth method to accept '${username}' (${role}), got ${response.status()}`, + ).toBe(200); + + const body = await response.json(); + return { policies: body.auth.policies || [], token: body.auth.client_token }; +} + +test("rbac: each role group maps its member to exactly its own policy", async ({ request }) => { + skipUnlessServiceEnabled("ldap"); + + for (const role of RBAC_ROLES) { + const { policies } = await login(request, role); + expect(policies, `'${role}' must receive the '${role}' policy`).toContain(role); + + for (const other of RBAC_ROLES.filter((candidate) => candidate !== role)) { + expect( + policies, + `'${role}' must not receive the '${other}' policy, got ${policies.join(", ")}`, + ).not.toContain(other); + } + } +}); + +test("rbac: the operator may write an application secret and the reader may not", async ({ request }) => { + skipUnlessServiceEnabled("ldap"); + + const secretPath = `${baseUrl}/v1/${kvMount}/data/playwright/rbac-probe`; + const payload = { data: { data: { probe: "operator-write" } } }; + + const operator = await login(request, "operator"); + const operatorWrite = await request.post(secretPath, { + headers: { "X-Vault-Token": operator.token }, + data: payload, + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); + expect( + [200, 204], + `the operator must be able to write an application secret, got ${operatorWrite.status()}`, + ).toContain(operatorWrite.status()); + + const reader = await login(request, "reader"); + const readerWrite = await request.post(secretPath, { + headers: { "X-Vault-Token": reader.token }, + data: payload, + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); + expect( + readerWrite.status(), + `the reader must be refused write access, got ${readerWrite.status()}`, + ).toBe(403); + + const readerRead = await request.get(secretPath, { + headers: { "X-Vault-Token": reader.token }, + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); + expect( + readerRead.status(), + `the reader must be able to read the secret the operator wrote, got ${readerRead.status()}`, + ).toBe(200); +}); + +test("rbac: only the administrator may read the ACL policy definitions", async ({ request }) => { + skipUnlessServiceEnabled("ldap"); + + const policyPath = `${baseUrl}/v1/sys/policies/acl/operator`; + + const administrator = await login(request, "administrator"); + const adminRead = await request.get(policyPath, { + headers: { "X-Vault-Token": administrator.token }, + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); + expect( + adminRead.status(), + `the administrator must be able to read a policy definition, got ${adminRead.status()}`, + ).toBe(200); + + for (const role of ["operator", "reader"]) { + const { token } = await login(request, role); + const response = await request.get(policyPath, { + headers: { "X-Vault-Token": token }, + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }); + expect( + response.status(), + `'${role}' must be refused the policy definitions, got ${response.status()}`, + ).toBe(403); + } +}); From 3f6a454a5378572ce26f5d0d94e9b35b1f06a855 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:40 +0200 Subject: [PATCH 071/145] test(web-app-openbao): aggregate the RBAC tier spec --- roles/web-app-openbao/files/playwright/playwright.spec.js | 1 + 1 file changed, 1 insertion(+) diff --git a/roles/web-app-openbao/files/playwright/playwright.spec.js b/roles/web-app-openbao/files/playwright/playwright.spec.js index 2e6b3c8328..a7682701d9 100644 --- a/roles/web-app-openbao/files/playwright/playwright.spec.js +++ b/roles/web-app-openbao/files/playwright/playwright.spec.js @@ -5,4 +5,5 @@ require("./test-administrator-persona"); require("./test-oidc-login"); require("./test-ldap-login"); require("./test-rbac-denial"); +require("./test-rbac-groups"); require("./test-seal-status"); From 025168e2f788e5bf20bea99a63f8961695516b88 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:55 +0200 Subject: [PATCH 072/145] fix(test-e2e-playwright): record video for hand-built browser contexts Cherry-picked from Alejandro Roman's 4287dee96b on infinito-nexus/core#429. Playwright applies the project's use.video setting only to the context behind the page fixture, so a spec that builds its own context through browser.newContext() reaches the artefact with a trace but no video. --- .../files/personas/utils/env.js | 24 +++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/roles/test-e2e-playwright/files/personas/utils/env.js b/roles/test-e2e-playwright/files/personas/utils/env.js index 2afb5609b7..0eea54eb46 100644 --- a/roles/test-e2e-playwright/files/personas/utils/env.js +++ b/roles/test-e2e-playwright/files/personas/utils/env.js @@ -132,6 +132,29 @@ function safeIsEnabled(name) { } } +/** + * Video-recording options for a hand-built browser context. + * + * Playwright applies the project's `use.video` setting only to the context the + * `page` fixture provides, so a context from `browser.newContext()` records + * nothing and its spec reaches the artefact with a trace but no video. Spread + * the result into the `newContext` options to follow the project setting. + * + * Args: + * testInfo: the spec's TestInfo; supplies the per-test output directory and + * the resolved project config. + * + * Returns: + * `{ recordVideo: { dir } }` when the project asks for video, else `{}`. + */ +function recordVideoOptions(testInfo) { + const configured = testInfo?.project?.use?.video; + const mode = + configured && typeof configured === "object" ? configured.mode : configured; + if (!mode || mode === "off") return {}; + return { recordVideo: { dir: testInfo.outputDir } }; +} + module.exports = { decodeDotenvQuoted, normalizeUrl, @@ -141,4 +164,5 @@ module.exports = { apiGetOnion, safeSkipUnlessEnabled, safeIsEnabled, + recordVideoOptions, }; From ac4609d93784f209d7f85124acccbcd1989ae526 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:35:59 +0200 Subject: [PATCH 073/145] style(sys-svc-compose): drop the narrative comment on the namespace label --- roles/sys-svc-compose/tasks/utils/network/create.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/roles/sys-svc-compose/tasks/utils/network/create.yml b/roles/sys-svc-compose/tasks/utils/network/create.yml index 6085953094..3e6c89ac78 100644 --- a/roles/sys-svc-compose/tasks/utils/network/create.yml +++ b/roles/sys-svc-compose/tasks/utils/network/create.yml @@ -39,8 +39,6 @@ - "com.docker.compose.network={{ lookup('shared_network_key', network_role_id) }}" - --label - "com.docker.compose.project={{ network_role_id | get_entity_name }}" - # Exception: without the stack namespace label `docker stack deploy` cannot see this - # network as one of its own, so it retries the create and dies on "already exists". - --label - "com.docker.stack.namespace={{ network_role_id | get_entity_name }}" - "{{ network_role_id | get_entity_name }}" From ce8ad5c97ebdcc48257e71d9bada4e7d5ce6e873 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:36:10 +0200 Subject: [PATCH 074/145] style(web-app-openbao): drop the narrative comments from the OIDC spec --- roles/web-app-openbao/files/playwright/test-oidc-login.js | 7 ------- 1 file changed, 7 deletions(-) diff --git a/roles/web-app-openbao/files/playwright/test-oidc-login.js b/roles/web-app-openbao/files/playwright/test-oidc-login.js index 599874e6bf..f0492c8ed8 100644 --- a/roles/web-app-openbao/files/playwright/test-oidc-login.js +++ b/roles/web-app-openbao/files/playwright/test-oidc-login.js @@ -19,8 +19,6 @@ const CALLBACK_PATH = "/ui/vault/auth/oidc/oidc/callback"; test.use({ ignoreHTTPSErrors: true }); -// Exception: a failed auth_url request renders as "Invalid role", indistinguishable from a -// misconfigured role, so probe it from the page to separate a browser problem from a server one. test("oidc: the browser can reach the auth_url endpoint the login form depends on", async ({ page }) => { skipUnlessServiceEnabled("sso"); await page.context().clearCookies(); @@ -52,12 +50,9 @@ test("oidc: the browser can reach the auth_url endpoint the login form depends o test("oidc: the administrator signs in through Keycloak and lands in the OpenBao UI", async ({ page }) => { skipUnlessServiceEnabled("sso"); - // Exception: the default per-test budget is smaller than this flow's own onion-scaled waits. test.setTimeout(resolveTimeout(240_000)); await page.context().clearCookies(); - // Exception: the provider is a second hidden service needing its own Tor circuit; building it - // inside the popup's navigation timeout makes this flaky (5.1 min cold vs 5.3 s warm). if (issuerUrl) { const warmup = await page.context().newPage(); await gotoOnion(warmup, issuerUrl, { waitUntil: "domcontentloaded" }).catch(() => {}); @@ -96,8 +91,6 @@ test("oidc: the administrator signs in through Keycloak and lands in the OpenBao ) .toBe(200); - // Exception: clicking before the component's own debounced role fetch settles opens no popup - // at all, and nothing in the DOM distinguishes the two states — hence short retries. let popup = null; for (let attempt = 1; attempt <= 4 && !popup; attempt += 1) { const popupPromise = page From 2149255aaa899c6ec34be8ddc8760bfe2554a0e7 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:36:15 +0200 Subject: [PATCH 075/145] style(web-app-openbao): drop the narrative comment from the LDAP spec --- roles/web-app-openbao/files/playwright/test-ldap-login.js | 2 -- 1 file changed, 2 deletions(-) diff --git a/roles/web-app-openbao/files/playwright/test-ldap-login.js b/roles/web-app-openbao/files/playwright/test-ldap-login.js index 7c848b16be..0ad6f03805 100644 --- a/roles/web-app-openbao/files/playwright/test-ldap-login.js +++ b/roles/web-app-openbao/files/playwright/test-ldap-login.js @@ -38,8 +38,6 @@ test("ldap: a user in no OpenBao role group gets no privileged policy", async ({ { data: { password: biberPassword }, failOnStatusCode: false, timeout: resolveTimeout(30_000) }, ); - // Exception: rejection and admission-with-`default` both satisfy the contract; only a - // privileged policy is a failure. if (response.status() === 200) { const body = await response.json(); const policies = body.auth.policies || []; From 694ee29f46d9899be106c2c1aaac7994b585d08d Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:36:24 +0200 Subject: [PATCH 076/145] style(web-app-openbao): drop the narrative comment from the RBAC denial spec --- roles/web-app-openbao/files/playwright/test-rbac-denial.js | 2 -- 1 file changed, 2 deletions(-) diff --git a/roles/web-app-openbao/files/playwright/test-rbac-denial.js b/roles/web-app-openbao/files/playwright/test-rbac-denial.js index 506720856b..681f98ea5f 100644 --- a/roles/web-app-openbao/files/playwright/test-rbac-denial.js +++ b/roles/web-app-openbao/files/playwright/test-rbac-denial.js @@ -31,8 +31,6 @@ test("rbac: an invalid token is refused on a privileged path", async ({ request ).toContain(response.status()); }); -// Exception: the listener serves metrics unauthenticated for the internal scrape, so the -// proxy must never expose them publicly. test("metrics: /v1/sys/metrics is not served through the public proxy", async ({ request }) => { const response = await request.get(`${baseUrl}/v1/sys/metrics?format=prometheus`, { failOnStatusCode: false, From c2620a7087efcbd71822fa7ee485883e6f4fb564 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:36:31 +0200 Subject: [PATCH 077/145] style(web-app-openbao): drop the narrative comment from meta/server.yml --- roles/web-app-openbao/meta/server.yml | 1 - 1 file changed, 1 deletion(-) diff --git a/roles/web-app-openbao/meta/server.yml b/roles/web-app-openbao/meta/server.yml index 49281e02d6..1842942c0b 100644 --- a/roles/web-app-openbao/meta/server.yml +++ b/roles/web-app-openbao/meta/server.yml @@ -1,5 +1,4 @@ --- -# Exception: OpenBao answers `/` with a 307 into its UI, which the health checker does not expect by default. status_codes: default: - 200 From 4a206f6d52d63728b4eb47219c28dfd7c8fae8d8 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:36:41 +0200 Subject: [PATCH 078/145] style(web-app-openbao): drop the narrative comment from the group-alias helper --- roles/web-app-openbao/tasks/utils/group_alias.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/roles/web-app-openbao/tasks/utils/group_alias.yml b/roles/web-app-openbao/tasks/utils/group_alias.yml index 02173b2a22..546edd114b 100644 --- a/roles/web-app-openbao/tasks/utils/group_alias.yml +++ b/roles/web-app-openbao/tasks/utils/group_alias.yml @@ -1,7 +1,4 @@ --- -# Exception: a group alias is keyed by (name, mount_accessor) and creating a duplicate errors, -# so the alias already on the group is compared before writing. -# # Param: openbao_group - one entry of the registered group-id loop (item = role name, stdout = group id) # Param: alias_mount - auth mount path, e.g. "oidc/" or "ldap/" # Param: alias_name - identity-provider group name the claim will carry From 71f2a0c414cbf57166b447c251ece4c594d52362 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:36:45 +0200 Subject: [PATCH 079/145] style(web-app-openbao): drop the narrative comment from 00_core.yml --- roles/web-app-openbao/tasks/00_core.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/roles/web-app-openbao/tasks/00_core.yml b/roles/web-app-openbao/tasks/00_core.yml index 88554898ff..c44d5b44b5 100644 --- a/roles/web-app-openbao/tasks/00_core.yml +++ b/roles/web-app-openbao/tasks/00_core.yml @@ -1,6 +1,4 @@ --- -# Exception: the config templates notify `compose-up`, so they MUST be rendered after -# sys-stk-backend has produced compose.yml and before the handler flush. - include_tasks: utils/once/flag.yml - name: "🧩 load docker for {{ application_id }}" From cbda5724c88a01379070a43e49607662521fccda Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:36:50 +0200 Subject: [PATCH 080/145] style(web-app-openbao): drop the narrative comments from 01_init.yml --- roles/web-app-openbao/tasks/01_init.yml | 2 -- 1 file changed, 2 deletions(-) diff --git a/roles/web-app-openbao/tasks/01_init.yml b/roles/web-app-openbao/tasks/01_init.yml index ce7044e7f2..8f5eb7fc6b 100644 --- a/roles/web-app-openbao/tasks/01_init.yml +++ b/roles/web-app-openbao/tasks/01_init.yml @@ -14,7 +14,6 @@ changed_when: false retries: 30 delay: 5 - # Exception: exit 2 means sealed and exit 1 means not yet initialised; both prove the API answers. until: openbao_status_raw.rc in [0, 2] or 'not yet initialized' in openbao_status_raw.stderr failed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" @@ -104,6 +103,5 @@ ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} sh -c 'bao token revoke -self >/dev/null 2>&1; rm -f ~/.bao-token' - # Exception: teardown of this run's own short-lived token, not configuration drift. changed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" From 47217af7a1d3a2ad5378164f87a2ce1823b2077b Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:36:55 +0200 Subject: [PATCH 081/145] style(web-app-openbao): drop the narrative comment from 03_policies.yml --- roles/web-app-openbao/tasks/03_policies.yml | 3 --- 1 file changed, 3 deletions(-) diff --git a/roles/web-app-openbao/tasks/03_policies.yml b/roles/web-app-openbao/tasks/03_policies.yml index 06f3d90b4b..84c28ad92a 100644 --- a/roles/web-app-openbao/tasks/03_policies.yml +++ b/roles/web-app-openbao/tasks/03_policies.yml @@ -7,9 +7,6 @@ register: openbao_kv_mount changed_when: "'OPENBAO_CHANGED' in openbao_kv_mount.stdout" -# Exception: `bao policy read` reformats the HCL, so the rendered file cannot be diffed against -# it. Comparing the server's own read-back before and after the write reports whether the write -# actually altered state. `set -e` keeps a failed write a task failure rather than a false change. - name: "📜 Write the human RBAC policies" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} From 241b01f94a66ca79b040693f7be376f6253bcdc6 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:37:01 +0200 Subject: [PATCH 082/145] style(web-app-openbao): drop the narrative comments from 04_auth_methods.yml --- roles/web-app-openbao/tasks/04_auth_methods.yml | 4 ---- 1 file changed, 4 deletions(-) diff --git a/roles/web-app-openbao/tasks/04_auth_methods.yml b/roles/web-app-openbao/tasks/04_auth_methods.yml index a02b4ee2f0..346a452543 100644 --- a/roles/web-app-openbao/tasks/04_auth_methods.yml +++ b/roles/web-app-openbao/tasks/04_auth_methods.yml @@ -22,8 +22,6 @@ args: stdin: "{{ OPENBAO_OIDC_CLIENT_SECRET }}" stdin_add_newline: false - # Exception: the write always runs so a rotated client secret lands, but OpenBao redacts - # the secret on read, so a real change cannot be detected and none is claimed. changed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" @@ -74,8 +72,6 @@ args: stdin: "{{ LDAP.BIND_CREDENTIAL }}" stdin_add_newline: false - # Exception: as with the OIDC config, the bind password is write-only, so the write always - # runs to carry a rotation but claims no change it cannot observe. changed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" From 6cd99b402f506d88671cd11ca7a13e0a46e5a6a5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:37:05 +0200 Subject: [PATCH 083/145] style(web-app-openbao): drop the narrative comment from the administrator policy --- roles/web-app-openbao/templates/policies/administrator.hcl.j2 | 1 - 1 file changed, 1 deletion(-) diff --git a/roles/web-app-openbao/templates/policies/administrator.hcl.j2 b/roles/web-app-openbao/templates/policies/administrator.hcl.j2 index 7b2272494a..5e5b0489a4 100644 --- a/roles/web-app-openbao/templates/policies/administrator.hcl.j2 +++ b/roles/web-app-openbao/templates/policies/administrator.hcl.j2 @@ -38,7 +38,6 @@ path "sys/seal-status" { capabilities = ["read"] } -# Exception: without these the web UI renders an empty secrets list. path "sys/internal/ui/mounts" { capabilities = ["read"] } From 2f1db60f14484913b6080922fa1134144a5d5af5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:37:09 +0200 Subject: [PATCH 084/145] style(web-app-openbao): drop the narrative comment from the operator policy --- roles/web-app-openbao/templates/policies/operator.hcl.j2 | 1 - 1 file changed, 1 deletion(-) diff --git a/roles/web-app-openbao/templates/policies/operator.hcl.j2 b/roles/web-app-openbao/templates/policies/operator.hcl.j2 index 085f346172..a288de8760 100644 --- a/roles/web-app-openbao/templates/policies/operator.hcl.j2 +++ b/roles/web-app-openbao/templates/policies/operator.hcl.j2 @@ -22,7 +22,6 @@ path "sys/seal-status" { capabilities = ["read"] } -# Exception: without these the web UI renders an empty secrets list. path "sys/internal/ui/mounts" { capabilities = ["read"] } From 793b32f7420bff06cd560ace0cc3dd2f7ad3d6b5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:37:14 +0200 Subject: [PATCH 085/145] style(web-app-openbao): drop the narrative comment from the reader policy --- roles/web-app-openbao/templates/policies/reader.hcl.j2 | 1 - 1 file changed, 1 deletion(-) diff --git a/roles/web-app-openbao/templates/policies/reader.hcl.j2 b/roles/web-app-openbao/templates/policies/reader.hcl.j2 index bd4850630d..f7be1a2352 100644 --- a/roles/web-app-openbao/templates/policies/reader.hcl.j2 +++ b/roles/web-app-openbao/templates/policies/reader.hcl.j2 @@ -14,7 +14,6 @@ path "sys/seal-status" { capabilities = ["read"] } -# Exception: without these the web UI renders an empty secrets list. path "sys/internal/ui/mounts" { capabilities = ["read"] } From a9afc5b45cc1a4bd48baa41ec9192047f6321383 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:37:19 +0200 Subject: [PATCH 086/145] style(web-app-openbao): drop the narrative comments from the server config template --- roles/web-app-openbao/templates/openbao.hcl.j2 | 2 -- 1 file changed, 2 deletions(-) diff --git a/roles/web-app-openbao/templates/openbao.hcl.j2 b/roles/web-app-openbao/templates/openbao.hcl.j2 index 9b25f8d5b3..044b0a15b8 100644 --- a/roles/web-app-openbao/templates/openbao.hcl.j2 +++ b/roles/web-app-openbao/templates/openbao.hcl.j2 @@ -2,7 +2,6 @@ ui = true log_level = "{{ OPENBAO_LOG_LEVEL }}" -# Exception: raft and mlock are mutually exclusive upstream. disable_mlock = true storage "raft" { @@ -31,5 +30,4 @@ telemetry { api_addr = "{{ OPENBAO_BASE_URL }}" -# Exception: loopback, not the container name — swarm renames every task, and a single-replica raft node only advertises to itself. cluster_addr = "http://127.0.0.1:{{ container_port | int + 1 }}" From c4e6cfb49029f6053decd73bc7ff6008c3211181 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:38:07 +0200 Subject: [PATCH 087/145] docs(web-app-openbao): record the RBAC tier coverage and the role-local alert rules --- roles/web-app-openbao/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index 30848da8bb..98cca2096c 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -115,6 +115,8 @@ docker run --rm -it \ - **Visiting `/ui/vault/logout` does not end the session.** Measured against a live instance: after that navigation the UI returns to `/ui/vault/secrets` still authenticated, because the Ember app restores its token from browser storage. Use the UI's own logout control, or clear site data, when you actually need the session gone. - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. +- **The RBAC tiers are covered by two dedicated accounts.** `meta/users.yml` declares `baooperator` (`roles: [operator]`) and `baoreader` (`roles: [reader]`), which the standard RBAC provisioning joins to `/web-app-openbao/` in Keycloak and to the matching LDAP role group. `files/playwright/test-rbac-groups.js` logs each of the three accounts in through the LDAP auth method and asserts both the policy set it receives and what it can actually do: the operator writes an application secret and the reader is refused, and only the administrator reads back an ACL policy definition. `operator` and `reader` are declared by no other role's `meta/rbac.yml`, so the two accounts hold no privileges anywhere else. +- **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). ### Backup and restore From c6bf74aa0fdba6835b17ceeeb10bb0bdc7d6cbe1 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 18:38:11 +0200 Subject: [PATCH 088/145] docs(requirements): close the RBAC tier criterion and relocate the seal alert --- docs/requirements/032-web-app-openbao.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index e7d000a69a..a34f73638a 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -276,7 +276,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] `services.prometheus.native_metrics.enabled: true` and `templates/prometheus.yml.j2` exist; Prometheus scrapes `/v1/sys/metrics?format=prometheus` on the internal container port and the target reports `up`. Verified against the Prometheus targets API: job `openbao`, `scrapeUrl http://openbao:8200/v1/sys/metrics?format=prometheus`, `health: "up"`, `lastError: ""`. - [x] The metrics endpoint is not reachable through `sys-svc-proxy` from outside the role-local docker network. It **was** publicly readable (HTTP 200 through the proxy) until a `location = /v1/sys/metrics { deny all; }` was added in the role's `templates/proxy.conf.j2`; `test-rbac-denial.js` now asserts the endpoint never answers 200 on the public surface. -- [ ] A sealed instance is detectable: `/v1/sys/health` returns 503 (asserted by `test-seal-status.js`) and an alert rule fires on that condition. *An `OpenBaoSealed` alert now exists in the shared `web-app-prometheus` alert template, gated on openbao being a native-metrics app. The health-code contract is asserted by the spec suite and the healthy path was confirmed live (200 in both compose and swarm), but the **firing** condition was never observed — that needs an instance deliberately sealed, which the AppRole-only admin model makes awkward to arrange in a test.* **Correction from implementation:** OpenBao keeps Vault's metric namespace, so the gauge is `vault_core_unsealed`, not `openbao_core_unsealed` — and it is unusable as an alert source here: on a healthy, demonstrably unsealed node it is emitted as `vault_core_unsealed{cluster=""} 0`, so `== 0` would fire permanently. Use the health endpoint as the primary signal, with `vault_core_active{cluster="…"} == 1` as the metric-side corroboration. The alert rule itself is still outstanding. +- [ ] A sealed instance is detectable: `/v1/sys/health` returns 503 (asserted by `test-seal-status.js`) and an alert rule fires on that condition. *An `OpenBaoSealed` alert now ships with the role itself, in `templates/prometheus/alert_rules.yml.j2`, which the `alert_rule_roles` lookup concatenates into the Prometheus rule file. The health-code contract is asserted by the spec suite and the healthy path was confirmed live (200 in both compose and swarm), but the **firing** condition was never observed — that needs an instance deliberately sealed, which the AppRole-only admin model makes awkward to arrange in a test.* **Correction from implementation:** OpenBao keeps Vault's metric namespace, so the gauge is `vault_core_unsealed`, not `openbao_core_unsealed` — and it is unusable as an alert source here: on a healthy, demonstrably unsealed node it is emitted as `vault_core_unsealed{cluster=""} 0`, so `== 0` would fire permanently. Use the health endpoint as the primary signal, with `vault_core_active{cluster="…"} == 1` as the metric-side corroboration. The alert rule itself is still outstanding. - [x] The container healthcheck uses `sealedcode=200&uninitcode=200` so a freshly created, not-yet-initialised container becomes healthy enough for the init task to run, while the monitoring probe keeps the strict codes. Both halves asserted by `test-seal-status.js`; the deploy's init step only ran because the pre-init container reported healthy. ### Logging @@ -299,6 +299,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] The three baseline personas are covered per the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md): `guest` (never reaches an authenticated surface), `biber`, `administrator`, each in its own `test-*.js` module aggregated by `playwright.spec.js`, with SSO/LDAP steps guarded by `skipUnlessServiceEnabled`. `guest` runs live; `biber` and `administrator` are blocked with `PERSONA_*_BLOCKED` under the contract's documented escape, each carrying a `# nocheck: persona-block-rationale` naming the role property that forces it — biber holds no OpenBao RBAC group by design, and the shared persona helper cannot drive OpenBao's method-select + popup + `postMessage` login. The administrator's OIDC journey is asserted instead by `test-oidc-login.js`, and biber's denial by `test-ldap-login.js`. - [x] A spec asserts the RBAC denial from the criteria above (a non-member gets no administrative surface). +- [x] A spec covers every RBAC tier, not only the denial. `meta/users.yml` declares `baooperator` and `baoreader`, whose `roles` lists put them in the operator and reader groups through the existing RBAC provisioning, and `test-rbac-groups.js` asserts for all three accounts that the policy set matches the tier and that the granted capabilities differ as the policies say: the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. - [x] `templates/playwright.env.j2` exposes every env var the specs read. (Enforced by `test_env_keys_used.py`, which also fails on a declared-but-unread key.) - [x] `make compose-playwright role=web-app-openbao` exits 0 with no stub tests and a clean logged-out final state. Latest deploy run: **13 passed, 2 skipped, 0 failed (30.8 s)**, no retries. From 484ebb31aced3ded2db2347bcfaf21edad04b772 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 20:54:38 +0200 Subject: [PATCH 089/145] fix(web-app-openbao): move the RBAC test accounts to the inventory so they can log in Declaring baooperator and baoreader in the role's meta/users.yml provisioned them correctly into LDAP and Keycloak, but every login returned 403: a user with no explicit password inherits "{{ 42 | strong_password }}", which is re-evaluated through secrets.choice on each render with nothing persisted, so the password written by ldap_passwd and the one rendered into playwright.env were different strings. --- roles/web-app-openbao/meta/users.yml | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/roles/web-app-openbao/meta/users.yml b/roles/web-app-openbao/meta/users.yml index f3f5e016b8..6c31213ebc 100644 --- a/roles/web-app-openbao/meta/users.yml +++ b/roles/web-app-openbao/meta/users.yml @@ -1,12 +1,2 @@ --- administrator: {} -baooperator: - description: OpenBao RBAC coverage account holding only the operator role - username: baooperator - roles: - - operator -baoreader: - description: OpenBao RBAC coverage account holding only the reader role - username: baoreader - roles: - - reader From 822f143680c1a0000051e3a6a8ae51ce502e589c Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 20:54:56 +0200 Subject: [PATCH 090/145] test(inventory): declare the OpenBao operator and reader RBAC accounts next to biber Test accounts that something authenticates as need a literal password, which only the inventory provides; the role-local meta/users.yml fallback is regenerated on every render. --- inventories/development/default.yml | 24 ++++++++++++++++++++++++ 1 file changed, 24 insertions(+) diff --git a/inventories/development/default.yml b/inventories/development/default.yml index f026649b6e..7c94b52d44 100644 --- a/inventories/development/default.yml +++ b/inventories/development/default.yml @@ -57,6 +57,30 @@ users: roles: [] tokens: {} authorized_keys: [] + baooperator: + username: "baooperator" + firstname: "Bao" + lastname: "Operator" + email: "baooperator@infinito.example" + password: "AnotherUnsecurePassword!2" + uid: 10002 + gid: 10002 + roles: + - operator + tokens: {} + authorized_keys: [] + baoreader: + username: "baoreader" + firstname: "Bao" + lastname: "Reader" + email: "baoreader@infinito.example" + password: "AnotherUnsecurePassword!3" + uid: 10003 + gid: 10003 + roles: + - reader + tokens: {} + authorized_keys: [] SIZE_PERCENT_CLEANUP_DISC_SPACE: 99 # Exception: set this high because the DinD CI environment in the GitHub workflow can already sit at ~92% disk usage. WORKSTATION_USER: biber From 7e0120bf2e144cf694d327ada40aa475fe9c1616 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 20:55:00 +0200 Subject: [PATCH 091/145] docs(web-app-openbao): record why the RBAC test accounts live in the inventory --- roles/web-app-openbao/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index 98cca2096c..991d2d3dd8 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -115,7 +115,7 @@ docker run --rm -it \ - **Visiting `/ui/vault/logout` does not end the session.** Measured against a live instance: after that navigation the UI returns to `/ui/vault/secrets` still authenticated, because the Ember app restores its token from browser storage. Use the UI's own logout control, or clear site data, when you actually need the session gone. - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. -- **The RBAC tiers are covered by two dedicated accounts.** `meta/users.yml` declares `baooperator` (`roles: [operator]`) and `baoreader` (`roles: [reader]`), which the standard RBAC provisioning joins to `/web-app-openbao/` in Keycloak and to the matching LDAP role group. `files/playwright/test-rbac-groups.js` logs each of the three accounts in through the LDAP auth method and asserts both the policy set it receives and what it can actually do: the operator writes an application secret and the reader is refused, and only the administrator reads back an ACL policy definition. `operator` and `reader` are declared by no other role's `meta/rbac.yml`, so the two accounts hold no privileges anywhere else. +- **The RBAC tiers are covered by two test accounts, declared where `biber` is.** `inventories/development/default.yml` declares `baooperator` (`roles: [operator]`) and `baoreader` (`roles: [reader]`), which the standard RBAC provisioning joins to `/web-app-openbao/` in Keycloak and to the matching LDAP role group. They live in the CI inventory rather than in `meta/users.yml` for one concrete reason: a user with no explicit `password` falls back to `"{{ 42 | strong_password }}"`, and that expression is re-evaluated on every render (`secrets.choice`, nothing persisted), so the password written into LDAP and the password rendered into `playwright.env` are different strings and the login returns 403. Only an account with a literal password in the inventory can be logged in as. The same reasoning applies to any role-declared account that something needs to authenticate as. `files/playwright/test-rbac-groups.js` logs each of the three accounts in through the LDAP auth method and asserts both the policy set it receives and what it can actually do: the operator writes an application secret and the reader is refused, and only the administrator reads back an ACL policy definition. `operator` and `reader` are declared by no other role's `meta/rbac.yml`, so the two accounts hold no privileges anywhere else. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). ### Backup and restore From 86fcff42f193ec18be9be719df9a4f8e588f47e1 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 20:55:08 +0200 Subject: [PATCH 092/145] docs(requirements): correct where the RBAC test accounts are declared --- docs/requirements/032-web-app-openbao.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index a34f73638a..61db5a9139 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -299,7 +299,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] The three baseline personas are covered per the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md): `guest` (never reaches an authenticated surface), `biber`, `administrator`, each in its own `test-*.js` module aggregated by `playwright.spec.js`, with SSO/LDAP steps guarded by `skipUnlessServiceEnabled`. `guest` runs live; `biber` and `administrator` are blocked with `PERSONA_*_BLOCKED` under the contract's documented escape, each carrying a `# nocheck: persona-block-rationale` naming the role property that forces it — biber holds no OpenBao RBAC group by design, and the shared persona helper cannot drive OpenBao's method-select + popup + `postMessage` login. The administrator's OIDC journey is asserted instead by `test-oidc-login.js`, and biber's denial by `test-ldap-login.js`. - [x] A spec asserts the RBAC denial from the criteria above (a non-member gets no administrative surface). -- [x] A spec covers every RBAC tier, not only the denial. `meta/users.yml` declares `baooperator` and `baoreader`, whose `roles` lists put them in the operator and reader groups through the existing RBAC provisioning, and `test-rbac-groups.js` asserts for all three accounts that the policy set matches the tier and that the granted capabilities differ as the policies say: the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. +- [x] A spec covers every RBAC tier, not only the denial. `inventories/development/default.yml` declares `baooperator` and `baoreader` next to `biber`, whose `roles` lists put them in the operator and reader groups through the existing RBAC provisioning, and `test-rbac-groups.js` asserts for all three accounts that the policy set matches the tier and that the granted capabilities differ as the policies say: the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. **Correction from implementation:** the accounts were first declared in the role's `meta/users.yml`, which deployed them correctly but made every login return 403 — a user with no explicit `password` inherits `"{{ 42 | strong_password }}"`, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so LDAP and `playwright.env` received different strings. Test accounts that must authenticate therefore belong in the inventory, where `biber` already lives. - [x] `templates/playwright.env.j2` exposes every env var the specs read. (Enforced by `test_env_keys_used.py`, which also fails on a declared-but-unread key.) - [x] `make compose-playwright role=web-app-openbao` exits 0 with no stub tests and a clean logged-out final state. Latest deploy run: **13 passed, 2 skipped, 0 failed (30.8 s)**, no retries. From c21002780e36c3935e9d71d908652b85b8b83d02 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 21:15:09 +0200 Subject: [PATCH 093/145] Revert "test(inventory): declare the OpenBao operator and reader RBAC accounts next to biber" The shared CI inventory is the wrong home for one role's test accounts. The stable password they need comes from a role-generated credential instead. --- inventories/development/default.yml | 24 ------------------------ 1 file changed, 24 deletions(-) diff --git a/inventories/development/default.yml b/inventories/development/default.yml index 7c94b52d44..f026649b6e 100644 --- a/inventories/development/default.yml +++ b/inventories/development/default.yml @@ -57,30 +57,6 @@ users: roles: [] tokens: {} authorized_keys: [] - baooperator: - username: "baooperator" - firstname: "Bao" - lastname: "Operator" - email: "baooperator@infinito.example" - password: "AnotherUnsecurePassword!2" - uid: 10002 - gid: 10002 - roles: - - operator - tokens: {} - authorized_keys: [] - baoreader: - username: "baoreader" - firstname: "Bao" - lastname: "Reader" - email: "baoreader@infinito.example" - password: "AnotherUnsecurePassword!3" - uid: 10003 - gid: 10003 - roles: - - reader - tokens: {} - authorized_keys: [] SIZE_PERCENT_CLEANUP_DISC_SPACE: 99 # Exception: set this high because the DinD CI environment in the GitHub workflow can already sit at ~92% disk usage. WORKSTATION_USER: biber From dbb8503eeb27c32311911d996d0ce2c744476b00 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 21:15:53 +0200 Subject: [PATCH 094/145] feat(web-app-openbao): generate the RBAC tier account passwords as role credentials A user with no explicit password inherits the strong_password fallback, which is re-evaluated on every render with nothing persisted, so ldap_passwd and playwright.env can receive different strings and the bind returns 403. Same mechanism web-app-zammad uses for api_bot_password. --- roles/web-app-openbao/meta/secrets.yml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/roles/web-app-openbao/meta/secrets.yml b/roles/web-app-openbao/meta/secrets.yml index 8bc1ec482a..0629dd6ea2 100644 --- a/roles/web-app-openbao/meta/secrets.yml +++ b/roles/web-app-openbao/meta/secrets.yml @@ -14,3 +14,16 @@ credentials: description: SecretID of the Ansible AppRole, written to auth/approle/role/ansible/custom-secret-id at bootstrap. algorithm: alphanumeric validation: non_empty_string + rbac_operator_password: + description: Local password of the role-internal `baooperator` user, whose only RBAC role is `operator`. It backs the LDAP-bind login the RBAC tier spec + drives. Declared here rather than left to the users fallback because a user without an explicit password inherits the strong_password fallback expression, + which is regenerated on every render, so the value written to LDAP and the value handed to the test differ and the bind returns 403. Alphanumeric for the + same reason as web-app-zammad's api_bot_password. + algorithm: alphanumeric + validation: "^[A-Za-z0-9]{32,}$" + length: 48 + rbac_reader_password: + description: Local password of the role-internal `baoreader` user, whose only RBAC role is `reader`. Same rationale as rbac_operator_password. + algorithm: alphanumeric + validation: "^[A-Za-z0-9]{32,}$" + length: 48 From 6e96d45e6626070c09cafa161750d7a6366daf0c Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 21:16:00 +0200 Subject: [PATCH 095/145] fix(web-app-openbao): keep the RBAC tier accounts role-local with a stable password The password resolves to the role's generated credential when OpenBao is deployed and falls back to the previous random default otherwise, so no other deployment changes. --- roles/web-app-openbao/meta/users.yml | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/roles/web-app-openbao/meta/users.yml b/roles/web-app-openbao/meta/users.yml index 6c31213ebc..a1f68ca41f 100644 --- a/roles/web-app-openbao/meta/users.yml +++ b/roles/web-app-openbao/meta/users.yml @@ -1,2 +1,16 @@ --- administrator: {} +baooperator: + description: OpenBao RBAC coverage account holding only the operator role + username: baooperator + roles: + - operator + password: "{{ (applications | default({}, true)).get('web-app-openbao', {}).get('secrets', {}).get('credentials', {}).get('rbac_operator_password') | default(42 + | strong_password, true) }}" +baoreader: + description: OpenBao RBAC coverage account holding only the reader role + username: baoreader + roles: + - reader + password: "{{ (applications | default({}, true)).get('web-app-openbao', {}).get('secrets', {}).get('credentials', {}).get('rbac_reader_password') | default(42 + | strong_password, true) }}" From 8eeb2029321dea66ef225cce3d75d5110436a65d Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 21:16:06 +0200 Subject: [PATCH 096/145] docs(web-app-openbao): record the credential-backed RBAC account passwords --- roles/web-app-openbao/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index 991d2d3dd8..d73fbf2c7d 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -115,7 +115,7 @@ docker run --rm -it \ - **Visiting `/ui/vault/logout` does not end the session.** Measured against a live instance: after that navigation the UI returns to `/ui/vault/secrets` still authenticated, because the Ember app restores its token from browser storage. Use the UI's own logout control, or clear site data, when you actually need the session gone. - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. -- **The RBAC tiers are covered by two test accounts, declared where `biber` is.** `inventories/development/default.yml` declares `baooperator` (`roles: [operator]`) and `baoreader` (`roles: [reader]`), which the standard RBAC provisioning joins to `/web-app-openbao/` in Keycloak and to the matching LDAP role group. They live in the CI inventory rather than in `meta/users.yml` for one concrete reason: a user with no explicit `password` falls back to `"{{ 42 | strong_password }}"`, and that expression is re-evaluated on every render (`secrets.choice`, nothing persisted), so the password written into LDAP and the password rendered into `playwright.env` are different strings and the login returns 403. Only an account with a literal password in the inventory can be logged in as. The same reasoning applies to any role-declared account that something needs to authenticate as. `files/playwright/test-rbac-groups.js` logs each of the three accounts in through the LDAP auth method and asserts both the policy set it receives and what it can actually do: the operator writes an application secret and the reader is refused, and only the administrator reads back an ACL policy definition. `operator` and `reader` are declared by no other role's `meta/rbac.yml`, so the two accounts hold no privileges anywhere else. +- **The RBAC tiers are covered by two role-local accounts, and they need an explicit password.** `meta/users.yml` declares `baooperator` (`roles: [operator]`) and `baoreader` (`roles: [reader]`), which the standard RBAC provisioning joins to `/web-app-openbao/` in Keycloak and to the matching LDAP role group. Each one's `password` resolves to a credential this role generates (`rbac_operator_password` / `rbac_reader_password` in `meta/secrets.yml`), **not** to the users fallback. That matters: a user with no explicit password inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings — and the LDAP bind then returns 403. The same applies to any role-declared account something has to authenticate as; `web-app-zammad` solves it the same way with `api_bot_password`. The expression falls back to the random default when OpenBao is not in the deployment, so the two accounts stay exactly as they were for every other role. `files/playwright/test-rbac-groups.js` logs each of the three accounts in through the LDAP auth method and asserts both the policy set it receives and what it can actually do: the operator writes an application secret and the reader is refused, and only the administrator reads back an ACL policy definition. `operator` and `reader` are declared by no other role's `meta/rbac.yml`, so the two accounts hold no privileges anywhere else. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). ### Backup and restore From 58aa57680e229bf8f4241980cfaf04265cc17793 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 21:16:14 +0200 Subject: [PATCH 097/145] docs(requirements): correct how the RBAC tier accounts get their password --- docs/requirements/032-web-app-openbao.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index 61db5a9139..7ebc8616aa 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -299,7 +299,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] The three baseline personas are covered per the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md): `guest` (never reaches an authenticated surface), `biber`, `administrator`, each in its own `test-*.js` module aggregated by `playwright.spec.js`, with SSO/LDAP steps guarded by `skipUnlessServiceEnabled`. `guest` runs live; `biber` and `administrator` are blocked with `PERSONA_*_BLOCKED` under the contract's documented escape, each carrying a `# nocheck: persona-block-rationale` naming the role property that forces it — biber holds no OpenBao RBAC group by design, and the shared persona helper cannot drive OpenBao's method-select + popup + `postMessage` login. The administrator's OIDC journey is asserted instead by `test-oidc-login.js`, and biber's denial by `test-ldap-login.js`. - [x] A spec asserts the RBAC denial from the criteria above (a non-member gets no administrative surface). -- [x] A spec covers every RBAC tier, not only the denial. `inventories/development/default.yml` declares `baooperator` and `baoreader` next to `biber`, whose `roles` lists put them in the operator and reader groups through the existing RBAC provisioning, and `test-rbac-groups.js` asserts for all three accounts that the policy set matches the tier and that the granted capabilities differ as the policies say: the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. **Correction from implementation:** the accounts were first declared in the role's `meta/users.yml`, which deployed them correctly but made every login return 403 — a user with no explicit `password` inherits `"{{ 42 | strong_password }}"`, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so LDAP and `playwright.env` received different strings. Test accounts that must authenticate therefore belong in the inventory, where `biber` already lives. +- [x] A spec covers every RBAC tier, not only the denial. `meta/users.yml` declares `baooperator` and `baoreader`, whose `roles` lists put them in the operator and reader groups through the existing RBAC provisioning, and `test-rbac-groups.js` asserts for all three accounts that the policy set matches the tier and that the granted capabilities differ as the policies say: the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. **Correction from implementation:** the two accounts first carried no `password`, which deployed them correctly but made every login return 403 — a user without one inherits the `strong_password` fallback, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so `ldap_passwd` and `playwright.env` could receive different strings. The fix keeps the accounts role-local and gives each a password sourced from a generated role credential (`rbac_operator_password`, `rbac_reader_password`), the same mechanism `web-app-zammad` uses for `api_bot_password`. - [x] `templates/playwright.env.j2` exposes every env var the specs read. (Enforced by `test_env_keys_used.py`, which also fails on a declared-but-unread key.) - [x] `make compose-playwright role=web-app-openbao` exits 0 with no stub tests and a clean logged-out final state. Latest deploy run: **13 passed, 2 skipped, 0 failed (30.8 s)**, no retries. From 85bd62de3028cbd730b949d0a014bf6331dc1a62 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 23:28:26 +0200 Subject: [PATCH 098/145] Revert "feat(web-app-openbao): generate the RBAC tier account passwords as role credentials" meta/*.yml is static schema, not a runtime template, and a lookup into the applications tree from there was the most runtime-dependent expression in any meta file. The accounts take ansible_become_password instead. --- roles/web-app-openbao/meta/secrets.yml | 13 ------------- 1 file changed, 13 deletions(-) diff --git a/roles/web-app-openbao/meta/secrets.yml b/roles/web-app-openbao/meta/secrets.yml index 0629dd6ea2..8bc1ec482a 100644 --- a/roles/web-app-openbao/meta/secrets.yml +++ b/roles/web-app-openbao/meta/secrets.yml @@ -14,16 +14,3 @@ credentials: description: SecretID of the Ansible AppRole, written to auth/approle/role/ansible/custom-secret-id at bootstrap. algorithm: alphanumeric validation: non_empty_string - rbac_operator_password: - description: Local password of the role-internal `baooperator` user, whose only RBAC role is `operator`. It backs the LDAP-bind login the RBAC tier spec - drives. Declared here rather than left to the users fallback because a user without an explicit password inherits the strong_password fallback expression, - which is regenerated on every render, so the value written to LDAP and the value handed to the test differ and the bind returns 403. Alphanumeric for the - same reason as web-app-zammad's api_bot_password. - algorithm: alphanumeric - validation: "^[A-Za-z0-9]{32,}$" - length: 48 - rbac_reader_password: - description: Local password of the role-internal `baoreader` user, whose only RBAC role is `reader`. Same rationale as rbac_operator_password. - algorithm: alphanumeric - validation: "^[A-Za-z0-9]{32,}$" - length: 48 From ef4b83014d640a00371d793bf8678437e06a7e0b Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 23:28:58 +0200 Subject: [PATCH 099/145] fix(web-app-openbao): give the RBAC tier accounts ansible_become_password Matches user-administrator and web-app-semaphore, the only other roles that set a password in meta/users.yml, and the key the meta lint allow-lists. It adds no exposure: user-administrator already assigns that password to the administrator account, which holds the OpenBao administrator policy through the same LDAP groups, so both accounts are strictly less privileged than one that already shares the secret. --- roles/web-app-openbao/meta/users.yml | 6 ++---- 1 file changed, 2 insertions(+), 4 deletions(-) diff --git a/roles/web-app-openbao/meta/users.yml b/roles/web-app-openbao/meta/users.yml index a1f68ca41f..d51384492a 100644 --- a/roles/web-app-openbao/meta/users.yml +++ b/roles/web-app-openbao/meta/users.yml @@ -5,12 +5,10 @@ baooperator: username: baooperator roles: - operator - password: "{{ (applications | default({}, true)).get('web-app-openbao', {}).get('secrets', {}).get('credentials', {}).get('rbac_operator_password') | default(42 - | strong_password, true) }}" + password: "{{ ansible_become_password }}" baoreader: description: OpenBao RBAC coverage account holding only the reader role username: baoreader roles: - reader - password: "{{ (applications | default({}, true)).get('web-app-openbao', {}).get('secrets', {}).get('credentials', {}).get('rbac_reader_password') | default(42 - | strong_password, true) }}" + password: "{{ ansible_become_password }}" From 3ebc68685798bc3c0b06e9e1bcc61eae3e95f1ba Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 23:29:45 +0200 Subject: [PATCH 100/145] docs(web-app-openbao): correct how the RBAC tier accounts get their password --- roles/web-app-openbao/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index d73fbf2c7d..e4f763e85c 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -115,7 +115,7 @@ docker run --rm -it \ - **Visiting `/ui/vault/logout` does not end the session.** Measured against a live instance: after that navigation the UI returns to `/ui/vault/secrets` still authenticated, because the Ember app restores its token from browser storage. Use the UI's own logout control, or clear site data, when you actually need the session gone. - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. -- **The RBAC tiers are covered by two role-local accounts, and they need an explicit password.** `meta/users.yml` declares `baooperator` (`roles: [operator]`) and `baoreader` (`roles: [reader]`), which the standard RBAC provisioning joins to `/web-app-openbao/` in Keycloak and to the matching LDAP role group. Each one's `password` resolves to a credential this role generates (`rbac_operator_password` / `rbac_reader_password` in `meta/secrets.yml`), **not** to the users fallback. That matters: a user with no explicit password inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings — and the LDAP bind then returns 403. The same applies to any role-declared account something has to authenticate as; `web-app-zammad` solves it the same way with `api_bot_password`. The expression falls back to the random default when OpenBao is not in the deployment, so the two accounts stay exactly as they were for every other role. `files/playwright/test-rbac-groups.js` logs each of the three accounts in through the LDAP auth method and asserts both the policy set it receives and what it can actually do: the operator writes an application secret and the reader is refused, and only the administrator reads back an ACL policy definition. `operator` and `reader` are declared by no other role's `meta/rbac.yml`, so the two accounts hold no privileges anywhere else. +- **The RBAC tiers are covered by two role-local accounts, and they need an explicit password.** `meta/users.yml` declares `baooperator` (`roles: [operator]`) and `baoreader` (`roles: [reader]`), which the standard RBAC provisioning joins to `/web-app-openbao/` in Keycloak and to the matching LDAP role group. Both set `password: "{{ ansible_become_password }}"`, as `user-administrator` and `web-app-semaphore` do. Leaving the field out does **not** work: a user without an explicit password inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings — and the LDAP bind then returns 403. This costs no extra exposure: `user-administrator` already gives the `administrator` account that same password, and that account holds the OpenBao *administrator* policy through the same LDAP group machinery, so these two are strictly less privileged than one that already shares the secret. `files/playwright/test-rbac-groups.js` logs each of the three accounts in through the LDAP auth method and asserts both the policy set it receives and what it can actually do: the operator writes an application secret and the reader is refused, and only the administrator reads back an ACL policy definition. `operator` and `reader` are declared by no other role's `meta/rbac.yml`, so the two accounts hold no privileges anywhere else. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). ### Backup and restore From c9e6a0fddc43da2ba58a20b5f5fa437aa37aae1f Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Fri, 11 Sep 2026 23:29:57 +0200 Subject: [PATCH 101/145] docs(requirements): correct how the RBAC tier accounts get their password --- docs/requirements/032-web-app-openbao.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index 7ebc8616aa..4e48c826f2 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -299,7 +299,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] The three baseline personas are covered per the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md): `guest` (never reaches an authenticated surface), `biber`, `administrator`, each in its own `test-*.js` module aggregated by `playwright.spec.js`, with SSO/LDAP steps guarded by `skipUnlessServiceEnabled`. `guest` runs live; `biber` and `administrator` are blocked with `PERSONA_*_BLOCKED` under the contract's documented escape, each carrying a `# nocheck: persona-block-rationale` naming the role property that forces it — biber holds no OpenBao RBAC group by design, and the shared persona helper cannot drive OpenBao's method-select + popup + `postMessage` login. The administrator's OIDC journey is asserted instead by `test-oidc-login.js`, and biber's denial by `test-ldap-login.js`. - [x] A spec asserts the RBAC denial from the criteria above (a non-member gets no administrative surface). -- [x] A spec covers every RBAC tier, not only the denial. `meta/users.yml` declares `baooperator` and `baoreader`, whose `roles` lists put them in the operator and reader groups through the existing RBAC provisioning, and `test-rbac-groups.js` asserts for all three accounts that the policy set matches the tier and that the granted capabilities differ as the policies say: the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. **Correction from implementation:** the two accounts first carried no `password`, which deployed them correctly but made every login return 403 — a user without one inherits the `strong_password` fallback, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so `ldap_passwd` and `playwright.env` could receive different strings. The fix keeps the accounts role-local and gives each a password sourced from a generated role credential (`rbac_operator_password`, `rbac_reader_password`), the same mechanism `web-app-zammad` uses for `api_bot_password`. +- [x] A spec covers every RBAC tier, not only the denial. `meta/users.yml` declares `baooperator` and `baoreader`, whose `roles` lists put them in the operator and reader groups through the existing RBAC provisioning, and `test-rbac-groups.js` asserts for all three accounts that the policy set matches the tier and that the granted capabilities differ as the policies say: the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. **Correction from implementation:** the two accounts first carried no `password`, which deployed them correctly but made every login return 403 — a user without one inherits the `strong_password` fallback, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so `ldap_passwd` and `playwright.env` could receive different strings. The fix keeps the accounts role-local and gives each `password: "{{ ansible_become_password }}"`, matching `user-administrator` and `web-app-semaphore`. It adds no exposure, because `user-administrator` already assigns that password to the `administrator` account, which holds the OpenBao administrator policy through the same LDAP groups. - [x] `templates/playwright.env.j2` exposes every env var the specs read. (Enforced by `test_env_keys_used.py`, which also fails on a declared-but-unread key.) - [x] `make compose-playwright role=web-app-openbao` exits 0 with no stub tests and a clean logged-out final state. Latest deploy run: **13 passed, 2 skipped, 0 failed (30.8 s)**, no retries. From 0053b07f2c0ec671bd835062aea55dc902d97855 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 00:17:07 +0200 Subject: [PATCH 102/145] fix(web-app-openbao): drop the dedicated RBAC tier accounts baooperator and baoreader existed only to hold one RBAC group each, and a role-declared account has no stable password until the inventory pins one, so they needed a workaround to be logged in as at all. Moving biber through the groups covers the same three tiers with an account whose password was never broken, and additionally proves the grant and the revoke rather than a steady state. --- roles/web-app-openbao/meta/users.yml | 12 ------------ 1 file changed, 12 deletions(-) diff --git a/roles/web-app-openbao/meta/users.yml b/roles/web-app-openbao/meta/users.yml index d51384492a..6c31213ebc 100644 --- a/roles/web-app-openbao/meta/users.yml +++ b/roles/web-app-openbao/meta/users.yml @@ -1,14 +1,2 @@ --- administrator: {} -baooperator: - description: OpenBao RBAC coverage account holding only the operator role - username: baooperator - roles: - - operator - password: "{{ ansible_become_password }}" -baoreader: - description: OpenBao RBAC coverage account holding only the reader role - username: baoreader - roles: - - reader - password: "{{ ansible_become_password }}" From 49201309d5586795a8e744de02e92e6b74d4cfbf Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 00:17:20 +0200 Subject: [PATCH 103/145] test(web-app-openbao): retire the account-based RBAC tier spec --- .../files/playwright/test-rbac-groups.js | 128 ------------------ 1 file changed, 128 deletions(-) delete mode 100644 roles/web-app-openbao/files/playwright/test-rbac-groups.js diff --git a/roles/web-app-openbao/files/playwright/test-rbac-groups.js b/roles/web-app-openbao/files/playwright/test-rbac-groups.js deleted file mode 100644 index f00fd8fe57..0000000000 --- a/roles/web-app-openbao/files/playwright/test-rbac-groups.js +++ /dev/null @@ -1,128 +0,0 @@ -const { test, expect } = require("@playwright/test"); -const { normalizeBaseUrl, decodeDotenvQuotedValue } = require("./personas"); -const { skipUnlessServiceEnabled } = require("./service-gating"); -const { resolveTimeout } = require("./timeouts"); - -const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); -const kvMount = decodeDotenvQuotedValue(process.env.OPENBAO_KV_MOUNT || "infinito"); - -const ACCOUNTS = { - administrator: { - username: decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""), - password: decodeDotenvQuotedValue(process.env.ADMIN_PASSWORD || ""), - }, - operator: { - username: decodeDotenvQuotedValue(process.env.BAO_OPERATOR_USERNAME || ""), - password: decodeDotenvQuotedValue(process.env.BAO_OPERATOR_PASSWORD || ""), - }, - reader: { - username: decodeDotenvQuotedValue(process.env.BAO_READER_USERNAME || ""), - password: decodeDotenvQuotedValue(process.env.BAO_READER_PASSWORD || ""), - }, -}; - -const RBAC_ROLES = Object.keys(ACCOUNTS); - -test.use({ ignoreHTTPSErrors: true }); - -async function login(request, role) { - const { username, password } = ACCOUNTS[role]; - const response = await request.post( - `${baseUrl}/v1/auth/ldap/login/${encodeURIComponent(username)}`, - { data: { password }, failOnStatusCode: false, timeout: resolveTimeout(30_000) }, - ); - expect( - response.status(), - `expected the LDAP auth method to accept '${username}' (${role}), got ${response.status()}`, - ).toBe(200); - - const body = await response.json(); - return { policies: body.auth.policies || [], token: body.auth.client_token }; -} - -test("rbac: each role group maps its member to exactly its own policy", async ({ request }) => { - skipUnlessServiceEnabled("ldap"); - - for (const role of RBAC_ROLES) { - const { policies } = await login(request, role); - expect(policies, `'${role}' must receive the '${role}' policy`).toContain(role); - - for (const other of RBAC_ROLES.filter((candidate) => candidate !== role)) { - expect( - policies, - `'${role}' must not receive the '${other}' policy, got ${policies.join(", ")}`, - ).not.toContain(other); - } - } -}); - -test("rbac: the operator may write an application secret and the reader may not", async ({ request }) => { - skipUnlessServiceEnabled("ldap"); - - const secretPath = `${baseUrl}/v1/${kvMount}/data/playwright/rbac-probe`; - const payload = { data: { data: { probe: "operator-write" } } }; - - const operator = await login(request, "operator"); - const operatorWrite = await request.post(secretPath, { - headers: { "X-Vault-Token": operator.token }, - data: payload, - failOnStatusCode: false, - timeout: resolveTimeout(30_000), - }); - expect( - [200, 204], - `the operator must be able to write an application secret, got ${operatorWrite.status()}`, - ).toContain(operatorWrite.status()); - - const reader = await login(request, "reader"); - const readerWrite = await request.post(secretPath, { - headers: { "X-Vault-Token": reader.token }, - data: payload, - failOnStatusCode: false, - timeout: resolveTimeout(30_000), - }); - expect( - readerWrite.status(), - `the reader must be refused write access, got ${readerWrite.status()}`, - ).toBe(403); - - const readerRead = await request.get(secretPath, { - headers: { "X-Vault-Token": reader.token }, - failOnStatusCode: false, - timeout: resolveTimeout(30_000), - }); - expect( - readerRead.status(), - `the reader must be able to read the secret the operator wrote, got ${readerRead.status()}`, - ).toBe(200); -}); - -test("rbac: only the administrator may read the ACL policy definitions", async ({ request }) => { - skipUnlessServiceEnabled("ldap"); - - const policyPath = `${baseUrl}/v1/sys/policies/acl/operator`; - - const administrator = await login(request, "administrator"); - const adminRead = await request.get(policyPath, { - headers: { "X-Vault-Token": administrator.token }, - failOnStatusCode: false, - timeout: resolveTimeout(30_000), - }); - expect( - adminRead.status(), - `the administrator must be able to read a policy definition, got ${adminRead.status()}`, - ).toBe(200); - - for (const role of ["operator", "reader"]) { - const { token } = await login(request, role); - const response = await request.get(policyPath, { - headers: { "X-Vault-Token": token }, - failOnStatusCode: false, - timeout: resolveTimeout(30_000), - }); - expect( - response.status(), - `'${role}' must be refused the policy definitions, got ${response.status()}`, - ).toBe(403); - } -}); From 840f59e080a908a885c4d34081f94140034d4b05 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 00:17:33 +0200 Subject: [PATCH 104/145] test(web-app-openbao): stop aggregating the retired tier spec --- roles/web-app-openbao/files/playwright/playwright.spec.js | 1 - 1 file changed, 1 deletion(-) diff --git a/roles/web-app-openbao/files/playwright/playwright.spec.js b/roles/web-app-openbao/files/playwright/playwright.spec.js index a7682701d9..2e6b3c8328 100644 --- a/roles/web-app-openbao/files/playwright/playwright.spec.js +++ b/roles/web-app-openbao/files/playwright/playwright.spec.js @@ -5,5 +5,4 @@ require("./test-administrator-persona"); require("./test-oidc-login"); require("./test-ldap-login"); require("./test-rbac-denial"); -require("./test-rbac-groups"); require("./test-seal-status"); From df4be52aa6ee15f26279fbfebf42d42e46cc843e Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 00:17:42 +0200 Subject: [PATCH 105/145] test(web-app-openbao): drop the env keys the retired tier spec consumed --- roles/web-app-openbao/templates/playwright.env.j2 | 10 ---------- 1 file changed, 10 deletions(-) diff --git a/roles/web-app-openbao/templates/playwright.env.j2 b/roles/web-app-openbao/templates/playwright.env.j2 index 9f9e2976c0..7e661d027c 100644 --- a/roles/web-app-openbao/templates/playwright.env.j2 +++ b/roles/web-app-openbao/templates/playwright.env.j2 @@ -4,14 +4,6 @@ ADMIN_USERNAME={{ lookup('users', 'administrator').username | dotenv_quote }} APP_BASE_URL={{ lookup('tls', application_id, 'url.base') }} -BAO_OPERATOR_PASSWORD={{ lookup('users', 'baooperator').password | dotenv_quote }} - -BAO_OPERATOR_USERNAME={{ lookup('users', 'baooperator').username | dotenv_quote }} - -BAO_READER_PASSWORD={{ lookup('users', 'baoreader').password | dotenv_quote }} - -BAO_READER_USERNAME={{ lookup('users', 'baoreader').username | dotenv_quote }} - BIBER_PASSWORD={{ lookup('users', 'biber').password | dotenv_quote }} BIBER_USERNAME={{ lookup('users', 'biber').username | dotenv_quote }} @@ -37,8 +29,6 @@ OIDC_ISSUER_URL={{ OIDC.CLIENT.ISSUER_URL | dotenv_quote }} OPENBAO_BASE_URL={{ lookup('tls', application_id, 'url.base') | dotenv_quote }} -OPENBAO_KV_MOUNT={{ OPENBAO_KV_MOUNT | dotenv_quote }} - # nocheck: persona-block-rationale - OpenBao's UI login is not a link-then-Keycloak journey the shared persona helper can drive: the Ember auth form requires selecting the OIDC method, resolving the role through POST /v1/auth/oidc/oidc/auth_url, then completing Keycloak in a popup that posts its result back via postMessage (see upstream ui/app/components/auth-jwt.js). The administrator's full OIDC journey is asserted by the role-local files/playwright/test-oidc-login.js instead; with SSO off there is no human login path at all, since OpenBao offers no username/password auth. PERSONA_ADMINISTRATOR_BLOCKED=true From 07d5a7180b23339f3e3c789399e9f0f3a14b019c Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 00:18:00 +0200 Subject: [PATCH 106/145] docs(web-app-openbao): record the role-declared account password trap --- roles/web-app-openbao/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index e4f763e85c..4e2d82f411 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -115,7 +115,7 @@ docker run --rm -it \ - **Visiting `/ui/vault/logout` does not end the session.** Measured against a live instance: after that navigation the UI returns to `/ui/vault/secrets` still authenticated, because the Ember app restores its token from browser storage. Use the UI's own logout control, or clear site data, when you actually need the session gone. - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. -- **The RBAC tiers are covered by two role-local accounts, and they need an explicit password.** `meta/users.yml` declares `baooperator` (`roles: [operator]`) and `baoreader` (`roles: [reader]`), which the standard RBAC provisioning joins to `/web-app-openbao/` in Keycloak and to the matching LDAP role group. Both set `password: "{{ ansible_become_password }}"`, as `user-administrator` and `web-app-semaphore` do. Leaving the field out does **not** work: a user without an explicit password inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings — and the LDAP bind then returns 403. This costs no extra exposure: `user-administrator` already gives the `administrator` account that same password, and that account holds the OpenBao *administrator* policy through the same LDAP group machinery, so these two are strictly less privileged than one that already shares the secret. `files/playwright/test-rbac-groups.js` logs each of the three accounts in through the LDAP auth method and asserts both the policy set it receives and what it can actually do: the operator writes an application secret and the reader is refused, and only the administrator reads back an ACL policy definition. `operator` and `reader` are declared by no other role's `meta/rbac.yml`, so the two accounts hold no privileges anywhere else. +- **A role-declared account cannot be logged in as unless it sets a password.** The tiers were first covered by two role-local accounts, `baooperator` and `baoreader`, which deployed and joined their RBAC groups correctly but returned 403 on every LDAP bind: a user declared without an explicit `password` inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings. Both accounts are gone now — the tiers are covered by moving `biber` through the groups instead — but the trap applies to any future role-declared account something has to authenticate as. Until [`feature/ai-agent-employees`](https://github.com/kevinveenbirkenbach/infinito-nexus-core/tree/refs/heads/feature/ai-agent-employees) lands (`f4398e1c5` pins a password per declared user at inventory creation), such an account needs an explicit `password:`, as `user-administrator` and `web-app-semaphore` do. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). ### Backup and restore From d6cc7c376b122927be91607727267ea80ec5b3bc Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 00:18:10 +0200 Subject: [PATCH 107/145] docs(requirements): restate the RBAC tier criterion around the LAM group rotation --- docs/requirements/032-web-app-openbao.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index 4e48c826f2..5cbf8f3e0c 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -299,7 +299,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] The three baseline personas are covered per the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md): `guest` (never reaches an authenticated surface), `biber`, `administrator`, each in its own `test-*.js` module aggregated by `playwright.spec.js`, with SSO/LDAP steps guarded by `skipUnlessServiceEnabled`. `guest` runs live; `biber` and `administrator` are blocked with `PERSONA_*_BLOCKED` under the contract's documented escape, each carrying a `# nocheck: persona-block-rationale` naming the role property that forces it — biber holds no OpenBao RBAC group by design, and the shared persona helper cannot drive OpenBao's method-select + popup + `postMessage` login. The administrator's OIDC journey is asserted instead by `test-oidc-login.js`, and biber's denial by `test-ldap-login.js`. - [x] A spec asserts the RBAC denial from the criteria above (a non-member gets no administrative surface). -- [x] A spec covers every RBAC tier, not only the denial. `meta/users.yml` declares `baooperator` and `baoreader`, whose `roles` lists put them in the operator and reader groups through the existing RBAC provisioning, and `test-rbac-groups.js` asserts for all three accounts that the policy set matches the tier and that the granted capabilities differ as the policies say: the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. **Correction from implementation:** the two accounts first carried no `password`, which deployed them correctly but made every login return 403 — a user without one inherits the `strong_password` fallback, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so `ldap_passwd` and `playwright.env` could receive different strings. The fix keeps the accounts role-local and gives each `password: "{{ ansible_become_password }}"`, matching `user-administrator` and `web-app-semaphore`. It adds no exposure, because `user-administrator` already assigns that password to the `administrator` account, which holds the OpenBao administrator policy through the same LDAP groups. +- [ ] A spec covers every RBAC tier, not only the denial: `biber` starts in no group and reaches no privileged surface, then an administrator adds him to each of the three RBAC groups in turn through LAM, and after every change he logs in again and the spec asserts the policy set and the capability that tier is meant to grant — the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. Removing him between rounds proves the revoke as well as the grant. **Correction from implementation:** this was first built with two dedicated accounts, `baooperator` and `baoreader`, which deployed and joined their groups correctly but returned 403 on every LDAP bind — a user declared without an explicit `password` inherits the `strong_password` fallback, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so `ldap_passwd` and `playwright.env` could receive different strings. Moving `biber` between the groups drops both accounts, needs no password workaround, and covers strictly more: the static members could only ever show a steady state, never a grant or a revoke taking effect. - [x] `templates/playwright.env.j2` exposes every env var the specs read. (Enforced by `test_env_keys_used.py`, which also fails on a declared-but-unread key.) - [x] `make compose-playwright role=web-app-openbao` exits 0 with no stub tests and a clean logged-out final state. Latest deploy run: **13 passed, 2 skipped, 0 failed (30.8 s)**, no retries. From 4ca69f12447ad2e54eaae9698dc68e9087cf36b2 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 10:22:21 +0200 Subject: [PATCH 108/145] fix(web-app-lam): allow the inline script that drives LAM's asynchronous tools script-src-elem.unsafe-inline was off, so the progress driver LAM renders inline was blocked by the CSP. LDAP import/export and Multi edit posted their job, rendered "Status: in progress" and never polled it, leaving both tools hanging for every user. Found while driving Multi edit from a Playwright spec; the directory was verifiably unmodified afterwards, so it was not a display artefact. --- roles/web-app-lam/meta/csp.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-lam/meta/csp.yml b/roles/web-app-lam/meta/csp.yml index 5ef6fc16a5..8f479faa84 100644 --- a/roles/web-app-lam/meta/csp.yml +++ b/roles/web-app-lam/meta/csp.yml @@ -3,7 +3,7 @@ flags: style-src-attr: unsafe-inline: true # nocheck: csp-unsafe Reason: historical, TODO: re-verify it is really required and does not open attack surface script-src-elem: - unsafe-inline: false + unsafe-inline: true # nocheck: csp-unsafe Reason: LAM renders the progress driver of its asynchronous tools as an inline script; with this off, LDAP import/export and multi edit post their job, display "Status: in progress" and never poll it, so both tools hang for every user unsafe-eval: true # nocheck: csp-unsafe Reason: historical, TODO: re-verify it is really required and does not open attack surface script-src-attr: unsafe-inline: true # nocheck: csp-unsafe Reason: historical, TODO: re-verify it is really required and does not open attack surface From 1a54f10e0e28bb878742c8b3540072786b63924c Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 10:22:35 +0200 Subject: [PATCH 109/145] test(web-app-openbao): deploy LAM in variant 0 so the RBAC rotation has an admin UI --- roles/web-app-openbao/meta/variants.yml | 3 +++ 1 file changed, 3 insertions(+) diff --git a/roles/web-app-openbao/meta/variants.yml b/roles/web-app-openbao/meta/variants.yml index c66952f1c0..ba5eecbbfb 100644 --- a/roles/web-app-openbao/meta/variants.yml +++ b/roles/web-app-openbao/meta/variants.yml @@ -11,6 +11,9 @@ ldap: enabled: true shared: true + lam: # nocheck: variants-services-match + enabled: true + shared: true logout: enabled: true shared: true From 15ad629375c1c23b81af0b052599bcc7acf4c0c7 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 10:22:41 +0200 Subject: [PATCH 110/145] test(web-app-openbao): expose LAM and the RBAC DN templates to Playwright --- .../web-app-openbao/templates/playwright.env.j2 | 16 ++++++++++++++++ 1 file changed, 16 insertions(+) diff --git a/roles/web-app-openbao/templates/playwright.env.j2 b/roles/web-app-openbao/templates/playwright.env.j2 index 7e661d027c..930d5a7e6a 100644 --- a/roles/web-app-openbao/templates/playwright.env.j2 +++ b/roles/web-app-openbao/templates/playwright.env.j2 @@ -10,6 +10,20 @@ BIBER_USERNAME={{ lookup('users', 'biber').username | dotenv_quote }} CANONICAL_DOMAIN={{ lookup('domain', application_id) | dotenv_quote }} +{% if lookup('config', application_id, 'services.lam.enabled', false) | bool %} +LAM_BASE_URL={{ lookup('tls', 'web-app-lam', 'url.base') | dotenv_quote }} + +LAM_OAUTH2_FRONTED={{ lookup('sso', 'web-app-lam', 'is_proxy_gated') | string | lower }} + +LDAP_ADMIN_PASSWORD={{ LDAP.BIND_CREDENTIAL | dotenv_quote }} + +LDAP_RBAC_GROUP_DN_TEMPLATE={{ ('cn=' ~ application_id ~ '-,' ~ LDAP.DN.OU.ROLES) | dotenv_quote }} + +LDAP_USER_DN_TEMPLATE={{ (LDAP.USER.ATTRIBUTES.ID ~ '=,' ~ LDAP.DN.OU.USERS) | dotenv_quote }} + +{% endif %} +LAM_SERVICE_ENABLED={{ lookup('config', application_id, 'services.lam.enabled', false) | string | lower }} + # nocheck: playwright-service-gate - CSS injection is asserted by the central CSP helper, not a per-persona surface CSS_SERVICE_ENABLED={{ lookup('config', application_id, 'services.css.enabled') | string | lower }} @@ -29,6 +43,8 @@ OIDC_ISSUER_URL={{ OIDC.CLIENT.ISSUER_URL | dotenv_quote }} OPENBAO_BASE_URL={{ lookup('tls', application_id, 'url.base') | dotenv_quote }} +OPENBAO_KV_MOUNT={{ OPENBAO_KV_MOUNT | dotenv_quote }} + # nocheck: persona-block-rationale - OpenBao's UI login is not a link-then-Keycloak journey the shared persona helper can drive: the Ember auth form requires selecting the OIDC method, resolving the role through POST /v1/auth/oidc/oidc/auth_url, then completing Keycloak in a popup that posts its result back via postMessage (see upstream ui/app/components/auth-jwt.js). The administrator's full OIDC journey is asserted by the role-local files/playwright/test-oidc-login.js instead; with SSO off there is no human login path at all, since OpenBao offers no username/password auth. PERSONA_ADMINISTRATOR_BLOCKED=true From 436525ca246cbb87d5d980054b9aacc4a987d9bd Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 10:22:51 +0200 Subject: [PATCH 111/145] test(web-app-openbao): prove every RBAC tier by moving biber through the groups in LAM biber starts in no group and reaches no privileged surface. For each tier the administrator signs into LAM, adds him to the role group through Multi edit, and the spec asserts both the policy set he receives and the capability that tier grants, then removes him and asserts the revoke. One account covers all three tiers and the grant and revoke are observed rather than assumed. LAM's group list renders posixGroup entries only, while the RBAC role groups are groupOfNames under ou=roles, so Multi edit is the surface that reaches them. It applies the change asynchronously, so the spec polls the resulting policy set rather than LAM's own progress wording. --- .../files/playwright/test-rbac-groups.js | 215 ++++++++++++++++++ 1 file changed, 215 insertions(+) create mode 100644 roles/web-app-openbao/files/playwright/test-rbac-groups.js diff --git a/roles/web-app-openbao/files/playwright/test-rbac-groups.js b/roles/web-app-openbao/files/playwright/test-rbac-groups.js new file mode 100644 index 0000000000..faab72ca6e --- /dev/null +++ b/roles/web-app-openbao/files/playwright/test-rbac-groups.js @@ -0,0 +1,215 @@ +const { test, expect } = require("@playwright/test"); +const { normalizeBaseUrl, decodeDotenvQuotedValue, gotoOnion } = require("./personas"); +const { skipUnlessServiceEnabled, isServiceEnabled } = require("./service-gating"); +const { resolveTimeout } = require("./timeouts"); + +const baseUrl = normalizeBaseUrl(process.env.OPENBAO_BASE_URL || ""); +const kvMount = decodeDotenvQuotedValue(process.env.OPENBAO_KV_MOUNT || ""); +const biberUsername = decodeDotenvQuotedValue(process.env.BIBER_USERNAME || ""); +const biberPassword = decodeDotenvQuotedValue(process.env.BIBER_PASSWORD || ""); +const lamBaseUrl = normalizeBaseUrl(process.env.LAM_BASE_URL || ""); +const lamOauth2Fronted = String(process.env.LAM_OAUTH2_FRONTED || "").toLowerCase() === "true"; +const adminUsername = decodeDotenvQuotedValue(process.env.ADMIN_USERNAME || ""); +const adminPassword = decodeDotenvQuotedValue(process.env.ADMIN_PASSWORD || ""); +const ldapAdminPassword = decodeDotenvQuotedValue(process.env.LDAP_ADMIN_PASSWORD || ""); +const groupDnTemplate = decodeDotenvQuotedValue(process.env.LDAP_RBAC_GROUP_DN_TEMPLATE || ""); +const userDnTemplate = decodeDotenvQuotedValue(process.env.LDAP_USER_DN_TEMPLATE || ""); + +const RBAC_ROLES = ["administrator", "operator", "reader"]; +const PROBE_PATH = `${kvMount}/data/playwright/rbac-probe`; +const POLICY_PATH = "sys/policies/acl/operator"; + +const groupDn = (role) => groupDnTemplate.replace("", role); +const biberDn = () => userDnTemplate.replace("", biberUsername); +const groupCn = (role) => groupDn(role).replace(/^cn=/, "").split(",")[0]; +const rootSuffix = () => groupDn("administrator").split(",").slice(-2).join(","); + +test.use({ ignoreHTTPSErrors: true }); + +async function signInToLam(page) { + await gotoOnion(page, `${lamBaseUrl}/lam/templates/login.php`, { waitUntil: "load" }); + + if (lamOauth2Fronted) { + const keycloakUsername = page.locator("input[name='username'], input#username").first(); + if (await keycloakUsername.isVisible().catch(() => false)) { + await keycloakUsername.fill(adminUsername); + await page.locator("input[name='password'], input#password").first().fill(adminPassword); + await page + .locator("button[type='submit'], input[name='login'], input[type='submit']") + .first() + .click({ timeout: resolveTimeout(30_000) }); + await page.waitForLoadState("networkidle"); + } + } + + const lamPassword = page.locator("input[name='passwd'], input#passwd").first(); + await expect(lamPassword, "LAM must present its bind-password form").toBeVisible({ + timeout: resolveTimeout(30_000), + }); + await lamPassword.fill(ldapAdminPassword); + await page + .locator("button[type='submit'], input[type='submit']") + .first() + .click({ timeout: resolveTimeout(30_000) }); + await page.waitForLoadState("networkidle"); + + await expect( + page.locator("a[href*='list.php?type=user']").first(), + "the administrator must reach LAM's account menu after signing in", + ).toBeVisible({ timeout: resolveTimeout(30_000) }); +} + +// LAM's group list only renders posixGroup entries, and the RBAC role groups are +// groupOfNames under ou=roles, so the members are edited through the Multi edit +// tool, which applies an attribute operation to every entry matching a filter. +async function setGroupMembership(page, role, operation) { + await gotoOnion(page, `${lamBaseUrl}/lam/templates/tools/multiEdit.php`, { waitUntil: "load" }); + await page.waitForLoadState("networkidle"); + + await page.locator("select#suffix").selectOption(rootSuffix()); + await page.locator("input#filter").fill(`(cn=${groupCn(role)})`); + await page.locator("select#op_0").selectOption(operation); + await page.locator("input#attr_0").fill("member"); + await page.locator("input#val_0").fill(biberDn()); + + await page.locator("#btn_applyChanges").click({ timeout: resolveTimeout(30_000) }); +} + +// LAM applies the change asynchronously, so the directory is only consistent +// some time after the click. Poll the effect rather than the tool's own wording. +async function expectTierAfterLam(request, role, present) { + await expect + .poll(async () => (await loginAsBiber(request)).policies.includes(role), { + timeout: resolveTimeout(60_000), + message: present + ? `LAM adding ${biberDn()} to ${groupCn(role)} must grant the '${role}' policy` + : `LAM removing ${biberDn()} from ${groupCn(role)} must revoke the '${role}' policy`, + }) + .toBe(present); + return loginAsBiber(request); +} + +async function loginAsBiber(request) { + const response = await request.post( + `${baseUrl}/v1/auth/ldap/login/${encodeURIComponent(biberUsername)}`, + { data: { password: biberPassword }, failOnStatusCode: false, timeout: resolveTimeout(30_000) }, + ); + expect( + response.status(), + `the LDAP auth method must accept ${biberUsername}, got ${response.status()}`, + ).toBe(200); + const body = await response.json(); + return { policies: body.auth.policies || [], token: body.auth.client_token }; +} + +const writeProbe = (request, token) => + request + .post(`${baseUrl}/v1/${PROBE_PATH}`, { + headers: { "X-Vault-Token": token }, + data: { data: { probe: "rbac" } }, + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }) + .then((r) => r.status()); + +const readProbe = (request, token) => + request + .get(`${baseUrl}/v1/${PROBE_PATH}`, { + headers: { "X-Vault-Token": token }, + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }) + .then((r) => r.status()); + +const readPolicy = (request, token) => + request + .get(`${baseUrl}/v1/${POLICY_PATH}`, { + headers: { "X-Vault-Token": token }, + failOnStatusCode: false, + timeout: resolveTimeout(30_000), + }) + .then((r) => r.status()); + +// What each tier must be able to do, read off templates/policies/*.hcl.j2. +const TIER_CAPABILITIES = { + administrator: async (request, token) => { + expect( + await readPolicy(request, token), + "the administrator tier must read a policy definition", + ).toBe(200); + }, + operator: async (request, token) => { + expect( + [200, 204], + "the operator tier must write an application secret", + ).toContain(await writeProbe(request, token)); + expect( + await readPolicy(request, token), + "the operator tier must not read policy definitions", + ).toBe(403); + }, + reader: async (request, token) => { + expect( + await writeProbe(request, token), + "the reader tier must not write an application secret", + ).toBe(403); + expect( + [200, 404], + "the reader tier must be permitted to read the application path", + ).toContain(await readProbe(request, token)); + }, +}; + +test.describe("rbac: the OpenBao tiers follow LDAP group membership", () => { + test.describe.configure({ mode: "serial" }); + + test("rbac: biber starts outside every OpenBao role group", async ({ request }) => { + skipUnlessServiceEnabled("ldap"); + + const { policies, token } = await loginAsBiber(request); + for (const role of RBAC_ROLES) { + expect(policies, `biber must hold no '${role}' policy before any group change`).not.toContain( + role, + ); + } + expect( + await readPolicy(request, token), + "a member of no role group must not read policy definitions", + ).toBe(403); + expect( + await writeProbe(request, token), + "a member of no role group must not write an application secret", + ).toBe(403); + }); + + for (const role of RBAC_ROLES) { + test(`rbac: LAM granting the ${role} group hands biber that tier, and revoking takes it back`, async ({ + page, + request, + }) => { + skipUnlessServiceEnabled("ldap"); + test.skip(!isServiceEnabled("lam"), "LAM not deployed (LAM_SERVICE_ENABLED=false)"); + test.setTimeout(resolveTimeout(180_000)); + + await signInToLam(page); + await setGroupMembership(page, role, "add"); + + const granted = await expectTierAfterLam(request, role, true); + for (const other of RBAC_ROLES.filter((candidate) => candidate !== role)) { + expect( + granted.policies, + `biber must not receive the '${other}' policy while only in the ${role} group`, + ).not.toContain(other); + } + await TIER_CAPABILITIES[role](request, granted.token); + + await setGroupMembership(page, role, "del"); + + const revoked = await expectTierAfterLam(request, role, false); + expect( + await readPolicy(request, revoked.token), + "a revoked member must not read policy definitions", + ).toBe(403); + }); + } +}); From 7a4f1104a605b33de872f680f2f57ffc0f19aef2 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 10:23:00 +0200 Subject: [PATCH 112/145] test(web-app-openbao): aggregate the RBAC tier rotation spec --- roles/web-app-openbao/files/playwright/playwright.spec.js | 1 + 1 file changed, 1 insertion(+) diff --git a/roles/web-app-openbao/files/playwright/playwright.spec.js b/roles/web-app-openbao/files/playwright/playwright.spec.js index 2e6b3c8328..a7682701d9 100644 --- a/roles/web-app-openbao/files/playwright/playwright.spec.js +++ b/roles/web-app-openbao/files/playwright/playwright.spec.js @@ -5,4 +5,5 @@ require("./test-administrator-persona"); require("./test-oidc-login"); require("./test-ldap-login"); require("./test-rbac-denial"); +require("./test-rbac-groups"); require("./test-seal-status"); From e8693e69b1c9637f1e3ed7c7ecbee0b142672108 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 10:23:17 +0200 Subject: [PATCH 113/145] docs(web-app-openbao): record the LAM-driven tier rotation and the stale-volume trap --- roles/web-app-openbao/README.md | 2 ++ 1 file changed, 2 insertions(+) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index 4e2d82f411..00524ea6b4 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -115,6 +115,8 @@ docker run --rm -it \ - **Visiting `/ui/vault/logout` does not end the session.** Measured against a live instance: after that navigation the UI returns to `/ui/vault/secrets` still authenticated, because the Ember app restores its token from browser storage. Use the UI's own logout control, or clear site data, when you actually need the session gone. - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. +- **The RBAC tiers are proven by moving `biber` through the groups, not by dedicated accounts.** `files/playwright/test-rbac-groups.js` signs the administrator into LAM, adds `biber` to `cn=web-app-openbao-,ou=roles` through LAM's Multi edit tool, logs `biber` into OpenBao over LDAP and asserts both the policy set and the capability that tier grants, then removes him again and asserts the revoke. One account covers all three tiers, and the grant and the revoke are observed rather than assumed — a permanently-assigned member can only ever show a steady state. LAM's *group list* cannot be used for this: it renders `posixGroup` entries, while the RBAC role groups are `groupOfNames` under `ou=roles`. LAM applies the change asynchronously, so the spec polls the resulting policy set instead of LAM's own progress wording. +- **Redeploying locally after the inventory regenerates leaves stale state that looks like a role bug.** The development inventory regenerates credentials per deploy while docker volumes survive, so anything that stores a credential *inside* its volume desynchronises: the raft store stays `initialized: true, sealed: true` when `seal_key` no longer matches, the Ansible AppRole reports `invalid role or secret ID` when it was pinned at an earlier bootstrap, and a surviving `mariadb_data` fails its healthcheck with `Access denied for user 'root'@'127.0.0.1'`. None of these are defects; CI never sees them because every run starts clean. Run `make compose-entity-purge apps=web-app-openbao` (or the affected entity) before redeploying locally. - **A role-declared account cannot be logged in as unless it sets a password.** The tiers were first covered by two role-local accounts, `baooperator` and `baoreader`, which deployed and joined their RBAC groups correctly but returned 403 on every LDAP bind: a user declared without an explicit `password` inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings. Both accounts are gone now — the tiers are covered by moving `biber` through the groups instead — but the trap applies to any future role-declared account something has to authenticate as. Until [`feature/ai-agent-employees`](https://github.com/kevinveenbirkenbach/infinito-nexus-core/tree/refs/heads/feature/ai-agent-employees) lands (`f4398e1c5` pins a password per declared user at inventory creation), such an account needs an explicit `password:`, as `user-administrator` and `web-app-semaphore` do. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). From c8f0c8f246b6b5bd630d9b19ecb53a22c2fda6f8 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 12 Sep 2026 10:23:25 +0200 Subject: [PATCH 114/145] docs(requirements): close the RBAC tier criterion and note the LAM CSP defect --- docs/requirements/032-web-app-openbao.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/requirements/032-web-app-openbao.md b/docs/requirements/032-web-app-openbao.md index 5cbf8f3e0c..97f9c014c0 100644 --- a/docs/requirements/032-web-app-openbao.md +++ b/docs/requirements/032-web-app-openbao.md @@ -299,7 +299,7 @@ cluster_addr = "http://127.0.0.1:8201" - [x] The three baseline personas are covered per the [Playwright contract](../contributing/artefact/files/role/playwright.specs.js.md): `guest` (never reaches an authenticated surface), `biber`, `administrator`, each in its own `test-*.js` module aggregated by `playwright.spec.js`, with SSO/LDAP steps guarded by `skipUnlessServiceEnabled`. `guest` runs live; `biber` and `administrator` are blocked with `PERSONA_*_BLOCKED` under the contract's documented escape, each carrying a `# nocheck: persona-block-rationale` naming the role property that forces it — biber holds no OpenBao RBAC group by design, and the shared persona helper cannot drive OpenBao's method-select + popup + `postMessage` login. The administrator's OIDC journey is asserted instead by `test-oidc-login.js`, and biber's denial by `test-ldap-login.js`. - [x] A spec asserts the RBAC denial from the criteria above (a non-member gets no administrative surface). -- [ ] A spec covers every RBAC tier, not only the denial: `biber` starts in no group and reaches no privileged surface, then an administrator adds him to each of the three RBAC groups in turn through LAM, and after every change he logs in again and the spec asserts the policy set and the capability that tier is meant to grant — the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. Removing him between rounds proves the revoke as well as the grant. **Correction from implementation:** this was first built with two dedicated accounts, `baooperator` and `baoreader`, which deployed and joined their groups correctly but returned 403 on every LDAP bind — a user declared without an explicit `password` inherits the `strong_password` fallback, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so `ldap_passwd` and `playwright.env` could receive different strings. Moving `biber` between the groups drops both accounts, needs no password workaround, and covers strictly more: the static members could only ever show a steady state, never a grant or a revoke taking effect. +- [x] A spec covers every RBAC tier, not only the denial: `biber` starts in no group and reaches no privileged surface, then an administrator adds him to each of the three RBAC groups in turn through LAM, and after every change he logs in again and the spec asserts the policy set and the capability that tier is meant to grant — the operator writes an application secret where the reader gets 403, the reader still reads it back, and only the administrator reads an ACL policy definition. Removing him between rounds proves the revoke as well as the grant. **Correction from implementation:** this was first built with two dedicated accounts, `baooperator` and `baoreader`, which deployed and joined their groups correctly but returned 403 on every LDAP bind — a user declared without an explicit `password` inherits the `strong_password` fallback, which `utils/templating/ansible.py` re-evaluates through `secrets.choice` on each render with nothing persisted, so `ldap_passwd` and `playwright.env` could receive different strings. Moving `biber` between the groups drops both accounts, needs no password workaround, and covers strictly more: the static members could only ever show a steady state, never a grant or a revoke taking effect. Building it also surfaced a defect in `web-app-lam`: `meta/csp.yml` set `script-src-elem.unsafe-inline: false`, which blocked the inline script LAM renders to drive its asynchronous tools, so LDAP import/export and Multi edit posted their job, displayed `Status: in progress` and never polled it — broken for every user, not just the spec. - [x] `templates/playwright.env.j2` exposes every env var the specs read. (Enforced by `test_env_keys_used.py`, which also fails on a declared-but-unread key.) - [x] `make compose-playwright role=web-app-openbao` exits 0 with no stub tests and a clean logged-out final state. Latest deploy run: **13 passed, 2 skipped, 0 failed (30.8 s)**, no retries. From 712de112835508343846a6788597297107343a77 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 02:14:25 +0200 Subject: [PATCH 115/145] feat(openbao): derive the seal key id and resolve the rotating credentials The platform regenerates generated-algorithm credentials on every deploy, so the seal key and AppRole this role is handed are not the ones the running instance holds. Derive current_key_id from the key itself -- upstream requires the identifier to match its key, and a derived id cannot drift out of step with one. Carry the previously applied credentials and the two rotation predicates alongside it, so the config render and the AppRole swap can both tell a rotation from a steady state. --- roles/web-app-openbao/vars/main.yml | 20 +++++++++++++++++++- 1 file changed, 19 insertions(+), 1 deletion(-) diff --git a/roles/web-app-openbao/vars/main.yml b/roles/web-app-openbao/vars/main.yml index ecaf9a74b7..d5caa506b1 100644 --- a/roles/web-app-openbao/vars/main.yml +++ b/roles/web-app-openbao/vars/main.yml @@ -20,7 +20,25 @@ OPENBAO_SEAL_KEY: "{{ lookup('config', application_id, 'secrets.credentials.seal OPENBAO_APPROLE_ROLE_ID: "{{ lookup('config', application_id, 'secrets.credentials.approle_role_id') }}" OPENBAO_APPROLE_SECRET_ID: "{{ lookup('config', application_id, 'secrets.credentials.approle_secret_id') }}" -OPENBAO_SEAL_KEY_ID: "infinito-1" +# Rationale: the platform rotates every generated credential on each deploy, so +# the key id is derived from the key instead of counted — upstream requires the +# identifier to match its key, and a derived id cannot drift out of step with one. +OPENBAO_SEAL_KEY_ID: "{{ (OPENBAO_SEAL_KEY | hash('sha256'))[:16] }}" + +# Rationale: what the previous deploy applied, read back before the config is +# rendered so the node can unseal with the old key and rewrap to the new one, and +# so the AppRole can still authenticate after its credentials rotated. +OPENBAO_APPLIED_STATE_HOST: "{{ [lookup('container', application_id, 'directories.instance'), '.applied.json'] | path_join }}" +OPENBAO_APPLIED: "{{ openbao_applied_state | default({}, true) }}" +OPENBAO_SEAL_KEY_PREVIOUS: "{{ OPENBAO_APPLIED.seal_key | default('', true) }}" +OPENBAO_SEAL_KEY_PREVIOUS_ID: "{{ (OPENBAO_SEAL_KEY_PREVIOUS | hash('sha256'))[:16] if OPENBAO_SEAL_KEY_PREVIOUS else '' }}" +OPENBAO_SEAL_ROTATING: "{{ (OPENBAO_SEAL_KEY_PREVIOUS | length > 0) and (OPENBAO_SEAL_KEY_PREVIOUS != OPENBAO_SEAL_KEY) }}" + +# Rationale: resolved from the login rather than from the state file — the +# SecretID OpenBao actually holds is whichever one was just accepted, and +# re-registering that same value answers 500, not a no-op. +OPENBAO_SECRET_ID_REGISTERED: "{{ openbao_secret_id_registered | default('', true) }}" +OPENBAO_SECRET_ID_ROTATING: "{{ (OPENBAO_SECRET_ID_REGISTERED | length > 0) and (OPENBAO_SECRET_ID_REGISTERED != OPENBAO_APPROLE_SECRET_ID) }}" OPENBAO_KV_MOUNT: "infinito" OPENBAO_ANSIBLE_APPROLE: "ansible" OPENBAO_PKI_MOUNT: "pki" From 5c8721ee90fb36bb97c5373008f7283b1c2274d3 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 02:14:32 +0200 Subject: [PATCH 116/145] feat(openbao): read the applied credentials before the container is rendered sys-stk-backend renders env.j2, which is what carries the previous seal key into the container. Reading the applied state after that include leaves the node with no way to decrypt its own store, so the read has to precede it. --- roles/web-app-openbao/tasks/00_core.yml | 21 +++++++++++++++++++++ 1 file changed, 21 insertions(+) diff --git a/roles/web-app-openbao/tasks/00_core.yml b/roles/web-app-openbao/tasks/00_core.yml index c44d5b44b5..fae44950be 100644 --- a/roles/web-app-openbao/tasks/00_core.yml +++ b/roles/web-app-openbao/tasks/00_core.yml @@ -1,6 +1,27 @@ --- - include_tasks: utils/once/flag.yml +# Exception: must precede sys-stk-backend — that role renders env.j2, which +# carries the previous seal key the node needs to unseal data the last deploy's +# key wrapped. Moving it after leaves the node sealed with no way back. +- name: "🔍 Read the credentials the previous deploy applied" + ansible.builtin.slurp: + src: "{{ OPENBAO_APPLIED_STATE_HOST }}" + register: openbao_applied_raw + failed_when: false + changed_when: false + delegate_to: "{{ STACK_HOST }}" + run_once: true + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🔧 Keep the previously applied credentials for the rotation path" + ansible.builtin.set_fact: + openbao_applied_state: >- + {{ (openbao_applied_raw.content | b64decode | from_json) + if (openbao_applied_raw.content is defined) + else {} }} + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + - name: "🧩 load docker for {{ application_id }}" include_role: name: sys-stk-backend From 4046dd2a0ad7c8bd273e5d2d03f50d2c5b8c45c3 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 02:14:48 +0200 Subject: [PATCH 117/145] feat(openbao): offer the previous seal key while the key is rotating The static seal supports n-1 to n rotation. Emitting the previous pair only while the two keys differ lets the node unseal with the old key and rewrap to the new one; a first deploy still renders a single-key stanza. --- roles/web-app-openbao/templates/openbao.hcl.j2 | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/roles/web-app-openbao/templates/openbao.hcl.j2 b/roles/web-app-openbao/templates/openbao.hcl.j2 index 044b0a15b8..9fc02c3bbc 100644 --- a/roles/web-app-openbao/templates/openbao.hcl.j2 +++ b/roles/web-app-openbao/templates/openbao.hcl.j2 @@ -21,6 +21,10 @@ listener "tcp" { seal "static" { current_key_id = "{{ OPENBAO_SEAL_KEY_ID }}" current_key = "env://OPENBAO_SEAL_KEY" +{% if OPENBAO_SEAL_ROTATING | bool %} + previous_key_id = "{{ OPENBAO_SEAL_KEY_PREVIOUS_ID }}" + previous_key = "env://OPENBAO_SEAL_KEY_PREVIOUS" +{% endif %} } telemetry { From 1a2bbb3c972be6db8aa23fb8ff740a9684c73a0b Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 02:14:59 +0200 Subject: [PATCH 118/145] feat(openbao): export the previous seal key to the container while rotating The seal stanza resolves both keys through env://, so the previous one has to reach the container's environment for the rotation to resolve at all. --- roles/web-app-openbao/templates/env.j2 | 3 +++ 1 file changed, 3 insertions(+) diff --git a/roles/web-app-openbao/templates/env.j2 b/roles/web-app-openbao/templates/env.j2 index 53c9d1feab..96de6576c4 100644 --- a/roles/web-app-openbao/templates/env.j2 +++ b/roles/web-app-openbao/templates/env.j2 @@ -1,4 +1,7 @@ {% include 'roles/sys-svc-container/templates/env.j2' %} OPENBAO_SEAL_KEY={{ OPENBAO_SEAL_KEY | dotenv_quote }} +{% if OPENBAO_SEAL_ROTATING | bool %} +OPENBAO_SEAL_KEY_PREVIOUS={{ OPENBAO_SEAL_KEY_PREVIOUS | dotenv_quote }} +{% endif %} BAO_ADDR=http://127.0.0.1:{{ container_port }} From e6e27b2dac6671cc80ff440368d4570ba9e880c2 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 02:15:14 +0200 Subject: [PATCH 119/145] feat(openbao): let the AppRole login survive its own credentials rotating A deploy is handed credentials the instance has not been told about yet, so the inventory login is expected to fail on every rotation. Try it, fall back to what the previous deploy recorded, and fail with a message naming the state file when neither is accepted -- a bare 403 here reads like an unrelated defect. Both logins are non-fatal so the guard is what reports the failure. Whichever one succeeded also identifies the SecretID the instance actually holds, which is the value the rotation has to destroy; taking it from the state file instead would destroy the wrong one whenever the inventory credentials still worked. --- roles/web-app-openbao/tasks/01_init.yml | 46 ++++++++++++++++++++++++- 1 file changed, 45 insertions(+), 1 deletion(-) diff --git a/roles/web-app-openbao/tasks/01_init.yml b/roles/web-app-openbao/tasks/01_init.yml index 8f5eb7fc6b..b22bf3ecb5 100644 --- a/roles/web-app-openbao/tasks/01_init.yml +++ b/roles/web-app-openbao/tasks/01_init.yml @@ -69,15 +69,56 @@ stdin_add_newline: false register: openbao_approle_login changed_when: false + failed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" +# Rationale: the credentials rotate every deploy, so the values this run was +# handed are not yet the ones pinned inside OpenBao. Fall back to what the +# previous deploy applied; 06_rotate.yml then re-pins to this run's values. +- name: "🔐 Log in with the AppRole the previous deploy applied" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao write -field=token auth/approle/login + role_id={{ OPENBAO_APPLIED.approle_role_id | quote }} secret_id=- + args: + stdin: "{{ OPENBAO_APPLIED.approle_secret_id }}" + stdin_add_newline: false + register: openbao_approle_login_previous + changed_when: false + failed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + when: + - openbao_approle_login.rc != 0 + - OPENBAO_APPLIED.approle_role_id is defined + +- name: "🔧 Keep whichever AppRole login succeeded" + ansible.builtin.set_fact: + openbao_approle_token: >- + {{ openbao_approle_login.stdout + if openbao_approle_login.rc == 0 + else openbao_approle_login_previous.stdout | default('') }} + openbao_secret_id_registered: >- + {{ OPENBAO_APPROLE_SECRET_ID + if openbao_approle_login.rc == 0 + else OPENBAO_APPLIED.approle_secret_id | default('') }} + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🛑 Fail when neither the current nor the previously applied AppRole is accepted" + ansible.builtin.fail: + msg: >- + Neither the AppRole credentials from the inventory nor the ones recorded in + {{ OPENBAO_APPLIED_STATE_HOST }} were accepted. The instance can only be + recovered from its volume backup plus the matching seal key. + when: (openbao_approle_token | trim) | length == 0 + - name: "🔧 Store the AppRole token in the container token helper" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec -i {{ OPENBAO_LOCAL_CID | quote }} bao login -no-print - args: - stdin: "{{ openbao_approle_login.stdout | trim }}" + stdin: "{{ openbao_approle_token | trim }}" stdin_add_newline: false changed_when: false delegate_to: "{{ OPENBAO_HOST_NODE }}" @@ -99,6 +140,9 @@ delegate_to: "{{ OPENBAO_HOST_NODE }}" when: OPENBAO_PKI_ENABLED | bool +- name: "♻️ Re-pin the AppRole and record what this deploy applied" + ansible.builtin.include_tasks: 06_rotate.yml + - name: "🧹 Drop the stored reconciliation token from the container" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} From b7a1223481ac5ea5343fcb50b273f0d628d52754 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 02:15:23 +0200 Subject: [PATCH 120/145] feat(openbao): re-pin the AppRole and record what each deploy applied Closes the rotation: the instance is moved onto this run's credentials and the result is written to .applied.json, which is what lets the next deploy log in before it re-pins in turn. The file sits in the compose project directory, which already holds the seal key in plaintext in .env/env, so this adds no new exposure class; it is written 0600 root-owned and last, after both pins, so a run that dies partway leaves the previous generation intact. custom-secret-id is not idempotent -- writing a SecretID the role already holds answers 500 'SecretID is already registered' rather than doing nothing -- so the swap is gated on the value having changed. The previously registered SecretID is destroyed afterwards, without which the rotation would add a credential and revoke none. --- roles/web-app-openbao/tasks/06_rotate.yml | 57 +++++++++++++++++++++++ 1 file changed, 57 insertions(+) create mode 100644 roles/web-app-openbao/tasks/06_rotate.yml diff --git a/roles/web-app-openbao/tasks/06_rotate.yml b/roles/web-app-openbao/tasks/06_rotate.yml new file mode 100644 index 0000000000..aa8fd31be8 --- /dev/null +++ b/roles/web-app-openbao/tasks/06_rotate.yml @@ -0,0 +1,57 @@ +--- +# Re-pin the AppRole to the credentials this deploy was handed, then record what +# was applied. The platform rotates every generated credential per deploy, so the +# values the next deploy receives will differ from these; the state file is what +# lets that deploy still authenticate before it re-pins in turn. +- name: "🔑 Re-pin the RoleID to the value this deploy was handed" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/approle/role/{{ OPENBAO_ANSIBLE_APPROLE | quote }}/role-id + role_id={{ OPENBAO_APPROLE_ROLE_ID | quote }} + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +# Exception: writing custom-secret-id with a SecretID the role already holds is a +# 500 "SecretID is already registered", not a no-op, so the swap has to be gated +# on the value actually having changed. +- name: "🔑 Register the SecretID this deploy was handed" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/approle/role/{{ OPENBAO_ANSIBLE_APPROLE | quote }}/custom-secret-id + secret_id=- + args: + stdin: "{{ OPENBAO_APPROLE_SECRET_ID }}" + stdin_add_newline: false + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + when: OPENBAO_SECRET_ID_ROTATING | bool + +- name: "🧹 Destroy the SecretID the previous deploy applied" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao write auth/approle/role/{{ OPENBAO_ANSIBLE_APPROLE | quote }}/secret-id/destroy + secret_id=- + args: + stdin: "{{ OPENBAO_SECRET_ID_REGISTERED }}" + stdin_add_newline: false + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + when: OPENBAO_SECRET_ID_ROTATING | bool + +- name: "💾 Record the credentials this deploy applied" + ansible.builtin.copy: + content: "{{ openbao_applied_now | to_nice_json }}\n" + dest: "{{ OPENBAO_APPLIED_STATE_HOST }}" + owner: root + group: root + mode: "0600" + vars: + openbao_applied_now: + seal_key: "{{ OPENBAO_SEAL_KEY }}" + approle_role_id: "{{ OPENBAO_APPROLE_ROLE_ID }}" + approle_secret_id: "{{ OPENBAO_APPROLE_SECRET_ID }}" + delegate_to: "{{ STACK_HOST }}" + run_once: true + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" From 2ad2009a9c961d36c4fe767c8268f13cd97ca570 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 02:15:45 +0200 Subject: [PATCH 121/145] test(openbao): pin the rotation invariants that fail silently Every one of these breaks into something that looks like ordinary drift rather than a defect: a state read moved after the backend include leaves the node sealed, a token helper fed the first login's stdout gets an empty string on every rotated deploy and fails later at the first write, an ungated SecretID swap answers 500 on every deploy that did not rotate, and a swap that never destroys the old SecretID revokes nothing at all. --- .../python/roles/web-app-openbao/__init__.py | 3 + .../test_credential_rotation.py | 263 ++++++++++++++++++ 2 files changed, 266 insertions(+) create mode 100644 tests/unit/python/roles/web-app-openbao/__init__.py create mode 100644 tests/unit/python/roles/web-app-openbao/test_credential_rotation.py diff --git a/tests/unit/python/roles/web-app-openbao/__init__.py b/tests/unit/python/roles/web-app-openbao/__init__.py new file mode 100644 index 0000000000..8ee16d7567 --- /dev/null +++ b/tests/unit/python/roles/web-app-openbao/__init__.py @@ -0,0 +1,3 @@ +from pathlib import Path + +PROJECT_ROOT: Path = Path(__file__).resolve().parents[5] diff --git a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py new file mode 100644 index 0000000000..8b35e11e22 --- /dev/null +++ b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py @@ -0,0 +1,263 @@ +"""Guards for the per-deploy credential rotation path of web-app-openbao. + +The platform regenerates generated-algorithm credentials on every deploy, so the +seal key and AppRole a run is handed differ from the ones the running instance +was configured with. Three things carry that transition, and each fails silently +rather than loudly when broken -- a sealed node and an unusable AppRole look +identical to ordinary drift. These tests pin them. +""" + +from __future__ import annotations + +import unittest + +from jinja2 import Environment, FileSystemLoader, StrictUndefined + +from utils.cache.files import read_text +from utils.cache.yaml import load_yaml_any + +from . import PROJECT_ROOT + +ROLE_DIR = PROJECT_ROOT / "roles/web-app-openbao" +TASKS_DIR = ROLE_DIR / "tasks" + +HCL_CONTEXT = { + "OPENBAO_LOG_LEVEL": "info", + "OPENBAO_DATA_DOCKER": "/openbao/file", + "OPENBAO_NAME": "openbao", + "container_port": "8200", + "OPENBAO_SEAL_KEY_ID": "0123456789abcdef", + "OPENBAO_SEAL_KEY_PREVIOUS_ID": "fedcba9876543210", + "OPENBAO_BASE_URL": "https://bao.example.com", +} + +ENV_CONTEXT = { + "OPENBAO_SEAL_KEY": "current-key", + "OPENBAO_SEAL_KEY_PREVIOUS": "previous-key", + "container_port": "8200", +} + + +def jinja_env(searchpath) -> Environment: + env = Environment( + loader=FileSystemLoader(str(searchpath)), undefined=StrictUndefined + ) + env.filters["bool"] = lambda value: bool(value) + env.filters["dotenv_quote"] = lambda value: f'"{value}"' + return env + + +def tasks_of(name: str) -> list[dict]: + return load_yaml_any(str(TASKS_DIR / name)) + + +class TestSealStanzaRotation(unittest.TestCase): + def _render(self, rotating: bool) -> str: + env = jinja_env(ROLE_DIR / "templates") + return env.get_template("openbao.hcl.j2").render( + OPENBAO_SEAL_ROTATING=rotating, **HCL_CONTEXT + ) + + def test_the_previous_key_is_offered_while_the_keys_differ(self): + rendered = self._render(True) + self.assertIn('previous_key_id = "fedcba9876543210"', rendered) + self.assertIn('previous_key = "env://OPENBAO_SEAL_KEY_PREVIOUS"', rendered) + + def test_no_previous_key_is_offered_on_a_first_deploy(self): + rendered = self._render(False) + self.assertNotIn("previous_key", rendered) + self.assertIn('current_key_id = "0123456789abcdef"', rendered) + + +class TestEnvCarriesThePreviousKey(unittest.TestCase): + """The shared sys-svc-container include needs a live Ansible context. + + Only the role's own lines are rendered here; the include is dropped, which + leaves the conditional under test intact. + """ + + def _render(self, rotating: bool) -> str: + source = read_text(str(ROLE_DIR / "templates/env.j2")) + body = "\n".join( + line for line in source.splitlines() if "{% include" not in line + ) + env = jinja_env(ROLE_DIR / "templates") + return env.from_string(body).render( + OPENBAO_SEAL_ROTATING=rotating, **ENV_CONTEXT + ) + + def test_the_previous_key_reaches_the_container_while_rotating(self): + rendered = self._render(True) + self.assertIn('OPENBAO_SEAL_KEY_PREVIOUS="previous-key"', rendered) + self.assertIn('OPENBAO_SEAL_KEY="current-key"', rendered) + + def test_no_previous_key_is_exported_on_a_first_deploy(self): + rendered = self._render(False) + self.assertNotIn("OPENBAO_SEAL_KEY_PREVIOUS", rendered) + self.assertIn('OPENBAO_SEAL_KEY="current-key"', rendered) + + +class TestAppliedStateIsReadBeforeTheConfigIsRendered(unittest.TestCase): + def test_the_state_read_precedes_the_backend_include(self): + """env.j2 is rendered by sys-stk-backend. + + Reading the applied state after that include leaves the container + without the previous seal key, and the node comes up sealed with no way + to decrypt its own store. + """ + tasks = tasks_of("00_core.yml") + state_read = next( + i + for i, task in enumerate(tasks) + if task.get("ansible.builtin.slurp", {}).get("src", "").endswith( + "OPENBAO_APPLIED_STATE_HOST }}" + ) + ) + backend = next( + i + for i, task in enumerate(tasks) + if task.get("include_role", {}).get("name") == "sys-stk-backend" + ) + self.assertLess(state_read, backend) + + +class TestAppRoleLoginToleratesRotation(unittest.TestCase): + def setUp(self): + self.tasks = tasks_of("01_init.yml") + + def test_the_inventory_login_does_not_abort_the_play(self): + login = next( + task + for task in self.tasks + if task.get("register") == "openbao_approle_login" + ) + self.assertIs(login.get("failed_when"), False) + + def test_a_fallback_login_uses_the_previously_applied_credentials(self): + fallback = next( + task + for task in self.tasks + if task.get("register") == "openbao_approle_login_previous" + ) + self.assertIs( + fallback.get("failed_when"), + False, + "a failing fallback must reach the guard, not abort with a raw 403", + ) + self.assertIn( + "OPENBAO_APPLIED.approle_role_id", + fallback["ansible.builtin.shell"], + ) + self.assertIn( + "OPENBAO_APPLIED.approle_secret_id", + fallback["args"]["stdin"], + ) + + def test_the_stored_token_is_whichever_login_succeeded(self): + """A regression here re-introduces the bug this path exists to fix. + + Storing openbao_approle_login.stdout feeds the token helper an empty + string on every rotated deploy, and reconciliation then fails on the + first write instead of on the login. + """ + store = next( + task + for task in self.tasks + if "bao login -no-print" in str(task.get("ansible.builtin.shell", "")) + ) + self.assertIn("openbao_approle_token", store["args"]["stdin"]) + + def test_neither_credential_being_accepted_fails_loudly(self): + self.assertTrue( + any("ansible.builtin.fail" in task for task in self.tasks), + "a rotated-out instance must fail with a message, not a bare 403", + ) + + def test_the_rotation_pass_runs_before_the_token_is_dropped(self): + rotate = next( + i + for i, task in enumerate(self.tasks) + if task.get("ansible.builtin.include_tasks", {}) == "06_rotate.yml" + ) + drop = next( + i + for i, task in enumerate(self.tasks) + if "token revoke -self" in str(task.get("ansible.builtin.shell", "")) + ) + self.assertLess(rotate, drop) + + +class TestTheSecretIdSwapIsGuarded(unittest.TestCase): + """Re-registering an unchanged SecretID answers 500, and the old one must go. + + `custom-secret-id` is not idempotent: writing a SecretID the role already + holds fails with `SecretID is already registered`, so an unguarded re-pin + breaks every deploy that did not rotate. Leaving the previous SecretID + registered is the opposite failure -- the rotation would revoke nothing. + """ + + def setUp(self): + self.tasks = tasks_of("06_rotate.yml") + + def _guarded(self, fragment: str) -> dict: + return next( + task + for task in self.tasks + if fragment in str(task.get("ansible.builtin.shell", "")) + ) + + def test_the_new_secret_id_is_registered_only_while_rotating(self): + task = self._guarded("custom-secret-id") + self.assertEqual(task["when"], "OPENBAO_SECRET_ID_ROTATING | bool") + self.assertIn("OPENBAO_APPROLE_SECRET_ID", task["args"]["stdin"]) + + def test_the_previous_secret_id_is_destroyed_while_rotating(self): + task = self._guarded("secret-id/destroy") + self.assertEqual(task["when"], "OPENBAO_SECRET_ID_ROTATING | bool") + self.assertIn("OPENBAO_SECRET_ID_REGISTERED", task["args"]["stdin"]) + + def test_the_role_id_is_re_pinned_unconditionally(self): + task = self._guarded("role-id") + self.assertNotIn("when", task) + + +class TestTheRegisteredSecretIdComesFromTheLogin(unittest.TestCase): + def test_it_is_the_credential_that_was_just_accepted(self): + """Deriving it from the state file instead would be wrong. + + On a deploy whose inventory credentials still work, the state file may + hold an older value; destroying that would leave the live SecretID + registered and revoke nothing. + """ + tasks = tasks_of("01_init.yml") + keep = next( + task + for task in tasks + if "openbao_secret_id_registered" + in task.get("ansible.builtin.set_fact", {}) + ) + expression = keep["ansible.builtin.set_fact"]["openbao_secret_id_registered"] + self.assertIn("openbao_approle_login.rc == 0", expression) + self.assertIn("OPENBAO_APPROLE_SECRET_ID", expression) + self.assertIn("OPENBAO_APPLIED.approle_secret_id", expression) + + +class TestTheAppliedStateIsRecorded(unittest.TestCase): + def test_every_credential_the_next_deploy_needs_is_written(self): + tasks = tasks_of("06_rotate.yml") + record = next(task for task in tasks if "ansible.builtin.copy" in task) + applied = record["vars"]["openbao_applied_now"] + self.assertEqual( + sorted(applied), + ["approle_role_id", "approle_secret_id", "seal_key"], + ) + self.assertEqual(record["ansible.builtin.copy"]["mode"], "0600") + self.assertEqual(record["ansible.builtin.copy"]["owner"], "root") + + def test_the_record_is_the_last_thing_the_rotation_does(self): + """Recording before the pins would claim credentials that never applied.""" + tasks = tasks_of("06_rotate.yml") + record = next( + i for i, task in enumerate(tasks) if "ansible.builtin.copy" in task + ) + self.assertEqual(record, len(tasks) - 1) From 3eaf133114b2d715be9a35ca24ddf5a169b164e4 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 02:15:56 +0200 Subject: [PATCH 122/145] docs(openbao): document the per-deploy credential rotation Replaces the note that treated a rotated seal key as local stale state to purge around. It is the platform's behaviour, the role now carries it, and what is left to know is where the window ends: two generations, with MODE_RESET the concrete way to fall out of it, since it removes DIR_COMPOSITIONS while the named volumes survive. --- roles/web-app-openbao/README.md | 12 ++++++++++-- 1 file changed, 10 insertions(+), 2 deletions(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index fcc4a81e88..e9027bfaf1 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -107,7 +107,7 @@ docker run --rm -it \ ## Developer Notes - **The seal key is the whole ballgame.** The `static` seal encrypts the raft root key with a 32-byte key generated into the ansible-vault-encrypted inventory as `secrets.credentials.seal_key` and injected as `env://OPENBAO_SEAL_KEY`. A backup of the raft volume is **undecryptable without that key**, and the key is *not* covered by `svc-bkp-secrets-2-local` (whose sources are `DIR_SECRETS`, the root CA and the ACME material). The inventory is its authoritative home — back the inventory up accordingly. Upstream considers a static seal appropriate only where an existing source of trust already holds the key; here that source is the encrypted inventory, at the same trust level as every other platform credential. -- **Recovery keys are generated and then deliberately discarded.** Initialising with an auto-seal yields *recovery* keys rather than unseal shares. They cannot decrypt the root key — only the seal key can — so they are not disaster-recovery material for the data; they only authorise `operator generate-root` and `operator rekey`. Persisting them would mean writing OpenBao-generated secrets back into the inventory, which is exactly what the AppRole bootstrap exists to avoid, so the play does not keep them. **The consequence is explicit: the Ansible AppRole is the only administrative path into this instance.** If its RoleID/SecretID are changed out-of-band inside OpenBao, there is no way back in and the instance must be rebuilt from the volume backup plus the seal key. Re-running the role does not repair that, because reconciliation itself authenticates with the AppRole. +- **Recovery keys are generated and then deliberately discarded.** Initialising with an auto-seal yields *recovery* keys rather than unseal shares. They cannot decrypt the root key — only the seal key can — so they are not disaster-recovery material for the data; they only authorise `operator generate-root` and `operator rekey`. Persisting them would mean writing OpenBao-generated secrets back into the inventory, which is exactly what the AppRole bootstrap exists to avoid, so the play does not keep them. **The consequence is explicit: the Ansible AppRole is the only administrative path into this instance.** If its RoleID/SecretID are changed out-of-band inside OpenBao, there is no way back in and the instance must be rebuilt from the volume backup plus the seal key. Re-running the role does not repair that, because reconciliation itself authenticates with the AppRole — the rotation fallback below only covers the credentials the role itself applied. - **Bootstrap leaves no stored root token.** The first deploy runs `bao operator init`, holds the root token in memory only, enables AppRole, pins both the RoleID and the SecretID to inventory-generated values via `auth/approle/role//role-id` and `.../custom-secret-id`, then revokes the root token. Every later deploy logs in with those inventory credentials, so nothing OpenBao generates is ever written back into the inventory. Tokens reach the CLI through stdin and the container token helper, never through process arguments, and the helper file is removed at the end of the run. - **Policies live outside the config directory.** OpenBao parses every file in `-config` as server configuration, so the RBAC policy `.hcl` files are mounted at `/openbao/policies/` instead of `/openbao/config/`. - **Storage path is `/openbao/file`, not `/openbao/data`.** The image runs as the non-root `openbao` user and pre-chowns only `/openbao/{config,logs,file}`. A fresh named volume inherits ownership from the image path, so raft can write there; a volume at `/openbao/data` would land root-owned and fail. @@ -117,7 +117,15 @@ docker run --rm -it \ - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. - **The RBAC tiers are proven by moving `biber` through the groups, not by dedicated accounts.** `files/playwright/test-rbac-groups.js` signs the administrator into LAM, adds `biber` to `cn=web-app-openbao-,ou=roles` through LAM's Multi edit tool, logs `biber` into OpenBao over LDAP and asserts both the policy set and the capability that tier grants, then removes him again and asserts the revoke. One account covers all three tiers, and the grant and the revoke are observed rather than assumed — a permanently-assigned member can only ever show a steady state. LAM's *group list* cannot be used for this: it renders `posixGroup` entries, while the RBAC role groups are `groupOfNames` under `ou=roles`. LAM applies the change asynchronously, so the spec polls the resulting policy set instead of LAM's own progress wording. -- **Redeploying locally after the inventory regenerates leaves stale state that looks like a role bug.** The development inventory regenerates credentials per deploy while docker volumes survive, so anything that stores a credential *inside* its volume desynchronises: the raft store stays `initialized: true, sealed: true` when `seal_key` no longer matches, the Ansible AppRole reports `invalid role or secret ID` when it was pinned at an earlier bootstrap, and a surviving `mariadb_data` fails its healthcheck with `Access denied for user 'root'@'127.0.0.1'`. None of these are defects; CI never sees them because every run starts clean. Run `make compose-entity-purge apps=web-app-openbao` (or the affected entity) before redeploying locally. +- **Every deploy rotates the credentials, and the role is built around that.** The platform regenerates generated-algorithm credentials on each deploy — that is how a changed Ansible password reaches the application instead of silently diverging from it. For OpenBao it means the `seal_key` and the AppRole `role_id`/`secret_id` a run is handed are *not* the ones the running instance was configured with, which would otherwise leave the node sealed (`initialized: true, sealed: true`) and the AppRole answering `invalid role or secret ID`. The role therefore carries the transition itself: + - `tasks/06_rotate.yml` records what each deploy actually applied in `.applied.json` inside the compose project directory, `0600`, root-owned. The directory already holds the seal key in plaintext in `.env/env`, so this adds no new exposure class. + - `tasks/00_core.yml` reads that file back **before** `sys-stk-backend` renders `env.j2` — the rendered env must already carry `OPENBAO_SEAL_KEY_PREVIOUS`, or the node comes up with no way to decrypt its own store. + - `templates/openbao.hcl.j2` emits the `previous_key_id`/`previous_key` pair only while the two keys differ. The static seal supports `n-1` → `n` rotation, so the node unseals with the old key and rewraps to the new one. + - `tasks/01_init.yml` attempts the AppRole login with the inventory credentials, falls back to the recorded ones, and fails loudly when neither is accepted. Whichever one was accepted is the SecretID OpenBao actually holds, so that is what `tasks/06_rotate.yml` destroys after registering the new one — rotation that only adds would revoke nothing. + - `auth/approle/role//custom-secret-id` is **not** idempotent: writing a SecretID the role already holds answers `500 … SecretID is already registered`. The swap is therefore gated on the value having actually changed, which is also why the bootstrap in `tasks/02_auth.yml` pins directly instead of sharing the rotation path — at that point nothing is registered yet. + - `current_key_id` is derived as a short hash of the key rather than a counter. Upstream requires the identifier to match its key; a derived id cannot drift out of step with one, and no deploy has to know which generation it is. + The record is written last, after both pins, so a run that dies partway leaves the previous generation intact and the next deploy still authenticates. The one uncovered window is a failure *between* the two pins — the RoleID re-pinned, the SecretID not, or the reverse — which leaves a pairing neither the inventory nor the state file describes. Two generations is the whole window. Anything that skips a rotation puts the store outside it, and the instance is then recoverable only from its volume backup plus the matching seal key. The concrete way to do that is `MODE_RESET`: `sys-svc-compose/tasks/01_reset.yml` removes `DIR_COMPOSITIONS` wholesale while the named volumes survive, so the raft store outlives the record of the key that wraps it. Purge the entity's volumes too when you reset — the same asymmetry breaks every entity that keeps a credential inside its volume. +- **Other entities still desynchronise the same way, and those are not defects.** A surviving `mariadb_data` fails its healthcheck with `Access denied for user 'root'@'127.0.0.1'` for exactly the reason above, without a rotation path of its own. CI never sees it because every run starts clean; locally, run `make compose-entity-purge apps=` before redeploying. - **A role-declared account cannot be logged in as unless it sets a password.** The tiers were first covered by two role-local accounts, `baooperator` and `baoreader`, which deployed and joined their RBAC groups correctly but returned 403 on every LDAP bind: a user declared without an explicit `password` inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings. Both accounts are gone now — the tiers are covered by moving `biber` through the groups instead — but the trap applies to any future role-declared account something has to authenticate as. Until [`feature/ai-agent-employees`](https://github.com/kevinveenbirkenbach/infinito-nexus-core/tree/refs/heads/feature/ai-agent-employees) lands (`f4398e1c5` pins a password per declared user at inventory creation), such an account needs an explicit `password:`, as `user-administrator` and `web-app-semaphore` do. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). From d5c26594f00973742af4fb14c0725e6b03682c48 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Tue, 15 Sep 2026 18:13:31 +0200 Subject: [PATCH 123/145] fix(openbao): satisfy ruff in the rotation tests lint-python runs ruff, which the pytest lint suite does not cover, so these slipped through locally: an unnecessary lambda around bool, and a Jinja Environment left on the implicit autoescape default. The suppression has to sit on the autoescape argument itself, which is where ruff anchors the violation. --- .../web-app-openbao/test_credential_rotation.py | 12 +++++++----- 1 file changed, 7 insertions(+), 5 deletions(-) diff --git a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py index 8b35e11e22..3af7af87f4 100644 --- a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py +++ b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py @@ -40,9 +40,11 @@ ENV_CONTEXT = { def jinja_env(searchpath) -> Environment: env = Environment( - loader=FileSystemLoader(str(searchpath)), undefined=StrictUndefined + loader=FileSystemLoader(str(searchpath)), + undefined=StrictUndefined, + autoescape=False, # noqa: S701 - renders HCL and dotenv, not markup ) - env.filters["bool"] = lambda value: bool(value) + env.filters["bool"] = bool env.filters["dotenv_quote"] = lambda value: f'"{value}"' return env @@ -109,9 +111,9 @@ class TestAppliedStateIsReadBeforeTheConfigIsRendered(unittest.TestCase): state_read = next( i for i, task in enumerate(tasks) - if task.get("ansible.builtin.slurp", {}).get("src", "").endswith( - "OPENBAO_APPLIED_STATE_HOST }}" - ) + if task.get("ansible.builtin.slurp", {}) + .get("src", "") + .endswith("OPENBAO_APPLIED_STATE_HOST }}") ) backend = next( i From b20ad3c375c2af30b912661ad337d9e1b9611f51 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 19 Sep 2026 20:07:23 +0200 Subject: [PATCH 124/145] docs(web-app-openbao): the declared-user password trap is closed The note told readers to set an explicit password until a commit landed that pins one at inventory creation. That commit is an ancestor of this branch, so the workaround it prescribed is no longer needed, and the branch URL it linked answers 404 now that the branch is merged and deleted. --- roles/web-app-openbao/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index e9027bfaf1..54461abc8e 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -126,7 +126,7 @@ docker run --rm -it \ - `current_key_id` is derived as a short hash of the key rather than a counter. Upstream requires the identifier to match its key; a derived id cannot drift out of step with one, and no deploy has to know which generation it is. The record is written last, after both pins, so a run that dies partway leaves the previous generation intact and the next deploy still authenticates. The one uncovered window is a failure *between* the two pins — the RoleID re-pinned, the SecretID not, or the reverse — which leaves a pairing neither the inventory nor the state file describes. Two generations is the whole window. Anything that skips a rotation puts the store outside it, and the instance is then recoverable only from its volume backup plus the matching seal key. The concrete way to do that is `MODE_RESET`: `sys-svc-compose/tasks/01_reset.yml` removes `DIR_COMPOSITIONS` wholesale while the named volumes survive, so the raft store outlives the record of the key that wraps it. Purge the entity's volumes too when you reset — the same asymmetry breaks every entity that keeps a credential inside its volume. - **Other entities still desynchronise the same way, and those are not defects.** A surviving `mariadb_data` fails its healthcheck with `Access denied for user 'root'@'127.0.0.1'` for exactly the reason above, without a rotation path of its own. CI never sees it because every run starts clean; locally, run `make compose-entity-purge apps=` before redeploying. -- **A role-declared account cannot be logged in as unless it sets a password.** The tiers were first covered by two role-local accounts, `baooperator` and `baoreader`, which deployed and joined their RBAC groups correctly but returned 403 on every LDAP bind: a user declared without an explicit `password` inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings. Both accounts are gone now — the tiers are covered by moving `biber` through the groups instead — but the trap applies to any future role-declared account something has to authenticate as. Until [`feature/ai-agent-employees`](https://github.com/kevinveenbirkenbach/infinito-nexus-core/tree/refs/heads/feature/ai-agent-employees) lands (`f4398e1c5` pins a password per declared user at inventory creation), such an account needs an explicit `password:`, as `user-administrator` and `web-app-semaphore` do. +- **A role-declared account cannot be logged in as unless it sets a password.** The tiers were first covered by two role-local accounts, `baooperator` and `baoreader`, which deployed and joined their RBAC groups correctly but returned 403 on every LDAP bind: a user declared without an explicit `password` inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings. Both accounts are gone now — the tiers are covered by moving `biber` through the groups instead. [`f4398e1c5`](https://github.com/infinito-nexus/core/commit/f4398e1c58f3ce7d8d429472b2a023033242d747) closed the trap at the source: inventory creation now decides the value once, after the roles are resolved, so a declared user no longer carries a template that re-renders per use site. An explicit `password:`, as `user-administrator` and `web-app-semaphore` set, is therefore no longer required for a role-declared account to be authenticated as. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). ### Backup and restore From 2b51f1cb62e59073ba278e06ea16083b4851e69e Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 19 Sep 2026 20:58:05 +0200 Subject: [PATCH 125/145] fix(openbao): pin the seal key out of the per-deploy rotation The static seal carries one generation. Rotating the key on every deploy spends that slack continuously, so a run that skips a generation, or dies between the two AppRole pins, leaves the raft store readable only from a volume backup plus the key that wrapped it -- and the recovery keys that would authorise a rekey are discarded at init. Frequent rotation buys little here in return. The key lives in the vault-encrypted inventory either way, and whoever holds that inventory holds the AppRole too, so they reach the data without needing the seal key. An operator now rotates it deliberately by changing the inventory value; the previous_key path stays in place to carry that. The AppRole keeps rotating per deploy. It is the live authentication path, and the fallback login exists to absorb exactly that. --- roles/web-app-openbao/meta/secrets.yml | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/roles/web-app-openbao/meta/secrets.yml b/roles/web-app-openbao/meta/secrets.yml index 8bc1ec482a..bcf1f917fd 100644 --- a/roles/web-app-openbao/meta/secrets.yml +++ b/roles/web-app-openbao/meta/secrets.yml @@ -2,9 +2,12 @@ credentials: seal_key: description: 32-byte AES-256-GCM-96 key for the OpenBao static seal, injected as env://OPENBAO_SEAL_KEY. Every backup of the raft volume is undecryptable - without it; its authoritative home is the encrypted inventory. + without it; its authoritative home is the encrypted inventory. Pinned because the static seal carries only one generation, so a deploy that skips a + generation, or dies between the two AppRole pins, puts the store outside that window and leaves it recoverable only from a volume backup plus the key + that wraps it. Rotate it deliberately by changing the inventory value; openbao.hcl.j2 still renders the previous_key pair for that. algorithm: random_hex_32 validation: non_empty_string + rotatable: false approle_role_id: description: RoleID of the Ansible AppRole, written to auth/approle/role/ansible/role-id at bootstrap so later deploys authenticate without a stored root token. From 56c3da4a146da71b9abe6eb1cbbdc6e2ef8c9010 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 19 Sep 2026 20:58:14 +0200 Subject: [PATCH 126/145] test(openbao): pin which credentials rotate Nothing stopped the seal key drifting back into the per-deploy rotation, and the regression is invisible until a deploy is skipped and the node cannot unseal. Assert the pin on the seal key and its absence on both AppRole credentials. --- .../test_credential_rotation.py | 32 +++++++++++++++++-- 1 file changed, 30 insertions(+), 2 deletions(-) diff --git a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py index 3af7af87f4..554b6c0d26 100644 --- a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py +++ b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py @@ -1,8 +1,9 @@ """Guards for the per-deploy credential rotation path of web-app-openbao. The platform regenerates generated-algorithm credentials on every deploy, so the -seal key and AppRole a run is handed differ from the ones the running instance -was configured with. Three things carry that transition, and each fails silently +AppRole a run is handed differs from the one the running instance was configured +with. The seal key is pinned out of that and moves only when an operator changes +it, but the same transition then has to carry it. Each part fails silently rather than loudly when broken -- a sealed node and an unusable AppRole look identical to ordinary drift. These tests pin them. """ @@ -15,6 +16,7 @@ from jinja2 import Environment, FileSystemLoader, StrictUndefined from utils.cache.files import read_text from utils.cache.yaml import load_yaml_any +from utils.roles.mapping import ROLE_FILE_META_SECRETS from . import PROJECT_ROOT @@ -53,6 +55,32 @@ def tasks_of(name: str) -> list[dict]: return load_yaml_any(str(TASKS_DIR / name)) +class TestWhichCredentialsRotate(unittest.TestCase): + """The seal key survives a deploy; the AppRole does not. + + Rotating the seal key on every deploy spends the static seal's single + generation of slack for nothing. A run that skips a generation, or dies + between the two AppRole pins, leaves the raft store readable only from a + volume backup plus the key that wrapped it, and the role keeps no recovery + keys to fall back on. The AppRole is the opposite case: it is the live + authentication path, and absorbing its rotation is what the fallback login + in 01_init.yml exists for. + """ + + def setUp(self): + self.credentials = load_yaml_any(str(ROLE_DIR / ROLE_FILE_META_SECRETS))[ + "credentials" + ] + + def test_the_seal_key_is_pinned(self): + self.assertIs(self.credentials["seal_key"].get("rotatable"), False) + + def test_both_approle_credentials_still_rotate(self): + for name in ("approle_role_id", "approle_secret_id"): + with self.subTest(credential=name): + self.assertIsNot(self.credentials[name].get("rotatable"), False) + + class TestSealStanzaRotation(unittest.TestCase): def _render(self, rotating: bool) -> str: env = jinja_env(ROLE_DIR / "templates") From fcfff81f640e764da1459a86078d107c8bfe8bd3 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 19 Sep 2026 20:58:25 +0200 Subject: [PATCH 127/145] docs(openbao): correct what the rotation record covers The header claimed every generated credential rotates per deploy. The seal key no longer does; it is recorded for the deploy that follows an operator changing it. --- roles/web-app-openbao/tasks/06_rotate.yml | 8 +++++--- 1 file changed, 5 insertions(+), 3 deletions(-) diff --git a/roles/web-app-openbao/tasks/06_rotate.yml b/roles/web-app-openbao/tasks/06_rotate.yml index aa8fd31be8..f7ed0ba83a 100644 --- a/roles/web-app-openbao/tasks/06_rotate.yml +++ b/roles/web-app-openbao/tasks/06_rotate.yml @@ -1,8 +1,10 @@ --- # Re-pin the AppRole to the credentials this deploy was handed, then record what -# was applied. The platform rotates every generated credential per deploy, so the -# values the next deploy receives will differ from these; the state file is what -# lets that deploy still authenticate before it re-pins in turn. +# was applied. The platform rotates the AppRole per deploy, so the values the +# next deploy receives will differ from these; the state file is what lets that +# deploy still authenticate before it re-pins in turn. The seal key is pinned +# and normally repeats, and is recorded for the deploy that follows an operator +# changing it. - name: "🔑 Re-pin the RoleID to the value this deploy was handed" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} From 48333bb50a5637b3d3e362c9409141647e69d910 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 19 Sep 2026 20:59:50 +0200 Subject: [PATCH 128/145] docs(openbao): restate why the key id is derived The rationale rested on the key rotating every deploy, which it no longer does. Deriving the id from the key is what keeps it in step with the key, however rarely that happens. --- roles/web-app-openbao/vars/main.yml | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/roles/web-app-openbao/vars/main.yml b/roles/web-app-openbao/vars/main.yml index d5caa506b1..0de0720fe1 100644 --- a/roles/web-app-openbao/vars/main.yml +++ b/roles/web-app-openbao/vars/main.yml @@ -20,9 +20,9 @@ OPENBAO_SEAL_KEY: "{{ lookup('config', application_id, 'secrets.credentials.seal OPENBAO_APPROLE_ROLE_ID: "{{ lookup('config', application_id, 'secrets.credentials.approle_role_id') }}" OPENBAO_APPROLE_SECRET_ID: "{{ lookup('config', application_id, 'secrets.credentials.approle_secret_id') }}" -# Rationale: the platform rotates every generated credential on each deploy, so -# the key id is derived from the key instead of counted — upstream requires the -# identifier to match its key, and a derived id cannot drift out of step with one. +# Rationale: the key id is derived from the key instead of counted — upstream +# requires the identifier to match its key, and a derived id cannot drift out of +# step with one however rarely an operator rotates the key. OPENBAO_SEAL_KEY_ID: "{{ (OPENBAO_SEAL_KEY | hash('sha256'))[:16] }}" # Rationale: what the previous deploy applied, read back before the config is From 9248d31e025d2d47f137ee57e4a62e674390c578 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sat, 19 Sep 2026 21:00:33 +0200 Subject: [PATCH 129/145] docs(openbao): document the pinned seal key Records why the key is held out of the rotation, how to rotate it deliberately, and what pinning costs. Also corrects the MODE_RESET warning: with the seal key pinned the store stays readable, and what a reset destroys is the AppRole record, leaving no administrative path back in. --- roles/web-app-openbao/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index 54461abc8e..a6be358039 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -107,6 +107,7 @@ docker run --rm -it \ ## Developer Notes - **The seal key is the whole ballgame.** The `static` seal encrypts the raft root key with a 32-byte key generated into the ansible-vault-encrypted inventory as `secrets.credentials.seal_key` and injected as `env://OPENBAO_SEAL_KEY`. A backup of the raft volume is **undecryptable without that key**, and the key is *not* covered by `svc-bkp-secrets-2-local` (whose sources are `DIR_SECRETS`, the root CA and the ACME material). The inventory is its authoritative home — back the inventory up accordingly. Upstream considers a static seal appropriate only where an existing source of trust already holds the key; here that source is the encrypted inventory, at the same trust level as every other platform credential. +- **The seal key is pinned, and that is deliberate.** `meta/secrets.yml` marks it `rotatable: false`, so the per-deploy rotation skips it. The static seal carries one generation of slack, and rotating on every deploy spends all of it continuously: a run that skips a generation, or dies between the two AppRole pins, leaves the raft store readable only from a volume backup plus the key that wrapped it — and the recovery keys that would otherwise authorise a rekey were discarded at init. The key gains little from frequent rotation in exchange, because it lives in the vault-encrypted inventory either way, and an attacker holding that inventory holds the AppRole too and does not need the seal key to reach the data. Rotate it deliberately instead, by changing `secrets.credentials.seal_key` in the inventory: `templates/openbao.hcl.j2` then renders the `previous_key_id`/`previous_key` pair, the node unseals with the old key and rewraps to the new one, and the next deploy sees them equal again. The cost of pinning is that a leaked inventory backup stays valid against future volume snapshots instead of ageing out in two deploys. - **Recovery keys are generated and then deliberately discarded.** Initialising with an auto-seal yields *recovery* keys rather than unseal shares. They cannot decrypt the root key — only the seal key can — so they are not disaster-recovery material for the data; they only authorise `operator generate-root` and `operator rekey`. Persisting them would mean writing OpenBao-generated secrets back into the inventory, which is exactly what the AppRole bootstrap exists to avoid, so the play does not keep them. **The consequence is explicit: the Ansible AppRole is the only administrative path into this instance.** If its RoleID/SecretID are changed out-of-band inside OpenBao, there is no way back in and the instance must be rebuilt from the volume backup plus the seal key. Re-running the role does not repair that, because reconciliation itself authenticates with the AppRole — the rotation fallback below only covers the credentials the role itself applied. - **Bootstrap leaves no stored root token.** The first deploy runs `bao operator init`, holds the root token in memory only, enables AppRole, pins both the RoleID and the SecretID to inventory-generated values via `auth/approle/role//role-id` and `.../custom-secret-id`, then revokes the root token. Every later deploy logs in with those inventory credentials, so nothing OpenBao generates is ever written back into the inventory. Tokens reach the CLI through stdin and the container token helper, never through process arguments, and the helper file is removed at the end of the run. - **Policies live outside the config directory.** OpenBao parses every file in `-config` as server configuration, so the RBAC policy `.hcl` files are mounted at `/openbao/policies/` instead of `/openbao/config/`. @@ -117,14 +118,14 @@ docker run --rm -it \ - **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. - **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. - **The RBAC tiers are proven by moving `biber` through the groups, not by dedicated accounts.** `files/playwright/test-rbac-groups.js` signs the administrator into LAM, adds `biber` to `cn=web-app-openbao-,ou=roles` through LAM's Multi edit tool, logs `biber` into OpenBao over LDAP and asserts both the policy set and the capability that tier grants, then removes him again and asserts the revoke. One account covers all three tiers, and the grant and the revoke are observed rather than assumed — a permanently-assigned member can only ever show a steady state. LAM's *group list* cannot be used for this: it renders `posixGroup` entries, while the RBAC role groups are `groupOfNames` under `ou=roles`. LAM applies the change asynchronously, so the spec polls the resulting policy set instead of LAM's own progress wording. -- **Every deploy rotates the credentials, and the role is built around that.** The platform regenerates generated-algorithm credentials on each deploy — that is how a changed Ansible password reaches the application instead of silently diverging from it. For OpenBao it means the `seal_key` and the AppRole `role_id`/`secret_id` a run is handed are *not* the ones the running instance was configured with, which would otherwise leave the node sealed (`initialized: true, sealed: true`) and the AppRole answering `invalid role or secret ID`. The role therefore carries the transition itself: +- **Every deploy rotates the AppRole, and the role is built around that.** The platform regenerates generated-algorithm credentials on each deploy — that is how a changed Ansible password reaches the application instead of silently diverging from it. For OpenBao it means the `role_id`/`secret_id` a run is handed are *not* the ones the running instance was configured with, which would otherwise leave the AppRole answering `invalid role or secret ID`. The role therefore carries the transition itself: - `tasks/06_rotate.yml` records what each deploy actually applied in `.applied.json` inside the compose project directory, `0600`, root-owned. The directory already holds the seal key in plaintext in `.env/env`, so this adds no new exposure class. - `tasks/00_core.yml` reads that file back **before** `sys-stk-backend` renders `env.j2` — the rendered env must already carry `OPENBAO_SEAL_KEY_PREVIOUS`, or the node comes up with no way to decrypt its own store. - `templates/openbao.hcl.j2` emits the `previous_key_id`/`previous_key` pair only while the two keys differ. The static seal supports `n-1` → `n` rotation, so the node unseals with the old key and rewraps to the new one. - `tasks/01_init.yml` attempts the AppRole login with the inventory credentials, falls back to the recorded ones, and fails loudly when neither is accepted. Whichever one was accepted is the SecretID OpenBao actually holds, so that is what `tasks/06_rotate.yml` destroys after registering the new one — rotation that only adds would revoke nothing. - `auth/approle/role//custom-secret-id` is **not** idempotent: writing a SecretID the role already holds answers `500 … SecretID is already registered`. The swap is therefore gated on the value having actually changed, which is also why the bootstrap in `tasks/02_auth.yml` pins directly instead of sharing the rotation path — at that point nothing is registered yet. - `current_key_id` is derived as a short hash of the key rather than a counter. Upstream requires the identifier to match its key; a derived id cannot drift out of step with one, and no deploy has to know which generation it is. - The record is written last, after both pins, so a run that dies partway leaves the previous generation intact and the next deploy still authenticates. The one uncovered window is a failure *between* the two pins — the RoleID re-pinned, the SecretID not, or the reverse — which leaves a pairing neither the inventory nor the state file describes. Two generations is the whole window. Anything that skips a rotation puts the store outside it, and the instance is then recoverable only from its volume backup plus the matching seal key. The concrete way to do that is `MODE_RESET`: `sys-svc-compose/tasks/01_reset.yml` removes `DIR_COMPOSITIONS` wholesale while the named volumes survive, so the raft store outlives the record of the key that wraps it. Purge the entity's volumes too when you reset — the same asymmetry breaks every entity that keeps a credential inside its volume. + The record is written last, after both pins, so a run that dies partway leaves the previous generation intact and the next deploy still authenticates. The one uncovered window is a failure *between* the two pins — the RoleID re-pinned, the SecretID not, or the reverse — which leaves a pairing neither the inventory nor the state file describes, and no recovery keys are kept to get back in. Two generations is the whole window for the AppRole. Discarding the state file while the volume survives puts the instance outside it: OpenBao still holds the previous SecretID, the inventory has already rotated past it, and the fallback has nothing left to read. `MODE_RESET` does exactly that — `sys-svc-compose/tasks/01_reset.yml` removes `DIR_COMPOSITIONS` wholesale while the named volumes persist. The store itself stays readable, because the seal key is pinned and still comes from the inventory, but no administrative path into it remains. Purge the entity's volumes too when you reset — the same asymmetry breaks every entity that keeps a credential inside its volume. - **Other entities still desynchronise the same way, and those are not defects.** A surviving `mariadb_data` fails its healthcheck with `Access denied for user 'root'@'127.0.0.1'` for exactly the reason above, without a rotation path of its own. CI never sees it because every run starts clean; locally, run `make compose-entity-purge apps=` before redeploying. - **A role-declared account cannot be logged in as unless it sets a password.** The tiers were first covered by two role-local accounts, `baooperator` and `baoreader`, which deployed and joined their RBAC groups correctly but returned 403 on every LDAP bind: a user declared without an explicit `password` inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings. Both accounts are gone now — the tiers are covered by moving `biber` through the groups instead. [`f4398e1c5`](https://github.com/infinito-nexus/core/commit/f4398e1c58f3ce7d8d429472b2a023033242d747) closed the trap at the source: inventory creation now decides the value once, after the roles are resolved, so a declared user no longer carries a template that re-renders per use site. An explicit `password:`, as `user-administrator` and `web-app-semaphore` set, is therefore no longer required for a role-declared account to be authenticated as. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). From 7affa3d163c69031235b18c2a31268339bfab88c Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sun, 20 Sep 2026 01:08:56 +0200 Subject: [PATCH 130/145] feat(openbao): add a break-glass path through the recovery key Mints a root token with bao operator generate-root and hands over to 02_auth.yml, which re-pins both AppRole halves to the inventory values. The four protocol steps run as separate tasks because the token comes back XOR-wrapped with a one-time password and the image carries neither python nor jq, so the CLI performs each step and Ansible parses the JSON -- the same split the rest of the role already uses for bao status. --- roles/web-app-openbao/tasks/utils/recover.yml | 84 +++++++++++++++++++ 1 file changed, 84 insertions(+) create mode 100644 roles/web-app-openbao/tasks/utils/recover.yml diff --git a/roles/web-app-openbao/tasks/utils/recover.yml b/roles/web-app-openbao/tasks/utils/recover.yml new file mode 100644 index 0000000000..1b3e0ceab5 --- /dev/null +++ b/roles/web-app-openbao/tasks/utils/recover.yml @@ -0,0 +1,84 @@ +--- +# Break-glass: mint a root token from the recovery key and rebuild the AppRole +# from the inventory. Reached only when neither the credentials this deploy was +# handed nor the ones the previous deploy recorded are accepted, which without +# this path ends the run with no administrative way into a node whose data is +# still perfectly readable. +# +# The protocol is four steps and the token it yields is XOR-wrapped with a +# one-time password, so each step runs through the bao CLI and Ansible parses +# the result -- the image carries no python or jq to do it in-container. +# +# The last step hands over to 02_auth.yml, which re-pins both halves to the +# inventory values and revokes the root token it was given. The AppRole this +# deploy holds is therefore live on return, and 06_rotate.yml finds nothing +# left to swap. + +# Exception: an attempt left behind by an interrupted run makes -init fail with +# "root generation already in progress". +- name: "🧹 Cancel any root generation left in progress" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao operator generate-root -cancel + changed_when: false + failed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + +- name: "🎲 Generate the one-time password the root token is wrapped with" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao operator generate-root -generate-otp + register: openbao_root_otp + changed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🚑 Begin root generation" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao operator generate-root -format=json + -init -otp={{ openbao_root_otp.stdout | trim | quote }} + register: openbao_root_attempt + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🔑 Authorise the attempt with the recovery key" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao operator generate-root -format=json + -nonce={{ (openbao_root_attempt.stdout | from_json).nonce | quote }} - + args: + stdin: "{{ OPENBAO_RECOVERY_KEY }}" + stdin_add_newline: false + register: openbao_root_update + changed_when: true + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🛑 Fail when the recovery key did not complete root generation" + ansible.builtin.fail: + msg: >- + The recovery key held for {{ application_id }} was rejected. Neither + AppRole is accepted either, so this instance has no administrative path + left and must be rebuilt from its volume backup plus the seal key. + when: not ((openbao_root_update.stdout | from_json).complete | default(false) | bool) + +- name: "🔓 Unwrap the generated root token" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao operator generate-root + -decode={{ (openbao_root_update.stdout | from_json).encoded_token | quote }} + -otp={{ openbao_root_otp.stdout | trim | quote }} + register: openbao_root_decoded + changed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🔧 Hold the generated root token in memory only" + ansible.builtin.set_fact: + openbao_root_token: "{{ openbao_root_decoded.stdout | trim }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + +- name: "🔑 Rebuild the Ansible AppRole from the inventory credentials" + ansible.builtin.include_tasks: ../02_auth.yml From b129189f50ace5dba132eabf3fb23a32a62d8fd5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sun, 20 Sep 2026 01:22:34 +0200 Subject: [PATCH 131/145] feat(openbao): resolve the recovery key from the platform token store DIR_SECRETS is the durable home a role actually has. The inventory would be the natural one, but no role writes host_vars -- the CLI writes it before the play. DIR_SECRETS outlives MODE_RESET and a replaced swarm manager, both of which take .applied.json with them, and svc-bkp-secrets-2-local backs it up. --- roles/web-app-openbao/vars/main.yml | 13 +++++++++++++ 1 file changed, 13 insertions(+) diff --git a/roles/web-app-openbao/vars/main.yml b/roles/web-app-openbao/vars/main.yml index 0de0720fe1..1564c6e4bc 100644 --- a/roles/web-app-openbao/vars/main.yml +++ b/roles/web-app-openbao/vars/main.yml @@ -31,6 +31,19 @@ OPENBAO_SEAL_KEY_ID: "{{ (OPENBAO_SEAL_KEY | hash('sha256'))[:16] }}" OPENBAO_APPLIED_STATE_HOST: "{{ [lookup('container', application_id, 'directories.instance'), '.applied.json'] | path_join }}" OPENBAO_APPLIED: "{{ openbao_applied_state | default({}, true) }}" OPENBAO_SEAL_KEY_PREVIOUS: "{{ OPENBAO_APPLIED.seal_key | default('', true) }}" + +# Rationale: the recovery key comes from OpenBao at init rather than from the +# inventory generator, and sys-token-store is where the platform keeps exactly +# that class of secret. DIR_SECRETS outlives both MODE_RESET and a manager +# replacement, each of which takes .applied.json with it, and +# svc-bkp-secrets-2-local already covers it. +OPENBAO_RECOVERY_USER_KEY: "administrator" +OPENBAO_RECOVERY_TOKEN_APP: "{{ application_id }}-recovery" +OPENBAO_RECOVERY_KEY: >- + {{ (sys_token_store_users | default({}, true)) + .get(OPENBAO_RECOVERY_USER_KEY, {}) + .get('tokens', {}) + .get(OPENBAO_RECOVERY_TOKEN_APP, '') }} OPENBAO_SEAL_KEY_PREVIOUS_ID: "{{ (OPENBAO_SEAL_KEY_PREVIOUS | hash('sha256'))[:16] if OPENBAO_SEAL_KEY_PREVIOUS else '' }}" OPENBAO_SEAL_ROTATING: "{{ (OPENBAO_SEAL_KEY_PREVIOUS | length > 0) and (OPENBAO_SEAL_KEY_PREVIOUS != OPENBAO_SEAL_KEY) }}" From f1c217b685f1186a15c12d78d847d6db98e521c9 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sun, 20 Sep 2026 01:22:46 +0200 Subject: [PATCH 132/145] feat(openbao): keep the recovery key and fall back to it bao operator init issued a recovery key on the first deploy and the play read only root_token from the JSON, dropping it. That left the AppRole as the sole administrative path, so losing both halves of it ended the deploy for good on an instance whose data was still readable. The key is now persisted at init and loaded before the logins that may need it. When neither the inventory AppRole nor the recorded one is accepted, the run rebuilds the AppRole from it instead of failing. The hard fail remains for the case where there is no key either. The three first-deploy tasks move into a block: a third sibling sharing the same when is what test_no_repeated_when_runs rejects. --- roles/web-app-openbao/tasks/01_init.yml | 93 +++++++++++++++++++------ 1 file changed, 73 insertions(+), 20 deletions(-) diff --git a/roles/web-app-openbao/tasks/01_init.yml b/roles/web-app-openbao/tasks/01_init.yml index b22bf3ecb5..93a405d416 100644 --- a/roles/web-app-openbao/tasks/01_init.yml +++ b/roles/web-app-openbao/tasks/01_init.yml @@ -25,22 +25,39 @@ if (openbao_status_raw.stdout | default('') | trim) | length > 0 else false }} -- name: "🔐 Initialise OpenBao (first deploy only)" - ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node - container exec {{ OPENBAO_LOCAL_CID | quote }} - bao operator init -format=json -recovery-shares=1 -recovery-threshold=1 - register: openbao_init_raw - changed_when: openbao_init_raw.rc == 0 - delegate_to: "{{ OPENBAO_HOST_NODE }}" - no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" +- name: "🔐 Initialise OpenBao and keep what only the first deploy ever sees" when: not (openbao_initialized | bool) + block: + - name: "🔐 Initialise OpenBao (first deploy only)" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec {{ OPENBAO_LOCAL_CID | quote }} + bao operator init -format=json -recovery-shares=1 -recovery-threshold=1 + register: openbao_init_raw + changed_when: openbao_init_raw.rc == 0 + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" -- name: "🔧 Hold the bootstrap root token in memory only" - ansible.builtin.set_fact: - openbao_root_token: >- - {{ (openbao_init_raw.stdout | from_json).root_token }} - no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" - when: not (openbao_initialized | bool) + - name: "🔧 Hold the bootstrap root token in memory only" + ansible.builtin.set_fact: + openbao_root_token: >- + {{ (openbao_init_raw.stdout | from_json).root_token }} + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + + # Exception: the only authority here that never rotates. Without it the + # AppRole is the sole administrative path, so losing the recorded pair -- + # a reset, a replaced manager, a run that dies between the two pins in + # 06_rotate.yml -- locks the instance out for good while its data stays + # perfectly readable. + - name: "💾 Persist the recovery key through sys-token-store" + ansible.builtin.include_role: + name: sys-token-store + tasks_from: write.yml + vars: + sys_token_store_user_key: "{{ OPENBAO_RECOVERY_USER_KEY }}" + sys_token_store_app: "{{ OPENBAO_RECOVERY_TOKEN_APP }}" + sys_token_store_token: >- + {{ (openbao_init_raw.stdout | from_json).recovery_keys_b64 | first }} + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" - name: "⏳ Wait until the static seal has auto-unsealed the node" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node @@ -59,6 +76,12 @@ include_tasks: 02_auth.yml when: not (openbao_initialized | bool) +- name: "🧩 Load the recovery key from the platform token store" + ansible.builtin.include_role: + name: sys-token-store + tasks_from: 01_load.yml + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + - name: "🔐 Log in with the Ansible AppRole" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec -i {{ OPENBAO_LOCAL_CID | quote }} @@ -93,24 +116,54 @@ - openbao_approle_login.rc != 0 - OPENBAO_APPLIED.approle_role_id is defined +# Rationale: the AppRole is the only administrative path into the instance, so +# losing both halves of it used to end the deploy permanently. The recovery key +# does not rotate, which is what makes it a way back. +- name: "🚑 Rebuild the AppRole from the recovery key when neither login is accepted" + when: + - openbao_approle_login.rc != 0 + - (openbao_approle_login_previous.rc | default(1)) != 0 + - (OPENBAO_RECOVERY_KEY | trim) | length > 0 + block: + - name: "🚑 Mint a root token and re-pin the AppRole" + ansible.builtin.include_tasks: utils/recover.yml + + - name: "🔐 Log in with the rebuilt AppRole" + ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node + container exec -i {{ OPENBAO_LOCAL_CID | quote }} + bao write -field=token auth/approle/login + role_id={{ OPENBAO_APPROLE_ROLE_ID | quote }} secret_id=- + args: + stdin: "{{ OPENBAO_APPROLE_SECRET_ID }}" + stdin_add_newline: false + register: openbao_approle_login_recovered + changed_when: false + failed_when: false + delegate_to: "{{ OPENBAO_HOST_NODE }}" + no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" + - name: "🔧 Keep whichever AppRole login succeeded" ansible.builtin.set_fact: openbao_approle_token: >- {{ openbao_approle_login.stdout if openbao_approle_login.rc == 0 - else openbao_approle_login_previous.stdout | default('') }} + else (openbao_approle_login_previous.stdout + if (openbao_approle_login_previous.rc | default(1)) == 0 + else openbao_approle_login_recovered.stdout | default('')) }} openbao_secret_id_registered: >- {{ OPENBAO_APPROLE_SECRET_ID - if openbao_approle_login.rc == 0 + if (openbao_approle_login.rc == 0 + or (openbao_approle_login_recovered.rc | default(1)) == 0) else OPENBAO_APPLIED.approle_secret_id | default('') }} no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" -- name: "🛑 Fail when neither the current nor the previously applied AppRole is accepted" +- name: "🛑 Fail when no AppRole and no recovery key are accepted" ansible.builtin.fail: msg: >- - Neither the AppRole credentials from the inventory nor the ones recorded in - {{ OPENBAO_APPLIED_STATE_HOST }} were accepted. The instance can only be - recovered from its volume backup plus the matching seal key. + Neither the AppRole credentials from the inventory, nor the ones recorded + in {{ OPENBAO_APPLIED_STATE_HOST }}, nor a recovery key in the platform + token store were accepted. The instance can only be recovered from its + volume backup plus the matching seal key. when: (openbao_approle_token | trim) | length == 0 - name: "🔧 Store the AppRole token in the container token helper" From 19fddfddaaa843d8d7f221bdf622a8d4c8cb5feb Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sun, 20 Sep 2026 01:23:02 +0200 Subject: [PATCH 133/145] test(openbao): pin the recovery key and the path it opens Six assertions: the key is persisted on the deploy that creates it, it is loaded before the logins that may need it, recovery is gated on both logins having failed and a key existing, the rebuilt AppRole is the inventory one rather than the recorded one, the recovery path ends by re-pinning, and a rejected key fails loudly. Recovering to the recorded pair instead of the inventory pair would strand the next deploy the same way, which is why that one is asserted directly. --- .../test_credential_rotation.py | 92 +++++++++++++++++++ 1 file changed, 92 insertions(+) diff --git a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py index 554b6c0d26..164b2a90cd 100644 --- a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py +++ b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py @@ -251,6 +251,98 @@ class TestTheSecretIdSwapIsGuarded(unittest.TestCase): self.assertNotIn("when", task) +class TestTheRecoveryKeyIsKeptAndUsable(unittest.TestCase): + """The one authority that does not rotate, and the way back it buys. + + `bao operator init` issues a recovery key once and nothing else ever + reissues it. Dropping it leaves the AppRole as the only administrative + path, so a reset, a replaced manager, or a run that dies between the two + pins in 06_rotate.yml locks the instance out for good while its data stays + readable. It goes to sys-token-store rather than the inventory: no role + writes host_vars, and DIR_SECRETS outlives both DIR_COMPOSITIONS and the + node, and is covered by svc-bkp-secrets-2-local. + """ + + def setUp(self): + self.tasks = tasks_of("01_init.yml") + + def _init_block(self) -> list[dict]: + return next( + task["block"] + for task in self.tasks + if "block" in task and "openbao_initialized" in str(task.get("when", "")) + ) + + def test_the_key_is_persisted_on_the_deploy_that_creates_it(self): + store = next( + task + for task in self._init_block() + if task.get("ansible.builtin.include_role", {}).get("name") + == "sys-token-store" + ) + self.assertEqual( + store["ansible.builtin.include_role"]["tasks_from"], "write.yml" + ) + self.assertIn("recovery_keys_b64", store["vars"]["sys_token_store_token"]) + + def test_the_key_is_loaded_before_the_logins_that_may_need_it(self): + load = next( + i + for i, task in enumerate(self.tasks) + if task.get("ansible.builtin.include_role", {}).get("tasks_from") + == "01_load.yml" + ) + login = next( + i + for i, task in enumerate(self.tasks) + if task.get("register") == "openbao_approle_login" + ) + self.assertLess(load, login) + + def test_recovery_runs_only_when_both_logins_failed_and_a_key_exists(self): + block = next( + task + for task in self.tasks + if "openbao_approle_login_recovered" in str(task) + ) + conditions = " ".join(block["when"]) + self.assertIn("openbao_approle_login.rc != 0", conditions) + self.assertIn("openbao_approle_login_previous.rc", conditions) + self.assertIn("OPENBAO_RECOVERY_KEY", conditions) + + def test_the_rebuilt_approle_is_the_one_the_inventory_holds(self): + """Recovering to the recorded pair would strand the deploy again. + + 02_auth.yml re-pins both halves to the inventory values, so the login + that proves recovery worked has to use those, not the applied ones. + """ + block = next( + task + for task in self.tasks + if "openbao_approle_login_recovered" in str(task) + ) + login = next( + task + for task in block["block"] + if task.get("register") == "openbao_approle_login_recovered" + ) + self.assertIn("OPENBAO_APPROLE_ROLE_ID", login["ansible.builtin.shell"]) + self.assertIn("OPENBAO_APPROLE_SECRET_ID", login["args"]["stdin"]) + + def test_the_recovery_path_ends_by_rebuilding_the_approle(self): + tasks = tasks_of("utils/recover.yml") + self.assertEqual( + tasks[-1]["ansible.builtin.include_tasks"], + "../02_auth.yml", + "minting a root token without re-pinning leaves the deploy no better off", + ) + + def test_a_rejected_recovery_key_fails_loudly(self): + tasks = tasks_of("utils/recover.yml") + guard = next(task for task in tasks if "ansible.builtin.fail" in task) + self.assertIn("complete", guard["when"]) + + class TestTheRegisteredSecretIdComesFromTheLogin(unittest.TestCase): def test_it_is_the_credential_that_was_just_accepted(self): """Deriving it from the state file instead would be wrong. From 67ac5a3c59d67f9cb0d370f7b8aeabbc317a4533 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sun, 20 Sep 2026 01:23:13 +0200 Subject: [PATCH 134/145] docs(openbao): document the kept recovery key Replaces the note that explained why the keys were discarded. Records where the key lives and why there rather than the inventory, what it does and does not protect, and the two costs: in swarm DIR_SECRETS is the NFS export, and the key is root-equivalent. Also corrects the MODE_RESET warning a second time. A reset is no longer a lockout, because the key survives it and rebuilds the AppRole. --- roles/web-app-openbao/README.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index a6be358039..f764f1d49c 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -108,7 +108,8 @@ docker run --rm -it \ - **The seal key is the whole ballgame.** The `static` seal encrypts the raft root key with a 32-byte key generated into the ansible-vault-encrypted inventory as `secrets.credentials.seal_key` and injected as `env://OPENBAO_SEAL_KEY`. A backup of the raft volume is **undecryptable without that key**, and the key is *not* covered by `svc-bkp-secrets-2-local` (whose sources are `DIR_SECRETS`, the root CA and the ACME material). The inventory is its authoritative home — back the inventory up accordingly. Upstream considers a static seal appropriate only where an existing source of trust already holds the key; here that source is the encrypted inventory, at the same trust level as every other platform credential. - **The seal key is pinned, and that is deliberate.** `meta/secrets.yml` marks it `rotatable: false`, so the per-deploy rotation skips it. The static seal carries one generation of slack, and rotating on every deploy spends all of it continuously: a run that skips a generation, or dies between the two AppRole pins, leaves the raft store readable only from a volume backup plus the key that wrapped it — and the recovery keys that would otherwise authorise a rekey were discarded at init. The key gains little from frequent rotation in exchange, because it lives in the vault-encrypted inventory either way, and an attacker holding that inventory holds the AppRole too and does not need the seal key to reach the data. Rotate it deliberately instead, by changing `secrets.credentials.seal_key` in the inventory: `templates/openbao.hcl.j2` then renders the `previous_key_id`/`previous_key` pair, the node unseals with the old key and rewraps to the new one, and the next deploy sees them equal again. The cost of pinning is that a leaked inventory backup stays valid against future volume snapshots instead of ageing out in two deploys. -- **Recovery keys are generated and then deliberately discarded.** Initialising with an auto-seal yields *recovery* keys rather than unseal shares. They cannot decrypt the root key — only the seal key can — so they are not disaster-recovery material for the data; they only authorise `operator generate-root` and `operator rekey`. Persisting them would mean writing OpenBao-generated secrets back into the inventory, which is exactly what the AppRole bootstrap exists to avoid, so the play does not keep them. **The consequence is explicit: the Ansible AppRole is the only administrative path into this instance.** If its RoleID/SecretID are changed out-of-band inside OpenBao, there is no way back in and the instance must be rebuilt from the volume backup plus the seal key. Re-running the role does not repair that, because reconciliation itself authenticates with the AppRole — the rotation fallback below only covers the credentials the role itself applied. +- **The recovery key is kept, and it is the way back.** `bao operator init` issues it once, on the first deploy, and nothing ever reissues it — which is exactly what makes it useful: it is the only authority here that does not rotate. `tasks/01_init.yml` persists it through `sys-token-store`, the platform role for "tokens that already exist (e.g. created by bootstrap processes)". That puts it in `DIR_SECRETS`, which outlives both `MODE_RESET` (it removes `DIR_COMPOSITIONS` only) and a replaced swarm manager (`/opt/compose` is manager-local), each of which takes `.applied.json` with it. `svc-bkp-secrets-2-local` already backs that directory up. The inventory would be the more natural home, but no role writes `host_vars` — the inventory is written by the CLI, before the play. When neither the inventory AppRole nor the recorded one is accepted, `tasks/utils/recover.yml` mints a root token from the key via `bao operator generate-root` and hands over to `tasks/02_auth.yml`, which re-pins both halves to the inventory values. The deploy then continues instead of ending. Two costs are real: in swarm `DIR_SECRETS` is the NFS export, so the key is cluster-readable rather than manager-local, and a recovery key authorises `generate-root` and `rekey` — whoever reads that file owns the instance. It sits beside `tokens.yml` and `databases.csv`, which are the same class of secret. +- **A recovery key cannot decrypt the store, so it is not a backup of the seal key.** Initialising with an auto-seal yields *recovery* keys rather than unseal shares. They cannot decrypt the root key — only the seal key can — so they are not disaster-recovery material for the data; they authorise `operator generate-root` and `operator rekey`, nothing more. Losing the seal key still means the raft volume is unreadable, whatever else survives. What the recovery key buys is administrative access, which is a different failure and the one the role could not previously survive: a RoleID/SecretID changed out-of-band inside OpenBao used to leave no way back in at all. - **Bootstrap leaves no stored root token.** The first deploy runs `bao operator init`, holds the root token in memory only, enables AppRole, pins both the RoleID and the SecretID to inventory-generated values via `auth/approle/role//role-id` and `.../custom-secret-id`, then revokes the root token. Every later deploy logs in with those inventory credentials, so nothing OpenBao generates is ever written back into the inventory. Tokens reach the CLI through stdin and the container token helper, never through process arguments, and the helper file is removed at the end of the run. - **Policies live outside the config directory.** OpenBao parses every file in `-config` as server configuration, so the RBAC policy `.hcl` files are mounted at `/openbao/policies/` instead of `/openbao/config/`. - **Storage path is `/openbao/file`, not `/openbao/data`.** The image runs as the non-root `openbao` user and pre-chowns only `/openbao/{config,logs,file}`. A fresh named volume inherits ownership from the image path, so raft can write there; a volume at `/openbao/data` would land root-owned and fail. @@ -125,7 +126,7 @@ docker run --rm -it \ - `tasks/01_init.yml` attempts the AppRole login with the inventory credentials, falls back to the recorded ones, and fails loudly when neither is accepted. Whichever one was accepted is the SecretID OpenBao actually holds, so that is what `tasks/06_rotate.yml` destroys after registering the new one — rotation that only adds would revoke nothing. - `auth/approle/role//custom-secret-id` is **not** idempotent: writing a SecretID the role already holds answers `500 … SecretID is already registered`. The swap is therefore gated on the value having actually changed, which is also why the bootstrap in `tasks/02_auth.yml` pins directly instead of sharing the rotation path — at that point nothing is registered yet. - `current_key_id` is derived as a short hash of the key rather than a counter. Upstream requires the identifier to match its key; a derived id cannot drift out of step with one, and no deploy has to know which generation it is. - The record is written last, after both pins, so a run that dies partway leaves the previous generation intact and the next deploy still authenticates. The one uncovered window is a failure *between* the two pins — the RoleID re-pinned, the SecretID not, or the reverse — which leaves a pairing neither the inventory nor the state file describes, and no recovery keys are kept to get back in. Two generations is the whole window for the AppRole. Discarding the state file while the volume survives puts the instance outside it: OpenBao still holds the previous SecretID, the inventory has already rotated past it, and the fallback has nothing left to read. `MODE_RESET` does exactly that — `sys-svc-compose/tasks/01_reset.yml` removes `DIR_COMPOSITIONS` wholesale while the named volumes persist. The store itself stays readable, because the seal key is pinned and still comes from the inventory, but no administrative path into it remains. Purge the entity's volumes too when you reset — the same asymmetry breaks every entity that keeps a credential inside its volume. + The record is written last, after both pins, so a run that dies partway leaves the previous generation intact and the next deploy still authenticates. The one uncovered window is a failure *between* the two pins — the RoleID re-pinned, the SecretID not, or the reverse — which leaves a pairing neither the inventory nor the state file describes. Two generations is the whole window for the AppRole, and discarding the state file while the volume survives puts the instance outside it: OpenBao still holds the previous SecretID, the inventory has already rotated past it, and the fallback has nothing left to read. `MODE_RESET` does exactly that — `sys-svc-compose/tasks/01_reset.yml` removes `DIR_COMPOSITIONS` wholesale while the named volumes persist. Neither case is fatal any more: the store stays readable because the seal key is pinned, and the recovery key in `DIR_SECRETS` survives the reset and rebuilds the AppRole. Purge the entity's volumes too when you reset — the same asymmetry breaks every entity that keeps a credential inside its volume, and those have no recovery key. - **Other entities still desynchronise the same way, and those are not defects.** A surviving `mariadb_data` fails its healthcheck with `Access denied for user 'root'@'127.0.0.1'` for exactly the reason above, without a rotation path of its own. CI never sees it because every run starts clean; locally, run `make compose-entity-purge apps=` before redeploying. - **A role-declared account cannot be logged in as unless it sets a password.** The tiers were first covered by two role-local accounts, `baooperator` and `baoreader`, which deployed and joined their RBAC groups correctly but returned 403 on every LDAP bind: a user declared without an explicit `password` inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings. Both accounts are gone now — the tiers are covered by moving `biber` through the groups instead. [`f4398e1c5`](https://github.com/infinito-nexus/core/commit/f4398e1c58f3ce7d8d429472b2a023033242d747) closed the trap at the source: inventory creation now decides the value once, after the roles are resolved, so a declared user no longer carries a template that re-renders per use site. An explicit `password:`, as `user-administrator` and `web-app-semaphore` set, is therefore no longer required for a role-declared account to be authenticated as. - **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). From 0071a6cde9d63bd7f055ddd04f85cf281e656ddb Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Sun, 20 Sep 2026 17:54:18 +0200 Subject: [PATCH 135/145] docs(mcp): regenerate the audit report after the baserow bump The merge from main moved web-app-baserow to 2.3.4 without regenerating the report the version is rendered into, so test_mcp_audit_report saw a document that no longer matched a fresh generation. Produced by make mcp-audit. --- docs/contributing/design/role/services/mcp/audit.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/contributing/design/role/services/mcp/audit.md b/docs/contributing/design/role/services/mcp/audit.md index f9b15c0fa0..16d629ee0f 100644 --- a/docs/contributing/design/role/services/mcp/audit.md +++ b/docs/contributing/design/role/services/mcp/audit.md @@ -13,7 +13,7 @@ not the version the MCP surface was verified against. | role | version | source | transport | auth | subject | scope | tools | implementation | consumers | blocker | | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | | `svc-db-qdrant` | v1.19.1 | [link](https://s.infinito.nexus/code) | streamable_http | bearer_token | service_account (`mcp-svc-db-qdrant`) | read-only | 2 | adapter | `web-app-flowise`, `web-app-hermes`, `web-app-openclaw`, `web-app-openwebui` | — | -| `web-app-baserow` | 2.3.3 | [link](https://baserow.io/user-docs/mcp-server) | streamable_http | bearer_token | service_account (`mcp-web-app-baserow`) | read-only | 4 | adapter | `web-app-hermes`, `web-app-openclaw`, `web-app-openwebui` | — | +| `web-app-baserow` | 2.3.4 | [link](https://baserow.io/user-docs/mcp-server) | streamable_http | bearer_token | service_account (`mcp-web-app-baserow`) | read-only | 4 | adapter | `web-app-hermes`, `web-app-openclaw`, `web-app-openwebui` | — | | `web-app-checkmk` | 2.4.0p32 | [link](https://s.infinito.nexus/code) | streamable_http | bearer_token | service_account (`mcp-web-app-checkmk`) | read-only | 3 | adapter | `web-app-hermes`, `web-app-openclaw`, `web-app-openwebui` | — | | `web-app-fider` | stable | [link](https://s.infinito.nexus/code) | streamable_http | bearer_token | service_account (`mcp-web-app-fider`) | read-only | 3 | adapter | `web-app-hermes`, `web-app-openclaw`, `web-app-openwebui` | — | | `web-app-flowise` | 24-slim | [link](https://github.com/FlowiseAI/Flowise/tree/flowise%403.1.4/packages/server/src/routes/custom-mcp-servers) | — | — | — | read-only | — | native | — | — | From e06a62e19b1ed7140e6d714c46cf2a5fb5db4bca Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 21 Sep 2026 00:28:38 +0200 Subject: [PATCH 136/145] style(openbao): drop the narrative exception comments Review had these removed across the role once already, and the 11 September pass cleared every file that was flagged. The seal-key pin and the recovery-key work then reintroduced the same pattern in nine new places. Also removes two that predate the review and were never flagged: the healthcheck note in meta/services.yml and the entrypoint note in compose.yml.j2. Both are the same narrative-exception shape. Param docs, file headers and the plain contract notes in the Playwright specs stay: separate categories, none of them objected to. What the deleted comments recorded lives in the role README. --- roles/web-app-openbao/meta/services.yml | 1 - roles/web-app-openbao/tasks/00_core.yml | 3 --- roles/web-app-openbao/tasks/01_init.yml | 11 -------- .../web-app-openbao/tasks/04_auth_methods.yml | 1 - roles/web-app-openbao/tasks/06_rotate.yml | 11 +++----- roles/web-app-openbao/tasks/utils/recover.yml | 21 +++++----------- .../web-app-openbao/templates/compose.yml.j2 | 1 - roles/web-app-openbao/templates/proxy.conf.j2 | 1 - roles/web-app-openbao/vars/main.yml | 18 ++----------- .../test_credential_rotation.py | 25 ++++++++----------- 10 files changed, 22 insertions(+), 71 deletions(-) diff --git a/roles/web-app-openbao/meta/services.yml b/roles/web-app-openbao/meta/services.yml index bfe825ad8b..f020bbea99 100644 --- a/roles/web-app-openbao/meta/services.yml +++ b/roles/web-app-openbao/meta/services.yml @@ -67,7 +67,6 @@ openbao: http: 8093 healthcheck: flavor: wget - # Exception: an uninitialised container must report healthy or the init task can never reach it. path: "v1/sys/health?uninitcode=200&sealedcode=200&standbyok=true" run_after: - web-app-keycloak diff --git a/roles/web-app-openbao/tasks/00_core.yml b/roles/web-app-openbao/tasks/00_core.yml index fae44950be..1201b61e79 100644 --- a/roles/web-app-openbao/tasks/00_core.yml +++ b/roles/web-app-openbao/tasks/00_core.yml @@ -1,9 +1,6 @@ --- - include_tasks: utils/once/flag.yml -# Exception: must precede sys-stk-backend — that role renders env.j2, which -# carries the previous seal key the node needs to unseal data the last deploy's -# key wrapped. Moving it after leaves the node sealed with no way back. - name: "🔍 Read the credentials the previous deploy applied" ansible.builtin.slurp: src: "{{ OPENBAO_APPLIED_STATE_HOST }}" diff --git a/roles/web-app-openbao/tasks/01_init.yml b/roles/web-app-openbao/tasks/01_init.yml index 93a405d416..2f268a1271 100644 --- a/roles/web-app-openbao/tasks/01_init.yml +++ b/roles/web-app-openbao/tasks/01_init.yml @@ -43,11 +43,6 @@ {{ (openbao_init_raw.stdout | from_json).root_token }} no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" - # Exception: the only authority here that never rotates. Without it the - # AppRole is the sole administrative path, so losing the recorded pair -- - # a reset, a replaced manager, a run that dies between the two pins in - # 06_rotate.yml -- locks the instance out for good while its data stays - # perfectly readable. - name: "💾 Persist the recovery key through sys-token-store" ansible.builtin.include_role: name: sys-token-store @@ -96,9 +91,6 @@ delegate_to: "{{ OPENBAO_HOST_NODE }}" no_log: "{{ MASK_CREDENTIALS_IN_LOGS | bool }}" -# Rationale: the credentials rotate every deploy, so the values this run was -# handed are not yet the ones pinned inside OpenBao. Fall back to what the -# previous deploy applied; 06_rotate.yml then re-pins to this run's values. - name: "🔐 Log in with the AppRole the previous deploy applied" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec -i {{ OPENBAO_LOCAL_CID | quote }} @@ -116,9 +108,6 @@ - openbao_approle_login.rc != 0 - OPENBAO_APPLIED.approle_role_id is defined -# Rationale: the AppRole is the only administrative path into the instance, so -# losing both halves of it used to end the deploy permanently. The recovery key -# does not rotate, which is what makes it a way back. - name: "🚑 Rebuild the AppRole from the recovery key when neither login is accepted" when: - openbao_approle_login.rc != 0 diff --git a/roles/web-app-openbao/tasks/04_auth_methods.yml b/roles/web-app-openbao/tasks/04_auth_methods.yml index 346a452543..ac49e7ab29 100644 --- a/roles/web-app-openbao/tasks/04_auth_methods.yml +++ b/roles/web-app-openbao/tasks/04_auth_methods.yml @@ -134,6 +134,5 @@ loop_var: openbao_group vars: alias_mount: "ldap/" - # Rationale: the RBAC LDAP layout contract encodes the application into the group CN. alias_name: "{{ application_id }}-{{ openbao_group.item }}" when: OPENBAO_LDAP_ENABLED | bool diff --git a/roles/web-app-openbao/tasks/06_rotate.yml b/roles/web-app-openbao/tasks/06_rotate.yml index f7ed0ba83a..481f61505f 100644 --- a/roles/web-app-openbao/tasks/06_rotate.yml +++ b/roles/web-app-openbao/tasks/06_rotate.yml @@ -1,10 +1,8 @@ --- # Re-pin the AppRole to the credentials this deploy was handed, then record what -# was applied. The platform rotates the AppRole per deploy, so the values the -# next deploy receives will differ from these; the state file is what lets that -# deploy still authenticate before it re-pins in turn. The seal key is pinned -# and normally repeats, and is recorded for the deploy that follows an operator -# changing it. +# was applied. The AppRole rotates per deploy, so the state file is what lets the +# next one authenticate before it re-pins in turn. The seal key is pinned and +# normally repeats; it is recorded for the deploy after an operator changes it. - name: "🔑 Re-pin the RoleID to the value this deploy was handed" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} @@ -13,9 +11,6 @@ changed_when: true delegate_to: "{{ OPENBAO_HOST_NODE }}" -# Exception: writing custom-secret-id with a SecretID the role already holds is a -# 500 "SecretID is already registered", not a no-op, so the swap has to be gated -# on the value actually having changed. - name: "🔑 Register the SecretID this deploy was handed" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec -i {{ OPENBAO_LOCAL_CID | quote }} diff --git a/roles/web-app-openbao/tasks/utils/recover.yml b/roles/web-app-openbao/tasks/utils/recover.yml index 1b3e0ceab5..821ea6e762 100644 --- a/roles/web-app-openbao/tasks/utils/recover.yml +++ b/roles/web-app-openbao/tasks/utils/recover.yml @@ -1,21 +1,12 @@ --- -# Break-glass: mint a root token from the recovery key and rebuild the AppRole -# from the inventory. Reached only when neither the credentials this deploy was -# handed nor the ones the previous deploy recorded are accepted, which without -# this path ends the run with no administrative way into a node whose data is -# still perfectly readable. +# Break-glass: mint a root token from the recovery key, then hand to 02_auth.yml +# to re-pin the AppRole to the inventory values. Reached only when neither the +# credentials this deploy was handed nor the recorded ones are accepted. # -# The protocol is four steps and the token it yields is XOR-wrapped with a -# one-time password, so each step runs through the bao CLI and Ansible parses -# the result -- the image carries no python or jq to do it in-container. -# -# The last step hands over to 02_auth.yml, which re-pins both halves to the -# inventory values and revokes the root token it was given. The AppRole this -# deploy holds is therefore live on return, and 06_rotate.yml finds nothing -# left to swap. +# Each protocol step runs through the bao CLI and Ansible parses the result: the +# token comes back XOR-wrapped with a one-time password and the image carries +# neither python nor jq to unwrap it in-container. -# Exception: an attempt left behind by an interrupted run makes -init fail with -# "root generation already in progress". - name: "🧹 Cancel any root generation left in progress" ansible.builtin.shell: > # nocheck: container-exec-resolver runtime-resolved CID, delegated to the hosting node container exec {{ OPENBAO_LOCAL_CID | quote }} diff --git a/roles/web-app-openbao/templates/compose.yml.j2 b/roles/web-app-openbao/templates/compose.yml.j2 index 5f36f67fe0..637acc3d86 100644 --- a/roles/web-app-openbao/templates/compose.yml.j2 +++ b/roles/web-app-openbao/templates/compose.yml.j2 @@ -7,7 +7,6 @@ {% include 'roles/sys-svc-container/templates/base.yml.j2' %} {{ lookup('container_ports', ['openbao', 'http', DOCKER_BIND_HOST]) | indent(4) }} {{ lookup('container_healthcheck', service_name) | indent(4) }} - # Exception: no -config here; the image entrypoint appends -config=/openbao/config itself. command: - "server" {{ lookup('container_depends_on', application_id) | indent(4) }} diff --git a/roles/web-app-openbao/templates/proxy.conf.j2 b/roles/web-app-openbao/templates/proxy.conf.j2 index 75be4dbb20..97d5a2c23c 100644 --- a/roles/web-app-openbao/templates/proxy.conf.j2 +++ b/roles/web-app-openbao/templates/proxy.conf.j2 @@ -17,7 +17,6 @@ server {% include 'roles/sys-front-tls/templates/nginx_ssl_header.j2' %} - # Exception: the listener serves metrics unauthenticated for the internal scrape; this keeps them off the public surface. location = /v1/sys/metrics { deny all; } diff --git a/roles/web-app-openbao/vars/main.yml b/roles/web-app-openbao/vars/main.yml index 1564c6e4bc..8117f89516 100644 --- a/roles/web-app-openbao/vars/main.yml +++ b/roles/web-app-openbao/vars/main.yml @@ -20,23 +20,14 @@ OPENBAO_SEAL_KEY: "{{ lookup('config', application_id, 'secrets.credentials.seal OPENBAO_APPROLE_ROLE_ID: "{{ lookup('config', application_id, 'secrets.credentials.approle_role_id') }}" OPENBAO_APPROLE_SECRET_ID: "{{ lookup('config', application_id, 'secrets.credentials.approle_secret_id') }}" -# Rationale: the key id is derived from the key instead of counted — upstream -# requires the identifier to match its key, and a derived id cannot drift out of -# step with one however rarely an operator rotates the key. OPENBAO_SEAL_KEY_ID: "{{ (OPENBAO_SEAL_KEY | hash('sha256'))[:16] }}" -# Rationale: what the previous deploy applied, read back before the config is -# rendered so the node can unseal with the old key and rewrap to the new one, and -# so the AppRole can still authenticate after its credentials rotated. OPENBAO_APPLIED_STATE_HOST: "{{ [lookup('container', application_id, 'directories.instance'), '.applied.json'] | path_join }}" OPENBAO_APPLIED: "{{ openbao_applied_state | default({}, true) }}" OPENBAO_SEAL_KEY_PREVIOUS: "{{ OPENBAO_APPLIED.seal_key | default('', true) }}" +OPENBAO_SEAL_KEY_PREVIOUS_ID: "{{ (OPENBAO_SEAL_KEY_PREVIOUS | hash('sha256'))[:16] if OPENBAO_SEAL_KEY_PREVIOUS else '' }}" +OPENBAO_SEAL_ROTATING: "{{ (OPENBAO_SEAL_KEY_PREVIOUS | length > 0) and (OPENBAO_SEAL_KEY_PREVIOUS != OPENBAO_SEAL_KEY) }}" -# Rationale: the recovery key comes from OpenBao at init rather than from the -# inventory generator, and sys-token-store is where the platform keeps exactly -# that class of secret. DIR_SECRETS outlives both MODE_RESET and a manager -# replacement, each of which takes .applied.json with it, and -# svc-bkp-secrets-2-local already covers it. OPENBAO_RECOVERY_USER_KEY: "administrator" OPENBAO_RECOVERY_TOKEN_APP: "{{ application_id }}-recovery" OPENBAO_RECOVERY_KEY: >- @@ -44,12 +35,7 @@ OPENBAO_RECOVERY_KEY: >- .get(OPENBAO_RECOVERY_USER_KEY, {}) .get('tokens', {}) .get(OPENBAO_RECOVERY_TOKEN_APP, '') }} -OPENBAO_SEAL_KEY_PREVIOUS_ID: "{{ (OPENBAO_SEAL_KEY_PREVIOUS | hash('sha256'))[:16] if OPENBAO_SEAL_KEY_PREVIOUS else '' }}" -OPENBAO_SEAL_ROTATING: "{{ (OPENBAO_SEAL_KEY_PREVIOUS | length > 0) and (OPENBAO_SEAL_KEY_PREVIOUS != OPENBAO_SEAL_KEY) }}" -# Rationale: resolved from the login rather than from the state file — the -# SecretID OpenBao actually holds is whichever one was just accepted, and -# re-registering that same value answers 500, not a no-op. OPENBAO_SECRET_ID_REGISTERED: "{{ openbao_secret_id_registered | default('', true) }}" OPENBAO_SECRET_ID_ROTATING: "{{ (OPENBAO_SECRET_ID_REGISTERED | length > 0) and (OPENBAO_SECRET_ID_REGISTERED != OPENBAO_APPROLE_SECRET_ID) }}" OPENBAO_KV_MOUNT: "infinito" diff --git a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py index 164b2a90cd..4beccfb7b3 100644 --- a/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py +++ b/tests/unit/python/roles/web-app-openbao/test_credential_rotation.py @@ -58,13 +58,11 @@ def tasks_of(name: str) -> list[dict]: class TestWhichCredentialsRotate(unittest.TestCase): """The seal key survives a deploy; the AppRole does not. - Rotating the seal key on every deploy spends the static seal's single - generation of slack for nothing. A run that skips a generation, or dies - between the two AppRole pins, leaves the raft store readable only from a - volume backup plus the key that wrapped it, and the role keeps no recovery - keys to fall back on. The AppRole is the opposite case: it is the live - authentication path, and absorbing its rotation is what the fallback login - in 01_init.yml exists for. + The static seal carries one generation of slack, and rotating every deploy + spends it continuously: a run that skips a generation leaves the raft store + readable only from a volume backup plus the key that wrapped it. The + AppRole is the opposite case -- the live authentication path, whose + rotation the fallback login in 01_init.yml exists to absorb. """ def setUp(self): @@ -254,13 +252,12 @@ class TestTheSecretIdSwapIsGuarded(unittest.TestCase): class TestTheRecoveryKeyIsKeptAndUsable(unittest.TestCase): """The one authority that does not rotate, and the way back it buys. - `bao operator init` issues a recovery key once and nothing else ever - reissues it. Dropping it leaves the AppRole as the only administrative - path, so a reset, a replaced manager, or a run that dies between the two - pins in 06_rotate.yml locks the instance out for good while its data stays - readable. It goes to sys-token-store rather than the inventory: no role - writes host_vars, and DIR_SECRETS outlives both DIR_COMPOSITIONS and the - node, and is covered by svc-bkp-secrets-2-local. + `bao operator init` issues a recovery key once and nothing reissues it. + Dropping it leaves the AppRole as the only administrative path, so a reset, + a replaced manager, or a run that dies between the two pins in + 06_rotate.yml locks the instance out for good while its data stays + readable. sys-token-store holds it rather than the inventory, which no role + writes. """ def setUp(self): From e300d3a0dc12b0e4b7e2470ac083785cd42971c5 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 21 Sep 2026 17:45:02 +0200 Subject: [PATCH 137/145] refactor(prometheus): filter alert rules by activation, not per fragment The lookup filtered on group_names, so each fragment had to re-check whether its app was actually scraped. Deployed and scraped are different conditions: metrics can be off on a deployed role, and a rule on a series nothing scrapes alerts on absent() forever. Intersecting with native_metrics_apps moves that check to one place, and every fragment from here on is plain rules with no guard. --- plugins/lookup/alert_rule_roles.py | 21 ++++++++++++++++----- 1 file changed, 16 insertions(+), 5 deletions(-) diff --git a/plugins/lookup/alert_rule_roles.py b/plugins/lookup/alert_rule_roles.py index 05096c9ca5..dd4e139c8e 100644 --- a/plugins/lookup/alert_rule_roles.py +++ b/plugins/lookup/alert_rule_roles.py @@ -2,6 +2,7 @@ from __future__ import annotations from typing import Any +from ansible.plugins.loader import lookup_loader from ansible.plugins.lookup import LookupBase from utils.cache import ROLES_DIR @@ -11,14 +12,19 @@ ALERT_RULES_TEMPLATE = ("templates", "prometheus", "alert_rules.yml.j2") class LookupModule(LookupBase): """ - Return a sorted list of deployed role IDs that ship their own Prometheus - alert rules, i.e. every role in ``group_names`` that has a template at - ``roles//templates/prometheus/alert_rules.yml.j2``. + Return a sorted list of role IDs whose Prometheus alert rules belong in + this deploy: every role that ships a template at + ``roles//templates/prometheus/alert_rules.yml.j2`` and is actually + scraped, i.e. appears in ``native_metrics_apps``. Keeps alert rules at the role that owns the metric instead of collecting them in web-app-prometheus: a role declares what "unhealthy" means for itself, and the Prometheus role only concatenates what the play enabled. + Filtering on activation here rather than inside each fragment is what keeps + the fragments free of boilerplate. A rule that fires on a series nothing + scrapes is worse than no rule: it alerts on ``absent()`` forever. + Usage in a template: {% for role_id in lookup('alert_rule_roles') %} {% include 'roles/' + role_id + '/templates/prometheus/alert_rules.yml.j2' %} @@ -39,11 +45,16 @@ class LookupModule(LookupBase): raise ValueError("alert_rule_roles lookup takes no positional terms") vars_ = variables or getattr(self._templar, "available_variables", {}) or {} - group_names: list[str] = vars_.get("group_names", []) or [] + + scraped: list[str] = lookup_loader.get( + "native_metrics_apps", + loader=self._loader, + templar=getattr(self, "_templar", None), + ).run([], variables=vars_)[0] result = [ role_id - for role_id in sorted(set(group_names)) + for role_id in sorted(set(scraped)) if (ROLES_DIR / role_id).joinpath(*ALERT_RULES_TEMPLATE).exists() ] From 71e4437634471af6c92998b25f236ab77244f242 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 21 Sep 2026 17:45:13 +0200 Subject: [PATCH 138/145] refactor(openbao): drop the activation guard from the alert fragment The lookup now returns only scraped roles, so the fragment is rules and nothing else. --- roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 | 2 -- 1 file changed, 2 deletions(-) diff --git a/roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 b/roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 index b2ce3a9243..f944990828 100644 --- a/roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 +++ b/roles/web-app-openbao/templates/prometheus/alert_rules.yml.j2 @@ -1,4 +1,3 @@ -{% if 'web-app-openbao' in lookup('native_metrics_apps') %} - name: openbao_seal interval: 30s rules: @@ -13,4 +12,3 @@ OpenBao has not reported an active, unsealed core for more than 2 minutes. Secrets, PKI issuance and machine logins are unavailable while it is sealed. Check the container and confirm the static seal key reached its environment. -{% endif %} From 603233f89459f6ffbed235197a84ff2e78510c58 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 21 Sep 2026 17:45:26 +0200 Subject: [PATCH 139/145] test(prometheus): pin that an unscraped role contributes no alert rules Stubs native_metrics_apps and asserts the intersection, including the case the refactor exists for: a role that ships a fragment and is deployed but is not scraped must not reach the rule file. --- .../plugins/lookup/test_alert_rule_roles.py | 73 ++++++++++++------- 1 file changed, 46 insertions(+), 27 deletions(-) diff --git a/tests/unit/python/plugins/lookup/test_alert_rule_roles.py b/tests/unit/python/plugins/lookup/test_alert_rule_roles.py index 7de5d73d55..09953d117e 100644 --- a/tests/unit/python/plugins/lookup/test_alert_rule_roles.py +++ b/tests/unit/python/plugins/lookup/test_alert_rule_roles.py @@ -1,13 +1,20 @@ """Unit tests for plugins/lookup/alert_rule_roles.py. Mocks the Ansible-side machinery and asserts the plugin returns exactly the -deployed roles that ship a ``templates/prometheus/alert_rules.yml.j2``. +scraped roles that ship a ``templates/prometheus/alert_rules.yml.j2``. + +Activation is delegated to ``native_metrics_apps`` rather than re-derived here, +so these tests stub that lookup and assert the intersection. A role that is +deployed but not scraped must not contribute rules: its alert would fire on +``absent()`` forever. """ from __future__ import annotations +import tempfile import unittest import unittest.mock as mock +from pathlib import Path import plugins.lookup.alert_rule_roles as module from plugins.lookup.alert_rule_roles import LookupModule @@ -39,19 +46,26 @@ def _seed_roles(tmp_path, roles_with_template, roles_without_template=()): class TestAlertRuleRolesLookup(unittest.TestCase): def setUp(self): - self._tmp = mock.patch.object(module, "ROLES_DIR") - self.roles_dir = self._tmp.start() - self.addCleanup(self._tmp.stop) + self._roles_dir = mock.patch.object(module, "ROLES_DIR") + self.roles_dir = self._roles_dir.start() + self.addCleanup(self._roles_dir.stop) + + def _patch_scraped(self, scraped): + """Stand in for the native_metrics_apps lookup.""" + inner = mock.MagicMock() + inner.run.return_value = [list(scraped)] + loader = mock.patch.object(module, "lookup_loader") + started = loader.start() + started.get.return_value = inner + self.addCleanup(loader.stop) def test_raises_when_positional_terms_supplied(self): + self._patch_scraped([]) lm = _make_lookup({"group_names": []}) with self.assertRaises(ValueError): lm.run(["unexpected"], variables=lm._templar.available_variables) - def test_returns_only_deployed_roles_that_ship_a_template(self): - import tempfile - from pathlib import Path - + def test_returns_only_scraped_roles_that_ship_a_template(self): with tempfile.TemporaryDirectory() as td: tmp_path = Path(td) _seed_roles( @@ -60,35 +74,40 @@ class TestAlertRuleRolesLookup(unittest.TestCase): roles_without_template=["web-app-matomo"], ) module.ROLES_DIR = tmp_path + self._patch_scraped(["web-app-matomo", "web-app-openbao"]) - vars_ = { - "group_names": [ - "web-app-matomo", - "web-app-openbao", - "web-app-never-deployed", - ] - } - lm = _make_lookup(vars_) - self.assertEqual(lm.run([], variables=vars_), [["web-app-openbao"]]) + lm = _make_lookup({}) + self.assertEqual(lm.run([], variables={}), [["web-app-openbao"]]) + + def test_a_deployed_role_that_is_not_scraped_contributes_no_rules(self): + """The whole point of moving the filter out of the fragments. + + svc-db-postgres ships a template here but native_metrics_apps does not + list it, so its rules must not reach the file. + """ + with tempfile.TemporaryDirectory() as td: + tmp_path = Path(td) + _seed_roles(tmp_path, roles_with_template=["svc-db-postgres"]) + module.ROLES_DIR = tmp_path + self._patch_scraped([]) + + lm = _make_lookup({}) + self.assertEqual(lm.run([], variables={}), [[]]) def test_result_is_sorted_and_deduplicated(self): - import tempfile - from pathlib import Path - with tempfile.TemporaryDirectory() as td: tmp_path = Path(td) _seed_roles(tmp_path, roles_with_template=["b-role", "a-role"]) module.ROLES_DIR = tmp_path + self._patch_scraped(["b-role", "a-role", "b-role"]) - vars_ = {"group_names": ["b-role", "a-role", "b-role"]} - lm = _make_lookup(vars_) - self.assertEqual(lm.run([], variables=vars_), [["a-role", "b-role"]]) - - def test_returns_empty_list_without_group_names(self): - import tempfile - from pathlib import Path + lm = _make_lookup({}) + self.assertEqual(lm.run([], variables={}), [["a-role", "b-role"]]) + def test_returns_empty_list_when_nothing_is_scraped(self): with tempfile.TemporaryDirectory() as td: module.ROLES_DIR = Path(td) + self._patch_scraped([]) + lm = _make_lookup({}) self.assertEqual(lm.run([], variables={}), [[]]) From 6a01b190b026a7d79c283532ebad47f0545514a4 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 21 Sep 2026 17:45:34 +0200 Subject: [PATCH 140/145] chore(openbao): promote the role to beta Twelve of twelve deploy jobs green across compose and swarm, clearnet and tor, on every distro in the rotation. --- roles/web-app-openbao/meta/services.yml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/roles/web-app-openbao/meta/services.yml b/roles/web-app-openbao/meta/services.yml index f020bbea99..1f6c913ce8 100644 --- a/roles/web-app-openbao/meta/services.yml +++ b/roles/web-app-openbao/meta/services.yml @@ -71,7 +71,7 @@ openbao: run_after: - web-app-keycloak - svc-db-openldap - lifecycle: alpha + lifecycle: beta # nocheck: playwright-service-flag # nocheck: variants-coverage — sso-variant-tor requires the onion on in every variant, so the false polarity has no variant left to live in From 08855ff9f0400d013e52b759a64fdb0f5e2db659 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 21 Sep 2026 17:45:45 +0200 Subject: [PATCH 141/145] docs(openbao): cut the developer notes down to what gets read Eighteen bullets to ten. Merges the four seal-and-recovery notes into two, folds the three path traps into one, compresses the rotation block from six sub-bullets to a paragraph, and drops the note on role-declared account passwords entirely: the commit it was waiting on has landed, so the workaround it described is no longer needed. Nothing operational is lost. The seal key still says a raft backup is undecryptable without the inventory, and the recovery key still says where it lives and what it cannot do. --- roles/web-app-openbao/README.md | 34 ++++++++++----------------------- 1 file changed, 10 insertions(+), 24 deletions(-) diff --git a/roles/web-app-openbao/README.md b/roles/web-app-openbao/README.md index f764f1d49c..d5226a022f 100644 --- a/roles/web-app-openbao/README.md +++ b/roles/web-app-openbao/README.md @@ -106,30 +106,16 @@ docker run --rm -it \ ## Developer Notes -- **The seal key is the whole ballgame.** The `static` seal encrypts the raft root key with a 32-byte key generated into the ansible-vault-encrypted inventory as `secrets.credentials.seal_key` and injected as `env://OPENBAO_SEAL_KEY`. A backup of the raft volume is **undecryptable without that key**, and the key is *not* covered by `svc-bkp-secrets-2-local` (whose sources are `DIR_SECRETS`, the root CA and the ACME material). The inventory is its authoritative home — back the inventory up accordingly. Upstream considers a static seal appropriate only where an existing source of trust already holds the key; here that source is the encrypted inventory, at the same trust level as every other platform credential. -- **The seal key is pinned, and that is deliberate.** `meta/secrets.yml` marks it `rotatable: false`, so the per-deploy rotation skips it. The static seal carries one generation of slack, and rotating on every deploy spends all of it continuously: a run that skips a generation, or dies between the two AppRole pins, leaves the raft store readable only from a volume backup plus the key that wrapped it — and the recovery keys that would otherwise authorise a rekey were discarded at init. The key gains little from frequent rotation in exchange, because it lives in the vault-encrypted inventory either way, and an attacker holding that inventory holds the AppRole too and does not need the seal key to reach the data. Rotate it deliberately instead, by changing `secrets.credentials.seal_key` in the inventory: `templates/openbao.hcl.j2` then renders the `previous_key_id`/`previous_key` pair, the node unseals with the old key and rewraps to the new one, and the next deploy sees them equal again. The cost of pinning is that a leaked inventory backup stays valid against future volume snapshots instead of ageing out in two deploys. -- **The recovery key is kept, and it is the way back.** `bao operator init` issues it once, on the first deploy, and nothing ever reissues it — which is exactly what makes it useful: it is the only authority here that does not rotate. `tasks/01_init.yml` persists it through `sys-token-store`, the platform role for "tokens that already exist (e.g. created by bootstrap processes)". That puts it in `DIR_SECRETS`, which outlives both `MODE_RESET` (it removes `DIR_COMPOSITIONS` only) and a replaced swarm manager (`/opt/compose` is manager-local), each of which takes `.applied.json` with it. `svc-bkp-secrets-2-local` already backs that directory up. The inventory would be the more natural home, but no role writes `host_vars` — the inventory is written by the CLI, before the play. When neither the inventory AppRole nor the recorded one is accepted, `tasks/utils/recover.yml` mints a root token from the key via `bao operator generate-root` and hands over to `tasks/02_auth.yml`, which re-pins both halves to the inventory values. The deploy then continues instead of ending. Two costs are real: in swarm `DIR_SECRETS` is the NFS export, so the key is cluster-readable rather than manager-local, and a recovery key authorises `generate-root` and `rekey` — whoever reads that file owns the instance. It sits beside `tokens.yml` and `databases.csv`, which are the same class of secret. -- **A recovery key cannot decrypt the store, so it is not a backup of the seal key.** Initialising with an auto-seal yields *recovery* keys rather than unseal shares. They cannot decrypt the root key — only the seal key can — so they are not disaster-recovery material for the data; they authorise `operator generate-root` and `operator rekey`, nothing more. Losing the seal key still means the raft volume is unreadable, whatever else survives. What the recovery key buys is administrative access, which is a different failure and the one the role could not previously survive: a RoleID/SecretID changed out-of-band inside OpenBao used to leave no way back in at all. -- **Bootstrap leaves no stored root token.** The first deploy runs `bao operator init`, holds the root token in memory only, enables AppRole, pins both the RoleID and the SecretID to inventory-generated values via `auth/approle/role//role-id` and `.../custom-secret-id`, then revokes the root token. Every later deploy logs in with those inventory credentials, so nothing OpenBao generates is ever written back into the inventory. Tokens reach the CLI through stdin and the container token helper, never through process arguments, and the helper file is removed at the end of the run. -- **Policies live outside the config directory.** OpenBao parses every file in `-config` as server configuration, so the RBAC policy `.hcl` files are mounted at `/openbao/policies/` instead of `/openbao/config/`. -- **Storage path is `/openbao/file`, not `/openbao/data`.** The image runs as the non-root `openbao` user and pre-chowns only `/openbao/{config,logs,file}`. A fresh named volume inherits ownership from the image path, so raft can write there; a volume at `/openbao/data` would land root-owned and fail. -- **`command:` is just `server`.** The image entrypoint appends `-config=/openbao/config` itself; passing an explicit `-config` duplicates the flag. -- **CLI OIDC login is out of scope.** The `redirect_uris` lookup registers `https:///*` per SSO consumer, which covers the UI and API callbacks but not the CLI's `http://localhost:8250/oidc/callback`. Use AppRole for non-interactive access. -- **Visiting `/ui/vault/logout` does not end the session.** Measured against a live instance: after that navigation the UI returns to `/ui/vault/secrets` still authenticated, because the Ember app restores its token from browser storage. Use the UI's own logout control, or clear site data, when you actually need the session gone. -- **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's `vault_*` metric namespace, and on a healthy unsealed node this gauge is emitted as `vault_core_unsealed{cluster=""} 0` — an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed) as the seal signal; `vault_core_active{cluster="…"} == 1` corroborates from the metrics side. -- **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume and `service_update_order = stop-first` — raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing the same directory. -- **The RBAC tiers are proven by moving `biber` through the groups, not by dedicated accounts.** `files/playwright/test-rbac-groups.js` signs the administrator into LAM, adds `biber` to `cn=web-app-openbao-,ou=roles` through LAM's Multi edit tool, logs `biber` into OpenBao over LDAP and asserts both the policy set and the capability that tier grants, then removes him again and asserts the revoke. One account covers all three tiers, and the grant and the revoke are observed rather than assumed — a permanently-assigned member can only ever show a steady state. LAM's *group list* cannot be used for this: it renders `posixGroup` entries, while the RBAC role groups are `groupOfNames` under `ou=roles`. LAM applies the change asynchronously, so the spec polls the resulting policy set instead of LAM's own progress wording. -- **Every deploy rotates the AppRole, and the role is built around that.** The platform regenerates generated-algorithm credentials on each deploy — that is how a changed Ansible password reaches the application instead of silently diverging from it. For OpenBao it means the `role_id`/`secret_id` a run is handed are *not* the ones the running instance was configured with, which would otherwise leave the AppRole answering `invalid role or secret ID`. The role therefore carries the transition itself: - - `tasks/06_rotate.yml` records what each deploy actually applied in `.applied.json` inside the compose project directory, `0600`, root-owned. The directory already holds the seal key in plaintext in `.env/env`, so this adds no new exposure class. - - `tasks/00_core.yml` reads that file back **before** `sys-stk-backend` renders `env.j2` — the rendered env must already carry `OPENBAO_SEAL_KEY_PREVIOUS`, or the node comes up with no way to decrypt its own store. - - `templates/openbao.hcl.j2` emits the `previous_key_id`/`previous_key` pair only while the two keys differ. The static seal supports `n-1` → `n` rotation, so the node unseals with the old key and rewraps to the new one. - - `tasks/01_init.yml` attempts the AppRole login with the inventory credentials, falls back to the recorded ones, and fails loudly when neither is accepted. Whichever one was accepted is the SecretID OpenBao actually holds, so that is what `tasks/06_rotate.yml` destroys after registering the new one — rotation that only adds would revoke nothing. - - `auth/approle/role//custom-secret-id` is **not** idempotent: writing a SecretID the role already holds answers `500 … SecretID is already registered`. The swap is therefore gated on the value having actually changed, which is also why the bootstrap in `tasks/02_auth.yml` pins directly instead of sharing the rotation path — at that point nothing is registered yet. - - `current_key_id` is derived as a short hash of the key rather than a counter. Upstream requires the identifier to match its key; a derived id cannot drift out of step with one, and no deploy has to know which generation it is. - The record is written last, after both pins, so a run that dies partway leaves the previous generation intact and the next deploy still authenticates. The one uncovered window is a failure *between* the two pins — the RoleID re-pinned, the SecretID not, or the reverse — which leaves a pairing neither the inventory nor the state file describes. Two generations is the whole window for the AppRole, and discarding the state file while the volume survives puts the instance outside it: OpenBao still holds the previous SecretID, the inventory has already rotated past it, and the fallback has nothing left to read. `MODE_RESET` does exactly that — `sys-svc-compose/tasks/01_reset.yml` removes `DIR_COMPOSITIONS` wholesale while the named volumes persist. Neither case is fatal any more: the store stays readable because the seal key is pinned, and the recovery key in `DIR_SECRETS` survives the reset and rebuilds the AppRole. Purge the entity's volumes too when you reset — the same asymmetry breaks every entity that keeps a credential inside its volume, and those have no recovery key. -- **Other entities still desynchronise the same way, and those are not defects.** A surviving `mariadb_data` fails its healthcheck with `Access denied for user 'root'@'127.0.0.1'` for exactly the reason above, without a rotation path of its own. CI never sees it because every run starts clean; locally, run `make compose-entity-purge apps=` before redeploying. -- **A role-declared account cannot be logged in as unless it sets a password.** The tiers were first covered by two role-local accounts, `baooperator` and `baoreader`, which deployed and joined their RBAC groups correctly but returned 403 on every LDAP bind: a user declared without an explicit `password` inherits the `strong_password` fallback expression, which is re-evaluated on every render (`secrets.choice`, nothing persisted), so the value `svc-db-openldap` writes with `ldap_passwd` and the value `playwright.env.j2` renders can be different strings. Both accounts are gone now — the tiers are covered by moving `biber` through the groups instead. [`f4398e1c5`](https://github.com/infinito-nexus/core/commit/f4398e1c58f3ce7d8d429472b2a023033242d747) closed the trap at the source: inventory creation now decides the value once, after the roles are resolved, so a declared user no longer carries a template that re-renders per use site. An explicit `password:`, as `user-administrator` and `web-app-semaphore` set, is therefore no longer required for a role-declared account to be authenticated as. -- **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup and concatenated into the Prometheus rule file, so the seal alert lives next to the thing it watches. The fragment must start at the `groups:` item level (two-space-indented `- name:` entries). +- **The seal key is the whole ballgame.** The `static` seal wraps the raft root key with `secrets.credentials.seal_key`, injected as `env://OPENBAO_SEAL_KEY`. A backup of the raft volume is **undecryptable without it**, and `svc-bkp-secrets-2-local` does not cover it — the encrypted inventory is its only home, so back the inventory up accordingly. It is pinned `rotatable: false`: the static seal carries one generation of slack, and spending that every deploy makes any skipped generation unrecoverable. Rotate it deliberately by changing the inventory value; `openbao.hcl.j2` then emits the `previous_key`/`previous_key_id` pair, the node unseals with the old key and rewraps to the new one. +- **The recovery key is the way back in.** `bao operator init` issues it once and nothing reissues it, which is what makes it useful — it is the only authority here that does not rotate. It is persisted through `sys-token-store` (`DIR_SECRETS`, backed up, and it survives both `MODE_RESET` and a replaced manager, each of which takes `.applied.json` with it). When neither the inventory AppRole nor the recorded one is accepted, `tasks/utils/recover.yml` mints a root token from it and re-pins the AppRole. It cannot decrypt the store — only the seal key can — so it is not a substitute for backing up the inventory. +- **Every deploy rotates the AppRole.** The `role_id`/`secret_id` a run is handed are not the ones the running instance holds. `tasks/01_init.yml` tries the inventory pair, falls back to the pair recorded in `.applied.json`, then `tasks/06_rotate.yml` re-pins to this run's values and destroys the old SecretID. The record is written last, so a run that dies partway leaves the previous generation intact. `custom-secret-id` is **not** idempotent — re-writing a held SecretID answers `500` — so the swap is gated on the value having changed. +- **Bootstrap leaves no stored root token.** The first deploy holds it in memory, pins both AppRole halves to inventory values, then revokes it. Tokens reach the CLI through stdin and the container token helper, never through process arguments, and the helper file is removed at the end of the run. +- **Paths that look wrong and are not.** Policies mount at `/openbao/policies/`, because OpenBao parses *every* file under `-config` as server configuration. Storage is `/openbao/file`, not `/openbao/data`: the image pre-chowns only `/openbao/{config,logs,file}` for the non-root `openbao` user, so a volume elsewhere lands root-owned and raft cannot write. `command:` is just `server` — the entrypoint appends `-config` itself, and passing it duplicates the flag. +- **Do not alert on `vault_core_unsealed`.** OpenBao keeps Vault's metric namespace, and a healthy unsealed node emits `vault_core_unsealed{cluster=""} 0`, so an alert on `== 0` fires permanently. Use `/v1/sys/health` (503 while sealed); `vault_core_active{cluster="…"} == 1` corroborates. +- **Alert rules ship with the role.** `templates/prometheus/alert_rules.yml.j2` is discovered by the `alert_rule_roles` lookup, which returns only roles that are actually scraped, so the fragment carries no activation guard of its own. It must start at the `groups:` item level. +- **Swarm:** single replica, `placement: manager`, `nfs: false` on the raft volume, `service_update_order = stop-first`. Raft's BoltDB store is mmap + `fsync` based, so it must stay node-local and must never have two tasks writing one directory. +- **CLI OIDC login is out of scope.** `redirect_uris` registers `https:///*`, covering UI and API callbacks but not the CLI's `http://localhost:8250/oidc/callback`. Use AppRole for non-interactive access. +- **Test notes.** The RBAC tiers are proven by moving `biber` through the groups with LAM's Multi edit tool and asserting both the grant and the revoke, rather than by dedicated accounts. `/ui/vault/logout` does not end a session — the Ember app restores its token from browser storage — so use the UI's own logout control when the session must actually be gone. ### Backup and restore From 9261ac9b0119e38019f6d6e2e6531988dc2ebeb2 Mon Sep 17 00:00:00 2001 From: evangelostsak Date: Mon, 21 Sep 2026 18:35:42 +0200 Subject: [PATCH 142/145] docs(web-app-lam): record why script-src-elem cannot be tightened The previous reason named only multi edit, which made the relaxation look specific to one tool and newly needed. Neither is true. LAM emits inline