Every kcadm call in the per-app mapper scripts sent stderr to /dev/null, so a failing call surfaced as rc 1 with empty stdout and empty stderr - no cause at all. CI run 32119014832 burned 3h23m on exactly that empty result.
stdout redirects, the tolerated-failure guards and every failed_when stay untouched: stdout is these scripts' return channel and the callers gate on its length.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
kcadm keeps its session inside the container, so a swarm task that gets rescheduled comes back unauthenticated. The cross-node rescue re-resolved the node but never logged in again, so all five recovery attempts died with "No server specified" -- the recovery path could not recover, by construction. Observed in CI run 32119014832 on FusionDirectory#0 and Fediwall#2 with an identical signature; a planned replacement survives today only because the realm import re-includes login.yml right after it.
Reuse utils/login.yml for the recovery leg: it already performs the same resolve and additionally re-authenticates and waits for health.
Drop the health=healthy filter on the login resolve. It contradicted the wait-for-healthy task right after it: with no healthy container the lookup returned an empty id, and inspecting an empty id can never report healthy, turning that wait into a 300s dead end whose empty-but-defined fact also defeats the default() in vars/main.yml. It also let login target a different container than the one updates.yml resolves unfiltered a moment later. aa6e156a6 relaxed the sibling resolver to status=running for the same reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
unsafe-inline re-admits the inline execution an XSS needs and unsafe-eval re-admits string-to-code, so a CSP relaxation must be verified against the running app rather than copied from another role. The new lint requires a nocheck marker plus a Reason on the flag's own line, so a marker cannot drift to the neighbouring directive.
The 197 pre-existing grants across 70 roles are checked out with a TODO: they were never verified and still have to be.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The cosmos generator linked a consumer to its provider only through an
'<role>' in group_names reference or an explicit provides: name. A hard
dependency has neither -- pinning the provider on unconditionally is
exactly what removes the Jinja reference the generator was reading -- so
web-svc-asset's bond to web-svc-file vanished from the diagram the moment
it became binding. Match a service key against the provider's own primary
key from the registry, in both directions.
Services the role turns off drew a solid 1:1 edge, claiming a mandatory
relationship for something that carries nothing at deploy time. They now
render as a red 0..0 edge; edge merging gained a real precedence so a
provider consumed through two keys stays live when only one is off.
Named colour, not hex: a '#' opens an entity code in the mermaid lexer and
turns the whole diagram into a parse error on GitHub.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role sat behind the oauth2-proxy, which authenticates at the edge and hands
the app nothing: SuiteCRM kept serving its own Angular login after the Keycloak
round trip, so no session ever existed and the app had no logout control. CI run
31927586060 is the only run in which the role's Playwright spec ever executed,
and it failed exactly there - "no in-app logout control reachable on the current
authenticated surface" at personas/utils/logout.js.
SuiteCRM 8.9.1 ships nbgrp/onelogin-saml-bundle and treats AUTH_TYPE=saml as a
first-class firewall in config/packages/security.php:249, so the app can own its
session without a line of PHP from us. flavor: saml turns the proxy off by
itself - is_proxy_gated is `enabled AND flavor == 'oauth2'` - which also drops
the redis session store and the auth_request clamp in the vhost. The role
declares no allowed_groups, so no edge ACL is lost.
Four settings are not cosmetic; each is a failure reproduced against a live
Keycloak 26.7.1 until the login completed:
- SAML_COMPRESS_REQUESTS: the HTTP-Redirect binding requires DEFLATE, and
Keycloak rejects an uncompressed AuthnRequest with a ZipException.
- SAML_USE_ATTRIBUTE_FRIENDLY_NAME=false: the bundle defaults to matching
attributes by friendlyName, which Keycloak's mappers do not set, so not a
single attribute arrives.
- APP_DEBUG is forced to 0 whenever SAML is on: public/legacy/modules/Users/
User.php:944 reads $_POST['ut'] unguarded while creating the user, and with
debug on Symfony turns that warning into an exception the authenticator
reports as a failed login - the account is created and the FIRST login of
every new user fails anyway.
- SAML_SP_ENTITY_ID must equal the Keycloak client's clientId.
The IdP signing certificate is read from the realm descriptor at boot rather
than templated in, so a key rotation needs no redeploy. It uses the same public
issuer URL the browser is redirected to - one address, not two - with the SOCKS
proxy espocrm established for onion issuers. An unreachable IdP or a descriptor
without a certificate aborts the start; both paths are verified, as is the skip
when the variable is absent on LDAP/native deployments.
Verified end to end locally against Keycloak 26.7.1, MariaDB 12.3 and SuiteCRM
8.9.1 from the upstream tarball: session-status reports active:true for the
Keycloak user, auto-created on the first attempt against an empty users table.
web-app-keycloak/TODO.md listed this role for the trusted-header bridge. Nothing
of it was implemented, and SuiteCRM exposes no header entry point at all
(REMOTE_USER / X-Forwarded-User / X-Auth-Request appear nowhere in core/ or
public/legacy), so it moves to the same exclusion as jitsi: use the native
mechanism the app already ships.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The realm carried exactly one client, the shared OIDC one, and no SAML client
existed anywhere in the repo - `flavor: saml` was in the resolver's enum with
zero consumers. Apps that cannot speak OIDC therefore had no way to
authenticate against Keycloak at all; they could only be fronted by the
oauth2-proxy, which authenticates at the edge and leaves the app itself without
a session.
KEYCLOAK_SAML_APPS collects every deployed app whose services.sso.flavor is
saml. It filters stepwise - value.services, then .sso, then .flavor, each
guarded by `defined` - because Ansible's undefined is a Marker(StrictUndefined,
Tripwire): resolving that chain in one selectattr raises for every app without
an sso block instead of yielding undefined, which would have broken the realm
render for every deployment in the repo.
The include is unconditional and the loop emits a leading comma per client, so
the realm needs no branch on the list being empty. That matters: an Ansible
`>-` value that failed to convert would be a non-empty string and a truthiness
guard would then splice a broken clients array.
The client's shape is not guesswork - every attribute comes from a failure
reproduced against a live Keycloak 26.7.1 until a SuiteCRM SAML login completed:
defaultClientScopes is emptied because the built-in role_list scope emits one
Role attribute per realm role and the SP rejects the assertion with "Found an
Attribute element with duplicated Name"; saml.client.signature is false because
the SP does not sign its AuthnRequests; the username mapper's attribute name
comes from OIDC.ATTRIBUTES.USERNAME rather than a hardcoded claim.
The new test renders the complete realm - every include, the role's own
ldap_groups_filter, StrictUndefined - and parses it as JSON for zero, one and
several SAML apps. The file had no test before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bare KC_HOSTNAME combined with KC_HOSTNAME_STRICT=false lets Keycloak derive scheme and port from the incoming request. The browser reaches the realm through the proxy on port 80 while container-to-container calls reach it on keycloak:8080, so the same realm mints two different issuers and userinfo rejects the front-channel token with 'Invalid token issuer'.
OIDC.URL carries the scheme, which Keycloak uses verbatim. It is also the value group_vars/all/11_oidc.yml:19 already hands every client as the expected issuer, so server and clients now agree.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.
Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.
The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.
Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7a70d27b7 made a variant that switches SSO on also switch the onion on, because
that pairing is where the platform breaks in ways clearnet never shows. The same
argument reaches further than SSO: a Secure cookie is dropped, an https identity
provider refuses to embed, and crypto.subtle is undefined for any feature on a
non-secure origin, not only for a login. A variant that never runs the onion is
not testing the deployment users get.
So the rule now covers every variant of a role that declares a tor block.
Variant 1 stays exempt - it is the minimal, partner-free round every role
declares and the one place the stack is exercised without the onion at all - and
a role whose meta/services.yml pins tor.enabled to a literal false is skipped
outright, because that declaration already says the role has no onion and needs
no comment repeating it.
That found 137 variants. 101 were variant 1. Of the rest, 28 are the LDAP-only
auth-isolation rounds, which gain real coverage from running LDAP over the onion
and are flipped on here. Four cannot: svc-net-tor is the provider itself, and
nextcloud's variant 8 pairs the clearnet-only PeerTube. Both carry a marker
naming the reason; opentalk and coturn need none, since their services.yml pins
the literal false the rule now reads.
Flipping them exposed a contract that contradicted itself. test_auth_isolation
and test_non_baseline_explicit_disables demanded `shared: false` beside
`enabled: false`, while variants-static-override flagged that same line as a
static override - which is why all 133 of them carried a nocheck marker. The
demand only makes sense where `shared` is dynamic: a literal base value cannot
leak through deep-merge, so restating it protects nothing. Both contracts now
ask for it only in that case, and the 133 lines are gone.
The result is a stricter rule with fewer lines: 189 insertions against 193
deletions.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles 107
passed / 610 subtests. The rule is checked in both directions - it reported 137
offenders before the resolution and none after.
Not measured: 28 additional onion rounds cost matrix wall-clock, against swarm
jobs already capped at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.
It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and f1dff50fb established both the reasoning and the precedent when it
dropped xwiki and bigbluebutton from 1 to 0.5: bridged partners with their own
role and their own spec, whose bond declaration was the outlier rather than
their placement.
This also unblocks the variant-level tor contract. Requiring a non-baseline
variant to pin the onion on while tor still counted as tightly coupled would
have pulled variant 0's closure requirements along with it.
Verified: tests/lint 336 passed / 307 subtests and tests/integration/roles 107
passed / 610 subtests, including test_resource_budget over every variant's
deduplicated footprint and the bond-aware test_order closure.
Not measured, and the honest cost: f1dff50fb's own caveat applies unchanged
here. Whether redistributing a service out of the baseline closure buys back
wall-clock is not established; role-runtimes has the per-role seconds and no run
has been made with this shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/volumes.yml says where a volume lands inside its container. Wherever a
second file spelled the same path out, the two drifted independently: the mount
moves, the consumer keeps pointing at the old place, and nothing fails until the
process looks for content that is not there. e031b7f61 fixed exactly this for
web-app-moodle; this generalises it and makes the rule hold.
The lookup gains what a caller needs. `targets` maps each mounting service to
its container path, and `target` is the one path an entry mounts at -- present
only when every mount agrees. An entry that lands at different paths per service
omits `target` rather than picking one, and the caller names the service:
"{{ lookup('volume', application_id, 'prometheus_config').target }}"
"{{ lookup('volume', 'web-app-prometheus', 'host_sys').targets['node-exporter'] }}"
66 literals across 48 role files now derive. Replacement only happened where the
path stood alone -- /srv/app inside /srv/app/data is a different path -- and
every rewritten file was re-parsed, Jinja and YAML alike. Three sites the tool
refused rather than guess were done by hand: zammad's path sat in a Jinja list
literal, seaweedfs' inside a quoted string feeding a filter, and openresty's
/var/run/docker.sock is not a mount target at all but the socket on the host,
which only happens to spell the same string. That one carries the single
nocheck.
tests/lint/repository/spot/test_volumes_path_single_spot.py keeps it that way.
roles/*/files/ is out of scope by construction: it is copied verbatim into an
image and executed there, so a Dockerfile or entrypoint has no Ansible to
evaluate a lookup with.
One defect this found, and it was in the tooling rather than the roles: deciding
ambiguity AFTER filtering short paths made web-app-prometheus' host_sys look
single-valued -- it mounts /sys for cadvisor and /host/sys for node-exporter,
and /sys is below the length filter. The template would have rendered an
undefined `.target`. Both the converter and the rule now decide ambiguity over
every target and filter afterwards, and that call reads targets['node-exporter'].
Verified by resolving rather than by reading: all 70 accesses in the diff were
run against the real registry and every one yields a non-empty path equal to the
literal it replaced. make test-lint: 328 passed.
Not verified: no Ansible render, so a call site whose surrounding context cannot
build the applications registry would still fail at deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A variant entry is baked into host_vars raw, not deep-merged, so its top-level
key IS the inventory path. keycloak and lam still pinned their Playwright
administrator_password under credentials, while f713ba835 moved both roles'
vars/main.yml to secrets.credentials. The write and the read had been pointing
at different places ever since.
Nothing failed, which is the problem: the read resolves, because the generator
had produced a random password at the new path, so the deploy came up with a
secret nobody knows and the admin sign-in the pin exists for broke with no
trace. The inventory validator could not catch it either -- it widens the legal
keys with the variant overlays themselves, so the stale root was legal exactly
because the variant declared it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml held credential definitions and their runtime values, but the
name said 'schema' and read like structural metadata next to services.yml and
volumes.yml. Nothing in the file is a schema in the sense the other meta files
use the word, and a contributor had to open it to learn it holds secrets.
62 role files renamed. ROLE_FILE_META_SCHEMA becomes ROLE_FILE_META_SECRETS and
keeps its role as the SPOT the path is built from.
Two readers bypassed that constant and spelled the name themselves:
utils/cache/applications.py built meta_dir / 'schema.yml' twice, and
plugins/lookup/application_gid.py carries the filename in the marker list it
detects application roles by. Left behind, the rename would not have raised —
credentials would have resolved empty and the role would have stopped counting
as an application.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
08_users/realm.yml seeded the realm from `lookup('users') | non_reserved_users`
while 08_rbac_groups_realm.yml built group memberships from the raw
`lookup('users')`. The membership builder has its own guards, but both are dead
against real data: across all 269 default users, none carries a `reserved` key
and none has an empty password. So membership was effectively unfiltered while
seeding filtered on `accounts`.
helpdesk and no-reply (accounts: [mailbox], roles: [bot]) were therefore
demanded as members of roles/web-app-mailu/bot but never created, and
ensure_group_path_members.sh exits 1 on the first miss. Four jobs -- Mattermost 1,
Friendica 1, Matrix 1, OpenProject 1 -- each burned five cross-node recovery
attempts on a data failure no reschedule could fix.
Rather than adding the filter at the site that forgot it, remove the ability to
forget: lookup('users_non_reserved') resolves users and applies the predicate in
one place, and all four consumers now read it -- the two Keycloak user tasks, the
RBAC group builder, and OPENLDAP_USERS, which had it right all along. The filter
stays as the pure predicate the lookup calls.
Reproduced offline against the real user model, byte-identical to the CI member
list: before, bot demanded helpdesk and no-reply; after, members are exactly the
four seeded accounts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
KEYCLOAK_REDIRECT_URIS derived the scheme from Keycloak's OWN TLS state and
stamped that single scalar onto every SSO consumer. With svc-net-tor
deployed Keycloak is onion-primary, so resolve_enabled returns False and
lookup('tls', ...) yields "http" -- which then went onto clearnet apps too.
Keycloak registered http://matrix.infinito.example/* while Synapse sends
https://.../_synapse/client/oidc/callback, and its redirect-URI match is
scheme-exact: 90 invalid_redirect_uri events, an error page instead of a
login form, and a 60s selector timeout that looked like a Playwright bug.
jitsi hit the same wall on https://meet.infinito.example/oauth2/callback.
Resolve per consumer instead, via resolve_enabled: an .onion is always
plaintext, a clearnet app follows its own server.tls.enabled, and only
then the global TLS_ENABLED. That override existed and was silently
discarded -- passing a pre-computed scheme string threw it away, which is
the same "what I am" instead of "what the consumer is" mistake one layer
up. resolve_enabled handles the onion case itself, so the filter's own
is_onion_domain branch is gone rather than duplicated.
Move it from a filter to a lookup while at it: the thing needs domains,
applications and TLS_ENABLED anyway, so resolving them inside collapses
the call site to lookup('redirect_uris') and drops
KEYCLOAK_REDIRECT_FEATURES, which listed the same path twice. A lookup
under plugins/lookup/ is also plainly importable, so the test loses the
sys.modules stubbing apparatus a role-local filter needed -- 60 lines net
smaller.
KEYCLOAK_WEB_ORIGINS is regex-derived from these URIs and inherits the
fix. Verified against the real applications view: 69 URIs, all https with
TLS_ENABLED true and all http with it false -- the same 69 the failing run
registered as plaintext.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Blocking a persona removes the only end-to-end proof that a role's auth chain
works. docs/contributing/artefact/files/role/playwright.specs.js.md:146 has
always required each flag to carry a one-line nocheck stating the role contract
that justifies it, plus a matching paragraph in the role's README or TODO.
Nothing enforced it, so 89 flags across 46 roles carried neither.
The lint mirrors test_has_spec's shape and uses is_suppressed_at with
mode="line-above", so the marker must sit directly above its flag -- a blank
line between them breaks the association, which is what the persona helper's
reader will look at. A literal =false blocks nothing and is exempt; the
sso-conditional ternary is not, because it blocks in every variant that pins
services.sso.enabled false.
Writing 89 reasons turned out to be the point. Five roles audited in one pass
blocked for five entirely different mechanisms, all of which read as the same
shrug behind a bare =true: pgAdmin's login id is the account e-mail rather than
the Keycloak username; phpMyAdmin authenticates as the MariaDB root account;
GitLab's seeded password belongs to the built-in root, not the administrator;
Mastodon's tootctl accounts create never sets a password; phpLDAPadmin binds
with an LDAP DN and a different secret. Further out: MediaWiki removes the form
that would accept the Keycloak secret ($wgPluggableAuth_EnableLocalLogin=false)
and bounces anonymous requests back into the IdP, so the unauthenticated
landing the contract asserts can never be observed; Mattermost Team Edition has
no native OIDC, so Keycloak rides the GitLab OAuth slot behind a relabelled
link; MinIO's console returns redirectRules: null and renders no SSO button at
all; XWiki's administrator is an XWikiUsers object with no password property.
Two flags turned out to be unjustifiable and are gone rather than annotated.
web-app-mailu gated PERSONA_GUEST_BLOCKED on services.dashboard.enabled, but
runGuestFlow does one anonymous goto and asserts no signed-in marker -- it
never touches the dashboard. A leftover from a helper version that entered
through the dashboard account menu. And web-app-mig's rationale claimed the
role is an in-app module of web-app-dashboard; it has its own compose stack,
healthcheck and canonical domain, and dashboard is merely a tile consumer.
Nine of the reasons say plainly that the block is not a role property but a gap
in the shared persona helpers -- decidim's sign-out is a data-method link the
generic click cannot activate, fider's SSO entry is a two-step modal, taiga,
wordpress, xwiki, odoo and penpot bind OIDC on a path the personas never visit,
jenkins blocks unconditionally although its OIDC variant looks runnable, and
opentalk's own spec proves both personas work. That list did not exist before;
it is the actual yield here, not the 89 comment lines.
45 pre-existing comments came into the comment lint's scope because this sweep
touched their files -- it only inspects files that differ from HEAD. 28 of them
were the banner "Service gate flags rendered for closure.", every one sitting
above a line that is not a service gate flag at all, orphaned by the
alphabetical re-sort these templates once got.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The mcp role sat in the administrator's unscoped roles list. That list carries
no application, and the RBAC builders admit any user whose roles contain the
name, so the single entry granted mcp in every deployed application at once.
It is replaced by application_roles, a mapping of application to roles, which
the user cache now reads and passes through. Both directory builders resolve
membership through one shared function, so the Keycloak and LDAP paths cannot
drift.
The member list never reached its consumer. mcp_group_members yielded username
strings while the Open WebUI reconciler calls member.get("email"), so the first
real grant would have crashed the deploy. It went unnoticed because the
reconciler returns early on an empty list and nobody held the role. The filter
now yields username and email, which is what the reconciler needs to match
accounts Open WebUI created from an OIDC claim, whose display name need not be
the username. A test drives the real filter output through the real script so
the two sides cannot diverge again.
The Playwright scenario walks the boundary end to end: biber is served no MCP
tools, the administrator adds him to the provider's LDAP mcp group in LAM, and
after signing in again the tools are served. LAM is reached only when it is
present in the inventory, so no service edge or variant is added for it.
The administrator spec inverts its last assertion. With the role scoped, an
administrator holds no application's mcp group, and BYPASS_ADMIN_ACCESS_CONTROL
is false, so the tools must not be served to them either.
modes.compose.enabled and modes.swarm.enabled were both false, and a
stack role is forced host=false, so all three test_<mode> columns were
false and the CI discovery filter (test_<mode> == true) dropped the role
from every matrix. It was resolved into the priority whitelist as a new
role and then silently filtered out again — svc-net-tor was only ever
exercised indirectly, as a dependency other roles pull in.
The flag is read solely by the discovery model and the README generator;
nothing in the deploy path consults it, so this has no runtime effect.
meta/categories.yml pins the whole net category to compose/swarm false,
which fits svc-net-wireguard-{core,plain,firewalled}: none of them ships
a compose template. svc-net-tor does, and is the only role in the
category that does, so it takes the category-modes exemption the lint
provides rather than bending the category default for three host roles.
Regenerated READMEs follow: svc-net-tor now carries its mode symbols in
every cosmos diagram that names it as a dependency.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repo's async convention is fire and forget: ASYNC_POLL is 0 whenever
ASYNC_ENABLED is true, and the swarm test lane forces it true through
extra vars. A task dispatched that way returns a stub, so a changed_when
or failed_when reading its own register evaluates against nothing and can
never report changed and never fail. Several tasks carried a
"not ASYNC_ENABLED and" prefix in front of those conditions, added to
avoid touching undefined rc and stdout, which turned the guard off
entirely rather than adapting it.
That is how the nextcloud plugin configuration stayed broken unseen: the
batch reported ok after 0.8s while its delegated commands kept running,
the next task read the half written configuration, and no failure in any
of them could surface.
Every async in the repo was decided individually rather than swept. Where
nothing consumes the result, the async stays and a one line rationale now
records what does not read it. Where a later task, a handler or a browser
does consume it, either the async goes or the result is collected.
Collecting is what the new reaper does. It waits for a whole batch, then
reports every broken job at once instead of aborting on the first, so one
run shows whether a single item or the whole set is broken. Deciding what
counts as broken lives in a filter plugin, because the distinction is
subtle enough to deserve tests: a module reports failure in msg, a command
in rc and stdout, an idempotent API answers "already exists" and means
success, and a job that cannot be read at all comes back marked finished
with no outcome, which would otherwise read as green.
Timing cost is roughly one to two minutes on a deploy measured in hours.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An untyped inspect searches every docker object kind, and a role's overlay network carries the same entity name as its main container, so the call can resolve the network and report a missing State key instead of naming the container. That is how the openresty assert died on every swarm job of run 30709070050.
Extend container-exec-resolver to cover inspect, stripping raw markers and quoted spans first so the flag walker survives Go template format strings. Add container-inspect-type, scoped to roles/ because the collision follows the entity naming convention and the outer act namespace carries no role networks. Sweep 40 call sites; suppress the delegate_to-resolved CIDs and the compose co-location probes with a stated reason.
Move the five container rules into tests/lint/ansible/shell/container/ and drop the container_ prefix from their filenames. Register both markers in the suppression docs, where the resolver marker was missing entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Twelve conflicts, resolved by intent rather than by side.
Six shared one shape: the incoming side adds the new rescue_index.sh call but
drops this branch's `# nocheck: shell-or-true` markers. The markers carry a
lint contract here, so the incoming structure was taken and the markers put
back — dedicated.sh, guide/one.sh, guide/host_deploy.sh, swarm/routine/00_one.sh
and trigger_units.sh, where the incoming version additionally writes the full
journal to a file instead of only echoing a tail.
web-app-matrix keeps HEAD's _synapse_extra_args accumulator (the superset: CA
mount plus the onion case) but adopts the incoming improvement inside it — the
inlined regex_replace gives way to lookup('subnet_gateway').
handlers/__init__.py drops rescue_diagnostics_dir from the old import group
(the incoming side moved it under .infinito.rescue, keeping both would import
it twice) and keeps tor_socks_port, a HEAD-only handler ORDERED_HANDLERS still
needs.
svc-bkp-volume-2-local takes the incoming side throughout: both branches landed
the same feature, but swarm-nfs evolved it — the knob moved from the global
STORAGE_BACKUP_SNAPSHOT_MODE to services.volume-2-local.snapshot_mode, plus the
subvolume helper, 04_snapshot_subject.yml, meta/packages.yml and baudolo 3.2.2.
Three files that left behind were removed: files/baudolo_snapshot.py and the two
worktree tests at their pre-refactor paths, all byte-identical to their moved
counterparts.
Six follow-on failures the merge produced, fixed at the root:
- STORAGE_BACKUP_SNAPSHOT_MODE lost its last reader with the config-lookup
design and is gone from group_vars.
- utils/github/variant_bundle_limits.py sat next to variant_bundles.py, which
the prefix lint forbids; the pair now lives in utils/github/variant/ as
bundles.py and bundle_limits.py, with the -m entry point, the ruff
per-file-ignore, the design-doc link and the mirroring test path updated.
- The incoming WordPress addon spec navigated with a raw page.goto and carried
three bare timeout literals; both route through gotoOnion and resolveTimeout
now, which is what makes navigation survive onion transport.
- Six unmarked `|| true` got a concrete reason each: docker.socket is absent on
distros that ship docker without it, the loop device is already detached when
the probe failed before attaching, and modprobe zfs may fail because the
/dev/zfs check decides.
make quality green across all four targets. The cold-domains budget test that
had to be waived here passes on its own now — 19d56f07e raises it to 100s.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The healthy replica threw `StorageUnavailableException: LDAP connection has been
closed` while the other two were down for an unrelated reason. slapd sets no
idletimeout, so it never closes a connection itself; what cuts them is the swarm
path, where an idle flow through a service VIP loses its conntrack entry without
either side seeing a FIN. The pool then hands the dead connection to the next
caller.
Capping the wait was the obvious remedy and is not available: reading the class
files of the pinned keycloak-ldap-federation jar, the provider knows
connectionPooling, connectionTimeout and readTimeout, and passes only
com.sun.jndi.ldap.connect.pool and .pool.protocol to JNDI. There is no
connectionPoolingTimeout - setting one would have parked a dead key in the
component config. Disabling the pool removes the failure class instead of
shortening the wait on a connection that is already gone; with importEnabled the
directory is not on a hot path, so the extra bind per operation costs little.
The flag lives in meta/services.yml and is read without a fallback, so an
inventory can put the pool back and a missing key fails the deploy rather than
resolving to a guess.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm branch turns on clustered Infinispan but never constrains the JGroups
transport, so Keycloak falls back to SITE_LOCAL. Every keycloak task is attached
to four 192.168.x overlays plus the node-local docker_gwbridge, all of them
site-local, so SITE_LOCAL cannot disambiguate and each task publishes whichever
interface enumerates first.
In run 30586213932 two of the three replicas published a 172.18.x gwbridge
address into the jdbc-ping table. gwbridge is per-node and unreachable from a
peer, so those JOINs timed out: ISPN000196 "failed to recover cluster state after
the current node became the coordinator", 25 Infinispan TimeoutExceptions, 17 of
them through StateTransferLockImpl. The cluster split, one replica froze on an
old topology and answered HTTP 500 on the auth endpoint, and the Playwright
sign-out assertion saw "We are sorry... An internal server error has occurred"
instead of the login form. Eight tests failed across three roles, because
web-app-dashboard and web-app-prometheus authenticate against the same Keycloak.
The matcher comes from the subnet_address_regex lookup, so the role's declared
192.168.24.0/24 yields match-address:192\.168\.24\.\d+ - its own overlay address
matches, gwbridge and the sibling overlays do not. The line sits inside the
existing swarm guard, so compose mode is untouched.
This complements 72e275d41 rather than replacing it: that one makes a broken
replica detectable and evicts it from the VIP, this one keeps it from breaking.
Not verified by me: that Keycloak 26.7.0 spells the option
KC_CACHE_EMBEDDED_NETWORK_BIND_ADDRESS and evaluates match-address as a Java
regex. That is read from the upstream documentation, not executed, and it is the
one assumption this change rests on. If it is wrong the container fails to start,
loudly, on the first swarm run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The tcp healthcheck ends its probe with `cat <&3 | grep -q 'HTTP/1'`, which
matches the status LINE. A backend answering HTTP/1.1 500 therefore passes the
probe exactly like a 200. Swarm's only mechanism for taking a task out of the
VIP is the healthcheck, so a Keycloak replica serving 500s stays in rotation
indefinitely, and with replicas: 3 it catches roughly every third request.
That is what failed run 30586213932's keycloak job. The Playwright artifact's
error context shows the sign-out assertion waiting on the login form while the
page rendered "We are sorry... An internal server error has occurred" on
/realms/master/protocol/openid-connect/auth, and the rescue artifact's container
log shows why the replica was broken: ISPN000196, failed to recover cluster
state after becoming coordinator, followed by 25 Infinispan TimeoutExceptions,
17 of them through StateTransferLockImpl. Eight tests failed across three roles
because web-app-dashboard and web-app-prometheus authenticate against the same
Keycloak.
The status grading lands in a new template rather than in tcp.yml.j2, because
that one has fifteen other consumers whose probe paths were not audited. They
keep the behaviour they have; only Keycloak moves. The new probe reads the
status line with head -n1 and accepts 2xx/3xx only.
Keycloak also moves from health/live to health/ready: liveness answers "the JVM
is up", which a broken replica satisfies, while readiness answers "requests can
be served", which is the distinction the VIP needs.
Readiness returns 503 through schema migration, so the strict probe needs a
start period that outlives it. Measured across all five replica logs of that
run, Keycloak reaches "started" in 13.0-13.2s including migration, so 180s is
roughly thirteen times the observed boot. The deploy gate that waits for
health=healthy allows 600 retries at 2s, i.e. 1200s, so the longer start period
consumes at most an eighth of its budget.
Tool availability verified in quay.io/keycloak/keycloak:26.7.0 (digest
0f198be292568439d700cdbfb893e69a6009bb43a94a06a945b1d3d506c76b13): bash, cat,
grep and head are all present, so the probe's head -n1 resolves.
This fixes detection, not the clustering fault behind it. Why the Infinispan
cluster fails to form is still unknown; a replica that never clusters is now
pulled from the VIP instead of serving errors, but if no replica clusters there
is nothing healthy to route to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, none resolved by picking a side wholesale:
- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
rc now counts as failure; the previous default(0) let an aborted Playwright
run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
which this branch's lint enforces, and adopted the incoming retry-inside-poll
for the OIDC button plus its networkidle wait.
The declared values were guesses that ran up to 60x the size of the image they belong to: seaweedfs asked for 30GB against a 104MB image, nextcloud for 20GB against 353MB. Because min_storage feeds the storage guard, the runner packer and the sufficient-storage filter, those numbers made hosts look too small and pruned caches that deploys did not need to lose.
Every service that declares min_storage together with image and version now carries the measured compressed transfer size of that image plus 200MB of headroom. Sizes were read from the registries through utils.docker.registry.manifest_transfer_size, the same path the headroom test uses, so the declarations and the gate agree by construction. Values are written in MB throughout, matching the existing 500MB / 100MB / 50MB style.
119 declarations across 89 roles. Range: 206MB (memcached) to 3473MB (ollama); four exceed 1GB. The 84 min_storage entries that name no image are untouched, since there is nothing to measure them against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the raw package modules at 76 call sites with package_install, moves
each role's distribution mapping into its own meta/packages.yml, and turns the
inventory PACKAGES variable from source-keyed literal names into a list of
declared package ids. The lint that forbids the raw modules is enabled here,
because the migration burns it down from 76 offenders to zero.
Removes sys-aur, sys-aur-install and sys-package. The AUR build user, the
passwordless pacman rule and the inventory-driven install are part of the plan
the action plugin executes, so the roles no longer carry them.
Declaring the mappings surfaced breakage the raw modules had hidden: on Debian
the `which` binary comes from debianutils, jrnl is in no Debian or Ubuntu
suite, chromium-browser is absent from Debian stable, JDK 11 is unobtainable on
RedHat, wireguard had no RedHat branch at all, qbittorrent was built from the
AUR although Arch ships it in extra, and desk-docker installed a Docker variant
that conflicts with the one sys-svc-container installs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_task_file_numbering.py: numbered task
files use exactly two leading digits, a numbered *_core.yml must be
00_core.yml, per directory the numbers are unique/consecutive and start
at 01 (00 with a core file), numbered includes appear in ascending
order, and shared helpers live in tasks/utils/ instead of the numbered
chain.
Apply it across every role: rename NN_core.yml to 00_core.yml with a
consecutive down-shift of the siblings, move letter-suffix children into
a subdirectory named after their parent task, relocate helpers included
from more than one place into tasks/utils/, and update every reference
(include_tasks, path_absolute, .ansible-lint-ignore, requirement docs).
Legacy narration comments in the touched files are deleted; the upstream
xwiki.properties reference block opts out via nocheck. The unused
MAILU_IP6_PUBLIC var (only ever named in now-removed comments) is dropped.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Resolves the runner Dockerfile to the lint-standard Acquire::Retries=3
and keeps the collabora TODO wording; hardens the merged tree for the
stricter lints (apt retries in the tor Dockerfile, shell-or-true
markers, comment cleanup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The compose-only guard left swarm sso-proxies resolving auth.<onion>
through the embedded DNS, which races container startup and
intermittently hands back a loopback that dead-ends in the task netns
(token exchange refused). The static extra_hosts entry wins in both
modes.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The provisioned become password doubles as the Keycloak administrator realm password (user-administrator/meta/users.yml) and the guide flow generated it 64-char alphanumeric, so kcadm set-password failed the realm passwordPolicy specialChars(1) on every fresh single-app deploy. Delegate generate_random_password to ValueGenerator.generate_strong_password, the same class-guaranteeing generator every other realm user already gets via the strong_password filter; the realm policy stays strict. Also drop the commented-out Trusted Hosts block from the realm template.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
