`retries: 30` with `delay: 10` only bounded the wait while each
`occ onlyoffice:documentserver --check` failed fast. onlyoffice v10.2.1
gives the health check and the version command 60 s each and the
conversion 120 s, so a document server that accepts connections but
does not answer held the deploy for up to two hours before it failed.
The retries now also stop once a deadline of
`lookup('timeout', 300)` seconds, set right before the check, has
passed. A per-attempt `timeout:` would not bound the total: ansible-core
2.21 leaves the `until` loop on the first timed-out attempt, so only a
limit below onlyoffice's 60 s would end a hanging server's loop.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The role writes onlyoffice's settings with one occ call per key,
DocumentServerUrl first and jwt_secret fourth. Nextcloud loads app
config once per process, so a cron.php that starts inside that batch
runs onlyoffice's EditorsCheck job, registered when the app was
enabled, with the document server URL but without the secret. The
document server answers "Invalid token", EditorsCheck stores it in
settings_error, and onlyoffice stops loading its Files integration;
EditorsCheck itself skips every later run while that error is set.
Seen in run 38455782184 (Nextcloud#4, swarm): cron.php started at
06:50:00 inside the batch that ran from 06:49:59 to 06:50:04, logged
"Invalid token" at 06:50:22, and both Playwright passes found .docx
files offered for download only.
DocumentServerUrl is now written last, so a process that cached the
config before the batch ended sees no URL and EditorsCheck returns
early. After the batch, `occ onlyoffice:documentserver --check` sets
settings_error from a fresh check: a redeploy clears an error latched
earlier, and a document server that stays unreachable fails the
deploy. A unit test keeps the URL last.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The api_key write was guarded by `(nextcloud_openai_litellm_api_key |
default('')) | length > 0`, so an empty credential SKIPPED the task. Nextcloud
kept whatever integration_openai api_key it already held, or none, and the
deploy reported success -- exactly the silently unauthenticated AI surface
requirement 036 forbids.
The three tasks now sit in one block gated on nextcloud_openai_use_litellm,
which also satisfies the repeated-when lint, and the shared consumer contract
runs first. That contract is what turns the empty-key case from a skip into a
failure, and it also asserts the base URL stays inside the deployment, which
this role never checked.
The key lookup drops its '' fourth argument and is strict again. Ordering
matters: the contract fails first with a readable message, so the strict lookup
is never the thing that raises.
Nextcloud was the last of the nine confirmed roles without the shared contract.
The occ call keeps the space form, `--value '{{ ... }}'`, deliberately. A
foreign commit on fork/feature/svc-net-tor (bbf4ca09cc) converts that to
`--value={{ ... | quote }}` because Symfony ArgvInput reads a token beginning
with a dash as an option. Adopting it here was considered and rejected on
measurement: shlex.quote leaves an sk-prefixed value completely unquoted, so
both forms emit an identical argv token and the change is inert at this call
site; git merge-file yields one conflict hunk either way, because the two sides
differ in the variable name rather than the quoting; and ansible's quote filter
maps None to '', which would turn a null credential into `--value=` -- an empty
write that exits 0 and matches changed_when, defeating the very guarantee this
commit adds. The equals form is still worth having where values are minted
externally, notably integration_openproject_provision.yml and
integration_suitecrm_provision.yml, and belongs in its own change covering
every call site rather than the one where the failure cannot fire.
Not verified: this role has not deployed since the change. On the branch where
Open WebUI is present, nextcloud_openai_use_litellm is false and no contract
runs at all; traffic stays in-cluster because Open WebUI is itself
gateway-backed, but criterion 4 is not asserted on that path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The readiness probe 5bd323999 added for the swarm path dialled the role's public
domain on port 80. wait_for opens a plain TCP socket from the controller, and
once 1d9d355bc gave the domain roles onion variant coverage that target became
a .onion authority, which needs Tor's SOCKS resolver and cannot be reached with
a socket. Variant 6 of run 31823845576 therefore spent the full budget on it -
"Timeout when waiting for lab.git.<onion>:80", elapsed: 600 - while the
preceding resolve_host_cid had already converged, so GitLab itself was up the
whole time.
The repository had already settled this. DOCKER_REACH_HOST
(group_vars/all/00_general.yml:47) is 127.0.0.1, swapped for the Docker bridge
gateway when the controller runs in a container, and every one of the other
eleven wait_for probes under roles/ uses it. The nearest neighbour is the same
task shape - same resolve preamble, same retries and delay - reaching its
partner as `host: DOCKER_REACH_HOST` with `ports.local.http`
(integration_openproject_provision.yml:12-17). This one was written from that
template with the host and port substituted, and the substitution is the defect.
Restoring the idiom keeps the readiness gate 5bd323999 wanted instead of
dropping it, so retries stays at 12.
The lint pins the shape, because the divergence is invisible in review: both
forms read as a plausible readiness probe, and the wrong one only fails where
the domain happens to be unroutable. It parses each wait_for block under roles/,
locates the host key by indentation, and requires DOCKER_REACH_HOST unless the
line carries `# nocheck: wait-for-reach-host` naming what the probe proves - the
opt-out a probe needs when it is verifying the ingress rather than a backend.
Checked against the original line, where it reports
integration_gitlab_provision.yml:14, and against the eleven correct probes,
which it accepts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Symfony ArgvInput reads a token beginning with a dash as an option, so passing the secret as a separate token after --value handed occ two tokens and occ reported the option as missing its value. Doorkeeper mints base64url secrets, so roughly one run in 43 produced one starting with a dash and the OpenProject provisioning died with: The "--value" option requires a value.
The equals form is unambiguous: Symfony splits at the first equals sign and takes everything after it as the value. The quote filter replaces the hand-written single quotes and stays shell-safe. Note that shlex leaves a leading-dash value unquoted, so the equals form is what carries the fix, not the quoting.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The integration_discourse settings page rewrites what the user types, and the
rewrite is unconditional. src/components/PersonalSettings.vue:217-224 upstream:
if (this.state.url !== '' && !this.state.url.startsWith('https://')) {
if (this.state.url.startsWith('http://')) {
this.state.url = this.state.url.replace('http://', 'https://')
An onion service carries no certificate and is served over plain http through
Tor, so on such a node the address is rewritten to an endpoint that does not
exist. Run 31658270172, job 94319317649 shows all three steps: the trace's
action log records the spec typing `http://forum.<onion>/`, the PUT to
/apps/integration_discourse/config carries `{"values":{"url":"https://forum.
<onion>/"}}`, and the connect navigation to `https://forum.<onion>//user-api-
key/new` fails with network status -1, which Playwright reports as
`Received string: "chrome-error://chromewebdata/"`.
The app ships compiled JS, so the fix has to reach the bundle rather than the
source. It is measured against the file the appstore actually installs --
nextcloud-releases/integration_discourse v3.3.0, whose
integration_discourse-personalSettings.mjs is the exact filename the trace
loaded from the container; the nextcloud/integration_discourse releases stop at
v2.0.1 in 2023 and still ship .js, which is why the earlier reading of that repo
was misleading.
Verified rather than argued:
- the anchor `startsWith("https://")&&(` occurs exactly once in that file
- after the patch the guard reads
`…startsWith("https://")&&!this.state.url.includes(".onion")&&(…`
and `node --check` accepts the bundle as valid ESM
- the anchor no longer exists afterwards, so a second application is a no-op:
idempotency is structural, not a marker check
- the task was run through a real ansible-playbook against a real copy of the
bundle with only the exec prefixes stubbed: first run changed=1 and patched,
second run changed=0, so changed_when reports honestly
- busybox sed (alpine:3, "This is not GNU sed version 4.0") produces the same
result as GNU sed, which matters because the image is 34-fpm-alpine
`occ app:getpath` locates the app instead of hardcoding custom_apps, which
appears nowhere in the repo and which the appstore may move. The `grep -q … ||
exit 0` makes the task both idempotent and safe when the file is absent.
Clearnet is untouched: the added condition is false for every non-onion URL, so
the upgrade still happens where TLS is available.
Not verified: what `app:getpath` returns in this installation. If the path is
wrong the task no-ops silently -- the same shape as the WordPress healthcheck,
where msmtp fails and a passing curl hides it. A `test -f` with a loud failure
would close that, and the layer behind the redirect has never been reached by a
green run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The stdin: value concatenating doorkeeper_allow_onion_redirect.rb with
ensure_nextcloud_storage.rb ran past the line budget; it now wraps before the
closing braces.
Whitespace only. yaml.safe_load of the file before and after the change returns
the same document, so the Ruby that reaches the container is byte-identical.
Not authored as part of the surrounding session work; committed separately so it
does not ride along inside an unrelated bundle.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
files/ now mirrors the role's own structure: php, ruby and shell, each split
into addons for the per-integration scripts that tasks/addons/ drives and utils
for what more than one caller shares. set_encrypted_appconfig.php serves gitlab,
mattermost and zammad; doorkeeper_allow_onion_redirect.rb serves zammad and
openproject; init-gate.sh belongs to the role itself. The partner directories
survive because two scripts are both named provision_oauth_app.rb.
NEXTCLOUD_INIT_GATE_FILE had to split. It was the source path under files/, the
name Dockerfile.j2 COPYs into /usr/local/bin, and the ENTRYPOINT, all at once;
moving the script would have produced /usr/local/bin/shell/utils/init-gate.sh.
The basename keeps that name and NEXTCLOUD_INIT_GATE_SRC carries the path.
The same commit closes two of the three faces of one defect. resolve_enabled
reports TLS off for an .onion domain, correctly -- the vhost is plaintext and
Tor carries the transport crypto -- so every redirect URI built from
lookup('tls', ...) is http, and Doorkeeper rejects it. Zammad raises out of
app.save! and fails the task at rc 1; OpenProject discards result.errors, keeps
the unsaved model whose plaintext_secret already exists, reports
oauth_application_created:true, and only dies three tasks later on a null
client_id. The prelude relaxes force_ssl_in_redirect_uri for .onion hosts and
delegates every other host to the previous setting, so clearnet is unchanged at
both partners; the OpenProject script now raises with the real validation
errors.
The third face is PeerTube, whose Keycloak callback is registered http and
requested https, and it stays open: fixing it means changing either PeerTube's
scheme, which is its federation identity, or the registration side.
Evidence: run 31658270172, jobs 94319317307 ("Redirect URI must be an HTTPS/SSL
URI.") and 94319317278 (changed:true, then length on NoneType).
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four readiness probes waited on lookup('domain', <app>) port 80 -- a
publication identity used as a machine address. With svc-net-tor deployed
that name is onion-only, because exclusive: true replaces the clearnet
domain rather than adding to it (utils/cache/domains.py).
Nextcloud 5 then burned its full 600s on zammad.helpdesk.<onion>:80. The task
carries no delegate_to but inherits apply: delegate_to: NEXTCLOUD_HOST_NODE
from 02_manager_ops.yml:46-50, which propagates through nested includes, so it
ran on a worker. svc-net-tor is placement: manager, so that worker had no
.onion split-DNS, no TransPort redirect and no :80 listener. wait_for swallows
socket.gaierror in a bare except and retries, so an unresolvable name is
indistinguishable from a slow service -- it just burns the timeout.
Point the probes at DOCKER_REACH_HOST and the service's own
ports.local.http instead. No name resolution, so the onion replacement cannot
reach it; node-local, so the inherited delegation stops mattering; and it hits
the service's own listener rather than the shared front proxy, which is what
the probe was supposed to measure in the first place.
Verified on the failing run's artifacts that this is node-independent in
swarm: zammad_railsserver.1 ran only on mgr-01, yet dockerd listens on *:8077
on all three nodes -- the ingress mesh. Only svc-prx-openresty and svc-net-tor
publish with mode: host; every other role, these four included, goes through
ingress.
Same swap for openproject, suitecrm and wordpress->discourse, which had the
identical construct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The gate probed the partner with container inspect on a bare name, which only resolves under compose co-location. In swarm GitLab sits on another node, the probe fails, and failed_when: false turns that into a silent skip: the deploy reports success while client_id/client_secret are never written, so the per-user Connect button leads nowhere. Nothing else writes those two values.
Mirrors integration_zammad: resolve node and container id first, wait for the partner endpoint, then run both execs against the resolved ids. Gate on IS_STACK_HOST like every other addon. The include had to become relative, or the stack-host lint cannot see the gate through a templated path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both waits are needed and check different things: wait_converged inside
resolve_host_cid asks swarm whether the task reached running, the wait_for asks
whether the service answers. Running the endpoint wait first meant a partner
that is not deployed at all paid both budgets in sequence - up to 600s of HTTP
timeout followed by 120x5s of convergence - before the play gave up.
Ordering the cheap service-level check first halves that without giving up
either guarantee. The resolver needs no HTTP: it runs container service ps,
container ps and set_fact, and its two outputs are first consumed by the rails
call further down, i.e. after both waits.
Task order only; nothing added, removed or reparameterised. 8b2cbcb04 applies
the same ordering to the Zammad twin, where the wait was newly introduced.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resolve_host_cid's convergence wait is swarm-only (wait_converged.yml:9), so in
compose mode nothing made sure Zammad answers before the rails provisioning ran
- the only budget there was the provisioning task's own 12x10s against a rails
server whose cold start can exceed it. integration_openproject_provision.yml
already guards its partner this way; this restores the symmetry.
The wait sits after the resolver rather than before it. Both waits are needed
and check different things - wait_converged asks swarm whether the task reached
running, the wait_for asks whether the service answers - but ordering the cheap
service-level check first means a partner that is not deployed at all fails
after one budget instead of two. The resolver itself needs no HTTP: it only runs
container service ps / container ps and set_fact.
Not executed: that lookup('domain', 'web-app-zammad') resolves to a name
answering on port 80 is taken from the openproject twin, not measured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The marker claimed to suppress container-exec-resolver, but that rule does not
fire on this task. Its extractor reads the first argument after `container exec`
and finds `-i`, so it returns no target; the container id is four lines further
down. The position was wrong too: the rule accepts a marker on the same or the
immediately preceding non-empty line, and three lines separated the two.
That leaves a suppression marker carrying a rationale, which is the shape the
comment policy exists to prevent. The reason it stated - that container_address
would expand to a manager-only subshell - is already in 04b8cfead's message.
tests/lint/ansible/shell/container/ stays green without it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
02_manager_ops.yml includes the plugin chain with `apply: delegate_to:
NEXTCLOUD_HOST_NODE`, so the addon hook runs wherever that node happens to be.
On a swarm worker the partner probe could not answer: it used
NEXTCLOUD_ZAMMAD_CONTAINER_ADDRESS, which expands to a
`resolve-container-id zammad railsserver` subshell, and that helper exits 64 off
a manager. The substitution came back empty, `container inspect ""` failed,
`failed_when: false` swallowed it, and the gate
`(nextcloud_zammad_partner_running.stdout | trim) == 'true'` was false.
The OAuth provisioning was therefore skipped, client_id and client_secret stayed
empty, and integration_zammad's personal view fell back to its access-token UI -
so the 'Connect to Zammad' control the spec waits for never existed. That is
what failed web-app-nextcloud 5 in run 30781723102.
The probe and its clause are gone; provisioning now resolves the railsserver's
hosting node and local container id through the standard
sys-svc-compose/utils/swarm/resolve_host_cid.yml and delegates the rails call
there. resolve_host_cid pins its own delegate_to on STACK_HOST and
<prefix>_HOST_NODE, so it is unaffected by the inherited one, and the trailing
occ tasks keep no delegate_to and still run on the Nextcloud node.
Losing the probe also loses its quiet skip. The addon is still gated on
services.zammad.enabled, i.e. 'web-app-zammad' in group_names
(meta/addons/integration_zammad.yml:2), so the exposure is a targeted deploy
where the partner is in the inventory but excluded from the run: wait_converged
then spends 120x5s before failing instead of skipping.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
main.yml now reads as the chronological sequence it describes: it includes
01_fullstack.yml, then the single IS_STACK_HOST-gated 02_manager_ops.yml,
whose substeps live in the matching 02_manager_ops/ directory numbered in
execution order. The plugin pipeline moves under its parent step so its
include resolves against its own directory instead of an inherited search
root.
The flush that deploys the stack moves out of the config step to the top of
02_manager_ops.yml. It previously ran after the swarm resolver, which then
waited 600 times for a service that did not exist yet and killed two CI jobs
with "no such service: nextcloud_nextcloud". The second flush and the
re-resolve behind it are removed: neither included file notifies a handler,
so they could never fire.
A new lint pins what made this refactor risky. A relative include is resolved
at runtime, so a typo or a moved file survives ansible-lint and
--syntax-check and only fails mid-deploy; worse, a bare name that matches
under two search roots silently shadows one file with the other.
Comments that only narrated code are dropped from the addon hooks and the
plugin install step.
NEXTCLOUD_DOCKER_EXEC resolved the container id through a subshell evaluated on the manager. When the manager holds no running replica, resolve_container_id.sh falls back to a cluster-wide lookup with no node filter and returns the id of a task on another node, which the local daemon does not know: 'No such container', sixty times, then the wait in 02_config.yml fails the deploy.
08_manager_ops.yml now resolves the hosting node and its node-local container id once via resolve_host_cid.yml and delegates every include there with apply:. Redefining NEXTCLOUD_DOCKER_EXEC in terms of NEXTCLOUD_LOCAL_CID lets all seventeen consumers inherit the fix without touching a task line. It re-resolves after the handler flush and after the upgrade, the two points where a rolling restart can move the task. Compose degenerates to delegate-to-self with the previous container name.
Four call sites built their exec inline because they need -u root or -e flags, and their subshell would have run on the worker where resolve-container-id exits 64 for want of a manager. Without converting them, delegation would have made things worse rather than better.
This is the outermost of four stacked defects and not the reason the job was red: the manager was empty because the object store had no free volume slots, which left a half-installed instance holding the init lock. Those layers are fixed separately. This one is a genuine cross-node bug that the rest of the repo already solved with the same pattern.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
07_admin_identities.yml registered 'occ ldap:search <admin>' with async/poll, and ASYNC_POLL is 0 whenever ASYNC_ENABLED is true. The register then holds an async job handle with no stdout, and thirteen lines later the set_fact parses exactly that stdout for the '(identity)' pattern. The list therefore collapsed to [NEXTCLOUD_ADMINISTRATOR_USERNAME] on every swarm run, the LDAP-mapped administrator_<n> never entered the admin group, and failed_when could not complain because it is keyed on the same global flag.
The symptom is a 403 on every admin-scoped route while user routes return 200. The Playwright trace of one session carries data-user=administrator_2981 and _oc_isadmin=false on BOTH - and _oc_isadmin comes straight from IGroupManager::isAdmin(), so it states the group membership rather than the route's verdict. The compose lane stays green only because it runs a synchronous pass first, where the same task does have stdout and adds administrator_3447; its later async pass degenerates identically.
Three tasks become synchronous: the identity resolution, the group:adduser loop that writes the membership, and the LDAP cache warm in addons/user_ldap.yml whose task name promises the users are present. Measured cost 2.533 s + 0.730 s against a 2:30:13 job. The 'not ASYNC_ENABLED and' guards stay: they hang on the global flag, not on the task being async, so removing async does not arm them.
The condition moves into NEXTCLOUD_USER_LDAP_ENABLED and the two tasks sharing it into one block.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An untyped inspect searches every docker object kind, and a role's overlay network carries the same entity name as its main container, so the call can resolve the network and report a missing State key instead of naming the container. That is how the openresty assert died on every swarm job of run 30709070050.
Extend container-exec-resolver to cover inspect, stripping raw markers and quoted spans first so the flag walker survives Go template format strings. Add container-inspect-type, scoped to roles/ because the collision follows the entity naming convention and the outer act namespace carries no role networks. Sweep 40 call sites; suppress the delegate_to-resolved CIDs and the compose co-location probes with a stated reason.
Move the five container rules into tests/lint/ansible/shell/container/ and drop the container_ prefix from their filenames. Register both markers in the suppression docs, where the resolver marker was missing entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud reached its OpenAI-compatible backend through Flowise, which
only forwarded to the LiteLLM sidecar that Flowise no longer runs. The
consumer flag, the activation condition and the endpoint now bind to
svc-ai-litellm directly, one hop shorter.
The endpoint switches from Flowise' public TLS URL to the internal
gateway address plus /v1, so a container-to-container call no longer
travels through the reverse proxy.
The api_key stops crossing a role boundary: it was read from
web-app-flowise credentials.llm_master_key, handing Nextcloud
administrative access to the whole gateway, and is now this role's own
per-consumer virtual key.
The litellm variant pin sits on variant 7 rather than 3 because adding
the gateway to variant 3 pushed that round past the matrix memory budget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
In swarm mode NEXTCLOUD_CONTAINER_ADDRESS is a subshell expression that resolves the live container id, but the additional-config loader task called container exec via command/argv, which passes the expression literally and fails with 'No such container'. Run it through the shell like every occ task does, and convert the encrypted-appconfig task that carried the same latent argv pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
On the DinD swarm nodes glibc resolves localhost to ::1 first and the IPv6 connect to the ingress-published port hangs until timeout, so the mattermost HTTP gate burned 30 attempts while 127.0.0.1 through the ingress DNAT answers fine (measured on an identically shaped node). Point nextcloud_mm_local_url at DOCKER_REACH_HOST, the repo's established reach pattern (gitea, shopware, nextcloud's own config wait), which the DinD nodes override to the bridge gateway; compose keeps 127.0.0.1.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Every readiness signal before the OAuth provisioning (mmctl local mode, the container healthcheck) probes Mattermost's local-mode unix socket, never the HTTP listener, and the provisioning curl was the deploy's first request against :8065, one-shot and unretried. On swarm the ingress-published port can transiently swallow that single request after a concurrent stack deploy reprograms the mesh (observed as curl 28, 30s, 0 bytes). Wait for /api/v4/system/ping on the delegate node before provisioning and retry the provisioning itself.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The db:migrate and settings tasks, and the nextcloud to openproject OAuth
provisioning, ran container exec into the openproject web container from the
manager. In swarm the web replica is scheduled on a worker, so the
manager-resolved container id is not local and the exec fails. Resolve the
web service's hosting node and node-local container id via resolve_host_cid
and delegate every exec to that node, matching the pattern the role's own
03_admin.yml and the mattermost integration already use. The nextcloud
running probe is dropped in favour of the convergence wait.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The Mattermost <-> Nextcloud OAuth provisioning ran its container probe and
mmctl steps on the manager, but in swarm the mattermost task runs on a
worker, so the probe found no container and the whole coupling was skipped.
The connect button then stayed disabled and the integration e2e timed out.
Resolve the mattermost hosting node and node-local container id via
resolve_host_cid and delegate every mmctl step, and the OAuth-app
provisioning, to that node, mirroring the pattern web-app-mattermost's own
role uses. The OAuth-app step keeps running bash on the host because the
image has no shell, but now on the mattermost node so its localhost URL
resolves. Drop the now-unused running probe and the
NEXTCLOUD_MATTERMOST_CONTAINER_ADDRESS var.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The provision include used a templated path, which the container-exec lint cannot follow, so its argv-form 'container exec' into the OpenProject container read as ungated and would fire on swarm workers (resolver exits 64). Switch to a static include the lint follows and carry IS_STACK_HOST on the include when, so workers skip the whole coupling as a unit. Also picks up the autoformat reflow of the diagnostics daemon-journal capture.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The mattermost and openproject integration probes plus the openproject provisioning execs addressed the partner by its bare compose container name. Under swarm the running task name differs, so container inspect found nothing, the partner-running probe returned false, and the bidirectional OAuth provisioning was skipped. Route them through the container_address lookup (bare name in compose, resolve-container-id subshell in swarm) so the probe resolves and provisioning fires in both deploy modes.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Reverts b9a5aa816. It rewrote the Mattermost OAuth provisioning to
'container exec ... bash -s', but the mattermost-team-edition:11.9.0 image
has no shell at all (verified: 'stat /bin/sh: no such file'), so the task
fails rc127 in BOTH swarm and compose (run 29715516546, web-app-nextcloud 3,
jobs 88269803451 + 88270969917). Restores the previous provisioning form.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The partner probe and both provisioning execs addressed the bare
compose container name zammad-railsserver, which does not resolve in
swarm; the probe silently skipped the whole coupling and the connect
control never rendered. Route them through the container_address
resolver with the shell module (unquoted, subshell expansion), gate the
provision include on IS_STACK_HOST, and retry the rails provision to
absorb Zammad convergence lag. Compose behavior is unchanged: the
lookup returns the bare name there.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The container_address lookup emits $(...) shell syntax in swarm and
documents that callers must use the shell module; the 22 occ tasks
still used command:, so the resolution reached docker literally and
every plugin/LDAP/integration task died on all swarm variants. Convert
them to shell: (args stay single-quoted or [A-Za-z0-9] slugs, so
compose semantics are unchanged).
NFS mounts use local_lock=flock, which makes the stock entrypoint's
init flock node-local: concurrent replicas each won "exclusive" locks
and shredded the rsync install (occ 255, converge livelock). A small
init gate now fronts the entrypoint: only swarm task slot 1 runs the
init, other slots wait for version parity plus config.php; without
TASK_SLOT (compose) it degenerates to a plain exec. The gate ships as
files/nextcloud-init-gate.sh, copied into the build context on the
stack host and referenced through the NEXTCLOUD_INIT_GATE_FILE SPOT.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Deploy GitLab from the official CNG CE images as six services: webservice (puma) and sidekiq scale to the topology default, workhorse is the sole HTTP entry, gitaly and a one-shot migrations job are pinned singletons, shell runs gitlab-sshd. The Omnibus container, its Dockerfile and the GITLAB_OMNIBUS_* config groups are removed. All rails/workhorse/shell/gitaly configs and shared secrets are pre-rendered by Ansible and mounted read-only; eleven credentials land in meta/schema.yml including an rsa_pem_2048 OIDC signing key (new value_generator algorithm). Postgres, redis and S3 object storage ride the existing lookups; OIDC, SMTP, the public SSH port and the root password are preserved. compose_volumes gains a per-volume nfs:false opt-out so the gitaly repositories volume stays node-local in swarm (upstream forbids git repositories on NFS); nfs_prep skips volumes without driver_opts accordingly. The Nextcloud gitlab integration execs the CNG rails runner. ruby_single_line dies with its only consumer; ruby_dq keeps test coverage. Ports (8181/8080/2222/8075/3808/9122) and service names resolve from single SPOT declarations in meta/services.yml via vars constants; the version pin sits on the webservice image pair so the updater can bump it. Live-validated in compose up to all six services healthy and migrations completed; fresh-install-only semantics for existing Omnibus data are documented.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The container-exec-resolver lint only saw the inline shell form, so the
`command: argv: [container, exec, <name>]` list form (two separate lines)
slipped through, which is how checkmk's bare name reached swarm. Extend the
scanner to parse argv sequences via yaml.compose_all with node line numbers,
and annotate the nine legitimate bare-name execs (mariadb recovery container,
matomo compose-only debug, probe-gated nextcloud partner provisioning, pixelfed
runtime-resolved CID, wordpress launcher-managed discourse) with
# nocheck: container-exec-resolver.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a 'timeout' lookup plugin that returns round(base * TIMEOUT_FACTOR)
(group_vars default 1, so value-identical to the old literals). One knob
in group_vars stretches every wait in the stack for a slow uplink:
TIMEOUT_FACTOR: 2 doubles them all.
- plugins/lookup/timeout.py + unit tests (identity, factor scaling,
string coercion, error paths).
- group_vars/all/17_resource.yml: TIMEOUT_FACTOR: 1.
- Convert all 34 literal 'timeout:' task keywords / module args across 25
task+handler files to "{{ lookup('timeout', <base>) | int }}".
- tests/lint/repository/yaml/test_timeout_uses_lookup.py enforces the
lookup in every task/handler timeout (opt-out: # nocheck: timeout-lookup).
- swarm.yml pre-deploy build handler: add the missing timeout (same
network-build hang vector as compose.yml's Build compose).
- Drop 13 pre-existing narration comments in the touched files (comment
lint enforces whole-file on any edited file).
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Conflicts resolved:
- .claude/.gitignore: union of ignore patterns
- plugins/lookup/rbac_group_path.py: drop both unused conflicting imports
(body fetches apps via lookup_loader)
- roles/sys-svc-compose/handlers/main.yml: keep HEAD's refactor (compose-up
handlers live in compose.yml, included per DEPLOYMENT_MODE)
- utils/cache/base.py: keep HEAD's merge_with_defaults/FILE_TOKENS imports
Post-merge fixes for main's newly-enabled lints:
- pyproject.toml: grandfather 5 files under the BLE001 ratchet
- justify 6 pre-existing security # noqa markers (S108/S311/S701)
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
odoo: sale_management.spec.js asserted `breadcrumb.or(appTitle)` without
`.first()`; when the Sales module installs correctly BOTH the menu brand
and the breadcrumb render, so `.or()` resolved to 2 elements and
`toBeVisible()` raised a strict-mode violation — failing the success case.
Add `.first()` (matching the same file's createQuotation locator).
nextcloud: the GitLab OAuth provisioning hard-`raise`d when no Organization
existed yet (default org is seeded asynchronously). Resolve/create it
defensively and add `retries/until` so a transient not-yet-seeded window no
longer fails the whole deploy.
jenkins: the deploy-lockdown service stop had no retry; under load systemd
intermittently refused with "Transaction is destructive". Add `retries/until`
so the transient contention is ridden out.
moodle: the shared `compose up` handler had no retry; a transient missing
external network (central DB stack still creating it) failed the deploy. Add
`retries/until`, matching the handler's existing compose-pull pattern.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_name_emoji_and_length.py: every Ansible task
name and GitHub Actions workflow/job/step name must start with an emoji (first
char not 0-9/A-Za-z) and be <= 80 chars.
- emit emoji names at the source: cli/build/include generator now writes
"🔧 setup <app>" / "🚿 flush handlers after <app>" so generated
tasks/groups/*-roles.yml satisfy the convention too
- convert existing task/step/job/workflow names to emoji-prefixed, choosing a
task-fitting icon per action (📦 install, ✅ ensure, 🔍 probe, 🥾 bootstrap,
🚀 deploy, 🧩 service-module load, ⬇️ checkout, 📂 include-routines, ...)
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolve 69 conflicts across ~70 files: union main's image-version bumps and new service fields (bond, resource limits) with the swarm branch's default_placement / NFS / IS_STACK_HOST config; retire the three per-deploy-type test workflows main still edited (consolidated into compose+swarm); adopt main's run-once 01_core dispatchers while re-applying each role's swarm guard. Fix the post-merge lint surfaced by main's stricter rules: bond + resource-budget keys, container-exec stack-host gating + resolver routing, leaky/dead proxy vars, suitecrm once-flag, xwiki variant nocheck, nextcloud oversized-block to gated include, matrix-mdad capped at 3g. make autoformat clean, make test green.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
integration_mattermost reads its OAuth client_id/client_secret via IAppConfig getAppValueString() (plain, relying on the sensitive-value auto-decrypt) - only the per-user access token is decrypted via ICrypto. Like integration_zammad, the shared set_encrypted_appconfig.php stored these ICrypto-encrypted (default mode), so mattermost read back the raw ciphertext as its client_id and the personal connect could not build a valid /oauth/authorize request. Pass NC_ENC_MODE=sensitive so getAppValueString() returns the plaintext client_id (gitlab keeps the icrypto default - it decrypts explicitly).
This latent break was never caught in CI because the mattermost variant (v2) was blocked by the v0 files_linkeditor failure and never ran.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
integration_zammad reads its OAuth client_id via IAppConfig::getValueString() (plain, relying on the sensitive-value auto-decrypt), unlike integration_gitlab/mattermost which decrypt explicitly via ICrypto getClearAppValue(). The shared set_encrypted_appconfig.php stored every value ICrypto-encrypted, so zammad read back the raw ciphertext as its client_id and the personal connect could not build the /oauth/authorize URL. Add an NC_ENC_MODE switch (default icrypto, unchanged for gitlab/mattermost) and have the zammad provisioning pass sensitive so getValueString() returns the plaintext client_id.
Verified in-container: sensitive round-trips to plaintext, icrypto still yields a pipe-separated ciphertext, and the live Zammad Doorkeeper app uid matches the stored client_id. This also restores idempotency: the ICrypto ciphertext contained a pipe, tripping the re-provision guard on every deploy; the sensitive plaintext does not.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The gitlab/openproject/suitecrm/zammad integration tasks gated their partner-running check on 'container exec <partner> true' (rc==0), which returns rc 127 on partner images lacking a true binary, wrongly skipping the integration. Switch to 'container inspect -f {{.State.Running}}' gated on stdout-trimmed == 'true' — the same probe already shipped for the mattermost integration in 8023e4b8b. inspect never enters the container, so the in-container PATH lookup that broke the old probe is irrelevant.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The nextcloud->mattermost OAuth provisioning failed because the partner-running probe used 'container exec <mm> true' (rc 127 — no true binary in the mattermost image), which gated provisioning off and left the OAuth app unprovisioned (spec failed on oauth_instance_url=null). Replace it with 'container inspect -f {{.State.Running}}' gated on stdout-trimmed == true, provision the OAuth app via a controller-run script using a short-lived admin PAT, and rewrite the spec as a behavioral connect-redirect check.
Mattermost personal access tokens are now enabled only when the consumer web-app-nextcloud is deployed (services.mattermost.user_access_token_enabled renders from 'web-app-nextcloud' in group_names) instead of being hardcoded on.
Verified: probe live in the DiD stack (inspect running -> rc0, missing -> graceful skip); the user-access-token chain renders True/False via a real Templar (nextcloud present -> True, absent -> False); service-flag integration tests pass.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The integration_zammad Connect flow reached the partner /oauth/authorize with a malformed client_id, so Zammad's Doorkeeper rejected the unknown client UID and 302-redirected to its homepage (the spec then captured the partner root, not the authorize page). The encrypted-appconfig helper stored the credentials via setValueString(..., lazy: true) WITHOUT sensitive: true, which mangled the client_id on read. Store it sensitive so the provisioned client_id round-trips intact and the OAuth authorize carries the correct UID.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Nextcloud runs pretty-URLs (front_controller_active), so each integration app emits its oauth-redirect WITHOUT /index.php at runtime. The partner OAuth apps were registered with only the /index.php variant, so exact-match validation rejected the real callback and the per-user connect silently failed while the deploy and Playwright specs went green. Register both variants (the no-/index.php one first). Build the list on the consumer side (Doorkeeper newline-join for gitlab, native JSON callback_urls array for mattermost, Doorkeeper for openproject) to avoid newline-in-argv.
SuiteCRM is unaffected (OAuth2 password grant, no redirect_uri); the Nextcloud->OpenProject client callback is an OpenProject URL and is left untouched. Also drop the Doorkeeper>=5 hashed-secret fallback in the gitlab provisioner so a missing plaintext secret fails loudly.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Provision a Doorkeeper OAuth application on the partner Zammad and write its client_id/client_secret into integration_zammad (Nextcloud-encrypted), so the per-user OAuth connect flow has the credentials the generic SPOT cannot supply. Register both the pretty-URL and /index.php oauth-redirect variants, and drop the hashed-secret fallback so a missing plaintext secret fails loudly instead of persisting an unusable hash.
Strengthen the integration_peertube spec: pin the admin 'instances' value to the exact deployed partner host (new PEERTUBE_BASE_URL) instead of merely 'not Nextcloud', and add a functional OCS search-providers check that fails when the PeerTube provider is not registered. PeerTube has no per-user OAuth (verified upstream: no client_id/oauth_instance_url config, no /oauth/authorize route), so the removed Connect->/oauth/authorize assertion tested a non-existent flow.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
