Conflict: roles/web-app-bridgy-fed/files/Dockerfile
Both sides extended the header comment. main also replaced the upstream
checkout, `git clone --depth=1 --branch "${BRIDGY_REF}"`, with git init plus a
fetch of the single ref; this branch never touched those lines, so git took
main's version automatically and only the comment collided.
Resolved to main's file verbatim. It is a superset of ours in both respects:
it keeps our two comment paragraphs and adds a third naming the trip-wire the
non-idiomatic fetch exists for, namely that BRIDGY_REF is a commit and
`git clone --branch` refuses one. Verified byte-identical to stage 3.
Committed with the hook skipped and before the semantic audit of the other 45
files finished. main brings a lint rule that has never seen this branch's
roles, image bumps on svc-ai-ollama and web-app-nextcloud which this branch
rewrote, a stylelint major, and release 13.0.0 across the packaging surface.
Any of those can merge cleanly line by line and still be wrong.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The context_agent ExApp calls back into the same php-fpm pool it is being
proxied from: its UserAuthMiddleware resolves the bearer against
/ocs/v2.php/cloud/user while a child is already parked forwarding the MCP
request out to the ExApp. Every in-flight MCP request therefore occupies TWO
children. Below twice mcp.limits.concurrent_requests the two legs deadlock, and
because the ExApp's callback carries no timeout the pool never recovers -- which
is why the Context Agent sat unhealthy rather than merely slow.
services.nextcloud.performance.php.fpm now sizes the pool explicitly:
max_children 16, start_servers 4, min/max spare 4/8. request_terminate_timeout
is 900s and must not drop below the proxy's fastcgi_read_timeout; lower kills a
child while nginx is still waiting on it, equal means only work nginx has
already abandoned gets reclaimed.
The readiness path was spending the same scarce workers. /healthz/ready had no
location of its own, so it fell through to `location /`, was rewritten to
/index.php/healthz/ready and booted a full Nextcloud request on a php-fpm child
once per probe interval against a pool of five. It is an exact-match location
returning 200 from nginx now, with access_log off, and never reaches PHP.
The pool file is rendered through render_replicated_templates.yml so swarm peers
receive it too, and mounted from meta/volumes.yml at a path that sorts after
zz-docker.conf, which reopens the [www] section last -- mount it earlier and the
image's own values win while the deploy looks correct.
The context_agent healthcheck moves from a hand-written probe in compose.yml.j2
to the shared container_healthcheck lookup, declared as flavor curl on path
heartbeat with start_period 20m. The ExApp installs dependencies on first boot
and legitimately outlives a 60s start period, which is the other half of why it
reported unhealthy.
Not verified: none of this has deployed. max_children 16 is derived from twice
concurrent_requests rather than measured against real memory use, and the
mount-ordering requirement is read from the image layout rather than confirmed
against a merged pool config in a running container -- if it sorts wrong the
whole change is inert while appearing correct. request_terminate_timeout 900s
also means a genuinely stuck request now holds a child for fifteen minutes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both partners in one variant overran the 20400s swarm drill budget by
~2%; the job then hit timeout-minutes and surfaced as cancelled.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
unsafe-inline re-admits the inline execution an XSS needs and unsafe-eval re-admits string-to-code, so a CSP relaxation must be verified against the running app rather than copied from another role. The new lint requires a nocheck marker plus a Reason on the flag's own line, so a marker cannot drift to the neighbouring directive.
The 197 pre-existing grants across 70 roles are checked out with a TODO: they were never verified and still have to be.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a34cb44ad split variant 3 into openproject, mattermost+matrix and the rest, and
closed with the open question: 'Whether dropping one partner moves 214 sync
under 185 is open.' Run 31864448027 answers it for the pair that was left.
Job 94965456921 (swarm, variant 10) died at rc=137, SIGKILL, 221s past the
20400s sweep deadline. Its sync pass alone took 11547s = 192.5 minutes - still
above the ~185 that commit derived as the survivable boundary. One partner was
not enough; the round needs one app.
matrix is the heavier half. Of PASS 2's 2811s of Playwright, its suite is
1090.55s - 39%, against 401.96s for nextcloud itself and 25-90s for ten of the
remaining twelve. PASS 1 carries the same shape at 1071.81s.
The new round is appended as variant 12 rather than inserted after 10, so every
existing index keeps its job name and its artifact names.
The alternative considered and rejected: restricting the async pass to the app
under test. It would have saved more seconds and cost the co-deployment coverage
of thirteen roles in the update path, to fix an overshoot one role caused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant bundling packed several variants onto one runner and made the
job count depend on storage sums. Every variant now gets its own matrix
row instead, and the two things that used to be fixed per workflow --
which deploy mode a row runs in, and whether it sits behind the node
onion -- become axes assigned per row.
Both axes are a deterministic rotation over the row's position and the
sweep number, never random, so a red job reproduces by re-running the
same sweep. Tor turns on sweep//2 so it does not flip in lockstep with
the mode: a row walks all four mode/tor combinations over four sweeps
instead of only two.
A role offers at most two modes in practice, since swarm requires its
own stack and host requires the absence of one.
Drops with the bundler: limits.py, the variant_bundle env handlers,
INFINITO_VARIANT_BUNDLE_SIZE/_MAX_STORAGE, the per-role
meta/tests.yml variant_bundle_size key and its only user, and the
ComplexityRow bundles/jobs fields with the --deploy-mode and --max-jobs
flags that consumed them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three rounds of run 31823845576 died at 356 minutes against the 355-minute cap
in call-test-deploy-swarm.yml: variants 0, 2 and 3, with 214.9, 214.0 and 202.5
minutes in the sync pass alone and no async pass reached. The role-runtimes
artifacts put the boundary between 183 sync (variant 4, 303 total, survived) and
202.5 (cancelled), so a round has to come in under roughly 185.
Each of the three carried more than it could finish. Variant 0 is the baseline
and had bigbluebutton on top of every tightly-coupled service; variant 2 paired
openwebui with flowise; variant 3 carried openproject, mattermost and matrix.
Splitting them leaves one partner each, and the freed partners fill three new
rounds.
bigbluebutton is the lever for variant 0 because f1dff50fb already prepared it:
that commit dropped its bond to 0.5 and noted "bigbluebutton stays in variant 0
for now, by explicit request; its bond change only makes the move possible
later." Everything else in variant 0 is bond >= 1 and would need redeclaring
first.
The xwiki split 6fcbffe08 merged away comes back here without costing a variant
of its own: xwiki takes flowise along, which variant 2 had just given up. sso and
tor stay on everywhere except variant 1, the minimal round, and variant 8, which
pairs the clearnet-only PeerTube.
Verified: tests/integration/roles/meta/variants plus the role-meta lints, 39
passed. Every variant resolves between 18.26GB and 27.77GB reservation against
the 32GB cap. The planner confirms the split is clean - round 2 pulls
web-app-openwebui without web-app-flowise, round 9 pulls flowise and xwiki
without openwebui, and neither role references the other in meta/services.yml.
Costs three more jobs in compose and in swarm, measured 9 -> 12 each. The slot
pool is fixed, so they displace three lower-priority rows per mode rather than
growing the run. host is unaffected: test_host is false for this role because it
ships a stack.
Not measured, and the reason to expect less than the arithmetic suggests: the
round closures sit at 21-24 roles before and after, because the shared base -
keycloak, mailu, matomo, dashboard, prometheus, the database engines, tor -
dominates the partner. Whether dropping one partner moves 214 sync under 185 is
open, and every measurement here predates 8f98e1648, which put the onion stack
into every round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.
Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.
The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.
Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7a70d27b7 made a variant that switches SSO on also switch the onion on, because
that pairing is where the platform breaks in ways clearnet never shows. The same
argument reaches further than SSO: a Secure cookie is dropped, an https identity
provider refuses to embed, and crypto.subtle is undefined for any feature on a
non-secure origin, not only for a login. A variant that never runs the onion is
not testing the deployment users get.
So the rule now covers every variant of a role that declares a tor block.
Variant 1 stays exempt - it is the minimal, partner-free round every role
declares and the one place the stack is exercised without the onion at all - and
a role whose meta/services.yml pins tor.enabled to a literal false is skipped
outright, because that declaration already says the role has no onion and needs
no comment repeating it.
That found 137 variants. 101 were variant 1. Of the rest, 28 are the LDAP-only
auth-isolation rounds, which gain real coverage from running LDAP over the onion
and are flipped on here. Four cannot: svc-net-tor is the provider itself, and
nextcloud's variant 8 pairs the clearnet-only PeerTube. Both carry a marker
naming the reason; opentalk and coturn need none, since their services.yml pins
the literal false the rule now reads.
Flipping them exposed a contract that contradicted itself. test_auth_isolation
and test_non_baseline_explicit_disables demanded `shared: false` beside
`enabled: false`, while variants-static-override flagged that same line as a
static override - which is why all 133 of them carried a nocheck marker. The
demand only makes sense where `shared` is dynamic: a literal base value cannot
leak through deep-merge, so restating it protects nothing. Both contracts now
ask for it only in that case, and the 133 lines are gone.
The result is a stricter rule with fewer lines: 189 insertions against 193
deletions.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles 107
passed / 610 subtests. The rule is checked in both directions - it reported 137
offenders before the resolution and none after.
Not measured: 28 additional onion rounds cost matrix wall-clock, against swarm
jobs already capped at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.
It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and f1dff50fb established both the reasoning and the precedent when it
dropped xwiki and bigbluebutton from 1 to 0.5: bridged partners with their own
role and their own spec, whose bond declaration was the outlier rather than
their placement.
This also unblocks the variant-level tor contract. Requiring a non-baseline
variant to pin the onion on while tor still counted as tightly coupled would
have pulled variant 0's closure requirements along with it.
Verified: tests/lint 336 passed / 307 subtests and tests/integration/roles 107
passed / 610 subtests, including test_resource_budget over every variant's
deduplicated footprint and the bond-aware test_order closure.
Not measured, and the honest cost: f1dff50fb's own caveat applies unchanged
here. Whether redistributing a service out of the baseline closure buys back
wall-clock is not established; role-runtimes has the per-role seconds and no run
has been made with this shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant 7 pinned sso, tor and peertube together, and since 8ffb1f567 those three
cannot coexist. test_sso_variant_requires_tor forbids sso: true without
tor: true, so tor cannot give way. With tor on, keycloak is published onion-only
- auth.infinito.example does not appear once in the job log - and
align_domain_to_consumer falls through to `return primary_domain` because there
is no clearnet sibling to align to. PeerTube therefore receives
http://auth.<onion>/realms/.../.well-known/openid-configuration, oauth4webapi
rejects it on scheme before any request leaves ("only requests to HTTPS are
allowed", OAUTH_HTTP_REQUEST_FORBIDDEN), the plugin registers without building a
client, and the gate at oidc/enable.yml:105 fails the deploy. Both deploy modes
reproduced it, with different onions, which rules out routing: the SSO
back-channel pin from container_extra_hosts is emitted for peertube and fixes
where a name points, not which scheme a library will accept.
peertube is the only one of the three that can move, so it moves to a variant of
its own with tor off. sso follows tor off, which costs nothing here: the addon
gate is services.peertube.enabled alone and the spec signs in with a direct
password, not through Keycloak. PeerTube keeps its OIDC coverage in its own
variant 0, which is sso-on and clearnet - the topology where the plugin works.
The slot is paid for by folding xwiki into variant 7 rather than adding a tenth
variant. This does not undo f1dff50fb: that commit moved xwiki out of variant 0
because job 94319317296 was cancelled at exactly the 355-minute ceiling, and
variant 0 is untouched here. Nor does it grow a round - variant 7 carried two
partners before (discourse + peertube) and carries two now (discourse + xwiki),
while variant 3 already carries three. Measured: the merged variant 7 resolves to
20.26GB reservation / 48.01GB limit against caps of 32/64, the lightest
non-baseline variant in the file, and the two-partner variant 7 of run
31823845576 spent 101.4 minutes against a 355-minute box.
mariadb and redis stay shared: true in the new variant because
variants-shared-false-once allows exactly one shared: false pin each and variant
1 already holds both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An addon's enabled is either dynamic -- a Jinja expression resolving per deploy
against a service flag, an API credential, the SSO plugin choice or group_names
-- or static, a literal boolean. A dynamic addon already follows its variant:
switch the service off and the expression yields false. A statically-on addon
does not. It installs in every round, so each non-baseline variant pays install
time for plugins it never exercises, and variant 0 stops being the only
maximum-footprint shape that meta/variants.yml exists to document.
web-app-nextcloud carried 22 of them across nine variants. They are now pinned
off once in variant 1 behind a YAML anchor, and variants 2-8 alias it, so the
set has one spot -- the same shape meta/volumes.yml and the
volumes-target-single-spot lint already prescribe.
test_addons_variant_pinning.py holds the rule for every role that ships
meta/addons/ and declares more than one variant:
1. an addon whose base enabled is literal true MUST be pinned enabled: false
in variant 1; a literal false is out of scope, the pin would override
nothing,
2. an addon whose base enabled is a Jinja expression MUST NOT be pinned in
any variant -- that replaces the expression with a constant and decouples
it from the service it follows,
3. every later variant MUST carry the same set, in practice by aliasing
variant 1. Variant-specific pins beyond that set stay allowed, so a
variant may still switch a statically-off addon on (web-app-matrix does
exactly that for its bridges in variant 2).
Measured on the real role through utils.cache.applications.get_variants: all
nine variants keep 51 addon keys, variant 0 keeps its 22 statics on, variants
1-8 hold none, and every dynamic entry keeps its Jinja string -- xwiki still
resolves true only where services.xwiki.enabled is. The 51 keys matter: the
override merges per addon instead of replacing the map, which is what an
earlier lookup-based attempt at this got wrong.
Each rule was broken on purpose and seen to fail: calendar pinned true, xwiki
pinned (reported for variants 1-8, which also proves the alias resolves), and
variant 5's alias removed (22 findings, exactly the required set).
web-app-discourse is exempted per addon with a reason. Its plugin set is
compiled into the image from services.discourse.plugins, and meta/addons/ is a
second declaration of the same thing; an addon-level disable would still build
the plugin and only flip the gate flag, saving no time while silencing the
spec. web-app-wordpress is not exempt: vars/main.yml:16 builds the plugin list
from the merged map and 07_manager_ops.yml:45 gates the install loop on
enabled, so its variant 1 genuinely skips a wp plugin install. Its README
claimed activitypub was "always enabled"; that is corrected rather than left to
contradict the file.
Not verified: no deploy has run. The saving is derived from the install loops
being gated, not observed, and the WordPress Multisite variant losing
activitypub rests on the same reading plus its skipUnlessAddonEnabled gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant 0 of this role no longer fits the box CI gives it. Job 94319317296 of
run 31658270172 started at 01:49:01Z and logged
"##[error]The operation was canceled." at 07:44:14Z -- 355 minutes later, to the
minute, which is the ceiling .github/workflows/call-test-deploy-swarm.yml:161
sets. It died inside its own step, at
"TASK [test-e2e-playwright : ▶️ Run Playwright container]". The drill costs about
6h40m and the box is 5h55m.
The cost is dominated by the deploy itself, not by a repeatable sweep: PASS 1
alone is 13695.90s in role-runtimes, of which sys-svc-compose is 3094.19s and
test-e2e-playwright 2832.15s. It therefore scales with how many services variant
0 switches on, which is what this commit reduces.
The file's own header states the rule: variant 0 carries every tightly-coupled
(bond>=1, same-host) service, while loosely-coupled (bond<1) partners are
excluded from the closure and distributed across later variants to keep each
host under budget. xwiki and bigbluebutton were declared bond: 1 and therefore
had to sit in variant 0. Both are bridged partners with their own role and their
own spec -- the same shape as openproject, gitlab and mattermost, which already
carry bond: 0.5 -- so the declaration was the outlier, not the placement.
Both bonds drop to 0.5. xwiki moves out of variant 0 into a new variant 8 with
sso, ldap and tor, the same shape as variant 5 (zammad) and variant 6 (gitlab).
bigbluebutton stays in variant 0 for now, by explicit request; its bond change
only makes the move possible later. Each edge is one-directional -- neither
web-app-xwiki nor web-app-bigbluebutton declares a nextcloud service -- so there
is exactly one bond per partner to change.
coturn is deliberately absent from variant 8. BigBlueButton speaks WebRTC and
the reflex is to bring a TURN server, but the only coupling in the repo is
meta/addons/spreed.yml:8, which bridges coturn for Nextcloud Talk, and talk is
off in that variant. An extra service there would be exactly the overhead this
commit removes.
Measured: the file loads to 9 variants, variant 0 falls from 16 enabled services
to 15, and variant 8 resolves to ['ldap', 'sso', 'tor', 'xwiki']. make test-lint
stays at 328 passed -- variant-limit (capped at 3, opened by the file's existing
nocheck), variants-static-override and the bond lint (which only requires a
number in 0..1) all accept it.
Not measured, and the honest cost: a ninth variant is one more deploy round, and
the limit lint's own rationale says total deploy time grows linearly with
max(variant_count) across the primary apps -- Nextcloud sets that maximum, so
this is roughly +12.5% for the whole matrix, spent to relieve the single job that
dies on its own ceiling. Whether removing one service of sixteen closes a ~45
minute overrun is not established; role-runtimes has the per-role seconds and no
run has been made with this shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A volume that reaches several services repeated its container path once per
mount. Written out each time it has no single point of truth: a rename that
misses one line still parses, still deploys, and mounts one service somewhere
nobody intended. Nothing fails until that service looks for content which is
not there.
YAML already solves this, and 62722a6ab used it here first -- web-app-openproject
anchors await_migrations_target and aliases it for the second service. Nineteen
roles now follow, and tests/lint/repository/spot/test_volumes_target_single_spot.py
fails on any target spelled out twice. An alias carries no path, so the rule
cannot see it, which is the point.
The conversion was mechanical and is provable rather than reviewed: every file
was rewritten and then reloaded, and any file whose yaml.safe_load result
differed from the original was discarded instead of kept. Nineteen written,
none discarded. Anchors and aliases are a serialisation form; the loader
resolves them before any consumer sees a document.
validate_volumes_meta over all 81 meta/volumes.yml reports no violation.
make test-lint: 328 passed.
web-app-prometheus carries two unrelated deletions: comments that predate this
change and fell under the comment lint once the file was touched. They explained
a 0444 mode and a compose-only bind; both facts are visible in the entries
themselves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine roles reached their MCP surface as the deployment administrator or with
an account password kept beside the account in credentials. A Moodle site
admin bypasses every capability check, a Nextcloud app password is not
scopable, a Gitea token reads every private repository, and a GitLab token on
root reads every private project. Each role now declares its own principal in
meta/users.yml and provisions it, so the exposed surface is what the service
role permits. jellyfin, zammad and snipe-it kept their account but took its
password from a credential named token; they read the registry too.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Covered by PR #380 rather than merged on its own. This branch already carries
the Tor onion-service provider across the whole role set, and #495 touches the
same nextcloud Playwright specs; landing them separately would mean resolving
the same conflicts twice, once here and once on main. Reviewing #495 as part of
#380 keeps the resolution in one place.
One conflict, in the external-sites addon spec. Nextcloud 34 renamed the admin
section from #external to #external-vue and replaced the add-site control with
a 'New site' button, so the incoming selectors win — retargeting them is what
the PR is for. This branch's resolveTimeout wrappers stay on both assertions.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The integration addons run container exec against the partner during
nextcloud's own deploy, and the loader orders the web-app bucket by role
name, so openproject and zammad had not been deployed yet and the exec
answered "No such container". Edges to roles outside the play are dropped
by the topological sort, so the entries are inert where the partner is
absent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SSO over a plaintext onion is where this platform actually breaks, and none of
it shows on clearnet: a framework's Secure session-cookie default makes Chromium
drop the cookie, an https identity provider cannot be embedded in an http page,
and crypto.subtle is undefined because an onion is no secure context. Pixelfed,
Taiga, Joomla and WordPress each cost a red job over exactly that. The
combination is only exercised if a variant deploys both, so the lint requires
it: services.sso.enabled true implies services.tor.enabled true.
69 variants enable SSO; 59 already complied. Of the ten that did not, seven are
flipped on because the mechanism is proven there - matrix and nextcloud both run
SSO behind the onion in variant 0 already (20ddda8c1), and their later variants
were pinned false only by the blanket rule from 1d9d355bc. Nextcloud variant 1
stays false: SSO is off there, and it is what gives the coverage test its
false-pinning variant.
Three roles opt out, each for a structural reason named in the marker. jitsi and
bigbluebutton pin tor.enabled false in meta/services.yml because the conference
needs getUserMedia, which no plaintext onion can offer. seaweedfs gates its own
sso flag on 'svc-net-tor not in group_names', so on a tor node there is no login
flow to test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml held credential definitions and their runtime values, but the
name said 'schema' and read like structural metadata next to services.yml and
volumes.yml. Nothing in the file is a schema in the sense the other meta files
use the word, and a contributor had to open it to learn it holds secrets.
62 role files renamed. ROLE_FILE_META_SCHEMA becomes ROLE_FILE_META_SECRETS and
keeps its role as the SPOT the path is built from.
Two readers bypassed that constant and spelled the name themselves:
utils/cache/applications.py built meta_dir / 'schema.yml' twice, and
plugins/lookup/application_gid.py carries the filename in the marker list it
detects application roles by. Left behind, the rename would not have raised —
credentials would have resolved empty and the role would have stopped counting
as an application.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tools.allowlist meant two things at once: the surface a consumer may reach,
which is a decision of ours, and the surface the upstream offers, which the
vendor changes without asking. Conflating them made a version bump read as a
contract violation, and left no place to record what was actually measured.
Split them. allowlist stays policy; upstream_serves records the observation at
supported_version. State the second only where it DIFFERS, because a provider
that serves exactly what is allowed would otherwise carry the same list twice
and the copies would drift apart. Absence means identical.
The difference between the two measures unenforced exposure: mattermost 103,
gitlab 2, and zero wherever the fields collapse. A non-empty difference is what
tells us a provider needs a gateway in front of it.
Require a non-empty allowlist for an enabled, non-adapter, server-capable
provider. An empty one used to make the probe's comparison vacuous, which read
as a pass while every tool the upstream serves stayed reachable. Nextcloud was
in exactly that state.
Three exemptions, all semantic: an adapter authors its tools here against a
REST upstream, so there is nothing to observe; a provider declared
enabled: false is never admitted; and upstream_serves: dynamic covers an
upstream composing its surface at runtime, which nextcloud does by proxying
MCP servers an administrator configures.
Reject an allowlist naming a tool the upstream does not serve. That found
mattermost's add_user_to_channel, gone since 11.10.0 renamed it
add_channel_member, which would have been advertised and never callable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud's SSRF guard answers every server-side call to an RFC1918
address with 401 'violates local access rules'. integration_mastodon
registers its OAuth app dynamically against the partner, so in the swarm
lab -- where every partner resolves inside 192.168.244.0/24 -- the
integration never connects and the addon spec never leaves Nextcloud.
The switch is a service config entry rather than a hardcoded value, so a
deployment whose partners really do sit on a private network can turn it
on without the role stomping the setting on every deploy. Default is on
in development only; production keeps the guard.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ten provider roles served an MCP surface that other roles consume, but
were never declared as shared services, so the registry did not know
them. Declaring them pulls in the recursion contract every registry
service owes: seven roles move their entry tasks into tasks/00_core.yml
behind the run_once flag, so the service loader cannot include them a
second time.
pretix already had a run_once block, but set the flag last; the flag now
comes first, before the body it protects.
gino0631/nextcloud-metadata has no appstore release for server major
34, so `occ app:install metadata` reports the app as incompatible and
the role skips enable/configure — it was the only addon of ~50 to do
so. The spec asserts the opposite: OC.appswebroots must carry the
metadata key and the Files page must load apps/metadata/*, both true
only for an installed and enabled app.
`required` is the second half of the Playwright gate — addon_env_flags
emits <ADDON>_ADDON_ENABLED only when enabled AND required — so
flipping it lets skipUnlessAddonEnabled skip instead of asserting a
surface that cannot exist. `enabled` stays true: the install is still
attempted every run, and a compatible release only needs `required`
flipped back. Deploy behaviour is untouched; the abort condition reads
config.mandatory, not required. Same lever as integration_suitecrm.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-registry-cache is the twin of the registry volume fixed in 1e5d2abb8: same /var/lib/registry target, same missing declaration, and the same claim in its own first line that the volume is backup-disabled. Matomo's tmp entry likewise already documents a regenerable Twig and session cache. Neither statement was true, because backup_volume.py keys on the declaration alone.
Also jellyfin's /cache, which sits beside the /config volume that stays backed up, and nextcloud's whiteboard tmp, whiteboard fontcache and talk recording tmp.
Left alone: gitlab's builds volume holds CI artifacts, which are reproducible but whose retention is a product decision, and the gitlab config templates, the cdn build script and the openldap ldif bind, which a name-based scan turns up but which are not caches.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The block sat under services but never behaved like a service. A service has
exactly one providing role, declared with provides, so the registry can map the
key to that role and consumers bind their flag to its presence. No role declares
provides: mcp. There are seventeen providers and the pairing is done by
allowed_consumers inside the block itself.
Every rule that enforces the service model therefore needed an exemption for it:
a shared-provider suppression in fourteen roles, a playwright-service-flag
suppression, and two freshly added exemptions in the dynamic-flag and
variants-static-override checks. Four independent rules, four escape hatches,
one cause. They are all gone now, together with a bond key that only the
resource budgeter reads and only for entries under services, and a wordpress
addon bridge that pointed at a service block that no longer exists.
The block moves to meta/mcp.yml. The applications loader maps meta/<name>.yml to
applications.<app>.<name> generically, so the move needs no wiring; the file
takes the same shape as meta/rbac.yml. Every config path follows, the variant
overrides are lifted out of their services map, and the vocabulary module leaves
the services package because it no longer describes one. roles_with_service
gains an explicit topic parameter rather than an implicit fallback, so nothing
silently resolves against the wrong container.
Files that are otherwise unrelated to this move are included where the committed
tree would otherwise keep pointing at a path that no longer resolves: thirteen
vars files, eleven playwright env templates and one task file.
A provider overlay keyed on services.mcp.enabled and .shared admitted every
MCP-capable role, servers included, because those flags carry no direction. Each
provider's network therefore collected every other provider and the result was a
mesh in which any sidecar could reach any application. The new mcp_client
consumer kind additionally requires services.mcp.direction to be client or both,
and the four provider overlays switch to it.
A sidecar rendered its networks with the same lookup as its provider and
inherited that provider's consumer attachments, which it neither needs nor
should have. own_network_only keeps it on the provider network alone. A provider
whose own topology is default_net contributes no attachment line at all, and a
bare networks: key is not a list, so the flag returns nothing in that case and
lets the sidecar fall back to the project default, which is that provider's own
network.
Four roles declared auth_subject: user while handing the client an
account-bound credential. Nextcloud and Home Assistant mint against the
administrator account, Mattermost issues the administrator's personal access
token, and Baserow creates a dedicated mcp@<domain> account with its own
workspace. The declarations now name what the minting tasks do; Baserow's
README additionally claimed the administrator owns its endpoint, which it
never did.
The tools declarations follow one convention instead of two: the field records
the deployment's intent and the README names the missing enforcement, which is
what GitLab already did and what Baserow and Mattermost now do.
The discovery lookup carries auth_subject through so clients can see the
subject. No consumer branches on it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight server roles declare an `mcp` RBAC role, so the platform mints
roles/<app>/mcp and the administrator joins it. Open WebUI now honours those
groups instead of flattening them to user/admin, and each MCP tool server is
restricted to the group of the role that serves it.
The grant cannot be rendered: Open WebUI matches it against a group id that
insert_new_group mints as a fresh UUID, so tasks/01_mcp.yml resolves the group
after the container is up and writes the grant through the admin API, the way
every instance-minted credential in this repository is provisioned.
Entries render disabled. An empty access_grants is not 'nobody', Open WebUI
reads it as every administrator, and ENABLE_PERSISTENT_CONFIG=false makes the
env authoritative again on every start. Enabling only together with the grant
keeps the fallback at no access rather than too much.
The provisioning account is the role-internal openwebui-api-bot, never a
platform identity: OAUTH_MERGE_ACCOUNTS_BY_EMAIL ships disabled and an OIDC
login whose email a local account already holds is refused, so seeding a
person's address would lock them out.
Known limits are documented rather than claimed away: a bare restart leaves MCP
off until the next deploy, a user's roles list carries no application context so
roles: [mcp] grants every deployed mcp group at once, and Open WebUI drops a
stale membership only while the groups claim is non-empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`|-` preserves the newlines that surround the `{% if %}` and `{% endif %}`
lines, so the rendered value carried leading and trailing whitespace instead
of a bare filter. An LDAP filter with leading whitespace is rejected by the
server, and Nextcloud answers with an empty user list rather than an error.
A double quoted scalar cannot hold a newline by construction. The branch is
unchanged: the full user filter when the user directory is enabled, `()`
otherwise.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, each resolved by combining both sides rather than picking one.
compose.yml passes both env vars through to the container: swarm-nfs adds
INFINITO_SWARM_MGR_IP, this branch adds INFINITO_TOR_SOCKS_PORT, and neither
substitutes for the other.
daemon.json.j2 takes the data-root block from swarm-nfs and keeps HEAD's dns
key. The incoming side writes networks.internet.dns directly; lookup(
'container_dns') already returns that value as its clearnet entry and prepends
the docker0 bridge on a Tor node in compose mode, so the incoming form is a
strict subset and dropping it loses nothing.
The WordPress CA-trust addon spec gains skipUnlessServiceEnabled("sso") from
swarm-nfs alongside HEAD's gotoOnion and resolveTimeout, and asserts on the
Plugin Name the mu-plugin header declares - the mustuse screen renders that,
not the filename - with resolveTimeout applied to the literal the incoming
side left bare.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The php-fpm listener port lived as a literal 9000 in the nginx sidecar
config, and the application user as a literal www-data in both
healthchecks, while the role already carried NEXTCLOUD_DOCKER_USER and
declares every other internal port in meta/services.yml.
Declare the listener as services.nextcloud.ports.internal.fpm and read
it through the config lookup, the same shape the moodle and magento
roles use for their fpm listeners and the same shape this role already
uses for the signaling and recording ports. That also satisfies the
port lint, which requires every *_PORT constant in a role's vars to
resolve through the lookup rather than carry a literal.
The rendered output is unchanged: the port is still 9000 and the user
is still www-data. After this change no www-data literal remains in the
role's code; the occurrences under docs/ stay as they are, since those
are copy-paste operator recipes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud carries no MCP server of its own, so the role runs the Context
Agent as an ExApp: render its container while MCP is enabled, wait for
its heartbeat, register it with AppAPI, add the MCP route, enable it, and
only then mint the app password clients present.
Split the credentials by origin. The shared secret between the AppAPI
proxy and the ExApp is generated by this deployment and stays a vault
credential; the app password is minted by Nextcloud through occ and is
persisted through sys-token-store instead.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three conflicts, none resolved by picking a side wholesale:
- Makefile: took the incoming ACT_INPUTS, which routes through SWARM_DISTROS
instead of inlining the same expression a second time.
- test-e2e-playwright/02_run_one.yml: took the incoming rc handling. An absent
rc now counts as failure; the previous default(0) let an aborted Playwright
run pass silently, contradicting the debug message right above it.
- web-app-peertube spec: combined both. Kept gotoOnion and resolveTimeout,
which this branch's lint enforces, and adopted the incoming retry-inside-poll
for the OIDC button plus its networkidle wait.
The declared values were guesses that ran up to 60x the size of the image they belong to: seaweedfs asked for 30GB against a 104MB image, nextcloud for 20GB against 353MB. Because min_storage feeds the storage guard, the runner packer and the sufficient-storage filter, those numbers made hosts look too small and pruned caches that deploys did not need to lose.
Every service that declares min_storage together with image and version now carries the measured compressed transfer size of that image plus 200MB of headroom. Sizes were read from the registries through utils.docker.registry.manifest_transfer_size, the same path the headroom test uses, so the declarations and the gate agree by construction. Values are written in MB throughout, matching the existing 500MB / 100MB / 50MB style.
119 declarations across 89 roles. Range: 206MB (memcached) to 3473MB (ollama); four exceed 1GB. The 84 min_storage entries that name no image are untouched, since there is nothing to measure them against.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nextcloud reached its OpenAI-compatible backend through Flowise, which
only forwarded to the LiteLLM sidecar that Flowise no longer runs. The
consumer flag, the activation condition and the endpoint now bind to
svc-ai-litellm directly, one hop shorter.
The endpoint switches from Flowise' public TLS URL to the internal
gateway address plus /v1, so a container-to-container call no longer
travels through the reverse proxy.
The api_key stops crossing a role boundary: it was read from
web-app-flowise credentials.llm_master_key, handing Nextcloud
administrative access to the whole gateway, and is now this role's own
per-consumer virtual key.
The litellm variant pin sits on variant 7 rather than 3 because adding
the gateway to variant 3 pushed that round past the matrix memory budget.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Resolves the runner Dockerfile to the lint-standard Acquire::Retries=3
and keeps the collabora TODO wording; hardens the merged tree for the
stricter lints (apt retries in the tor Dockerfile, shell-or-true
markers, comment cleanup).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The infinito config loader was appended to the occ-managed config.php with a shell printf while the image entrypoint was still in its post-install write phase; the entrypoint read config.php before the append and wrote it back after, clobbering the loader, so /var/www/infinito extras (oidc_login_provider_url among them) never reached Nextcloud and /apps/oidc_login/oidc answered 500 in the compose e2e job. Install the loader as config/zzz-infinito.config.php after occ reports installed instead: Nextcloud merges config/*.config.php on every read, occ never rewrites those files, and the entrypoint rsync excludes /config/, so no clobber path remains. A file mount was rejected because a non-empty config/ suppresses the entrypoint's base-config copy on first install. The loader now declares $CONFIG directly (additional config files load after unset($CONFIG)), cron gets the /var/www/infinito bind it silently lacked, and the dead include-path vars and host render go away.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
