default.env declares INFINITO_DOMAIN and everything is meant to derive from it.
Nothing checked that, so copies accumulated. A copy is a second source nobody
updates, and it fails in the least useful way: the code keeps running and
quietly talks about the wrong host. integration_mastodon is the worked example
-- it asserted the provisioned partner equalled a clearnet literal, so the spec
passed on a clearnet node and failed on an onion one while the product was
right in both.
tests/lint/repository/spot/test_no_hardcoded_primary_domain.py now fails on the
literal anywhere in tracked source. It reads the value it hunts for FROM
default.env, so the rule follows the SPOT instead of repeating it and would
otherwise be its own first violation.
Scope, and why each exclusion is not a loophole: default.env itself is the
source; .md is documentation; tests/ and scripts/tests/ hold fixtures that must
assert against a named host; inventories/ is where writing the real values is
the entire point; gitignored output carries the value because it correctly
derived it. Everything else is in.
Two real hardcodes are repaired rather than suppressed.
utils/domains/default_primary.py mirrors utils/roles/lifecycle.py -- one
function reading one key through parse_static_env -- and replaces the module
constant in utils/cleanup/nginx_vhosts.py and the argparse default in
cli/meta/domains/__main__.py. Both resolve to the same value as before.
Seven documentation sites become <primary-domain> placeholders instead of
carrying a nocheck: a Python docstring cannot host one without polluting the
docstring, and a placeholder does not rot when the default changes. The single
nocheck is scripts/system/tls/trust/wsl2.sh:61, where the marker is matched
verbatim against what an earlier run wrote into the Windows hosts file --
deriving it would orphan that block instead of replacing it.
Collateral, not intent: touching roles/sys-ctl-hlth-csp/files/script.py brought
it under the comment lint, which found five pre-existing narration comments.
Three deleted, `fn[:-5] # strip ".conf"` replaced by fn.removesuffix(".conf"),
and the root-user comment rewritten in the sanctioned Exception form. Its 33
tests stay green.
tests/lint/repository/test_no_hardcoded_container_name.py (05f425ea5) is the
same family for INFINITO_CONTAINER and could move into spot/ later; this change
leaves it where it is.
make test-lint: 326 passed. The rule found 18 sites before the cleanup and none
after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container_hostname is computed for sethostname(2) and clamped to 63 bytes, so a
v3 onion domain degrades it to the bare entity name. container_healthcheck
forwarded that value into the curl probe's Host header, and curl is the only
flavor that asserts a vhost identity on a loopback probe -- tcp and http_status
hardcode Host: localhost, wget and http send none. An app that validates Host
therefore rejected its own healthcheck: bookwyrm's Django answered 400 to
Host: bookwyrm and the container never turned healthy while serving fine over
its onion.
This was diagnosed once already. e9817f825 measured it A/B on pretix
(Host: pretix -> 400 x4 unhealthy, Host: localhost -> healthy), named reusing a
kernel hostname as an HTTP Host the underlying design bug, and deferred the
general fix because it would move six roles' probes at once. 9fb08b4ed then
pinned bookwyrm the same way. This is that deferred change, so there is no
third round of per-role pinning.
The header now comes from meta/services.yml and nowhere else. Of the eight
roles defining container_hostname, bookwyrm and pretix were already pinned and
bridgy-fed and postmarks use the tcp flavor, which never reads it. Of the
remaining four, chess never passed anything but its entity name and xwiki was
measured healthy while answering to Host: xwiki, a name in no vhost config; so
both are safe without a header. confluence and jira are pinned here rather than
argued about -- their Host is today the real domain on a clearnet deploy, and
Tomcat's ATL_PROXY_NAME being a proxyName and not an allow-list is reasoning,
not a measurement.
The templating contract moves from the play var to healthcheck.hostname, and a
new test fails if the fallback ever returns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight prose references still pointed at the pre-rename root, and one of them
did more than mislead a reader: the addons lint told an author to reach a
secret through lookup('config', application_id, 'credentials.<name>') at the
exact moment it rejected their literal, so the guidance produced the next
defect. The check itself only tests whether the value is templated, so no
enforcement changes here.
variants.md gains the reason rather than a corrected list. A variant entry is
baked into host_vars raw, which makes its top-level key the inventory path, so
a key no meta topic produces is silently discarded. The previous list drifted
precisely because it stated the set without stating why it was the set.
design/variants.md was wrong twice over: the old root, and an argument order in
lookup('config', 'credentials', '<provider>') that could never have resolved.
The erpnext key was checked against svc-db-mariadb/vars/main.yml rather than
prefixed on faith.
Left alone on purpose: test_credentials_roundtrip and the layout lint name
meta/schema.yml and schema/main.yml as history and as the legacy directory the
lint forbids. Both are correct.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The only shape check on meta/variants.yml rejected a single key, `compose`.
Its docstring stated the actual rule -- top keys must be payload roots -- and
never enforced it, and had itself gone stale: it still named `credentials`,
the root f713ba835 retired. That is how the previous commit's four blocks
passed every gate while writing where nothing reads.
The legal roots are now derived from the meta files instead of restated: every
roles/*/meta/*.yml stem, plus addons when that directory carries specs, minus
main and variants, which are mechanism rather than payload. A role adding a new
topic makes its root legal without touching this test, so the list cannot drift
from the loader again.
compose needs no special case anymore -- it is not a meta file, so it fails
like any other unknown key, as does a typo such as `servicse`. Verified by
running the pre-fix keycloak and lam variants through the new whitelist: all
four `credentials` blocks are rejected.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compare_application_keys skips generated credentials because no default exists
for a secret, but it recognised them by the pre-rename prefix. Once the
generator started writing them again, every credential arrived as
secrets.credentials.<key> and the exemption stopped matching, so validation
rejected three keys per credential -- the leaf and both intermediate nodes,
which recursive_keys emits separately -- and aborted the deploy before the
playbook ran. The condition now admits the bare secrets node and the
secrets.credentials subtree, leaving a sibling such as secrets.other flagged.
The check in the inventory manager stays on the bare prefix on purpose: it
walks meta/secrets.yml, whose own root key is credentials.
test_credentials_keys_are_skipped fed the validator the old shape, so it kept
passing while real inventories were being rejected.
Evidence: run 31655559799, job 94311023723,
"Missing default for web-app-dashboard: secrets.credentials.sso_proxy_cookie_secret".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four comment lines above the assertNotIn block restate what those three
assertions state executably, and unlike them they cannot fail when the claim
stops holding. test_comments_valid lints only changed targets, so the block
stayed invisible until the previous commit made the file one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_apply_vars_overrides_from_file_deep_merge_and_overwrite sat at module
level while still taking self, so pytest treated self as a fixture request and
errored the test out at collection instead of running it. The assertions it
carries for apply_vars_overrides_from_file have therefore never executed.
Re-indenting it into HostVarsTestCase makes it a real test again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The host_vars merger still wrote applications.<app>.credentials and read
"credentials" out of the generated snippet. f713ba835 moved the emitter to
secrets.credentials, so the read resolved to {} on every app: nothing was ever
copied, and _ensure_ruamel_map left an empty credentials: {} shell behind as
the only trace. Every credential generated since that commit was discarded on
its way into the inventory.
Both sides now go through credentials_map() and declared_credentials(), the
accessors the emitter already uses, so the two ends cannot drift again.
_ensure_ruamel_map was a copy of emit.ensure_map and is gone.
The failure surfaced in two shapes from the one cause. Templates reading
through lookup('config', ...) raise, because get() defaults to strict:
"Key 'secrets' not found in dict at 'secrets'" for
web-app-mastodon.secrets.credentials.secret_key_base, with the dumped payload
showing 'credentials': {} and no secrets node. Templates reading through
lookup('database', ...) stay quiet, because that plugin passes strict=False
with default="": friendica's MariaDB received an empty MYSQL_ROOT_PASSWORD and
refused to initialise.
The generator's own tests pinned the pre-rename shape in both the snippet and
the assertion, which is why they stayed green through the regression.
Evidence: run 31649221474, job 94303863545.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f713ba835 moved credentials to applications.<app>.secrets.credentials and
updated every role that reads them, but four lookup plugins kept addressing
the bare credentials.<key> root. They call get() with strict=False and
default="", so the miss was silent rather than loud: MariaDB received
MYSQL_ROOT_PASSWORD= and refused to initialise, which took friendica-database
down as an unhealthy dependency and failed compose up. Object-store consumers
and shared redis auth resolved to empty secrets for the same reason.
The five call sites now build the path from OVERRIDE_SECTION, the single point
of truth that commit introduced, instead of spelling it beside it.
The two lookup unit tests pinned the pre-rename fixture shape, which is why
they stayed green straight through the regression; their fixtures move to
secrets.credentials so they fail against the old code.
Evidence: run 31649386343, job 94302347256, artifact
rescue-diagnostics-instructions-web-app-friendica,
containers/infinito_nexus_debian/nested/.../friendica-database.log:29
"Database is uninitialized and password option is not specified".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_distros_spot globs **/*.md and friends across the whole project root and
reads every hit, so a named pipe, socket or device node with a matching suffix
crashes the check with PermissionError instead of reporting an offender. It now
skips anything that is not a regular file.
test_no_uncached_file_reads flagged three reads in the inventory reset tests.
Its suggested fix - the process-wide cached reader - would be wrong there: each
of those tests writes the file and reads it back in the same test, so a cached
read would serve the pre-write content and the assertion would stop testing
anything. They carry a nocheck marker instead, on the call line and short enough
that the formatter cannot wrap the expression away from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The adapter was the only image in the tree pinned by a hand-maintained digest,
and the pin was never recomputed by anything. It drifted twice: run 31374959721
lost seven of eight jobs to mirror/docker.io/python@sha256:6771159c... not
found, and the renewal to the newest upstream digest earlier today would have
reproduced it, because builds resolve FROM against the GHCR mirror while the
mirror is copied by tag and lags upstream by design.
The base is now FROM <image>:<version>, taken from the sidecar service in
meta/services.yml - the only file iter_role_images scans, so the tag the build
asks for is by construction the tag the mirror was told to copy. The version
moves from the moving 3.13-slim to 3.13.15-slim, which upstream stops rewriting
once published, and which the existing version drift job renews like every other
image.
With the build reading services.yml, the adapter block's digest, image and
version had no consumer left. They are removed together with the schema check
for the digest form, the warn-only drift test, and the manifest_digest helper
that existed only for it. What remains in the block describes the adapter rather
than the image it is built from.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cli build include` generated one tasks/groups/<group>-roles.yml per invokable category during setup, and the stage plays included those files. They were written by the in-container UID, which is why a dedicated clean-container-owned helper had to delete them from inside the container.
The stage plays now loop over lookup('group_roles', <group>) and include tasks/utils/setup/{group,role}.yml directly. ordered_roles() in utils/roles/order.py holds the run_after topological sort the generator used to perform at build time.
utils/roles/stage.py:_group_names() no longer globs the generated files; it reads get_invokable_paths(), the same categories.yml source those files were derived from.
Measured: handlers still flush after every role (ansible-core 2.20.2, three loop items, three separate handler runs), and resolving all 18 groups costs 0.60s cold against 1.4s for the generator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Provisioning fills a missing credential and leaves an existing one alone, so every deploy pass reads the value the first pass wrote and the update path is never exercised. Rotating the generated values makes a redeploy carry a secret that moved.
`infinito administration inventory credentials reset` drops every vault-encrypted leaf under applications.*.credentials.* (--schema) and the pinned password of every declared user (--users), then lets the existing generators refill the gaps, so no schema logic is duplicated. Plain credentials stay unencrypted in host_vars and are therefore left alone, as are users that exist only in the inventory and would get nothing back. --except exempts application ids and user keys, --backup keeps the pre-rotation file as <name>.YYYYMMDDTHHMMSSZ.backup, and --mirror realigns the other hosts of a multi-host inventory.
Adds the command only; no deploy path calls it yet. Measured against web-svc-coturn and web-app-flowise: eleven generated secrets move between two runs while ansible_become_password and the administrator password stay pinned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Reverts the twelve pins to 9662417a, the digest the GHCR mirror currently
carries for python:3.13-slim. The previous commit renewed them to the newest
Docker Hub digest, which is the wrong registry: builds resolve FROM against the
mirror whenever a mirror prefix is configured, and the mirror is copied by tag,
so it holds whatever the tag pointed at when the sync last ran. A pin ahead of
the mirror resolves to nothing there while a local build keeps working - run
31374959721 lost seven of its eight jobs to exactly that, as
mirror/docker.io/python@sha256:6771159c...: not found.
The external check measured Docker Hub and would therefore have kept advising
the change that breaks the build. It now resolves the source the build uses via
mirror_image, asks whether the pin exists there at all, and only falls back to
upstream when no mirror is configured.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every one of the eleven declared tools carried the placeholder schema
{"properties": {}, "type": "object"}. The adapter answers tools/list purely
from this contract, so each client saw parameterless tools: nothing could know
that read_post needs a post_id, and the call failed upstream instead of being
formed correctly. The schemas here were measured against the running upstream;
list_agents keeps an empty property set because upstream declares none.
Both pins of the file follow, as the adapter rehashes the contract at startup.
Add a lint binding the declared surface to the shipped one. Only the allowlist
is an equality: writer_allowlist is a superset by construction, because
passthrough.py rejects a whole contract as soon as one tool declares
mutating: true while mutations are off, and upstream_serves is an observation
that may legitimately exceed what the platform allows.
The suppression lookup both MCP lints need moves into suppress.py rather than
being written twice.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Friendica 0 and WordPress 0 of run 31543714113 never delivered their startup mail. An onion relay has no TLS certificate, so the resolver forces tls=false - and both port and auth hung off that flag: port collapsed to 25 and auth to off. Port 25 is the inbound MX and offers no SMTP AUTH at all, so the message reached Mailu unauthenticated with a local envelope sender and rspamd rejected it 554 5.7.1. Port and auth now hang off `external` instead: 465 with TLS, 587 without, and auth wherever a relay is external.
Moving the port alone would only relocate the failure. msmtp refuses to pick PLAIN or LOGIN itself on a cleartext link and aborts with EX_UNAVAILABLE, so the mechanism has to be named. The new `auth_mechanism` key resolves to off, on, or plain and sits in RESOLUTION_ORDER right after auth, whose result plus tls it reads.
It is one derived value rather than two template expressions because the two failing jobs run different msmtp instances: WordPress renders sys-svc-mail-msmtp/templates/msmtprc.conf.j2 through WORDPRESS_MSMTP_SRC and mounts it at /etc/msmtprc, while Friendica generates its own from SMTP_AUTH via the image script, whose line 32 writes `auth $smtp_auth` verbatim. Both now read the same key.
Verified against the shipped Mailu 2024.06 front rather than assumed. EHLO on 587 under TLS_FLAVOR=notls answers `250-AUTH PLAIN LOGIN`, so PLAIN is offered on the cleartext link; dovecot renders disable_plaintext_auth=no in that flavor. PORT_587 is set because socrate lists 587 outside PORTS_REQUIRING_TLS, unlike 465, which notls skips - so 587 is the only submission port available there. smtp and submission bind identically to MAILU_IP4_PUBLIC, so the path to 587 is the one that already reached 25.
This supersedes the premise of 563da9e5b, which disabled AUTH because the resolver had. That commit quoted a different msmtp error - "the server does not support authentication" - which was port 25 having none, not the relay refusing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three fields pinned artefacts and nothing recomputed any of them.
The adapter base digest was stale for all twelve provider roles: the tag
python:3.13-slim moved to ffb752e1 while every role still pinned 9662417a.
The GHCR mirror copies by tag, so the pinned manifest is one it was never
asked to copy - a mirrored build resolves FROM to nothing while a direct
build keeps working.
adapter.specification_sha256 matched no revision of the file it pins in ten
of twelve roles; three of the old values were byte-identical to the same
role's tools.schema_sha256, so they had been copied rather than computed.
The field is enforced nowhere at runtime, which is exactly why it needs a
check.
Add a hard lint for both offline pins, computing the tool digest with the
adapter's own policy.schema_digest so it cannot drift from the code that
enforces it in production, and a warn-only external check that reports a
base digest the upstream tag no longer resolves to.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Alphabetical make target order, and the integration test walked the project
tree itself instead of the cached helpers, which share one walk and one read
per file across the whole session.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Nine roles name their MCP principal by a literal key, and a typo there does
not fail loudly: the lookup yields an empty password and the application
answers with a rejected login, on the one variant that enables the surface.
The rule stays scoped to the mcp- principals because the registry is open by
design, so a repository-wide version would report the inventory's own
Playwright persona as missing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A user password is pinned once by inventory creation from a shell-safe
alphabet, because it travels through Ansible, a shell and container exec.
An application whose registration demands a special character then had no
way to say so, and roles worked around it with a second credential beside
the account they already owned. The account now carries algorithm and
validation itself, two roles disagreeing about one account abort instead of
one silently winning, and an unsatisfiable pair fails naming both halves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An adapter-fronted provider holds two unrelated credentials, and handing
the upstream token to the contract probe produces a 401 the log blames on
the adapter. The rule binds the probe include, the vars definition reading
credentials.mcp_bearer and the ADAPTER_BEARER the sidecar starts with to
one variable, so the three cannot drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both tests reached for the filesystem themselves: one built the project root
from Path(__file__).parents[4], which silently points at the wrong directory the
moment the file moves, and both called Path.read_text directly, bypassing the
cache every other reader shares. The role-defaults path was spelled out by hand
where ROLE_FILE_DEFAULTS_MAIN already names it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
SSO over a plaintext onion is where this platform actually breaks, and none of
it shows on clearnet: a framework's Secure session-cookie default makes Chromium
drop the cookie, an https identity provider cannot be embedded in an http page,
and crypto.subtle is undefined because an onion is no secure context. Pixelfed,
Taiga, Joomla and WordPress each cost a red job over exactly that. The
combination is only exercised if a variant deploys both, so the lint requires
it: services.sso.enabled true implies services.tor.enabled true.
69 variants enable SSO; 59 already complied. Of the ten that did not, seven are
flipped on because the mechanism is proven there - matrix and nextcloud both run
SSO behind the onion in variant 0 already (20ddda8c1), and their later variants
were pinned false only by the blanket rule from 1d9d355bc. Nextcloud variant 1
stays false: SSO is off there, and it is what gives the coverage test its
false-pinning variant.
Three roles opt out, each for a structural reason named in the marker. jitsi and
bigbluebutton pin tor.enabled false in meta/services.yml because the conference
needs getUserMedia, which no plaintext onion can offer. seaweedfs gates its own
sso flag on 'svc-net-tor not in group_names', so on a tor node there is no login
flow to test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A retrigger reproduces its source run's configuration, and the tor axis was
missing from it: rerunning a run dispatched with 'exclusive' silently fell back
to 'auto' and tested a different matrix than the one whose failures it was meant
to chase.
The mode is read from the job log's Inputs block, not from the run title. The
title renders it as a glyph, so recovering it there would mean translating a
symbol back into a word on every retrigger; the log already holds what the
operator picked, and inputs_from_jobs reads that same block for the priority
list today. LOG_INPUTS names the inputs that take this route.
That also removes a trap the glyph created. The tor segment renders through
format('{0} ', ...) with no opening literal, and run_name locates a value by
searching for its head - str.find('') matches at position 0, so the parser would
have handed back the title's own prefix as the tor value, plausibly and without
raising. segments() now skips a headless segment; its glyphs stay openings, so
they still terminate the value before them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The onion glyph alone left a job name ambiguous: no glyph meant either
'clearnet' or 'this mode has no tor axis'. Swarm and compose entries now open
with the onion or the globe, so the name states which of the two networks the
job tested. Host keeps neither — it shares the matrix builder but has no onion,
and a glyph there would claim something the job never did.
'enabled' becomes 'enforced' so the manual selector says what it does: tor on
every role that can take one, not tor on everything.
'exclusive' is the new mode. keep_tor_variants runs before assign_tor and cuts
each entry down to its tor-capable variants, dropping entries left with none, so
a run tests the onion and nothing else. A bundle can come out with a hole in it
— variants 0 and 2 capable, 1 not, yields '0,2' — which filter_plan_to_variants
already handles by set membership; both shapes are now pinned by tests.
Note the word is overloaded: services.tor.exclusive drops a deployed app's
clearnet vhost, while this mode only narrows what CI runs. The docstring says so
where a maintainer will look.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml held credential definitions and their runtime values, but the
name said 'schema' and read like structural metadata next to services.yml and
volumes.yml. Nothing in the file is a schema in the sense the other meta files
use the word, and a contributor had to open it to learn it holds secrets.
62 role files renamed. ROLE_FILE_META_SCHEMA becomes ROLE_FILE_META_SECRETS and
keeps its role as the SPOT the path is built from.
Two readers bypassed that constant and spelled the name themselves:
utils/cache/applications.py built meta_dir / 'schema.yml' twice, and
plugins/lookup/application_gid.py carries the filename in the marker list it
detects application roles by. Left behind, the rename would not have raised —
credentials would have resolved empty and the role would have stopped counting
as an application.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3a1ef70cb removed the rerun workflow's rows from the actions configuration
page and left two blank lines at the end of the file, which MD012 rejects.
That is what failed the Lint job on 777f7f0fa.
The two Python files carry the reflow ruff 0.16.2 produces; 6320da703
brought that version in with the main merge. Committing it keeps the tree
matching the formatter, since scripts/lint/python.sh runs ruff format in
write mode and would otherwise rewrite them on every run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_shared_db flipped every DB service to shared with no exemption, so
web-app-magento's deliberate embedded MariaDB 11.4 - declared shared:
false with a comment warning that no bot may move the major underneath
it - became dead config and setup:install hit the central engine's 12.3.2,
outside Magento's supported range. The run log names the override:
'swarm force-shared: web-app-magento.services.mariadb -> shared=true'.
Exempts a DB service whose role meta declares a literal shared: false AND
whose role is manager-pinned. The second condition is not decoration:
utils/storage/nfs.py keeps a manager-pinned role's volumes node-local, and
InnoDB over NFS is why the harness force-shares in the first place. Of the
two roles with a literal local DB, magento qualifies and bigbluebutton
(unpinned) does not, so its behaviour is unchanged.
derive_includes._force_shared_db_view documents itself as mirroring
force_shared_db, so it takes the same exemption - without it the dep walk
would compute magento's include closure against the central engine while
the deploy runs a local sidecar.
The sidecar carried no deploy block at all, so an exempted service would
have been free-floating with one task per node over a single-writer
datadir. The block now comes from a container_deploy lookup shared with
sys-svc-container/templates/deploy.yml.j2, which replaces the replicas
ternary and the duplicated placement constraint there. Its unit test
caught a real defect while being written: with compose_mode_force=swarm
over DEPLOYMENT_MODE=compose the delegated compose_replicas returned
empty, emitting a deploy: key with no child.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
22 commits, mostly dependabot and docker image-version bumps, plus two fixes:
5153f3030 loads keycloak-js 26 as an ES module, 87a053518 makes the coredns
Corefile readable to the container user.
One conflict, in roles/web-app-dashboard/templates/javascript/oidc.js.j2, resolved
as a union because the two sides touched the same line for unrelated reasons.
edb96e9ae had hoisted the asset URL into KC_JS_ASSET_URL and rewrote it onto the
onion mirror; 5153f3030 changed the path to lib/keycloak.js and declared a local
Keycloak binding, since the adapter now arrives as a module default export instead
of a window global. The onion rewrite stays at the conflict site, the local binding
is kept because loadKeycloakAdapter returns the constructor rather than setting a
global, and the path change moved to the constant's definition.
An import() fetch always runs in CORS mode, which is why 5153f3030 gave the CDN
aca_origin. The onion route reaches the same asset through web-svc-mirror, whose
proxy.conf.j2 already sets that header, so the onion flavor needs no counterpart.
make quality green on the merge result: test-external, test-integration, test-lint,
test-unit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tor coverage was decided per role variant and nowhere else: 1d9d355bc pins
services.tor.enabled true in variant 0 and false in every other variant, so an
operator had no way to steer it and a job name never said whether it ran behind
an onion. INFINITO_TOR now controls the axis from manual steer down to the
drill, and the entries that get an onion wear one in their label.
assign_tor decides per entry. 'enabled' gives tor to every capable entry,
'disabled' to none, 'auto' to every second one with the run number's parity
flipping which one starts — measured over the full matrix: 397 entries, 257
capable, 129 on even runs, 128 on odd, union 257, overlap 0. Two consecutive
runs therefore cover every capable entry exactly once.
tor_capable reads the entry's own variants, not just meta/services.yml.
Reading only the base config claimed an onion for 131 of 388 entries whose
variant had pinned the gate off — a third of the glyphs would have lied. A
bundle counts as capable when any variant it covers can use tor; a reactive
Jinja flag counts as capable, since that is the role saying 'onion when the
node is dark'.
The 'disable' token is the working half: it prunes svc-net-tor from the include
closure (utils/tests/swarm/derive_includes.py) and clears the service flags in
host_vars (cli/administration/inventory/provision/services_disabler.py). Both
halves are required — with only the first, extend_inventory walks the provider
back in. It runs after the variant overlay, so it also overrides a variant that
pinned the gate true.
Only swarm and compose join the axis. Host discovery shares the same matrix
builder, so without the allowlist its jobs would carry onion glyphs for a mode
that has no onion at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assert_surface decides which of two different readers a failure belongs to,
and nothing exercised it: contract.py only runs inside the probe container
against a live server, so the logic was verified by reading alone.
The load helper patches the environment before import because the module
reads MCP_URL, MCP_TRANSPORT and MCP_AUTH_HEADER at import time.
Five cases: a matching surface passes; a policy mismatch fails; drift is
reported before policy when the recorded surface is stale; a recorded
difference still fails policy while no gateway narrows it; and a provider
that records no difference is judged on policy alone, which is the case for
every role whose allowlist already equals what its upstream serves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
tools.allowlist meant two things at once: the surface a consumer may reach,
which is a decision of ours, and the surface the upstream offers, which the
vendor changes without asking. Conflating them made a version bump read as a
contract violation, and left no place to record what was actually measured.
Split them. allowlist stays policy; upstream_serves records the observation at
supported_version. State the second only where it DIFFERS, because a provider
that serves exactly what is allowed would otherwise carry the same list twice
and the copies would drift apart. Absence means identical.
The difference between the two measures unenforced exposure: mattermost 103,
gitlab 2, and zero wherever the fields collapse. A non-empty difference is what
tells us a provider needs a gateway in front of it.
Require a non-empty allowlist for an enabled, non-adapter, server-capable
provider. An empty one used to make the probe's comparison vacuous, which read
as a pass while every tool the upstream serves stayed reachable. Nextcloud was
in exactly that state.
Three exemptions, all semantic: an adapter authors its tools here against a
REST upstream, so there is nothing to observe; a provider declared
enabled: false is never admitted; and upstream_serves: dynamic covers an
upstream composing its surface at runtime, which nextcloud does by proxying
MCP servers an administrator configures.
Reject an allowlist naming a tool the upstream does not serve. That found
mattermost's add_user_to_channel, gone since 11.10.0 renamed it
add_channel_member, which would have been advertised and never callable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The adapter could only translate REST to MCP, so a provider whose upstream
already serves MCP tools had no way to sit behind it. Those providers are
exactly the ones with no enforcement point: their declared allowlist is a
statement about the upstream rather than a constraint on it, and a client
can reach any operation the upstream happens to serve by naming it.
Add the policy layer for that case. An MCP upstream exposes no HTTP method,
so a tool's blast radius cannot be inferred the way the REST layer infers
it from GET/HEAD; every tool must declare whether it mutates, and one that
does not say is refused at startup rather than assumed harmless.
Forwarding builds its own JSON-RPC envelope with a fixed method and a fresh
id. The client's envelope is never passed through, or a caller could reach
another method by nesting it in params. A single SSE frame is unwrapped,
because a streamable-http server may answer one call that way and plain
JSON parsing would choke on the data prefix.
Routing is deliberately total: a contract that omits upstream_kind is a
REST one, which is what every adapter renders today, so the ten existing
instances are untouched.
Report an upstream that grew a tool instead of filtering it forever, and
one that dropped a contracted tool, so drift is visible either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
swarmkit's default failure_action is pause, and the updater returns early
on a paused state without ever reaching its completion branch, so a
service that latches never leaves that state on its own. The gate every
role runs read replica counts and nothing else, so a latched update looked
exactly like a slow one and burned the full retry budget before failing
without a cause. That is what turned one corrupted openclaw task into a
25-minute silent timeout.
Read UpdateStatus before comparing replicas. The order carries the fix: a
rolled-back service reports N/N of the PREVIOUS spec, so a replica check
alone reports it as converged. Guard the format against a service with no
update in flight, since a bare .UpdateStatus.State nil-pointers and would
fail the whole multi-service inspect.
The handler has to honour the give-up or the gate keeps polling. It stops
only on the explicit give-up and still retries every other exit code, so a
timeout returning 124 does not turn into a failed deploy. That deviates
from wait_converged.yml, which stops on anything but 1; the deviation is
deliberate.
Stop treating rollback_completed as success in the per-service probe. It
is inert today because nothing here issues a rollback, and it is the guard
that keeps a silent revert from returning as a green deploy.
failure_action is deliberately left at the default. Rolling back would
revert a failed image bump while reporting success, and a daily cron opens
those bumps; it would also place the old binary against a schema ~28 roles
migrate on start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two callers polled the same stack_ready.sh with different budgets: the
handler read swarm_stack_converge_retries while post_deploy hardcoded
150. Eleven roles had set the override and were unprotected at the
second caller anyway. Both now read one SPOT in group_vars.
The gate also had no way to give up. A crash-looping espocrm burned 600
polls (20 min) before failing, while flowise was killed 45 s short of a
409 s image pull. has_fatal_task now exits 2 - and both callers stop at
once via 'until: rc in [0, 2]' - when a slot's newest attempt is
Rejected or Pending, or when a slot burned three attempts after the
service churned past swarm_stack_fatal_grace. Patience only grows where
nothing is broken.
report_tasks loses its 'awk !/error=$/' filter, which printed only rows
that HAVE an error and so stayed silent in exactly the no-error case
that occurred.
Measured on a live single-node swarm: a service failing 3x and
converging on the 4th returned rc 2 at t+15s before this change and rc 0
at t+32s after; a 3-replica stop-first rolling update gave 68/68 samples
rc 0; a permanent crash loop still reaches rc 2. The deploy epoch comes
from 'docker service inspect .UpdatedAt.Unix' rather than a /tmp marker,
which was reproduced contaminating a later deploy.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The hourly sweep fought the rest of the CI control plane instead of
helping it: it preempted itself (17 of the last 29 runs concluded
cancelled 1-2s after their successor was created, for 8s of work behind
a 74-80min queue wait) and shipped MAX_ATTEMPTS=0, which the script
reads as no ceiling while its own README promised the opposite. The 24h
age cutoff could not bound it either, because every rerun refreshes the
run's updatedAt and so keeps it inside the window.
Removes the workflow, the two scripts it was the sole consumer of, its
unit test, and the CI_RERUN_* documentation that would otherwise point
at a deleted file. The workflow is disabled in both origin and fork so
no sweep fires before this lands.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A retrigger carried the workspace and instructions toggles over from the
run title, so a suite that had already gone green was dispatched again
and spent runners on work with a known answer.
Derive each toggle from the source run's job record instead: a suite is
turned off only when every one of its jobs reports success. A suite that
was skipped, cancelled or never dispatched has not passed, so it is still
asked for; absence of a failure is not evidence of a success. One failing
shard keeps the whole suite on.
Match against the job path rather than the run title by dropping the
workflow-name segment, and fail open when nothing matches, so an unknown
job layout re-runs the suite rather than silently skipping it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The provisioner probed a stored token first, so a token that outlived a
password change kept the run green while the deployed password no longer
authenticated. Authenticate with the inventory password first and consult
the stored token only once that succeeds; when it fails, converge the
password onto the existing user.
The write targets /Users/Password?userId=<id>: omitting the query
parameter retargets the caller's own account, and an administrator acting
on another user is why no current password is required. Accept 200 as
well as 204, matching the sibling calls; demanding 204 alone would have
failed a successful convergence.
The repair branch is only reached when authentication failed and always
writes, so it now reports CHANGED. It previously printed UNCHANGED
whenever the user already existed, which told Ansible nothing had changed
while a credential had just been rotated.
configure-auth.sh no longer continues past an administrator it cannot
authenticate. Both call sites logged a warning and carried on, finishing
green with SSO and LDAP silently unconfigured.
Adds the role's first unit tests, covering token reuse, token staleness,
the convergence write and its rejection path.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Under swarm an unpinned role's volumes are rewritten onto the shared NFS
export while its services run one task per node. A role that declares no
RDBMS keeps its state in-process, so several replicas then write the same
SQLite file, Mnesia store or search index over NFS and corrupt it. This
already cost a deploy: openclaw ran three replicas against one shared
export and its state directory answered "file is not a database".
The existing unpinned-no-engine-volume lint states the right rule but
detects state through a hardcoded path list, so app-specific paths slip
through and each case gets pinned reactively after it fails. Add a lint
that joins three facts the roles already declare: the volume is NFS
backed (asked of the production swarm_nfs_backed, not re-derived), it is
mounted read-write, and the role runs no RDBMS. Share-safe payloads carry
a justification marker.
Replicas resolve service-first and fall back to the primary entity, and
a role whose swarm mode is disabled is skipped; both cases exist in tree.
Also correct the requirement and the design doc, which still described
the per-volume opt-in model that b09598dc8 replaced with placement
derived backing. Requirement 023 contradicted itself: line 76 already
described the implemented behaviour.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
