journalctl -n 10000 cut the window a CI job actually spans. Measured on a
busy host the journal runs at ~56 000 lines per boot-hour, so the cap
covered roughly the last ten minutes of a ninety-minute job and the
incident had scrolled out before the snapshot was taken. Capture the boot
in full, and give err its own file: priority is journald metadata rather
than text, so grepping the warning dump recovers only a third of it.
systemctl list-units calls a forking unit active/running while its guessed
main process is gone, and the containers carry no ps, so the unit table
alone cannot say whether a daemon is still there. service-state.txt adds
MainPID, NRestarts and ExecMainStatus per unit. sockets-proc.txt reads the
kernel tables directly for the containers that ship no iproute2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
occ prints 'Could not download app deck, it was not found on the appstore' for both a catalogue that carries no release and a catalogue it never reached. The filter matched only that sentence, so a cURL error 97 -- 'cannot complete SOCKS5 connection to garm3.nextcloud.com' -- classified as unavailable, which 01_install.yml:38-40 tolerates by design.
So the deploy went green without deck. addons/deck.spec.js then waited 946s on an app-shell locator that could never appear, and the job failed 53 minutes after the actual error, on a 404 page with an authenticated session. All seven install attempts had failed the same way, across 15 minutes of the retry loop.
unavailable is now only claimed when the catalogue was actually read: a transport marker in the same output disqualifies it, and failed_when fires where the error happens. The retry loop is untouched -- until: gates on ok, which this does not change, and failed_when only ever sees the last attempt. A passing sibling job proves the difference: its first deck attempt died on garm2 after 2m00s, its second returned rc 0 with 'deck 1.18.3 installed' 18s later, and nothing about that changes.
The filter had no tests at all. Seven now cover both directions, including that a reachable appstore without a release stays skippable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second glyph was emitted only for the modes that carry a tor axis, so host rows were the only ones reading with a single marker. LOCAL_GLYPH fills that position. 3.5 was already the glossary's symbol for host, bound to the word test_host, and symbol_glossary refuses to import when two words share a glyph -- so the axis binds the existing definition instead of writing the character out and stepping past that guarantee.
The variant now joins with # instead of a space: a display name and its shard were two whitespace-separated tokens, and decode_list splits on whitespace, so 'tor 1' parsed as two names. One label, one identifier.
encode builds it, _VARIANT_SUFFIX strips it and LABEL_RE reads it back -- all three now go through the same constant, which is what keeps the parser from drifting away from the builder it is the inverse of. Legacy titles are deliberately not accepted: a second tolerated format is exactly that drift. An old title now leaves the shard inside the name and decode returns None, so the job reads as unrecognised rather than as the wrong role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The email SPOT turns tls off for an .onion host -- Tor already carries the encryption and no authority issues a certificate for a v3 address -- and env.j2 renders TLS_FLAVOR=notls from that same value. Mailu's front then opens no implicit-TLS listener, while compose kept publishing 465, 993 and 995 anyway.
The swarm reachability probe reads the published set straight out of docker service inspect and demands a listener per port, so it failed on exactly those three while 25, 110, 143, 587, 4190 and 8007 answered. The stack was advertising endpoints it does not serve; the probe was right.
A role-local container_ports_mailu builds the block instead: protocols come from MAILU_PORTS, so declaring a port in the role meta is enough, and the implicit-TLS three drop out when tls is off. The terms are assembled in Python because a lookup call cannot spread a built list into positional terms -- writing the set out twice, once per TLS flavor, would let the two copies drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node identity is minted during the first distro leg's deploy, as root inside the DiD on the bind-mounted workspace, mode 0600. The workspace survives between legs, so the second leg's 'pack the repo once on the controller' runs tar as the runner user and aborts on all three key files with EACCES -- 13 seconds into the play, before anything of the role under test has run.
Five swarm jobs of run 31864448027 die exactly there: svc-db-postgres, svc-db-openldap, svc-db-redis, web-app-matomo and web-svc-cdn. First leg 3313-5778s and green, second leg 125-140s and dead, whichever distro it happens to be. The remaining swarm failures are not immune, only shadowed: they die in their first leg for unrelated reasons and never reach a second.
Ownership follows the tree, not the process that happened to mint it. The repair runs on the reuse path too, where ensure_node_onion returns early and a root-owned key set from an earlier run would otherwise never be corrected. Modes stay 0600. lchown so a symlink cannot redirect the change, and a path that vanishes mid-walk is skipped instead of aborting the provisioner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every clearnet matrix row carried disable=tor, including the row whose own app is the tor provider. init.py resolves that token to svc-net-tor, prunes it from the primary list and aborts with 'All primary apps disabled', so the job died at rc=1 having deployed nothing: the rescue snapshot captured 0 containers and 0 services.
Blanking the disable was not an option. disable is the only lever the deploy reads (tor: false reaches nothing but a glyph, a dedup key and a concurrency group), so a blanked row would have been byte-identical to the tor row while reporting a clearnet check. The provider's onion state is not a row property, so its rows now yield no clearnet state at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uri reports a connection error as status -1. The readiness gate only had an upper bound, so -1 satisfied it and the retry loop settled on the first attempt against a service that was not up yet - the suite then ran against a dead endpoint instead of waiting for it.
Same fix as 7511852bd1 on feature/svc-net-tor, found independently; that branch is not merged here and carries it alongside unrelated onion changes and a diverged variable naming. This adds the regression test it lacks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The applications: host_vars block is the config namespace keyed by role id, so it also carries system services (sys-ctl-hlth-csp, sys-svc-container). Those declare system_service_id and match no application role, and the resolver died on them instead of reporting their absence.
role_name now raises RoleNotFoundError and exits 3, distinct from argparse's usage error (2) and from a genuine failure (1). run_subprocess grew ok_returncodes so a caller can accept an expected non-zero status while every other failure still surfaces, and resolve_role_path returns None for that code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The negative control probed tools/list, which gitea answers with 404 "Invalid session ID" when no session exists - indistinguishable from a real refusal, while the same server serves every caller that runs initialize first. An open endpoint therefore passed as guarded. initialize is the only method that needs no session, so it reaches the auth layer.
A 4xx now refuses on its own regardless of the body; previously a clear HTTP refusal without a JSON-RPC body was classed unreachable and kept the caller retrying. 5xx stays retryable.
The unauthenticated handshake no longer leaks its session id into the authenticated pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CONFIG_INPUTS lost modes, sequencing and mode_fail_fast when the deploy
modes collapsed into one chunked sweep, but two mirrors of it did not
follow.
The --run help text enumerates what a retrigger carries over, so it was
telling the operator about three inputs the dispatch no longer has. The
job-log fixture described a run recording mode_fail_fast, a state the
dispatch can no longer produce, which left the log half of
config_from_run asserting on an input nobody writes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
report_failures matched job names against a raw role id
((?:web-app|web-svc|svc|sys)-[a-z0-9-]+), but job titles have carried
display names for a while, so the regex matched nothing and every failed
deploy on main went unreported. runs.py had the same shape and stopped
at the first mode glyph, which swallowed a caller path like
'z / Host / sys-front-proxy' into the role name.
Both now go through axes.parse_label, kept next to the function that
builds the label so the two cannot drift, and resolve the middle through
the display-name codec. The parser anchors on the LAST mode glyph so a
reusable-workflow caller path carrying one of its own cannot capture the
role.
The onion state is part of what identifies a job, because a priority row
runs the same mode and variant twice; artifact_name takes it so the
reporter looks for the artifact the run actually uploaded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One discovery query per run replaces one per mode, so the row basis, the
ranking and the budget cut all speak the same role#variant unit. The
result is deployed in serial chunk blocks rather than three parallel
mode matrices.
Chunk size is derived, not pinned: GitHub cancels a job queued for
INFINITO_CI_QUEUE_HOURS, so with INFINITO_CI_CONCURRENCY runners free a
chunk holds concurrency * floor(queue_hours / the deploy job's
timeout-minutes) rows. Assuming every job burns its full timeout is
deliberately pessimistic -- no job can outlast it -- so the drain
estimate can only overshoot.
The run job cap bounds the sweep as a whole; rows beyond it roll into
the next sweep, whose regular line starts at an offset so consecutive
sweeps walk the whole list instead of re-testing the same head. Priority
rows lead every sweep, never move with that offset, and force a chunk
boundary at the seam so a chunk is either all priority or all regular.
sequencing.py goes with the mode lines it ordered: chunking restarts the
queue clock at a controlled size, which is what serialising the modes
was approximating with three buckets of arbitrary size.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant bundling packed several variants onto one runner and made the
job count depend on storage sums. Every variant now gets its own matrix
row instead, and the two things that used to be fixed per workflow --
which deploy mode a row runs in, and whether it sits behind the node
onion -- become axes assigned per row.
Both axes are a deterministic rotation over the row's position and the
sweep number, never random, so a red job reproduces by re-running the
same sweep. Tor turns on sweep//2 so it does not flip in lockstep with
the mode: a row walks all four mode/tor combinations over four sweeps
instead of only two.
A role offers at most two modes in practice, since swarm requires its
own stack and host requires the absence of one.
Drops with the bundler: limits.py, the variant_bundle env handlers,
INFINITO_VARIANT_BUNDLE_SIZE/_MAX_STORAGE, the per-role
meta/tests.yml variant_bundle_size key and its only user, and the
ComplexityRow bundles/jobs fields with the --deploy-mode and --max-jobs
flags that consumed them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
67b6072bb tightened this test's assertion from "Skipping distro|budget
exhausted" to the headroom line alone, and in the same change shrank the
command and budget from sleep 3 / 5s to sleep 2 / 3s. That left the branch
reachable only when the first iteration measures exactly 2s.
distros.sh times with `date +%s`, whole seconds, so a measured duration
carries up to a second of boundary rounding on top of bash startup and two
`source scripts/meta/env/load.sh`. Run 31823654078 booked `sleep 2` as 3s,
took the `remaining <= 0` branch instead, and never printed the line the
test greps for:
>>> Duration: distro=debian took 3s (max_seen=3s)
[WARN] Global budget exhausted (remaining=0s). Stopping further distro runs.
It is a race and not a defect in the script: on the same commit manjaro and
centos both passed base/06_test.sh and moved on to 07_console.sh, and only
ubuntu lost it, twice within the one job.
Reaching the branch needs d < B < 2.15*d for a measured first duration d.
At 2 / 3s the window admitted d=2 alone. At 4 / 7s it admits 4, 5 and 6 -
three seconds of slack against the one second CI actually overshot by. The
assertion is unchanged, so the branch is still genuinely exercised rather
than relaxed, and the suite pays two more seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
508669809 booked a budget SIGKILL as skipped so the job went green. That is a
soft-skip: a kill is indistinguishable from the command hanging, so it hid a
real failure behind a green tick.
Fail on it again, and prevent it upstream instead. A distro now starts only
while the remaining budget covers the slowest run so far plus 30%. Fediwall run
31756400129 shows why the bare comparison was not enough: max_seen=6770s against
remaining=6878s let the distro start, and it took 6942s, 64s past the deadline.
max_seen is the maximum of the durations already observed in the same sweep -
a small sample over distros of genuinely different weight, drawn in randomised
order, so it underestimates systematically.
Removing the soft-skip also removed its diagnosis, so a killed distro now
reports both numbers without claiming a cause: the run only knows the exit code
and how much budget was left when it started.
The suite drops from 85s to 4.1s. The 68s case made a process ignore SIGTERM to
force the -k escalation, which exercised coreutils rather than this script; the
124 path reaches the same branch in a second.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ad9222630 wrapped each distro in timeout so a run that outlives the budget stops instead of drifting into the action timeout, and mapped rc 124 to skipped because stopping at the budget is the documented intent rather than a defect. It verified that timeout returns 124 for its own kill. It did not cover the other one: timeout -k escalates to SIGKILL when the command survives SIGTERM, and then reports 137, which is equally its own kill. An ansible run with live docker children survives SIGTERM routinely, so 137 is the common case rather than the exotic one.
The result was that the governor never fired as documented. Fediwall swarm and Nextcloud swarm 0 in run 31756400129 and 31756560361 both ended on [ERROR] Run failed for distro=arch (rc=137) with budget=20400s remaining=-60s. Fediwalls playbook had already printed PLAYBOOK RECAP, the stack had converged and every node port answered; it was booked as a failure 0.3 seconds later.
The two narrowing conditions carry the discrimination and are unchanged. dur >= remaining - 5 rejects a SIGKILL that arrives at an arbitrary time, such as an OOM kill mid-run, because a budget kill lands at the deadline by construction. passed > 0 still requires an earlier distro to have succeeded.
A unit case in tests/unit/scripts/tests/deploy/test_distros.py drives a distro that traps SIGTERM and sleeps. Without the change it reproduces the CI failure locally, exit 137 and Run failed for distro; with it the sweep exits 0 and the summary marks the distro skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle renders the same msmtprc as WordPress and mounts it into two services,
but nothing ever proved a message left the container: its healthcheck opens the
FPM port and nothing else. Moodle 0 of run 31658241728 passed green while mail
could not have worked -- the defect 2da285067 fixed for WordPress was invisible
here, and would have stayed invisible until a user reset a password.
WordPress proves delivery with the msmtp_curl flavor: the probe sends one mail
and writes a marker, and a task after convergence waits for that marker. Moodle
cannot use it, because curl cannot judge a port that answers FastCGI. Rather
than add a second msmtp_* class per liveness shape, the declaration now composes:
healthcheck:
flavor:
- msmtp
- connect
port_key: fpm
msmtp stops being a flavor and becomes a prefix -- shell that runs on the way
past and never decides liveness. connect joins the probes as the one that only
asks whether the port accepts a TCP connection. A declaration names exactly one
probe and any number of prefixes, as a string or a list; msmtp_curl stays as an
alias for [msmtp, curl], and the lint that validates declarations now shares
resolve_flavors with the lookup instead of keeping its own idea of a flavor.
The module became five responsibilities in one file, so it is a package:
probes.py answers which probes exist, prefixes.py what runs before them,
compose.py how a declaration becomes a rendered block. The curl helpers sit with
the probes and mail_branch with the prefix -- each next to its only caller,
rather than together in a module header.
Two defects in this work were found by measuring, not reading:
- the timing merge took the maximum across prefixes only, which cut Moodle's
start period from 15m to 120s and would have marked a heavy PHP app dead
while it was still installing. A prefix may now raise a timing and never
lower one, pinned by test.
- composing moved msmtp_curl's interval from 30s to 1m, because the composed
class derives from Curl where MsmtpCurl derived from Probe. That is the
interval every other curl probe uses, but it was not intended; it now has
its own test rather than riding along unseen.
Regression measured across the whole repo rather than argued: every declared
healthcheck was rendered with the pre-refactor code and with this one. 58 render
byte-identically, 1 is Moodle's intentionally changed declaration, and 2 differ
-- friendica and wordpress, in that interval alone, with the probe command
itself unchanged.
Coverage was checked in both directions. No test was removed: the four deleted
lines are an import, the registry literal, one class name in a url test, and the
helper, which now goes through compose() instead of reaching into PROBES -- a
tightening. New code had two gaps, both closed: connect was the only probe whose
emitted command no test pinned, and compose() raised KeyError for an unknown
name unobserved.
Not verified: no deploy has run any of these probes. The composed class is built
dynamically, so what it emits is proven for all 59 services while docker
accepting it is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every probe in this module targets 127.0.0.1 inside the container, so a request
a proxy picks up is wrong by definition. The Wget probe has said so since it was
written -- `wget --spider --proxy=off` -- while the four curl invocations said
nothing, and a container whose environment carries http_proxy would have sent
its own liveness check through it.
That environment is not hypothetical: 2d6a5fd18 puts http_proxy and https_proxy
into EspoCRM's env_file so the OIDC back-channel can reach an onion issuer. It
also sets no_proxy=localhost,127.0.0.1, so that role's probe is covered by hand.
The next role to route a container through a proxy and forget that line would
have had a healthy web app reported dead.
The flags now come from one place. CURL_NO_PROXY plus curl_argv() build the exec
form, and curl_shell() is the same argv run through shlex.quote. The two
spellings differ in exactly one character -- the exec form needs `*` bare
because docker runs no shell, the CMD-SHELL form needs it quoted or the shell
globs it -- and that difference was previously written out by hand at three
sites plus a fourth in a different shape. Deriving one from the other is what
keeps them from drifting; the Host-header branch moved along for the same
reason, since it too existed twice in two spellings.
The http fallback keeps its wget half untouched on purpose. That flavor exists
for images where it is unknown which tool ships, and busybox wget does not
accept --proxy=off; adding it there would have made the first branch fail on an
unknown option for exactly the images the fallback is for.
Verified by execution at each step:
- adding the flag broke five tests, which is every call site plus the hostname
variant -- the suite pins the argv and the shell string literally, so that
was the completeness check
- the refactor then passed those same 31 tests with no further edit, and the
four rendered probes were diffed against the pre-refactor output character
for character
- one of those tests executes the CMD-SHELL probe against stubbed msmtp and
curl, so the quoting survives a real shell rather than just a string compare
- the option is accepted where it will run: joomla:6.1.2-php8.3-apache, one of
the images configured for this flavor, ships curl 8.14.1 and answers
`curl -f --noproxy "*"` with exit 7 (refused, nothing listening) rather than
exit 2 (unknown option)
Not verified: the other images behind this flavor. --noproxy has existed since
curl 7.19 in 2008, but only one image was measured, and no probe built this way
has yet reported healthy in a live container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
extra_hosts is the one compose key a service may not carry twice. The onion SSO
back-channel pin was emitted by a Jinja partial that six roles included, while
six other services wrote the key by hand; nothing stopped a service from
receiving both. YAML then either rejects the document or keeps one list and
drops the other, silently.
The partial becomes plugins/lookup/container_extra_hosts.py, and every pin now
goes through it. Callers add their own entries with extra_hosts=[...], the way
container_volumes already takes extra_volumes=, and the lookup merges them after
the automatic pin and deduplicates in first-seen order. With nothing to emit it
returns an empty string.
The swarm address reasoning from 7b1839681 is preserved and now written down in
the plugin rather than only in that commit: the pin must NOT be host-gateway in
swarm, because that alias resolves per node while openresty publishes :80 with
mode: host under node.role == manager. It carries the stack host's address,
which reaches every task because the compose file is rendered once with the
manager as inventory host. The mode is read as compose_mode_force or
DEPLOYMENT_MODE, not the bare cluster mode, so a role with an override does not
lose its pin and its overlay at once.
Two roles kept a pin that now collides with the automatic one, and both are
resolved the same way -- the hand-written entry stays for clearnet and yields
the .onion case to the lookup, which picks the address per mode. In
web-app-mailu the three conditional entries move into a role-local
mailu_extra_hosts lookup exposed as MAILU_EXTRA_HOSTS, because an eight-line
conditional expression in the template argument is worse than the block it
replaced. It returns its entries as separate results, so vars/main.yml uses
query(); plain lookup() would join them into one comma-separated string.
tests/lint/ansible/jinja/test_compose_extra_hosts_use_lookup.py fails on any
literal extra_hosts: key in a role template, so the invariant survives the next
author who needs a pin. It follows test_compose_ports_use_container_ports_lookup:
the raw construct is forbidden, the lookup is not mandated per service -- a
service with no pins is correct without it, and the swarm branch raises when
facts are absent, which would turn a no-op into a failure surface for containers
that never speak OIDC.
Equivalence checked by rendering rather than by reading: under the trim_blocks +
lstrip_blocks the stack-host action forces for compose.yml.j2, both former call
site shapes (indented, and matrix's column 0) emit byte-identical text to the
new form when the block applies. The no-op case differs -- a whitespace-only
line where the include emitted none -- and yaml.safe_load returns the same
document for both.
31 unit tests; make test-lint 325 passed. Does not touch web-app-espocrm, whose
missing pin is a separate claim and gets its own commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
container_hostname is computed for sethostname(2) and clamped to 63 bytes, so a
v3 onion domain degrades it to the bare entity name. container_healthcheck
forwarded that value into the curl probe's Host header, and curl is the only
flavor that asserts a vhost identity on a loopback probe -- tcp and http_status
hardcode Host: localhost, wget and http send none. An app that validates Host
therefore rejected its own healthcheck: bookwyrm's Django answered 400 to
Host: bookwyrm and the container never turned healthy while serving fine over
its onion.
This was diagnosed once already. e9817f825 measured it A/B on pretix
(Host: pretix -> 400 x4 unhealthy, Host: localhost -> healthy), named reusing a
kernel hostname as an HTTP Host the underlying design bug, and deferred the
general fix because it would move six roles' probes at once. 9fb08b4ed then
pinned bookwyrm the same way. This is that deferred change, so there is no
third round of per-role pinning.
The header now comes from meta/services.yml and nowhere else. Of the eight
roles defining container_hostname, bookwyrm and pretix were already pinned and
bridgy-fed and postmarks use the tcp flavor, which never reads it. Of the
remaining four, chess never passed anything but its entity name and xwiki was
measured healthy while answering to Host: xwiki, a name in no vhost config; so
both are safe without a header. confluence and jira are pinned here rather than
argued about -- their Host is today the real domain on a clearnet deploy, and
Tomcat's ATL_PROXY_NAME being a proxyName and not an allow-list is reasoning,
not a measurement.
The templating contract moves from the play var to healthcheck.hostname, and a
new test fails if the fallback ever returns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compare_application_keys skips generated credentials because no default exists
for a secret, but it recognised them by the pre-rename prefix. Once the
generator started writing them again, every credential arrived as
secrets.credentials.<key> and the exemption stopped matching, so validation
rejected three keys per credential -- the leaf and both intermediate nodes,
which recursive_keys emits separately -- and aborted the deploy before the
playbook ran. The condition now admits the bare secrets node and the
secrets.credentials subtree, leaving a sibling such as secrets.other flagged.
The check in the inventory manager stays on the bare prefix on purpose: it
walks meta/secrets.yml, whose own root key is credentials.
test_credentials_keys_are_skipped fed the validator the old shape, so it kept
passing while real inventories were being rejected.
Evidence: run 31655559799, job 94311023723,
"Missing default for web-app-dashboard: secrets.credentials.sso_proxy_cookie_secret".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The four comment lines above the assertNotIn block restate what those three
assertions state executably, and unlike them they cannot fail when the claim
stops holding. test_comments_valid lints only changed targets, so the block
stayed invisible until the previous commit made the file one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_apply_vars_overrides_from_file_deep_merge_and_overwrite sat at module
level while still taking self, so pytest treated self as a fixture request and
errored the test out at collection instead of running it. The assertions it
carries for apply_vars_overrides_from_file have therefore never executed.
Re-indenting it into HostVarsTestCase makes it a real test again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The host_vars merger still wrote applications.<app>.credentials and read
"credentials" out of the generated snippet. f713ba835 moved the emitter to
secrets.credentials, so the read resolved to {} on every app: nothing was ever
copied, and _ensure_ruamel_map left an empty credentials: {} shell behind as
the only trace. Every credential generated since that commit was discarded on
its way into the inventory.
Both sides now go through credentials_map() and declared_credentials(), the
accessors the emitter already uses, so the two ends cannot drift again.
_ensure_ruamel_map was a copy of emit.ensure_map and is gone.
The failure surfaced in two shapes from the one cause. Templates reading
through lookup('config', ...) raise, because get() defaults to strict:
"Key 'secrets' not found in dict at 'secrets'" for
web-app-mastodon.secrets.credentials.secret_key_base, with the dumped payload
showing 'credentials': {} and no secrets node. Templates reading through
lookup('database', ...) stay quiet, because that plugin passes strict=False
with default="": friendica's MariaDB received an empty MYSQL_ROOT_PASSWORD and
refused to initialise.
The generator's own tests pinned the pre-rename shape in both the snippet and
the assertion, which is why they stayed green through the regression.
Evidence: run 31649221474, job 94303863545.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
f713ba835 moved credentials to applications.<app>.secrets.credentials and
updated every role that reads them, but four lookup plugins kept addressing
the bare credentials.<key> root. They call get() with strict=False and
default="", so the miss was silent rather than loud: MariaDB received
MYSQL_ROOT_PASSWORD= and refused to initialise, which took friendica-database
down as an unhealthy dependency and failed compose up. Object-store consumers
and shared redis auth resolved to empty secrets for the same reason.
The five call sites now build the path from OVERRIDE_SECTION, the single point
of truth that commit introduced, instead of spelling it beside it.
The two lookup unit tests pinned the pre-rename fixture shape, which is why
they stayed green straight through the regression; their fixtures move to
secrets.credentials so they fail against the old code.
Evidence: run 31649386343, job 94302347256, artifact
rescue-diagnostics-instructions-web-app-friendica,
containers/infinito_nexus_debian/nested/.../friendica-database.log:29
"Database is uninitialized and password option is not specified".
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
test_distros_spot globs **/*.md and friends across the whole project root and
reads every hit, so a named pipe, socket or device node with a matching suffix
crashes the check with PermissionError instead of reporting an offender. It now
skips anything that is not a regular file.
test_no_uncached_file_reads flagged three reads in the inventory reset tests.
Its suggested fix - the process-wide cached reader - would be wrong there: each
of those tests writes the file and reads it back in the same test, so a cached
read would serve the pre-write content and the assertion would stop testing
anything. They carry a nocheck marker instead, on the call line and short enough
that the formatter cannot wrap the expression away from it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`cli build include` generated one tasks/groups/<group>-roles.yml per invokable category during setup, and the stage plays included those files. They were written by the in-container UID, which is why a dedicated clean-container-owned helper had to delete them from inside the container.
The stage plays now loop over lookup('group_roles', <group>) and include tasks/utils/setup/{group,role}.yml directly. ordered_roles() in utils/roles/order.py holds the run_after topological sort the generator used to perform at build time.
utils/roles/stage.py:_group_names() no longer globs the generated files; it reads get_invokable_paths(), the same categories.yml source those files were derived from.
Measured: handlers still flush after every role (ansible-core 2.20.2, three loop items, three separate handler runs), and resolving all 18 groups costs 0.60s cold against 1.4s for the generator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Provisioning fills a missing credential and leaves an existing one alone, so every deploy pass reads the value the first pass wrote and the update path is never exercised. Rotating the generated values makes a redeploy carry a secret that moved.
`infinito administration inventory credentials reset` drops every vault-encrypted leaf under applications.*.credentials.* (--schema) and the pinned password of every declared user (--users), then lets the existing generators refill the gaps, so no schema logic is duplicated. Plain credentials stay unencrypted in host_vars and are therefore left alone, as are users that exist only in the inventory and would get nothing back. --except exempts application ids and user keys, --backup keeps the pre-rotation file as <name>.YYYYMMDDTHHMMSSZ.backup, and --mirror realigns the other hosts of a multi-host inventory.
Adds the command only; no deploy path calls it yet. Measured against web-svc-coturn and web-app-flowise: eleven generated secrets move between two runs while ansible_become_password and the administrator password stay pinned.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Friendica 0 and WordPress 0 of run 31543714113 never delivered their startup mail. An onion relay has no TLS certificate, so the resolver forces tls=false - and both port and auth hung off that flag: port collapsed to 25 and auth to off. Port 25 is the inbound MX and offers no SMTP AUTH at all, so the message reached Mailu unauthenticated with a local envelope sender and rspamd rejected it 554 5.7.1. Port and auth now hang off `external` instead: 465 with TLS, 587 without, and auth wherever a relay is external.
Moving the port alone would only relocate the failure. msmtp refuses to pick PLAIN or LOGIN itself on a cleartext link and aborts with EX_UNAVAILABLE, so the mechanism has to be named. The new `auth_mechanism` key resolves to off, on, or plain and sits in RESOLUTION_ORDER right after auth, whose result plus tls it reads.
It is one derived value rather than two template expressions because the two failing jobs run different msmtp instances: WordPress renders sys-svc-mail-msmtp/templates/msmtprc.conf.j2 through WORDPRESS_MSMTP_SRC and mounts it at /etc/msmtprc, while Friendica generates its own from SMTP_AUTH via the image script, whose line 32 writes `auth $smtp_auth` verbatim. Both now read the same key.
Verified against the shipped Mailu 2024.06 front rather than assumed. EHLO on 587 under TLS_FLAVOR=notls answers `250-AUTH PLAIN LOGIN`, so PLAIN is offered on the cleartext link; dovecot renders disable_plaintext_auth=no in that flavor. PORT_587 is set because socrate lists 587 outside PORTS_REQUIRING_TLS, unlike 465, which notls skips - so 587 is the only submission port available there. smtp and submission bind identically to MAILU_IP4_PUBLIC, so the path to 587 is the one that already reached 25.
This supersedes the premise of 563da9e5b, which disabled AUTH because the resolver had. That commit quoted a different msmtp error - "the server does not support authentication" - which was port 25 having none, not the relay refusing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Alphabetical make target order, and the integration test walked the project
tree itself instead of the cached helpers, which share one walk and one read
per file across the whole session.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml became meta/secrets.yml, but its content still landed at
applications.<app>.credentials — the one meta file whose payload did not sit
under a root named after the file. Every other topic obeys that rule:
meta/volumes.yml goes to volumes, meta/rbac.yml to rbac. Credentials now go to
secrets.credentials.
The path exists exactly once, in utils/manager/credential_key.py, and
OVERRIDE_SECTION is derived from its two segments rather than spelled beside
them, so the dotted form used by --set and the nested form used by the
inventory cannot drift. Six consumers read it: the loader, the generator, the
CLI override parser, the two emit helpers, and the four lookup validators.
--set now accepts exactly one key shape,
applications.<app>.secrets.credentials.<key>. The three short forms are
rejected instead of guessed at: the same short key names different credentials
on different apps. That also retired primary_app_id, which only existed to
disambiguate them.
Three defects surfaced while wiring this up. _apply_one_role_schema still wrote
to the old root, so an app's own credentials would have moved while its
providers' stayed behind. web-app-pgadmin reads through
lookup('applications', ...).credentials.<key>, an access path no regex over
lookup('config', ...) could see, and would have deployed with an empty postgres
password. And _META_NON_GENERIC_TOPICS still listed 'schema' — a bare topic
name, invisible to any path search — so the generic file-root loop assigned the
raw schema under secrets and the follow-up write landed inside the cached
document, truncating keycloak's four credentials to one for every later
reader.
tests/integration/ansible/lookups/config/test_credentials_roundtrip.py pins both
directions of the seam and the cache invariant that last defect broke.
Existing inventories still carry the pre-rename root; migrating them is
deliberately out of scope here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A user password is pinned once by inventory creation from a shell-safe
alphabet, because it travels through Ansible, a shell and container exec.
An application whose registration demands a special character then had no
way to say so, and roles worked around it with a second credential beside
the account they already owned. The account now carries algorithm and
validation itself, two roles disagreeing about one account abort instead of
one silently winning, and an unsatisfiable pair fails naming both halves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both tests reached for the filesystem themselves: one built the project root
from Path(__file__).parents[4], which silently points at the wrong directory the
moment the file moves, and both called Path.read_text directly, bypassing the
cache every other reader shares. The role-defaults path was spelled out by hand
where ROLE_FILE_DEFAULTS_MAIN already names it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A retrigger reproduces its source run's configuration, and the tor axis was
missing from it: rerunning a run dispatched with 'exclusive' silently fell back
to 'auto' and tested a different matrix than the one whose failures it was meant
to chase.
The mode is read from the job log's Inputs block, not from the run title. The
title renders it as a glyph, so recovering it there would mean translating a
symbol back into a word on every retrigger; the log already holds what the
operator picked, and inputs_from_jobs reads that same block for the priority
list today. LOG_INPUTS names the inputs that take this route.
That also removes a trap the glyph created. The tor segment renders through
format('{0} ', ...) with no opening literal, and run_name locates a value by
searching for its head - str.find('') matches at position 0, so the parser would
have handed back the title's own prefix as the tor value, plausibly and without
raising. segments() now skips a headless segment; its glyphs stay openings, so
they still terminate the value before them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The onion glyph alone left a job name ambiguous: no glyph meant either
'clearnet' or 'this mode has no tor axis'. Swarm and compose entries now open
with the onion or the globe, so the name states which of the two networks the
job tested. Host keeps neither — it shares the matrix builder but has no onion,
and a glyph there would claim something the job never did.
'enabled' becomes 'enforced' so the manual selector says what it does: tor on
every role that can take one, not tor on everything.
'exclusive' is the new mode. keep_tor_variants runs before assign_tor and cuts
each entry down to its tor-capable variants, dropping entries left with none, so
a run tests the onion and nothing else. A bundle can come out with a hole in it
— variants 0 and 2 capable, 1 not, yields '0,2' — which filter_plan_to_variants
already handles by set membership; both shapes are now pinned by tests.
Note the word is overloaded: services.tor.exclusive drops a deployed app's
clearnet vhost, while this mode only narrows what CI runs. The docstring says so
where a maintainer will look.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/schema.yml held credential definitions and their runtime values, but the
name said 'schema' and read like structural metadata next to services.yml and
volumes.yml. Nothing in the file is a schema in the sense the other meta files
use the word, and a contributor had to open it to learn it holds secrets.
62 role files renamed. ROLE_FILE_META_SCHEMA becomes ROLE_FILE_META_SECRETS and
keeps its role as the SPOT the path is built from.
Two readers bypassed that constant and spelled the name themselves:
utils/cache/applications.py built meta_dir / 'schema.yml' twice, and
plugins/lookup/application_gid.py carries the filename in the marker list it
detects application roles by. Left behind, the rename would not have raised —
credentials would have resolved empty and the role would have stopped counting
as an application.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3a1ef70cb removed the rerun workflow's rows from the actions configuration
page and left two blank lines at the end of the file, which MD012 rejects.
That is what failed the Lint job on 777f7f0fa.
The two Python files carry the reflow ruff 0.16.2 produces; 6320da703
brought that version in with the main merge. Committing it keeps the tree
matching the formatter, since scripts/lint/python.sh runs ruff format in
write mode and would otherwise rewrite them on every run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
force_shared_db flipped every DB service to shared with no exemption, so
web-app-magento's deliberate embedded MariaDB 11.4 - declared shared:
false with a comment warning that no bot may move the major underneath
it - became dead config and setup:install hit the central engine's 12.3.2,
outside Magento's supported range. The run log names the override:
'swarm force-shared: web-app-magento.services.mariadb -> shared=true'.
Exempts a DB service whose role meta declares a literal shared: false AND
whose role is manager-pinned. The second condition is not decoration:
utils/storage/nfs.py keeps a manager-pinned role's volumes node-local, and
InnoDB over NFS is why the harness force-shares in the first place. Of the
two roles with a literal local DB, magento qualifies and bigbluebutton
(unpinned) does not, so its behaviour is unchanged.
derive_includes._force_shared_db_view documents itself as mirroring
force_shared_db, so it takes the same exemption - without it the dep walk
would compute magento's include closure against the central engine while
the deploy runs a local sidecar.
The sidecar carried no deploy block at all, so an exempted service would
have been free-floating with one task per node over a single-writer
datadir. The block now comes from a container_deploy lookup shared with
sys-svc-container/templates/deploy.yml.j2, which replaces the replicas
ternary and the duplicated placement constraint there. Its unit test
caught a real defect while being written: with compose_mode_force=swarm
over DEPLOYMENT_MODE=compose the delegated compose_replicas returned
empty, emitting a deploy: key with no child.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
22 commits, mostly dependabot and docker image-version bumps, plus two fixes:
5153f3030 loads keycloak-js 26 as an ES module, 87a053518 makes the coredns
Corefile readable to the container user.
One conflict, in roles/web-app-dashboard/templates/javascript/oidc.js.j2, resolved
as a union because the two sides touched the same line for unrelated reasons.
edb96e9ae had hoisted the asset URL into KC_JS_ASSET_URL and rewrote it onto the
onion mirror; 5153f3030 changed the path to lib/keycloak.js and declared a local
Keycloak binding, since the adapter now arrives as a module default export instead
of a window global. The onion rewrite stays at the conflict site, the local binding
is kept because loadKeycloakAdapter returns the constructor rather than setting a
global, and the path change moved to the constant's definition.
An import() fetch always runs in CORS mode, which is why 5153f3030 gave the CDN
aca_origin. The onion route reaches the same asset through web-svc-mirror, whose
proxy.conf.j2 already sets that header, so the onion flavor needs no counterpart.
make quality green on the merge result: test-external, test-integration, test-lint,
test-unit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Tor coverage was decided per role variant and nowhere else: 1d9d355bc pins
services.tor.enabled true in variant 0 and false in every other variant, so an
operator had no way to steer it and a job name never said whether it ran behind
an onion. INFINITO_TOR now controls the axis from manual steer down to the
drill, and the entries that get an onion wear one in their label.
assign_tor decides per entry. 'enabled' gives tor to every capable entry,
'disabled' to none, 'auto' to every second one with the run number's parity
flipping which one starts — measured over the full matrix: 397 entries, 257
capable, 129 on even runs, 128 on odd, union 257, overlap 0. Two consecutive
runs therefore cover every capable entry exactly once.
tor_capable reads the entry's own variants, not just meta/services.yml.
Reading only the base config claimed an onion for 131 of 388 entries whose
variant had pinned the gate off — a third of the glyphs would have lied. A
bundle counts as capable when any variant it covers can use tor; a reactive
Jinja flag counts as capable, since that is the role saying 'onion when the
node is dark'.
The 'disable' token is the working half: it prunes svc-net-tor from the include
closure (utils/tests/swarm/derive_includes.py) and clears the service flags in
host_vars (cli/administration/inventory/provision/services_disabler.py). Both
halves are required — with only the first, extend_inventory walks the provider
back in. It runs after the variant overlay, so it also overrides a variant that
pinned the gate true.
Only swarm and compose join the axis. Host discovery shares the same matrix
builder, so without the allowlist its jobs would carry onion glyphs for a mode
that has no onion at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
assert_surface decides which of two different readers a failure belongs to,
and nothing exercised it: contract.py only runs inside the probe container
against a live server, so the logic was verified by reading alone.
The load helper patches the environment before import because the module
reads MCP_URL, MCP_TRANSPORT and MCP_AUTH_HEADER at import time.
Five cases: a matching surface passes; a policy mismatch fails; drift is
reported before policy when the recorded surface is stale; a recorded
difference still fails policy while no gateway narrows it; and a provider
that records no difference is judged on policy alone, which is the case for
every role whose allowlist already equals what its upstream serves.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The adapter could only translate REST to MCP, so a provider whose upstream
already serves MCP tools had no way to sit behind it. Those providers are
exactly the ones with no enforcement point: their declared allowlist is a
statement about the upstream rather than a constraint on it, and a client
can reach any operation the upstream happens to serve by naming it.
Add the policy layer for that case. An MCP upstream exposes no HTTP method,
so a tool's blast radius cannot be inferred the way the REST layer infers
it from GET/HEAD; every tool must declare whether it mutates, and one that
does not say is refused at startup rather than assumed harmless.
Forwarding builds its own JSON-RPC envelope with a fixed method and a fresh
id. The client's envelope is never passed through, or a caller could reach
another method by nesting it in params. A single SSE frame is unwrapped,
because a streamable-http server may answer one call that way and plain
JSON parsing would choke on the data prefix.
Routing is deliberately total: a contract that omits upstream_kind is a
REST one, which is what every adapter renders today, so the ten existing
instances are untouched.
Report an upstream that grew a tool instead of filtering it forever, and
one that dropped a contracted tool, so drift is visible either way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
swarmkit's default failure_action is pause, and the updater returns early
on a paused state without ever reaching its completion branch, so a
service that latches never leaves that state on its own. The gate every
role runs read replica counts and nothing else, so a latched update looked
exactly like a slow one and burned the full retry budget before failing
without a cause. That is what turned one corrupted openclaw task into a
25-minute silent timeout.
Read UpdateStatus before comparing replicas. The order carries the fix: a
rolled-back service reports N/N of the PREVIOUS spec, so a replica check
alone reports it as converged. Guard the format against a service with no
update in flight, since a bare .UpdateStatus.State nil-pointers and would
fail the whole multi-service inspect.
The handler has to honour the give-up or the gate keeps polling. It stops
only on the explicit give-up and still retries every other exit code, so a
timeout returning 124 does not turn into a failed deploy. That deviates
from wait_converged.yml, which stops on anything but 1; the deviation is
deliberate.
Stop treating rollback_completed as success in the per-service probe. It
is inert today because nothing here issues a rollback, and it is the guard
that keeps a silent revert from returning as a green deploy.
failure_action is deliberately left at the default. Rolling back would
revert a failed image bump while reporting success, and a daily cron opens
those bumps; it would also place the old binary against a schema ~28 roles
migrate on start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
