i8mativaso and i8matiso read the compose view, so a row only swarm or host
tests was missing from a matrix that looks like the whole picture: 243 rows
where the run has 256. Both now render i8maal, the union of the three modes,
which is exactly what discovery hands the chunker.
Rows no mode tests at all stay out. Of those 54, fifty-two carry
compose=swarm=host=false and most are pre-alpha or eol: listing them in a
table whose order IS the selection priority would claim a rank for rows that
never reach a chunk.
The per-mode views keep their place; i8maal joins them as the union.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two tokens naming one row is the case the axes exist for: a variant that
failed in compose and in swarm comes back as web-app-x#1@compose+tor and
web-app-x#1@swarm+tor, and both have to deploy. The applier could not express
it. It walked the rows, took the first pin that matched and broke out, so the
second token matched nothing, counted as a typo, and aborted the chunk's
discovery with 'matches no discovered row'.
Two CI runs died on it before the cause was clear, both reporting a token that
was perfectly valid.
Pins and rows are now paired both ways round: every pin that matches a row
yields its own entry. Two tokens that narrow a row identically collapse into
one, so a duplicate in the input cannot become two jobs racing for a single
artifact name, and a bare role name loses to any token that narrows the same
row -- asking for role#1@swarm should not also deploy a rotation-picked
variant of that role.
The old unit test paired two pins on two different variants, which is why it
stayed green; the collision on one variant now has its own.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Both were left by earlier commits and are collapsed by markdownlint-cli2 and shfmt respectively, so every make quality run left the tree dirty and the change would eventually have ridden along in an unrelated commit.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
runs.py stood at 568 lines against the 500-line cap from 4b37e123d. The git and gh helpers were the clearest seam in it: eight functions that know nothing about deploy runs and only drive subprocess and the gh CLI. They move to cli/administration/deploy/ci/gh.py, leaving 482 lines behind. The module is deliberately not named runs_gh.py, which would have formed a prefix group with runs.py and tripped the structure lint.
Re-exporting the moved names from runs.py does not survive the formatter: autoformat strips them as unused imports, and seven call sites in status/__main__.py and trigger/__main__.py would then raise AttributeError at runtime. The call sites therefore reach into gh directly, and the mock targets move with them -- patching a name in the module that no longer resolves it makes a test green without testing anything.
fetch_jobs stays reachable through runs, because find_last_deploy_run calls it internally and test_status.py covers both the URL and the branch path with a single patch. Splitting that name would have left one of the two paths unpatched.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The job titles in these fixtures still separated the variant with a space. Since 4dc89cf17 gave the variant a separator, utils/roles/display.py sets VARIANT_SEPARATOR to a hash and axes.LABEL_RE expects it, so parse_label matched but folded the variant into the role name -- Label(name="web-app-xwiki 0", variant="") -- and display_names().decode() then returned None. Every variant-carrying job dropped out of failed_roles, which is the one job class the function exists for.
That commit updated test_axes.py and test_display.py and missed this file. Real job titles from a run created before it still show the space form, which is why the drift stayed invisible.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
meta/services.yml pinned sso.flavor: oauth2, so an oauth2-proxy fronted the whole role. That gated the storefront too and left /admin behind a proxy the administrator persona could not log out of, while tasks/setup/oidc.yml -- the only OIDC wiring -- installed a literal placeholder plugin name behind an || true and wrote InfxOidc.config.* keys that no plugin reads. tasks/03_setup.yml and tasks/04_cleanup.yml were included by nothing, so none of it ever ran.
The flavor moves to oidc, which drops the proxy on its own because sso_proxy.yml keys off is_proxy_gated: the storefront becomes public and /admin authenticates through heptacom/shopware-platform-admin-open-auth against Keycloak. The plugin is baked into files/Dockerfile beside the existing composer require, installed and activated by tasks/setup/oidc.yml, and its Keycloak client row is written idempotently by files/ensure/oidc_client.php over the Admin API -- GET by provider, then PATCH or POST -- because the plugin keeps its configuration in the heptacom_admin_open_auth_client DAL entity, not in system_config.
The web service gains container_extra_hosts. With the proxy gone the token exchange runs inside the shopware container, which under swarm resolves clearnet only and could not reach the keycloak onion.
The LDAP plugin tasks are deleted rather than wired. No admin-LDAP extension for 6.7 is installable unattended -- the only real one is a paid Store extension needing a licence token -- and arming tasks/03_setup.yml would have fired the INFX_LDAP_PLUGIN placeholder on every deploy that ships OpenLDAP, since meta/services.yml derives ldap.enabled from group membership. Directory identities reach the backend through the Keycloak LDAP federation instead.
PERSONA_BIBER_BLOCKED becomes unconditional: biber has an account in no variant. test-admin-native.js keeps the authenticated journey and now drives the AdminOpenAuth provider link when SSO is on.
Verified by execution: the provisioning payload and both idempotency branches against a stub; composer resolving 9.1.0 against shopware/core 6.7.13.0 without conflict; the install gate rejecting a no-plugins-found run and accepting both success shapes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/__init__.py makes roles a Python package, so every pytest run leaves a roles/__pycache__ behind. The required_by coverage lint walked ROLES_DIR.iterdir() unfiltered and reported that cache as roles/__pycache__/meta/services.yml (missing), i.e. as a non-invokable role lacking required_by.
Enumeration now skips names starting with . or _. The only such entries under roles/ are .gitignore and __init__.py, both files that is_dir() already excluded, so no real role leaves the check: 256 -> 255 directories, the single loss being __pycache__. With the cache present the lint still fails on an injected roles/sys-svc-covtest/ and lists only that offender.
A sibling fix for the same class lives on other branches as 38c8b59fe, covering tests/integration/test_roles_have_meta_main.py and test_roles_naming.py -- different files, no overlap with this one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The converge gate polls STACK_NAME_PRIMARY_SERVICE_KEY. Four roles never
create that name and each was given a static `workload: node-local` to make
the gate skip itself:
21fe9e984 web-svc-cdn ships only cdn_build, never cdn_cdn
2c6c8b763 web-app-matrix names its services matrix_synapse and friends
76508c946 svc-runner deploys node-local compose projects per node
cb52a72f2 bigbluebutton phantom service, and the DR drill must still run
Four hand-written claims for one observable fact. The claim is also static
where the truth is not: web-app-discourse renders `services: {}` only while
its sidecars are shared, and job 94965456921 of run 31864448027 polled
discourse_discourse ninety times for a stack that had none. Declaring the
role node-local would have blinded the gate in the variants where its
sidecars are local and a stack does exist.
has_swarm_service asks the deployed stack for the exact name instead. Every
caller runs behind `if rc == 0`, so an absent service is the role's shape and
not a broken deploy. Exact-line matching, not docker's name filter, which
matches substrings.
No coverage is lost: 03_wait_converge cannot pass without that service, and
05/06/07 plus the backup drill only run once it has. The four roles keep the
skip they were given; discourse and every future case get it without a
declaration. The teardown branch in the drill keeps its meaning - stack rm
where the service exists, compose stop per node where it does not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
n8n defaults N8N_SECURE_COOKIE to true and then refuses to render its sign-in
form on a non-https origin, showing its own block page instead. Over an onion
address the origin is plain http, so job 95016928194 of run 31864526217 waited
out the full onion-scaled 300000ms on a form that could never appear.
The flag was set nowhere in the repo. It now follows N8N_WEB_PROTOCOL, the same
per-application fact that already drives N8N_PROTOCOL one line above and
WEBHOOK_URL one line below, so the three cannot desync. plugins/lookup/tls.py:102
derives that protocol from tls enabled, which makes this the same shape as
web-app-confluence's ATL_TOMCAT_SECURE.
On https it renders true, which is n8n's own default, so clearnet is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three groups drifted from the commands behind them.
The retrigger family still passed a mode scope that --failed no longer takes,
and two of them (i8cire, i8puci) pinned it to swarm permanently, so half of
every red run was dropped before the trigger ever saw it. Both retriggers now
force-cancel the source run before dispatching, taking that step from
fork/feature/ai-agent-employees (da7b80b89) and extending it to i8cifo, which
does not route through i8cifail and would otherwise keep the old jobs holding
runner slots while the new ones queue behind them. The cancel is deliberately
not chained with &&: the usual case is a finished run, where force-cancel
reports an error, and chaining would block the trigger exactly when it is
wanted.
i8matiso and i8mativaso rebuilt the discovery sort and filter by hand. They
now render through i8maco, which is the query module, so the order they print
is the order CI takes. This also removes the failure they had in a fresh
shell: INFINITO_DISCOVERY_SORT is only set by i8source, and without it the
hand-rolled --sort was empty and the CLI refused it.
--mode is spelled --modes, the name the parser actually declares; argparse was
accepting the abbreviation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A retrigger answered a red run with role names. Three things were lost on the
way, and each of them let the rerun test something other than what broke.
The mode was a scope: --failed swarm dropped whatever fell in compose, and two
aliases passed that scope permanently. Every mode is read now, and a leftover
scope argument is accepted and ignored rather than breaking the callers.
The selection was a role id, so the rerun deployed whichever variant, mode and
onion state the sweep rotation picked next -- not necessarily the combination
that failed. Each red job now contributes its own token
(web-app-x#0@swarm+tor); a role that failed in two modes comes back twice.
The carried configuration was a hand-written list of eight inputs against the
thirteen the form declares, so resume_from_chunk, sweep, workspace_track and
whitelist silently fell back to defaults. The list is read off
entry-manual-steer.yml now, and everything but the priority line travels
verbatim, values taken from the job log that records them all.
That rule is deliberate and total: it supersedes the suite-skip behaviour on
fork/feature/ai-agent-employees (b1ca39e9d), which turned workspace and
instructions off when the source run had passed them. A retrigger reproduces
its source; deciding which suites are worth re-running is not its job.
The integration test walks the workflow's own input declaration, so an input
added to the form without being carried fails here rather than in a run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The plan told you a row is being deployed, never that another row in the same
sweep already proves whatever it would prove. The complexity matrix has always
known -- covered_by holds the id of the earlier row that embeds this one -- but
the number stopped at the matrix, and 46 of 256 rows carry one.
The matrix entry now carries the row's discovery id and that covering id
through to the plan, which renders them as two columns and drops them again
before the matrix reaches the deploy jobs.
The id column had to change with it. It was a running count over the rendered
lines, so a covered_by of 1 pointed at a row that numbering does not have. It
is now the discovery id, the same number the complexity matrix prints, which is
what makes the reference resolvable inside the table it appears in.
Two tests indexed their columns by position and broke on the new one; they ask
_COLUMNS for the index now, so the next column does not break them again.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Two shell aliases rebuilt the discovery order by hand, and both got it wrong
in the same two ways: they passed INFINITO_DISCOVERY_SORT straight through,
dropping the clones-last prefix that keeps one representative per dna cluster
ahead of the budget cut, and they filtered on `compose` -- what a role can do
-- where discovery filters on `test_compose`, what CI actually tests. Measured
against the real query that is 13 rows of difference and a divergence from row
two onward.
The query module already owned both pieces. It now also renders the whole-role
view (--matrix --roles), which is the only thing the hand-rolled version could
do that this one could not, so nothing outside this module has a reason to
assemble a --sort or a --filter again.
A unit test pins the contract: the role view may drop the --variant flag and
nothing else, and the filter reads the tested column.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7538b8a0b turned this probe from isVisible() into waitFor() and claimed the
EspoCRM 1 and 2 failures with it. It was already in ce2db79ee, the commit run
31864526217 tested, and EspoCRM 1 failed anyway.
The wait now happens; it is just not onion-scaled. The trace of that run records
the probe as {'selector': "input[type='password']:visible >> nth=0", 'state':
'visible', 'timeout': '10000'} and it burns the full 10003.9 ms before erroring,
while the neighbouring wait at admin.js:145 carries 'timeout': '75000' from
resolveTimeout(15_000). This one call is the only timeout in the personas tree
that does not go through resolveTimeout, and utils/env.js:53-59 documents why
that matters: onion gotos deliberately return at DCL because "the caller's
explicit selector waits (onion-scaled) cover app boot".
They did not cover it here. All three attempts quit at DCL+10.1s with zero
/api/v1/* requests in the network log, i.e. App.start() never ran; four of the
six preload fonts were still in flight, the slowest completing 18.3s after DCL.
The base moves 10_000 -> 15_000 so the onion budget matches admin.js:145 rather
than inventing a third constant. Clearnet is unaffected: resolveTimeout
multiplies by 1 off-onion.
Not measured, and the reason to expect another round: DCL -> password field was
never observed on an onion circuit in that run, because every attempt quit
before it. 75s is sized from the parts, not from an observation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
roles/sys-svc-mail-msmtp/tasks/ca_bundle.yml stat'ed candidate bundle paths on
the Ansible target and wrote the winner into msmtprc. Moodle and WordSpress then
copied that file into a container, where the named path is a different file - or
none. Only update-ca-certificates populates a container's system bundle, and
that runs solely from the with-ca-trust.sh entrypoint, which compose_ca.py sets
only when 'docker image inspect' succeeded. On a fresh host the image is built
minutes later, so it never ran: job 94965457602 of run 31864448027 shows
Config.Entrypoint still moodle-entrypoint.sh while the CA sits mounted beside it,
and msmtp reporting 'The certificate issuer is unknown'.
An explicit tls_trust_file also outranks SSL_CERT_FILE, so the env injection the
override always performs could not compensate.
Name CA_TRUST.inject_cert_container when the CA is actually injected, and emit
no line at all otherwise - msmtp's own 'system' default is the right answer for
a letsencrypt relay, and it is what the container's stock bundle verifies. This
also removes a latent defect on letsencrypt: a RedHat host would have written
/etc/pki/... into a Debian container.
The removal of the probing apparatus is taken from 9a72d01eb on
fork/feature/ai-agent-employees, which diagnosed the wrong-machine measurement
correctly. Its own conclusion - drop the line entirely and rely on the trust
store - does not hold where the wrapper never runs, and in swarm it never does:
handlers/swarm.yml passes wrapper=False by design.
The three-term predicate behind all of this was hand-copied into four places.
lookup('ca_injected', application_id) is now its single source; the swarm and
compose gates and sys-svc-compose-ca's own var read it too.
compose_ca.py finally says when it degrades to mount-plus-env, naming the service
and the image it could not inspect.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a34cb44ad split variant 3 into openproject, mattermost+matrix and the rest, and
closed with the open question: 'Whether dropping one partner moves 214 sync
under 185 is open.' Run 31864448027 answers it for the pair that was left.
Job 94965456921 (swarm, variant 10) died at rc=137, SIGKILL, 221s past the
20400s sweep deadline. Its sync pass alone took 11547s = 192.5 minutes - still
above the ~185 that commit derived as the survivable boundary. One partner was
not enough; the round needs one app.
matrix is the heavier half. Of PASS 2's 2811s of Playwright, its suite is
1090.55s - 39%, against 401.96s for nextcloud itself and 25-90s for ten of the
remaining twelve. PASS 1 carries the same shape at 1071.81s.
The new round is appended as variant 12 rather than inserted after 10, so every
existing index keeps its job name and its artifact names.
The alternative considered and rejected: restricting the async pass to the app
under test. It would have saved more seconds and cost the co-deployment coverage
of thirteen roles in the update path, to fix an overshoot one role caused.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
a8dd7f150 rebound the handlers to listen topics and renamed them across 107
files; 'Build compose' became '🔨 Build compose'. This lint looked its two
build tasks up by name, so _by_name returned None and all three assertions
died on NoneType rather than reporting a missing task.
The register is the better key: the fallback's own 'when: <var> is failed'
already depends on it, so a task cannot lose its register without breaking
the thing the lint exists to guard. A name is prose the next refactor may
rewrite again.
The lookup now asserts instead of returning None, so the next drift names the
task it could not find.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Four tests were guarding two rules. Both duplicates are gone and what they
were meant to check now lives with the rule it belongs to.
The handler name/length pair: a 120-character limit sat over the same files
as an 80-character one, so it could never fire first, and outside that overlap
it matched `name:` keys that were no task at all -- a volume name, a var. It
found nothing in either half. Deleted; the surviving lint instead grows the
scope it was missing, `roles/*/handlers/**` and `playbooks/**`, because a
handler is a task, and the dispatch input descriptions, because a form line is
read like a job name.
The templated-handler-name pair is worse: the second one skipped every
document with `if not isinstance(doc, dict)`, and a handlers file is a list.
It has been vacuously green for its whole life. Its intent -- every
`handlers/*.yml`, not just `main.yml` -- is now what the surviving test does.
The notify/listen contract itself gains the rule that made the rename in
a8dd7f1 safe: a notify must land on a `listen:` topic, never on a display
name, and both spell out as [a-z_-]+. Following `import_tasks` chains by hand
is gone with it; scanning every project YAML sees the same handlers, plus the
ones a role hands to a generic renderer through a data structure.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An input description is what an operator reads in the dispatch form, one line
per input, exactly like a job or step name -- and it was drifting the other
way: eighteen of them opened with a bare word where every name in this
repository opens with an emoji, and sixteen ran past the line, the longest at
382 characters explaining four tor values inside a form field.
The descriptions now say what the input is; the workflow README says what the
values do. It gains the selection-token syntax and a table of the four tor
values, which is where a paragraph belongs when it outgrows its form field.
workflow_call inputs are left alone. They are wiring between workflows and are
rendered for nobody.
Two explanations did not survive the cut and are worth knowing: the mirror
cleanup no longer warns that the default GITHUB_TOKEN may lack the
delete:packages scope, and the workspace track no longer spells out that
'rotate' flips with the run number.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A handler name was doing two jobs at once: the line an operator reads in the
play output, and the identifier a `notify:` binds to. The two pull in opposite
directions -- the log wants "restart the SSH service, with diagnostics", the
binding wants a token that survives quoting, YAML flow style and a shell
round-trip -- and a name that changes for readability silently changes which
handler runs.
Every handler that a notify reached by name now carries a `listen:` with that
exact wording, so the binding is explicit before anything is renamed. The
topics themselves become identifiers: `compose up` reads `compose-up`,
`Restart NFS ganesha` reads `restart-nfs-ganesha`, 22 of them across both
sides of every notify. Where the topic is assembled at runtime, both branches
moved together: the conditional in sys-service carries two literals, and the
LDAP renderer interpolates `import-{{ folder }}-ldif-files`, which is what
makes those handlers reachable at all.
Freed from the binding, the names take the emoji prefix every other task line
already has.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A whitelist or priority entry was a role name and nothing else, so a run
could not ask for "this variant, on swarm, behind the onion". It either
covered everything the role offers or waited for the rotation to come
around to the combination that broke.
An entry is now a token that may carry the axes, either as
web-app-x#0,2@swarm+tor or as the job label pasted straight back out of
the run that failed. What a token leaves open still belongs to the line
it stands in: priority covers every remaining combination in one sweep,
whitelist keeps the rotation and its position in the matrix. A pin the
row cannot take aborts the matrix rather than dropping the row, because
a dropped row reports a green run for a combination that never ran.
The onion state spells out as +clearnet, not as a -tor suffix: svc-net-tor
would otherwise read as role svc-net on the clearnet.
Chunk membership still follows the discovery ranking, which is what
decides who makes the budget cut, but the rows inside a chunk are sorted
by display name, variant, mode and onion state, so a chunk's job list
reads like the plan table instead of like the sweep's rotation.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The dnsmasq and apt-resolution investigations both stalled on facts the
snapshot never recorded.
The snapshot is taken by docker exec inside the DiD container and the
recursion only descends, so the runner's own docker - where the resolver
runs as ${INFINITO_RUNNER_PREFIX}-coredns - was structurally invisible.
dedicated.sh now runs a second pass on the CI host, with RESCUE_SEEN
seeded with the DiD id so it does not collect the first snapshot twice.
For the rest: a process table, so the MainPID that service-state reports
can be checked against something; the dnsmasq drop-in directory, because
this project's config lives at /etc/dnsmasq.d/tor-onion.conf and never in
the main file; nsswitch.conf and hosts, without which a getent verdict is
ambiguous; one probe per name plus, where the distro ships dig or
nslookup, one per nameserver, so a refusing resolver is distinguishable
from an unreachable mirror; the full firewall table set, since a dropped
query is recorded in filter and -t nat was actively hiding it; routes and
neighbours; os-release, because the matrix is distro-shaped and a bundle
could not say which leg it came from; the runtime's networks, which hold
the forwarder behind every container's 127.0.0.11; free inodes, which a
df -h reads as free space; and the runtime's disk, volume and image
inventory, the last with full references, since an engine was once picked
from a registry host embedded in an image name.
ps runs without args: the bundle is uploaded as an artifact and argv
carries whatever credentials a process was invoked with - a real proxy
password showed up in the first live run of this capture.
Two exec budgets are corrected while the per-container cost grows to
three calls. The per-container journal was borrowing the 600s meant for a
host boot journal, so one wedged container could cost 840s; container
journals span a deploy, not a boot. The cheap questions get 30s, and the
one collection site that had no outer bound gets the timeout 1500 the
others already carry.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three captures read two sources into one file - devices.txt held
/proc/devices and /proc/misc, net-snmp.txt held snmp and netstat,
sockets-proc.txt held four socket tables - so a grep hit no longer named
a source. Derive the filename from the path instead: the twelve cat
captures become a list of paths, and source_name() turns each into its
own file. Nine names are unchanged; conntrack gains the two segments its
old name dropped, and it was pointing at /proc/net/conntrack while
reading /proc/net/stat/nf_conntrack.
The daemon and kill-marker journals sat under containers/ between the
per-container dumps with an underscore prefix to sort them to the top.
They are host journals; name them as such and put them beside the others.
Give every journalctl call its own budget. The boot journal is now the
largest file in the bundle by two orders of magnitude - 224 MB of 220 MB
total on a one-day host - and timing out writes the exception text into
the file instead of the log.
The nocheck marker on the probe read repairs the lint gate for the
capture test added in 58c06fc8d.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RESCUE_MAX_DEPTH did two jobs without saying so: it bounded real nesting,
and it terminated cycles. Real nesting terminates on its own - every level
is a container - so the cap only ever cost snapshots. Cycles do not
terminate: svc-runner (templates/compose.yml.j2:14) and svc-prx-openresty
(meta/volumes.yml:86) bind-mount the host's docker socket, so those
containers list the host's containers including themselves and would
descend into themselves without end.
Carry the ids already entered instead. Container ids are shared across a
socket and distinct across daemons, so this cuts exactly the cycle and
lets true nesting run to whatever depth it has. A container whose id will
not resolve is skipped rather than entered: without an identity the cycle
cannot be seen, and a missing nested snapshot costs less than a recursion
that never returns.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
getent exits 2 with neither stdout nor stderr when it cannot resolve a
name, so resolve-probe.txt landed in the bundle as zero bytes - the same
shape a capture takes when the collector never reached it. The one file
that answers 'could this host resolve anything' could not distinguish a
failure from its own absence. Record the exit status whenever a command
produced nothing.
write() swallowed OSError, which hides exactly the incident that produces
it: once the filesystem holding the bundle is full, every later file goes
missing without a reason. The job log is the only channel left at that
point, so announce it there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
journalctl -n 10000 cut the window a CI job actually spans. Measured on a
busy host the journal runs at ~56 000 lines per boot-hour, so the cap
covered roughly the last ten minutes of a ninety-minute job and the
incident had scrolled out before the snapshot was taken. Capture the boot
in full, and give err its own file: priority is journald metadata rather
than text, so grepping the warning dump recovers only a third of it.
systemctl list-units calls a forking unit active/running while its guessed
main process is gone, and the containers carry no ps, so the unit table
alone cannot say whether a daemon is still there. service-state.txt adds
MainPID, NRestarts and ExecMainStatus per unit. sockets-proc.txt reads the
kernel tables directly for the containers that ship no iproute2.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The proxy exists only so libcurl will speak to an .onion issuer at all -- it refuses such a host by name before any resolution (RFC 7686, error 6), as 20ddda8c1 measured. Everything else that crosses it is collateral, and proxyexclude is what keeps it out. apps.nextcloud.com was listed for exactly that reason.
But Guzzle evaluates the list per hop, against the host the handler is handed. The appstore answers with a redirect to a mirror, path identical, host swapped, and garm2/garm3.nextcloud.com were not listed -- so hop 1 went clearnet and succeeded while hop 2 went through Tor and died with error 97. All 24 garm URLs in the failing job are that one endpoint, /api/v1/apps.json; not one is a download.
A suffix entry covers the entry host and every mirror it redirects to, instead of chasing garm2, garm3, garm4 one at a time. The form is already in use in this same list: .githubusercontent.com carries the leading dot for the same reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
occ prints 'Could not download app deck, it was not found on the appstore' for both a catalogue that carries no release and a catalogue it never reached. The filter matched only that sentence, so a cURL error 97 -- 'cannot complete SOCKS5 connection to garm3.nextcloud.com' -- classified as unavailable, which 01_install.yml:38-40 tolerates by design.
So the deploy went green without deck. addons/deck.spec.js then waited 946s on an app-shell locator that could never appear, and the job failed 53 minutes after the actual error, on a 404 page with an authenticated session. All seven install attempts had failed the same way, across 15 minutes of the retry loop.
unavailable is now only claimed when the catalogue was actually read: a transport marker in the same output disqualifies it, and failed_when fires where the error happens. The retry loop is untouched -- until: gates on ok, which this does not change, and failed_when only ever sees the last attempt. A passing sibling job proves the difference: its first deck attempt died on garm2 after 2m00s, its second returned rc 0 with 'deck 1.18.3 installed' 18s later, and nothing about that changes.
The filter had no tests at all. Seven now cover both directions, including that a reachable appstore without a release stays skippable.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Funkwhale's sign-in input carries id=username-field, name=username and autocomplete=username -- and no type attribute at all. An attribute selector does not match an element whose attribute is absent, so input[type='text'], input[type='email'] found zero nodes and locator.fill burned its full 30s timeout, three times, for both personas. The password field beside it does carry type=password, which is why the assertion one line above passed and only this one failed.
Adding an autocomplete=username arm matches the field without widening into input:not([type]): Funkwhale's header search box is typeless too, and .first() would then be decided by DOM order alone.
Transport does not enter into it: a selector that matches no node matches none over any route, and the run was not starved either -- the trace's slowest request is 360ms with the /login navigation at 107ms. The locator has been iterated twice before, scoped to the form and unscoped again, without anyone reaching the missing attribute.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The second glyph was emitted only for the modes that carry a tor axis, so host rows were the only ones reading with a single marker. LOCAL_GLYPH fills that position. 3.5 was already the glossary's symbol for host, bound to the word test_host, and symbol_glossary refuses to import when two words share a glyph -- so the axis binds the existing definition instead of writing the character out and stepping past that guarantee.
The variant now joins with # instead of a space: a display name and its shard were two whitespace-separated tokens, and decode_list splits on whitespace, so 'tor 1' parsed as two names. One label, one identifier.
encode builds it, _VARIANT_SUFFIX strips it and LABEL_RE reads it back -- all three now go through the same constant, which is what keeps the parser from drifting away from the builder it is the inverse of. Legacy titles are deliberately not accepted: a second tolerated format is exactly that drift. An old title now leaves the shard inside the name and decode returns None, so the job reads as unrecognised rather than as the wrong role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3.4.2 stops the registry host and the tag from deciding which dump tool runs. On a swarm node that hosts the local registry and is named after the app under test, svc-db-mariadb-swarm-mgr-01:5000/postgres_custom read as MariaDB, mariadb-dump ran inside a Postgres container and exited 127, and the BackupException took svc-bkp-volume-2-local down with it -- which is what failed the MariaDB 0 job of run 31864448027 through its CLI test.
Verified on PyPI before the bump: pip index versions reports 3.4.2 as LATEST, so the pinned install resolves. Repo-wide this is the only pin of the package.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The email SPOT turns tls off for an .onion host -- Tor already carries the encryption and no authority issues a certificate for a v3 address -- and env.j2 renders TLS_FLAVOR=notls from that same value. Mailu's front then opens no implicit-TLS listener, while compose kept publishing 465, 993 and 995 anyway.
The swarm reachability probe reads the published set straight out of docker service inspect and demands a listener per port, so it failed on exactly those three while 25, 110, 143, 587, 4190 and 8007 answered. The stack was advertising endpoints it does not serve; the probe was right.
A role-local container_ports_mailu builds the block instead: protocols come from MAILU_PORTS, so declaring a port in the role meta is enough, and the implicit-TLS three drop out when tls is off. The terms are assembled in Python because a lookup call cannot spread a built list into positional terms -- writing the set out twice, once per TLS flavor, would let the two copies drift.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
02_performance deployed matomo with no disable set at all, so svc-net-tor joined the groups. It is exclusive, matomo follows it through tor.enabled in its own services meta, and the clearnet vhost disappears. The assertion two lines down then aborts in TLS before HTTP: 'curl: (35) TLS connect error: tlsv1 unrecognized name', reported as '[FAIL] https://matomo.infinito.example returned HTTP 000, expected 200'.
The counter-example sits in the same job: 01_minimal deploys dashboard with the group-conditional services disabled, tor among them, and its clearnet assertion returns 200. This suite tests the inventory, not the transport -- that a dedicated deploy replaces the previous app set. So it takes the route the sibling script already proves, rather than asserting against an onion address the runner never touched.
03_reuse stays untouched: it runs mode=update against the inventory 02 just built.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant 1 pinned nfs-server enabled false, so the matrix deployed the role that backs up the NFS export on a host where no export exists. The unit's own guard then refused the run and the job died: 'ERROR: /srv/nfs missing; this host is expected to serve the NFS export', followed by 'Failed to start svc-bkp-nfs-2-local.12.0.0.infinito.nexus.service'.
The guard is right; the variant was not. The role declares run_after svc-storage-nfs-server and reads its source through lookup('nfs_export_base') -- the export is the subject of the backup, not an optional service. The sibling role shows the legitimate shape: svc-storage-nfs-server toggles its consumer nfs_backup, and the server survives either way.
With no variant left to carry the false polarity, nfs-server takes the exemption the coverage lint documents for exactly this case: flags whose group-derived value is a hard invariant that variants must never override.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The node identity is minted during the first distro leg's deploy, as root inside the DiD on the bind-mounted workspace, mode 0600. The workspace survives between legs, so the second leg's 'pack the repo once on the controller' runs tar as the runner user and aborts on all three key files with EACCES -- 13 seconds into the play, before anything of the role under test has run.
Five swarm jobs of run 31864448027 die exactly there: svc-db-postgres, svc-db-openldap, svc-db-redis, web-app-matomo and web-svc-cdn. First leg 3313-5778s and green, second leg 125-140s and dead, whichever distro it happens to be. The remaining swarm failures are not immune, only shadowed: they die in their first leg for unrelated reasons and never reach a second.
Ownership follows the tree, not the process that happened to mint it. The repair runs on the reuse path too, where ensure_node_onion returns early and a root-owned key set from an earlier run would otherwise never be corrected. Modes stay 0600. lchown so a symlink cannot redirect the change, and a path that vanishes mid-walk is skipped instead of aborting the provisioner.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every clearnet matrix row carried disable=tor, including the row whose own app is the tor provider. init.py resolves that token to svc-net-tor, prunes it from the primary list and aborts with 'All primary apps disabled', so the job died at rc=1 having deployed nothing: the rescue snapshot captured 0 containers and 0 services.
Blanking the disable was not an option. disable is the only lever the deploy reads (tor: false reaches nothing but a glyph, a dedup key and a concurrency group), so a blanked row would have been byte-identical to the tor row while reporting a clearnet check. The provider's onion state is not a row property, so its rows now yield no clearnet state at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Measured against the running stack, not guessed:
- LAM sits behind the SSO proxy, so the first form is Keycloak's and rejects an LDAP bind DN as its username
- LAM offers its admin accounts as a select, so filling a text input never reached it and the bind was skipped silently
- hasText cannot see the value attribute of an input[type=submit], so the login and save buttons are addressed by role
- the vhost rewrites only "/" to "/lam/", and treeView.php base64-decodes ?dn
- the directory names people by uid=, so a cn= member DN pointed at nothing
- asserting the absence of "error" matches LAM's own hidden message templates, so the written value is checked instead
LDAP_ROLE_DN_BASE is wired through because the role groups live under ou=roles.
Unproven: the spec has not executed in any run - its title appears zero times in four deploy logs while sibling titles from the same file appear 24, 12 and 24 times. Everything from the tree navigation onward is therefore untested.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
policy-rc.d answers 101 in the container images, so installing the package never starts postfix, and the existing notify only fires when a template changed. A run without a config change on a freshly installed container therefore leaves postfix installed but dead, and local delivery fails silently.
The task is idempotent: systemd reports no change when postfix already runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A bare User.new(...).save! leaves the personal namespace unbuilt on this generation and dies with "Namespace can't be blank". Users::CreateService builds it and takes the organization, so the default organization is resolved first and passed as organization_id.
Three abort markers replace silent continuation: no default organization, a failed create, and a user still missing after a reported success.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Baserow finishes table creation in a background job that still holds the table lock, so the immediately following field write answers 409 ERROR_FAILED_TO_LOCK_TABLE_DUE_TO_CONFLICT - the API asking the client to retry. The spec treated it as a failure.
The retry is narrow on purpose: only that error code loops, any other 409 and every non-2xx still fails at once through readJson's response.ok() assertion, and the loop gives up after 60s.
Tradeoff: the spec no longer measures how fast the lock clears, so a regression that holds it far longer would pass silently. The 60s budget is not measured against the real lock duration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
uri reports a connection error as status -1. The readiness gate only had an upper bound, so -1 satisfied it and the retry loop settled on the first attempt against a service that was not up yet - the suite then ran against a dead endpoint instead of waiting for it.
Same fix as 7511852bd1 on feature/svc-net-tor, found independently; that branch is not merged here and carries it alongside unrelated onion changes and a diverged variable naming. This adds the regression test it lacks.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Gating it on FLOWISE_MCP_SERVERS rendered the block away whenever discovery came back empty, and a provider whose credential has not resolved drops out of that list without failing the run. The container then kept Flowise's default deny list, which covers 192.168.0.0/16 - the range the qdrant MCP sidecar resolves to - and the registry create answered 400 "Server URL is not allowed by policy".
The rendered env carried HTTP_SECURITY_CHECK zero times while CUSTOM_MCP_PROTOCOL from the same template appeared once, which is what proved the block never rendered.
mcp.enabled is a deliberate operator setting; a security posture must not hinge on a discovery result that is allowed to be empty.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The task execs into the container to read cn=module{0}.ldif. Under swarm the scheduler can replace the task between resolving the container id and running the exec, and the single attempt died on "Error response from daemon: container ... is not running". Compose has no such window, so the task carried no retry.
The address resolves inside the shell, so each attempt picks up the replacement. failed_when gives way to until: with both present it is ambiguous whether an attempt fails outright or retries, and an unmet until still fails the task after the last attempt. Same shape as the mariadb sibling.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
/healthz answers 200 with the database unreachable, so a container that cannot serve a single request still reports healthy. Recreating the shared postgres under a running n8n left exactly that state: the provisioning task then retried 30 times against a container the orchestrator called healthy, and took the whole round down with it.
Measured before and after the redeploy: healthz 200 / healthz-readiness 503 / rest-settings 503 "Database is not ready!", then 200 across all three.
1834987066 made the same switch for the owner-bootstrap probe; this closes the remaining call site.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
