install-lint provisioned chrome-headless-shell on every run, and `make lint`
runs install-lint once per lint target in parallel whenever its stamp is not
fresh. On CI the stamp never is (eslint lives in node_modules, not on PATH),
so one lint job provisioned 16 times (run 38455552981). As root, puppeteer's
--install-deps runs `apt-get satisfy` even for an installed browser, and two
of those at once fail on the dpkg lock; a failed provisioning then deleted the
shared browser cache while lint-mermaid could be rendering with it.
The browser is now provisioned only when it is not installed, and --force
drops its cache so `make install-lint-force` still reinstalls it with its
libraries.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
test_no_uncached_file_reads rejects Path.read_text in tests unless the file is
written in the test itself. The mermaid script test reads the call log its
stubs append to during the run, so that read carries `# nocheck: cache-read`.
The mermaid cli test is formatted with ruff.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The loop test only required three different parent images for three distros,
which a swap between two of them would still pass. Each iteration now also
records the owner and tag it sees, and its image must equal
pkgmgr_image(distro, owner, tag). With the loader of main the test fails on
arch with pkgmgr-debian:stable.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The loader copies INFINITO_DISTRO into a loader-local variable before it
compares the value with the one in .env. With INFINITO_DISTRO read directly in
that condition, `shellcheck -x scripts/meta/resolve/apps.sh` reported SC2153
for INFINITO_DISTROS, which apps.sh takes from the sourced .env; with the copy
it reports nothing, as on main.
The loader test checks that .env is unchanged with filecmp against the file it
was copied from, instead of reading it twice with Path.read_bytes, which
test_no_uncached_file_reads rejects.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The release backfill dispatches the highest version tag whose images are
missing. Every tag from v15.0.0 down to v9.0.2 has its images now, so since
2026-10-05 it picks v9.0.1 every day, and each of the nine runs failed in
"Build missing CI images / Image" at
`entry.sh: line 19: infinito: command not found`. The v9.0.1 tree cannot
build its own image; c680be12ea fixed that build in v9.0.2, and a tag cannot
take the fix.
highest_release_tag.sh now stops at v9.0.2: no tag below it is picked, so the
backfill reports nothing missing instead of retrying v9.0.1. v9.0.0 has the
same Dockerfile, Makefile and scripts/docker as v9.0.1.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The scheduled mirror job has failed every day since 2026-09-25 on the same two
images: quay.io/minio/minio and quay.io/minio/mc answer `unauthorized` to
anonymous pulls ("2/197 failed", run 38352569503). Both belong to
web-app-minio, an eol role outside the lifecycle envelope that default.env
tests, so no CI deploy pulls them.
iter_role_images now skips every role that is_maintained() reports as
unmaintained, the check the version updaters already apply to the same list.
On the tree the discovery yields 187 images instead of 192; the five it drops
belong to the four eol roles web-app-confluence, web-app-jira, web-app-minio
and web-app-phpldapadmin. The mirror sync, resolver and wait read this list,
and their descriptions and mirror.md now say so.
The roles directory is resolved first: is_maintained reads a relative path as
a role name and would keep every role.
475ab9b386 on feature/store-i18n takes the same approach with a lifecycle
reader of its own; main already has is_maintained for it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Eight compose-or-swarm rows over eight sweeps: each sweep must put swarm rows
on the meshed and the direct arm, and no compose row may carry the mesh. Fails
on the previous wants_vpn, which held every swarm row of a sweep on one arm.
The mode rotation picks swarm for a row when (position + sweep) is odd, so
every swarm row of a sweep shares one position parity. wants_vpn rotated on
(position + sweep // 4) % 2, the same digit, so all swarm rows of a default
run landed on one arm: a sweep was entirely meshed or entirely direct and
never compared the two.
Reading the position's second digit (position // 2) splits the swarm rows of
every sweep about evenly between the arms, and sweep // 4 still flips each
row on its own schedule. Compose rows are untouched: they never take the
mesh.
The any-number-of-meshes test zipped the specs against the planned meshes
without strict=, which ruff reports as B905 and the lint-python job failed on.
Both sequences come from the same declarations, so strict=True also turns a
plan that drops a mesh into a failure instead of a silently shorter loop.
Routes disk and /tmp cleanup through the cleanup skill and adds the repository rules: compose-project attribution, the shared cache stack that is never a candidate, act and swarm keep rules, teardown targets with what each destroys, and the targets that must never run as a cleanup step. Listed in the agent cheatsheet.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One conflict, in roles/web-app-n8n/files/python/provision/mcp.py.
Both sides fixed the same failure. This branch added records() to name the
shape a list endpoint returned instead of dying on 'str' object has no
attribute 'get', deliberately refusing to guess at an unwrapping rule for a
shape nobody had observed. Upstream observed it: e4078c05f provisions against
the n8n 2 API, and 75f024e71 and ad99ded58 page through api keys and workflows.
Upstream's side is a superset - its api_keys_labelled() and workflows_named()
call records() themselves, and its records() handles the paged
{"data": {"items": [...]}} that this branch would only have reported. Taken
whole.
One test moved with it: the nested-payload case asserted the old wording, and
upstream's message names the keys it found rather than the type.
PDS 0.4.0 (@atproto/pds 0.4.0-beta.7) signs the service-auth JWT it
attaches to proxied app.bsky.* calls with iss, aud and exp only
(createServiceJwt in dist/index.js of the image). The AppView requires
the lxm claim and rejects every such call with 401
BadJwtLexiconMethod, so social-app shows errorMessageView on Following
and Notifications. 0.4.66 passes the called method as lxm into every
service JWT it signs and sends every proxied call through the
SSRF-guarded proxy agent.
0.4.66 also replaces the whole-handle limit of 0.4.0 (30 characters
for the entire handle) with a limit of 18 on the part in front of the
service domain, so kc-administrator.<domain> passes for any domain
length, including kc-administrator.infinito.nexus (31).
- meta/services.yml: pin ghcr.io/bluesky-social/pds to 0.4.66.
- vars/main.yml: the public AppView URL is always https. Onion-primary
deployments took the protocol from the local TLS setting and sent the
service JWT to api.bsky.app over plain http. services.view.enabled is
evaluated as a bool.
- templates/env.j2: drop PDS_ADMIN_EMAIL, which neither 0.4.0 nor
0.4.66 reads. PDS_CONTACT_EMAIL_ADDRESS stays unset on purpose:
describeServer would publish the administrator's address.
- Administration.md: the account database only migrates forward; a
rollback restores the bluesky_pds_data backup together with the old
pin.
- Playwright: the broker login moves into _shared.js and waits for a
landing outside /oauth2/, because the bare base URL also matches the
intermediate /oauth2/callback hop. New scenarios check that the PDS
proxies getTimeline and listNotifications to the AppView with a JWT
it accepts, for the broker session (OIDC) and for a native session
(oauth2 disabled), and that social-app renders Following and
Notifications without errorMessageView or lxm page errors. The
rendering scenario is clearnet only: social-app boots only after
loading its age-assurance config from public.api.bsky.app.
Merge note for feature/store-i18n: 19b491b014 raises the pin to 0.4.21
and renames the contact-mail line to PDS_CONTACT_EMAIL_ADDRESS. Resolve
the pin to 0.4.66 (0.4.21 still signs without lxm, and the account
database cannot migrate back) and keep the line deleted. Take the union
of both sides in _shared.js and playwright.spec.js.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Two findings of the CodeRabbit review of #869.
jmap_call.yml counted notCreated and notUpdated as a refused call, but
not notDestroyed. A destroy that Stalwart rejected therefore passed, and
the post-load flush cleared stalwart_app_password_stale while the retired
app password stayed valid. A notDestroyed entry now fails the call,
except notFound, where the password is already gone. The password keeps
its retired description, so the next deploy collects it again and
retries the destroy.
tracing.yml took any Stdout tracer with an include policy for the ban
tracer, so one that includes other events kept the role from creating
its own and bans stayed off the container output. The selector now also
requires every event of STALWART_BAN_TRACER.
Verified with the real task files, a stubbed container command and
canned JMAP responses: a forbidden destroy now ends with failed=1, while
notFound and a clean destroy pass; with an unrelated include tracer the
role makes a second JMAP call that creates the ban tracer, with a
complete one it does not. The previous files pass the forbidden destroy
and skip the creation in both tracer cases.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The handlers after "Set systemctl service state" read Result and
ExecMainStatus and fail the deploy with "Unable to start service ..."
when either reports a failure. Most roles leave system_service_state
omitted (SYS_SERVICE_DEFAULT_STATE is omit outside MODE_DEBUG, and the
backup roles start their unit only in MODE_BACKUP), so the state task
changes nothing, and the two properties describe the unit's last run,
whenever it was.
Measured on a local stack: the volume backup unit had failed on
2026-10-12 in the CLI test of another deploy, when a container it had
stopped was recreated before it could start it again. The next day a
deploy of web-app-seaweedfs and web-app-prometheus, which did not run
the unit, read Result=exit-code and failed in the backup role.
The diagnostics now fire on a stale result only when the deploy set a
state itself. A failed state change still fails as before.
Verified in the stack container against a oneshot unit whose ExecStart
is /bin/false, left failed before each run, with the real handler file:
with the state omitted, main's handler fails with "Unable to start
service" and the new one passes; with state=started, which reruns the
unit, both fail. ansible lints and test_comments_valid pass.
Closes#914
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
meta/services.yml switches the SSO integration of SeaweedFS off on a Tor
node, but variant 0 pinned services.sso to a literal true (#868). A round
writes the variant pins into its inventory, the inventory value wins over
the expression in services.yml, and a Tor round therefore enabled SSO for
a frontend it does not deploy. Measured in a round inventory:
`sso {'enabled': True, 'shared': True}` with svc-net-tor in its groups.
The pin now carries the same gate as logout, dashboard and prometheus,
which #832 already fixed the same way. The sso block in services.yml is
marked for the variants-coverage and bond-baseline guards like the
frontend block, since no variant may force this consumer on a Tor node.
Verified on a local Tor round of variant 0: the round inventory holds
`{'enabled': "{{ 'svc-net-tor' not in group_names }}", 'shared': ...}`
for the sso block instead of True, and after the deploy only the
seaweedfs container runs; seaweedfs-sso-proxy and seaweedfs-redis, which
an earlier deploy had started, are gone, and the rendered compose file
holds no SSO service. tests/integration/roles/meta/variants and the
sso-variant-tor lint pass; without the two markers test_bond_baseline
and test_coverage fail on services.sso.
feature/store-i18n carries the same pin change for all four services in
d70e2bb967, so the two branches merge without a conflict in
meta/variants.yml.
Closes#868
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The SSO hook reported its start, every sign-in, every created account and
every error through this.logger (#852). n8n 2 calls an external hook with
a context that holds only dbCollections, so this.logger is undefined and
the optional chaining turned every one of those calls into a no-op: after
an SSO sign-in that created an account, the container log held no line
about it.
The hook now resolves the Logger service from n8n's dependency container
(@n8n/di and @n8n/backend-common, loaded through n8n's own require like
the other modules the hook uses) and writes every message through it. The
requires moved to the top of the ready hook so the first message can use
the logger.
Verified on the pinned image docker.n8n.io/n8nio/n8n:2.44.0, started
with EXTERNAL_HOOK_FILES pointing at the hook and N8N_FORWARD_AUTH_HEADER
set: with this hook the log carries "SSO middleware initializing with
header: X-Forwarded-Email" and "SSO middleware initialized successfully"
before "Editor is now accessible via:"; with the hook of main the same
start reaches the editor without a single SSO line.
Closes#852
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The readiness module http_ready rendered insecure_skip_verify from
TLS_MODE and lookup('ca_injected', application_id) (#851). The lookup is
true only when Prometheus itself has a domain, TLS and the self-signed
mode, so in a self-signed deployment where Prometheus runs in another
mode the exporter had no CA file and skipped verification. All targets
share the module, so a target with a publicly trusted certificate was
probed without verification too, and anyone on the probe's path could
answer for a failed application.
http_ready now always verifies. A second module, http_ready_private_ca,
skips verification, and only the targets that need it select it: those
in the self-signed mode while the CA is not mounted into Prometheus,
through the __param_module label, which overrides the job's module for
that target. The default self-signed deployment mounts the CA and keeps
every target on the verifying module.
Verified on a local self-signed Tor round, where the CA is mounted:
the rendered blackbox.yml holds http_ready with insecure_skip_verify:
false and http_ready_private_ca with true, and no target of the scrape
configuration carries the module label. All targets of that round are
http onion names, so the https path itself ran only in the unit tests:
tests/unit/python/roles/web-app-prometheus pass, among them a publicly
trusted target that keeps the verifying module while the CA is missing,
a self-signed target that switches to the private-CA module, and the
verifying module rendering insecure_skip_verify: false without the CA.
Against the templates of main the three new cases fail.
Closes#851
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A local deploy stopped four times in two days in "Ensure Docker apt key
and sources are consistent" (#849). apt_repo.sh downloads the Docker GPG
key on every run, and when download.docker.com refused the connection,
curl's four attempts were over within about six seconds and the script
returned 1, although the key from an earlier run was installed and is
all the following apt operations need.
install_key now fails only when no key is installed. With a key in
place, a refused download prints a warning and keeps it. A downloaded
key that differs still replaces the installed one, and the 0644 mode is
enforced in every case. The keyring path became an argument:
ensure-key-and-sanitize passes the keyring it already hands to
sanitize_sources, so the role's call still resolves to
/etc/apt/keyrings/docker.asc, and a test can point the script at a
temporary keyring.
Verified: tests/unit/python/roles/sys-svc-container/files/test_apt_repo.py
stubs curl and sudo. A refused download keeps the installed key and ends
with changed=0, a refused download without a key exits 1, and a new key
replaces the installed one. A copy of the script whose keep branch always
errors, the old behaviour, fails the first test. Two local deploys ran
the new script with rc 0 and an empty stderr and left the installed key
untouched. shellcheck is clean.
Not verified live: the refused download itself, which needs
download.docker.com to refuse a connection during a deploy. A host
without a key still needs the download and still fails when it is
refused.
Closes#849
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make lint runs install-lint once and then dispatches its fifteen targets
in parallel. scripts/lint/mermaid.sh provisioned chrome-headless-shell
inside that parallel phase with `npx puppeteer browsers install
--install-deps`, whose `apt-get satisfy` carries no lock timeout. In a
fresh stack container another target held the apt lock, the shared
libraries stayed missing, the script only warned, and all 289 Markdown
files then failed on libnspr4.so (#795).
utils/install/lint/mermaid_cli.py now provisions the browser right after
mmdc, inside the serial install-lint step, as root with --install-deps
and as a user without it, because puppeteer refuses to install libraries
without root. A failed install deletes the chrome-headless-shell cache
folder and retries once under a 900 second timeout per attempt; a cut-off
download otherwise leaves a folder without its executable, and puppeteer
refuses every later install of that build. The install-lint stamp counts
as fresh only while the browser executable exists, so a recreated
container provisions it again.
mermaid.sh no longer calls apt-get or npx. It renders one minimal diagram
first and, when that fails, exits with mmdc's output instead of reporting
every Markdown file.
Verified: tests/unit/python/scripts/lint/test_mermaid.py stubs mmdc,
npx and apt-get. With a browser that cannot load libnspr4.so the script
exits 1 with that line, before any Markdown file renders, and mmdc is the
only tool it calls. Against main's mermaid.sh the same test fails: it
calls apt-get and npx and reports the files one by one. The installer and
stamp cases pass, shellcheck is clean.
Not verified: a full make lint in a recreated stack container. The race
reproduced in one of two runs in the issue.
feature/store-i18n changed the provisioning in mermaid.sh with 2f5f52bd1a
and dc67bce5f0 (timeout, deleting a cut-off browser). Both behaviours
moved into mermaid_cli.py here, so its merge takes this version of
mermaid.sh.
Closes#795
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A compose deploy job labelled centos or arch built its stack image from
pkgmgr-debian and ran it as infinito_nexus_debian (#794). distros.sh
exports INFINITO_DISTRO per iteration and sources load.sh again, but the
loader returned early because INFINITO_ENV_LOADED and the .env mtime were
unchanged. A full reload would not have helped either: the debian values
were already exported, so the restore step kept them as caller intent.
load.sh now records the distro it loaded for. Another INFINITO_DISTRO
defeats the early return and drops INFINITO_CONTAINER,
INFINITO_PARENT_IMAGE and INFINITO_IMAGE, the three keys the env handlers
derive from the distro. When the requested distro also differs from the
one .env was generated for, the loader sources a set that
`python -m cli.meta.env --output` generates for it in a private temp
directory and leaves .env untouched, so a local multi-distro run does not
switch the checkout's distro. A caller value set for the loaded distro
still wins.
Verified: tests/unit/python/scripts/meta/env/test_load.py and the new
distros.sh case pass. Against main's loader both fail with the symptom of
the issue: INFINITO_DISTRO=centos next to infinito_nexus_debian and
pkgmgr-debian:stable, and distros.sh handing debian, arch and centos the
same debian parent. shellcheck is clean.
Consequence: CI jobs labelled arch, ubuntu, fedora or centos now build
and deploy that distro, so a failure that shows up there was masked
before.
feature/store-i18n carries 3f973eb137, which resolves the parent image
for scripts/image/push.sh on its own, and 05e4dbafcb, which inserts the
INFINITO_INSTANCE slot check into the block this commit restructures. The
merge keeps the slot check ahead of the distro check.
Closes#794
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
The three workspace jobs of main run 38318560960 failed in test-external on
one URL: https://de.ssl-tools.net/mailservers/ in
roles/web-app-mailu/docs/Testing.md answers HTTP 404, and ssl-tools.net now
serves an empty page. The doc links the Internet.nl email test instead, which
checks STARTTLS, DANE, SPF, DKIM and DMARC of a mail domain.
Verified: the repository's own probe (_probe_with_retry in
tests/external/repository/test_urls_reachable.py) returns "fail HTTP 404" for
the old URL and "ok HTTP 200" for https://internet.nl/test-mail/ and for the
unchanged http://testemail.de/, and _extract_urls finds both links in the
changed file.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Stalwart writes its log to a dated file in the data volume, which the rescue
diagnostics do not collect, so the failure of Baserow#0 in #869 showed the
banned address but not which account and listener tripped the ban. A Stdout
tracer limited to the four ban events and auth.failed now puts those lines
into `docker logs`, which the rescue already captures.
Verified on Stalwart v0.16.25: the tracer is accepted with these event ids and
buffered false, and after a settings reload a ban shows up on the container
output as "Banned due to authentication errors (security.authentication-ban)"
with listenerId, localPort, remoteIp and accountName.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The recover drill between the two CI passes restores /var/lib/infinito/secrets
with rsync --delete, so PASS 2 starts with an empty token store and the role
replaces the app password of every bot. It destroyed the old one at once,
while alertmanager, OpenWebUI and Baserow, rendered in PASS 1, kept sending
with it until the preload rendered them again 15 to 25 minutes later.
Stalwart counts every refused login against the client's address, which for
host-originated and cross-bridge clients is the Docker gateway 192.168.200.1.
After 100 refusals it banned that address for good, openresty's upstream,
alertmanager and the msmtp health check had their connections reset, and
Baserow#0 of #869 failed with exit 76. The rescue artifact shows 31 of those
refusals, all from alertmanager; Stalwart's own log, which names the account
behind the ban, was not collected.
A replaced app password is now renamed to
STALWART_BOT_APP_PASSWORD_RETIRED_DESCRIPTION once the new one is stored, and
destroyed in the service-loader post-load flush next to the deferred OIDC
restart, after the preload has rendered every consumer with the new one. Each
deploy also collects retired passwords an earlier deploy left behind, so a
deploy that stops before the flush neither replaces the password a second
time nor drops the one it handed out.
Verified: on Stalwart v0.16.25, 100 refused logins from one container address
banned it permanently (BlockedIp, reason authFailure, expiresAt null) and new
connections from it were reset, as in the CI log, and an app password's
description can be updated in place. A playbook running the changed task
files with JMAP stubbed (ansible-core 2.21.4) shows that a PASS 2 replacement
destroys the old ids only in the flush; that a deploy failing after the
Stalwart preload destroys nothing, while the next one mints nothing and
destroys only the retired ids; and that outside the service loader each bot's
old password is destroyed right after its new one is stored.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Run 38009212981 lost web-app-n8n#0 to this, after thirty retries of a
deterministic failure:
File "<stdin>", line 121, in api_key
AttributeError: 'str' object has no attribute 'get'
GET /rest/api-keys answered 200 with something that is not a list of records.
'(body or {}).get("data") if isinstance(body, dict) else body' hands a string
or a dict straight to a comprehension, which iterates it character by
character or key by key, and .get dies naming neither the endpoint nor the
shape.
records() accepts a bare list or {"data": [...]} and otherwise exits saying
which listing failed and what type arrived. It does not guess at an unwrapping
rule for a shape nobody has observed - two n8n version bumps just landed
upstream, so the next run reports the real shape in one line instead.
The message never carries the body: the credentials listing holds secrets.
Left alone: the create path extracts a single object the same way. It is the
same family, but it is not what failed.
report_failures asked `gh api --paginate` for the run's jobs with the jq
filter `[.jobs[] | {name, conclusion}]`. gh applies that filter to each page
and prints one array per page, so a run with more jobs than one page holds
produced two JSON documents, and json.loads stopped with "Extra data: line 2
column 1". Main run 38218402254 (113 jobs) failed its "Report main role
failures" job that way, and no issue was opened for its failed roles.
The listing now emits one job object per line and parses the lines one by
one, the same way logs/download/github.py lists jobs.
Verified: the old parse of the real output of run 38218402254 raises the
same "Extra data: line 2 column 1 (char 8491)" as CI; list_jobs on that run
returns all 113 jobs, 16 of them failed, grouped into 12 failed roles.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
