The shared pacman cache was the first writable CacheDir on every arch runner and lab node. Our arch image sets DisableSandbox, so libalpm downloads straight into that directory (util.c _alpm_download_dir_setup only makes a per-process temp dir with the sandbox on), and the XferCommand writes <pkg>.part there with curl -C -o %o; two nodes fetching the same package appended to the same .part file, and the second rename failed or left a corrupt package in the shared cache.
pacman.sh now gives each node its own download directory <shared>/pacman/node-XXXXXX/ (created once with mktemp, reused from pacman.conf on later runs) as the CacheDir between the read-only host layer and <shared>/pacman/, and writes a PostTransaction hook to pacman.d/hooks/infinito-shared-cache.hook that renames every finished package and signature into <shared>/pacman/; .part files stay for the next resume. Both directories sit in one bind mount, so each publish is an atomic rename(2). pacman.sh fails when pacman.conf sets HookDir, since pacman would then skip the hook silently. Requirement 065 and cache.md describe the three layers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parallel compose slots, swarm lab clusters and GUI lanes each kept their own pip, npm and pacman downloads, and a failed swarm run left its cluster (about 25 GB of node docker data plus the rescue bundle in the act container) on the root filesystem until someone ran make swarm-down. INFINITO_CACHE_SHARED_HOST_PATH (default <DIR_CACHE>/shared) is now mounted read-write at the same path into every compose slot and every swarm lab node, with PIP_CACHE_DIR and npm_config_cache below it; empty, the renders stay byte-identical to before. On a host whose os-release ID is exactly arch running an arch stack, INFINITO_CACHE_HOST_PACMAN_PATH mounts the host pacman cache read-only and scripts/docker/cache/pacman.sh makes it the first CacheDir with <shared>/pacman/ second, so new downloads land in the shared folder and the host cache stays untouched; on Manjaro and every other host pacman uses the shared folder alone.
make swarm-zombie without keep=true now copies the act job's rescue diagnostics and Playwright artefacts to the host and then releases the lab nodes, the lab network, the NFS export volume and the act outer container, on success and on failure; when the copy fails the nodes still go, the act container stays and the run exits naming make swarm-down. make roundtrip passes keep to every swarm step, make clean-cache removes the shared folder, and the disc-space cleanup skips pacman cache directories it cannot write. apt and dnf keep their per-container caches behind the package-cache proxy, and no docker data root is shared between daemons. Requirement 065.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI run 40979252521: the workspace compose jobs (ubuntu, centos, manjaro)
ran out of the 345-minute budget inside compose/02_performance. On
store-i18n matomo variant 1 pins web-app-docs, whose variant 0 enables
dashboard, matomo, prometheus and design, and prometheus pulls the AI
chain, so round 1 of the matomo-only deploy grew from 1 app (main) to
12; all three jobs were cancelled inside the web-app-docs Playwright run.
The step now disables docs next to tor, so round 1 deploys matomo alone
as on main and the following dashboard-unreachable check holds again.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The workspace swarm roundtrip for svc-db-postgres failed on every host
with "The lookup plugin 'tls' failed: NETWORK_MODE 'multi' needs
svc-net-tor with a node onion on this node" (CI run 40979252521, job
Workspace debian:13). roundtrip.sh (b2469ff948) pinned the swarm step to
multi whenever the validator accepts it and never looked at disable=,
while 03_roundtrip.sh passes a disable list that removes tor. The
hard-fail is the documented contract (network_modes.md: tor or multi on
a node without the provider fails the deploy); the request was wrong.
roundtrip.sh now pins clearnet when disable= holds the Tor key (read
from utils.github.variant.network.TOR_SERVICE), else multi when the role
takes it, else the rotation. 02_zombie.sh, which also disables tor,
passes network=clearnet instead of relying on the rotation drawing it.
TestRoundtripSwarmNetwork runs the real script against stubs; the old
pin logic fails three of its cases.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings in feature/updater accb22da75, which carries main 5995766f58.
Conflict resolutions:
- web-app-fider: main's tenant bootstrap (01_tenant.yml) runs first on
the stack host, OIDC setup moves to 02_setup_oidc.yml, design to
03_design.yml; flat includes so main's gate test reads them.
- sys-front-tls test_nginx_ssl_header: the tls fake maps the domain term
store-i18n passes back to its app.
- group_vars 04_webserver.yml, sys-svc-proxy CSP header: store-i18n's
webserver_resource lookup plus #1256's WEBSERVER_SECURITY_HEADERS and
the security_headers.lua include after the CSP block.
- web-app-nextcloud: updater's decoupled Talk stack (Janus, signaling,
NATS, recording builds) replaces aio-talk; store-i18n's per-network
Talk backends move from the wrapper script into signaling.conf.j2 urls
(NEXTCLOUD_HPB_SIGNALING_BACKEND_URLS from network_urls, canonical
first); IS_COMPOSE_MODE/IS_SWARM_MODE booleans kept; the integration
OpenProject digest stays delegated to STACK_HOST.
- users lookup docs, token-store write_many, cache README, meta info
lint: both sides combined.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Conflict in roles/web-app-gitea/tasks/01_manager_ops.yml: main (c59e2838d6, c2577560f8) moved the readiness probes inside the Gitea container on its hosting node, because a host port lands on the swarm ingress mesh; this line (381c3cd5bc) had reduced them to one wait for /api/v1/version = 200 and dropped the explicit `gitea migrate`, which raced Gitea's own migration on galaxyserver. The resolution keeps main's in-container probe as that single wait with `curl -fS` (200 required), after the resolve and with this line's 600 s budget scaled by lookup('timeout'), and drops main's app.ini wait, its migrate task and its post-admin readiness wait.
main's new tests/unit/python/roles/sys-front-tls/templates/test_nginx_ssl_header.py renders nginx_ssl_header.j2 without `domain`, which ssl_credentials.j2 needs on this line since 47160d755e; its render() now passes the vhost's domain.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The Validation step "Assert the base branch is fully merged into the PR
head" ran main_merged.sh, which fetches the live tip of main when the job
starts. PR runs wait in the queue for hours while image-update PRs land
on main several times an hour, so nearly every queued run failed
Validation on start and had to be re-pushed after another main merge
(PR #1265: behind_by=0 at push, behind_by=9 while its run still queued).
main_merged.sh takes a new INFINITO_MAIN_COMMIT override: when set, it
fetches that full commit SHA from the remote and requires HEAD to
contain it instead of the live tip; a set but malformed or empty value
exits 2. The Validation step passes github.event.pull_request.base.sha,
the main tip at event time, which the step's pull_request-only guard
always provides. The run tests the merge against that base, so the gate
still guarantees the tested tree contains it.
The local pre-push hook (make test-main-merged) leaves the override
unset and keeps checking the live tip: a push happens right now, so
lagging the current main is the real defect there.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Brings feature/updater 3966abdb16, including updater-lockdown-probe: the deferred probe host guard (aa88429c25), the per-host post-load queue (70ed8a2e98), the NFS probe noauto (0ece4fcbc4), the E2E prepull gate (ef20262813), the worktree swarm lab subnet fixes, the Tor port window and restart skip, the Prometheus Docker poller, the Stalwart JMAP client cache and the FusionDirectory PHP hardening mount.
Conflict resolutions:
- plugins/lookup/email.py: keep store-i18n's NETWORK_MODE forwarding into the provider view and add updater's provider carrier.
- roles/svc-net-tor/templates/test.env.j2: updater's ONION_PORT_TIMEOUT=130, store-i18n's ONION_SUFFIX.
- roles/svc-swarm-node/README.md: updater's ingress guard wording, without the generated Quick Setup and Credits sections.
- roles/test-e2e-cli/tasks/main.yml: updater's cli_role_id loop variable with store-i18n's async cli.once condition.
- roles/web-app-prometheus/templates/nginx/locations.conf.j2: store-i18n's entity_name filter with updater's container_service lookup.
- tasks/stages/01_constructor.yml: updater's order (svc-swarm-node after the preload context opens) under store-i18n's task name, whose loop also covers svc-cache.
Integration fixes:
- roles/sys-stk-front-proxy: git applied updater's IS_STACK_HOST guard to the "Freeze front proxy context" task, which would have left workers without front_proxy_* facts. main.yml stays as on store-i18n; the guard moves into the per-sibling vhost.yml split, where the probe facts and the queue sit behind IS_STACK_HOST and front_proxy_domain_conf_dst_resolved stays unguarded for the Deploy arguments. deferred_probe.yml takes updater's host guard verbatim.
- roles/svc-net-tor/tasks/00_core.yml: the restart probe tests IS_SWARM_MODE instead of a literal DEPLOYMENT_MODE comparison.
- roles/web-app-stalwart/tasks/jmap_call.yml: the cached client splits into stalwart_jmap_url and stalwart_jmap_user_password, so the exec argv references no fact that carries a credential.
- roles/sys-service-loader/tasks/list_or_shoot.yml: the two direct-shoot tasks share their preload condition in one block.
- tests: the Prometheus wiring test imports entity_name and pins the host resources its renders need; the email carrier tests use an onion-only provider, which on store-i18n is what drops TLS (a dual-stack provider keeps it).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The repository installs mbake unpinned and shipped no .bake.toml, so the
Makefile's final newline depended on the mbake version:
- mbake 1.4.6 (e.g. the pkgmgr venv that wins on a host PATH) defaults
to ensure_final_newline = false: `mbake format` strips the newline and
`mbake format --check` accepts the stripped file.
- mbake 1.5.0 (what pip installs today) rejects the same file with
"Missing final newline", which is what CI reported on PR #1255.
So a local `make autoformat` could strip the newline, the local
lint-makefile stayed green, and CI failed; 6f653ce633 and #1262 had to
restore the newline by hand.
The tracked .bake.toml sets only ensure_final_newline = true, so every
other formatter rule keeps its default. autoformat.sh and makefile.sh
pass it with --config, so the result no longer depends on the version
or on the directory mbake runs from. Checked with both versions: main's
Makefile passes `--check --config .bake.toml`, and a copy without the
final newline fails it with "Missing final newline".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The comment rule allows parameter docs and nocheck directives only, so
the five Exception comments this branch added go. What they explained:
- cli/administration/roles/marker.py: O_NONBLOCK keeps a FIFO planted
under the marker name from blocking the open; the fstat refuses it.
- cli/administration/roles/source.py: a git hook exports GIT_DIR,
GIT_INDEX_FILE and friends, which would point the scratch repository at
the core checkout; the askpass helpers and the detached session keep a
fetch that needs credentials from prompting.
- roles/web-app-bluesky/files/login-broker/server.js: the account map
lives in the broker's own volume, never in a Keycloak user attribute; a
bluesky_* attribute write under LDAP federation in WRITABLE mode
triggers a schema-violating sync push that breaks later OIDC logins.
- roles/web-app-yourls/files/php/upgrade_database.php: YOURLS reports
bootstrap and upgrade errors with die(), which exits 0; only the
shutdown handler turns them into a failure.
- scripts/tests/deploy/act/workflow.sh: act copies the checkout through
.gitignore, which drops the pulled custom roles, so the job binds the
checkout to keep them.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The repository installs mbake unpinned and shipped no .bake.toml, so the
Makefile's final newline depended on the mbake version:
- mbake 1.4.6 (e.g. the pkgmgr venv that wins on a host PATH) defaults
to ensure_final_newline = false: `mbake format` strips the newline and
`mbake format --check` accepts the stripped file.
- mbake 1.5.0 (what pip installs today) rejects the same file with
"Missing final newline", which is what CI reported on PR #1255.
So a local `make autoformat` could strip the newline, the local
lint-makefile stayed green, and CI failed; 6f653ce633 and #1262 had to
restore the newline by hand.
The tracked .bake.toml sets only ensure_final_newline = true, so every
other formatter rule keeps its default. autoformat.sh and makefile.sh
pass it with --config, so the result no longer depends on the version
or on the directory mbake runs from. Checked with both versions: main's
Makefile passes `--check --config .bake.toml`, and a copy without the
final newline fails it with "Missing final newline".
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
68 conflicts, resolved per hunk; resolutions agreed with the Updater
session were applied as agreed:
- svc-db-postgres (superuser password script and core) and
sys-svc-rdbms realign_dedicated_db: updater's argv/stdin redesign with
expand_argument_vars: false, nested into store-i18n's block structure.
- svc-db-redis 01_init: store-i18n's ACL password on stdin.
- sys-svc-proxy forwarded.conf.j2: store-i18n's handover-aware file,
without the forwarded_base include (Host and X-Forwarded-Proto would
be sent twice); forwarded_base serves the inline consumers, e.g.
web-app-seaweedfs nginx.conf.j2.
- basic.conf.j2 and web-app-openproject proxy.conf.j2: issuer redirect,
then handover, else updater's location/sso_acl.conf.j2; the
server-level webserver_extra_configuration hook stays.
- web-app-keycloak: updater's admin-REST settings step and renumbered
update steps; update/01_client.yml removed; store-i18n's design step
and LDAP block kept; a73435cc9d arrived through rename detection.
- web-app-nextcloud OpenProject provisioning: updater's rewrite, all
credentials on the Ruby runner's stdin.
- web-app-semaphore: store-i18n's canonical web root (a1d7835863).
- Role READMEs: no generated Cosmos/Architecture/Quick Setup/Credits
sections; desk-* paths resolved to the dsk-* renames.
Integration fixes for what merged cleanly but did not fit together:
- web-app-matrix ansible vars: drop updater's second
matrix_client_element_configuration_extension_json, which silently
overrode store-i18n's design-aware value.
- url.origin instead of a stripped url.base in 19 places store-i18n
added since 44aaa68924, same lookup targets (the double-slash lint).
- Package batch (18ea75049e, cb4d340de9): installs the deploy-start
batch covers are removed (caffeine, cursor, blender, npm in dsk-gnt-*,
with shared references), the rest carry batch: false and a nocheck
with the verified reason (PyPI fallback, before the batch, vendor
repository, configured flavor only, addon gate); dev-npm references
the root npm id.
- sys-token-store write_many.yml: store-i18n's change-gated block and
IS_SWARM_MODE.
- web-app-openproject: IS_COMPOSE_MODE in 02_migrate/run.yml; the
seeded-admin lock (ccbf8f168d) ported into 04_admin/provision.yml and
covered by the admin-state digest.
- Bucket policy (3b4ac40df5 and follow-ups) moved to web-svc-seaweedfs,
where store-i18n's object store lives; the S3 admin secret travels on
stdin instead of a task-level environment.
- act workflow.sh: no Exception-labelled comment.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
a7685e3b21 derived the worktree lab subnet as 10.244.<slot>.0/24, which
lies inside NETWORK_TOR_VIRTUAL_CIDR (10.240.0.0/12). On a swarm with
svc-net-tor the nft output chain redirects every TCP connection to that
range into Tor's TransPort, so the nodes could not reach each other: in
the slot-114 worktree `curl http://10.244.114.10:5000/v2/` timed out on
the manager itself and svc-prx-openresty died pushing to the stack
registry with Client.Timeout.
The prefix is now 10.160.<slot>, outside every range the platform
reserves in group_vars/all/08_networks.yml.
The lint now reads the reserved ranges from that file instead of a
hardcoded list: NETWORK_TOR_VIRTUAL_CIDR, NETWORK_MESH_POOL,
NETWORK_DOCKER_ADDRESS_POOLS, NETWORK_NESTED_DOCKER_ADDRESS_POOLS and
NETWORK_NESTED_DOCKER_BIP, plus the primary lab subnet and the worktree
INFINITO_SUBNET range. Against the 10.244 prefix it fails every slot on
the Tor range.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Step [2/9] of the DR drill starts svc-bkp-nfs-2-local on the NFS server,
which writes a full first generation of /srv/nfs into
/var/lib/infinito/backup/<machine-id>/backup-nfs-to-local on the node's
own root filesystem, i.e. the node container's writable layer on the
runner disk. Step [4/9] pulls that generation to the backup host. After
that nothing reads the NFS-side copy (the recovery restores from the
device into the restore root, and verify_recovered_marker.sh reads the
live export), and nothing deletes it: the drill never touched it and
sys-ctl-cln-bkps always keeps the last generation.
In run 40625679270 that left the size of the export on the runner disk
twice over while the [5/9] space gate measured the backup host:
shopware#0's NFS unit took the runner from 49105M to 37275M free and the
NFS node ended 13.1 GB above the control node, and SuiteCRM#0, BookWyrm#0,
Penpot#0, Funkwhale#0 and Snipe-IT#0 refused at "the sync holds the
...M pulled tree and its device copy at once ... must be free" (SuiteCRM:
8533M free, 20942M needed, 14563M of it the NFS-side generation).
The drill now removes ${DIR_BACKUPS}/${NFS_MID}/backup-nfs-to-local on
the NFS server right after the marker is confirmed on the backup host,
before the space gate measures.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
An inventory group web-app-cst-<name> with applications.<id>.custom: true and
a pinned source {url, version, path} is pulled into roles/ before the play,
read raw (Jinja, Ansible tags and credential URLs refused), placed through a
staging directory under an exclusive lock, and owned by a marker that names
its own directory, so a pull or roles-clean never replaces or removes a
directory it did not place and a failed pull leaves roles/ as it was.
Directory inventories, host lists and non-UTF-8 files behave as Ansible
reads them; generated test deploys pass --skip-pull. make roles-pull,
roles-clean and roles list, INFINITO_EXTRA_MOUNTS for local sources, the
web-app-cst category, the reserved prefix lint and a central
repository_files() that also yields the gitignored custom roles let every
lint scan them like in-tree roles. Supersedes the uncommitted 045 External
Roles draft and reuses its fetch, sync and mount design.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The nfs-ganesha drop-in sets Environment="OPTIONS=-L SYSLOG ...", but the
Debian and Ubuntu unit (upstream nfs-ganesha.service.debian8) also reads
EnvironmentFile=-/etc/default/nfs-ganesha, whose OPTIONS name
-L /var/log/ganesha/ganesha.log, and systemd lets EnvironmentFile=
override Environment=. So on those distros ganesha logged to a file with
no bound, while CentOS, Fedora and Arch already logged to the journal
(SystemMaxUse=256M on the lab node).
Under MODE_DEBUG ganesha.conf raised NFS_V4 to DEBUG since a085427236,
which writes a dozen lines per NFS compound. In odoo#1 of run
40625679270 (debian) the nfs-server node grew 12.6 GB above the control
node and the step aborted at "DISK EXHAUSTION IMMINENT (<6144M free on
/)". In the same run, nodes logging to the journal grew 0.8 to 1.6 GB
in four hours, nodes logging to the file 4.4 to 12.6 GB.
- The drop-in resets EnvironmentFile= before setting OPTIONS. The
Debian unit's ExecStart reads nothing else from that file (its journal
reports EPOCH, GNODEID, NUMACTL and NUMAOPTS as referenced but unset).
- NFS_V4 goes back to the default level: it was raised to chase the
probe-write bug that 9b7f25f2fe fixed. DISPATCH stays at DEBUG, since
nfs_Init_svc(), which binds 2049, logs under it (1aea8a472e).
- The swarm diagnostics read the last 600 journal lines of nfs-ganesha
instead of the file tail.
test_ganesha_log_levels and test_ganesha_unit_options pin the debug
components and the drop-in's OPTIONS source; against the previous
template and drop-in both fail.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
docker_dataroot.sh backs each node's ZFS pool with a sparse loop image
and created the pool without autotrim. ZFS never discarded freed blocks
to the loop device, so the image only grew: in WordPress#0 of run
40625679270 the runner still held 118G after the teardown had removed
every container ("Containers: 0") while only 47G of files remained, and
the pools held ~38G of live data. The step had already aborted at
"DISK EXHAUSTION IMMINENT (<6144M free on /)" with 139G of 145G used
(rc=75). The other ZFS swarm jobs of the run show the same df-minus-du
gap (wordpress1 73G, odoo1 73G, magento0 82G, weblate1 55G, gitlab2
44G); ext4 and btrfs jobs show none.
The pool is now created with -o autotrim=on, so ZFS discards freed
ranges to the loop device, which punches holes into the image file.
test_docker_dataroot runs the script's zfs path against stubs of every
host tool and asserts the single zpool create carries -o autotrim=on;
against the previous script that assertion fails.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
01_bootstrap.sh pins the five swarm nodes into the shared cache network
at 200 + INFINITO_INSTANCE * 5. That band fits a /24 only for instances
0 to 10; from instance 11 on the script aborted with "pushes the node
band past ...254", so no worktree past slot 10 could start a swarm
cluster while the package cache runs. Five nodes per slot cannot fit
255 slots in one /24 with any fixed formula.
Instances whose band fits keep their pinned addresses, so the primary
checkout and CI behave as before. Above that, the nodes join the cache
network without --ip and Docker assigns them from its pool (the upper
half of the subnet), the same way every worktree stack container joins
it. Nothing in the repository reads the nodes' cache-network addresses.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
scripts/system/worktree/up.sh pinned INFINITO_SWARM_LAB_SUBNET to
192.168.$((244 + slot)).0/24. From slot 12 on that third octet exceeds
255, so every worktree past slot 11 carries an invalid subnet
(192.168.358.0/24 for slot 114), and make swarm-zombie dies at the lab
network creation with `invalid subnet: netip.ParsePrefix(...)`.
The lab prefix is now 10.244.<slot>, valid and distinct for every slot
up to max_slot 255, and outside both the primary lab subnet
192.168.244.0/24 and the 172.30.0.0/16 worktree subnets. The host
daemon allocates nothing from 10.0.0.0/8.
A new lint renders the prefix for every slot and fails on an invalid,
repeated or overlapping subnet; against the old derivation it fails 247
of 255 slots.
Worktrees that were pinned before this fix keep their custom.env; the
ones at slot 12 or above need their INFINITO_SWARM_*_IP and
INFINITO_SWARM_LAB_SUBNET pins rewritten to 10.244.<slot>.x and a
make dotenv before they can run swarm.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2bf329cd3b put rescue_window.sh next to rescue_index.sh, and
test_prefix_grouped_files rejects two siblings sharing the `rescue`
prefix (quality-high pass 67). Both move to utils/rescue/ as index.sh and
window.sh, their tests to the mirrored tests/.../utils/rescue/, and every
caller follows: dedicated.sh, the swarm teardown in 00_one.sh, the
workspace workflow, and the swarm and diagnostics READMEs.
The budget-kill drill gave the stand-in a 1 s budget; under the full
parallel unit run the SIGTERM arrived before its EXIT trap was installed,
so nothing was captured and the test failed. The two kill drills now use
KILL_BUDGET = 3 s; their assertions are unchanged. A full `make test-unit`
passes them under load.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
distros.sh stops a distro at its budget with `timeout -k 60`: SIGTERM at
the deadline, SIGKILL 60 s later. dedicated.sh's cleanup trap then ran
the in-container capture under `timeout 1500` (266 s in Baserow#0 of run
40625679270) and the host capture under `timeout 900`, so the SIGKILL
always landed mid-capture, the snapshot was never copied out and the
rescue-diagnostics artifact was lost. The inner `timeout` also sat in its
own process group and outlived the kill, holding the step open.
- distros.sh keeps the grace in one place, KILL_GRACE_SECONDS, and
exports INFINITO_CI_DISTRO_KILL_EPOCH next to the deadline epoch.
- scripts/tests/deploy/utils/rescue_window.sh decides per trap step:
past the deadline, the step gets half of the time left until the
SIGKILL (at least 1 s) and an end epoch; before it, the nominal
timeout and no end epoch. Inside the trap `$?` is 0 on a budget kill,
so the deadline epoch is the only reliable signal.
- dedicated.sh runs both captures through `timeout --foreground` with
that window, keeping them in the trap's process group, and passes the
end epoch as RESCUE_DEADLINE.
- utils/diagnostics/container.py caps every command at RESCUE_DEADLINE,
writes `[skipped: past RESCUE_DEADLINE]` for what no longer fits and
hands the deadline to nested captures. Without the variable nothing
changes, so the swarm and workspace callers behave as before.
-k 60, the distro budget and every timeout-minutes are unchanged. Drill
tests replay the trap with the real helper and container.py: a budget
kill ends with rc 124 and the snapshot copied out, the old shape ends
with rc 137 and nothing copied, an early failure keeps the full capture.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
svc-bkp-nfs-2-local snapshots the export base (/srv/nfs), so a
generation's files/ root already holds infinito-state/. Without an
explicit target, `recover nfs` and `recover full` restored it onto
/srv/nfs/infinito-state: the tree came out as
infinito-state/infinito-state/<volume>, and `rsync --delete` wiped every
live volume directory and the backup root under the old target.
- paths.NFS_EXPORT_BASE = get_export_base(), the function behind
lookup('nfs_export_base') that also feeds the backup unit's source,
replaces NFS_EXPORT_STATE. NfsRecoverer defaults to it; an explicit
`snapshot:/abs` target still wins, a remote target is still prefixed
with its host. `recover full` inherits the default.
- recover_drill.sh seeds its marker at files/infinito-state/<volume>/
and asserts it under the export base, so the drill can no longer pass
on a nested layout.
- The help examples, schema.yml and the CLI README show the export base;
the documented subtree example would have nested too and now names the
matching snapshot subtree.
Reproduced on a temp tree with the role's real script.sh and recover.py
and the CLI's argv: the old default loses every live volume and the
backup root, the new one restores in place. The new tests fail 6 of 6
against the old default.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The NFS backup rsynced the whole export and honoured only one exclude, the
shared backup root. Volumes declared `backup: false` were copied anyway and
then pulled a second time by remote-2-local; in the swarm DR drill of
Magento#0 (run 40625679270) docs_sites alone (a full repo clone plus its
prepared and work trees) took about 15.5 GiB per copy and filled the
runner. The volume backup already skipped these through the backup_volume
lookup; only the NFS path ignored the declaration.
- utils/storage/nfs.py: volume_dir() names the state directory a volume
binds to (pinned name or key), shared with compose_volumes;
backup_disabled_dirs() returns every NFS-backed `backup: false` volume
dir that no backed-up volume also binds to.
- The role-local lookup backup_nfs_excludes feeds them into
BKP_NFS_2_LOCAL_EXCLUDES next to the backup root; script.sh takes any
number of excludes. Compose installs still pass only the backup root.
- recover.py protects every live state directory the snapshot does not
hold, so `rsync --delete` keeps the excluded volumes (model weights,
the docs tree, caches) and volumes created after the snapshot instead
of wiping them.
- The swarm DR drill restores BACKUP_NFS_VOLUME, the first NFS volume
not declared `backup: false`; PRIMARY_NFS_VOLUME stays for the drain
tests. BookWyrm now drills bookwyrm_media; decidim, fusiondirectory,
meta, coturn, svc-registry-cache and svc-registry-docker have nothing
backed up on NFS and skip the drill.
Measured: compose_volumes renders identically before and after for 291
roles across compose/swarm x local/nfs (1940 of 1940); the 15 excluded
directories are exactly the backup-disabled device dirs; the systemd unit
renders shell-safe on ansible-core 2.20 and 2.21 and stays byte-identical
in compose and swarm-local. A new unit test runs the real script.sh over
two generations and the real recover.py back into the export, and fails
without the restore protection.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A compose job that runs out of its distro budget is stopped by
`timeout -k 60` in distros.sh: SIGTERM at the deadline, SIGKILL 60 s
later. dedicated.sh's cleanup trap copied the generated inventory out of
the DinD container only after the rescue capture, which alone took 266 s
in Baserow#0 (run 40625679270), so the kill always landed first. The zip
step then failed with "No inventory found", never wrote INVENTORY_ZIP,
and the upload step failed again on an empty `path`; two follow-up errors
buried the real `rc=137`. Nextcloud#0 and #2 of the same run show the
same trio.
- The inventory copy moves unchanged to the top of the trap, ahead of
the Playwright copy and the rescue capture; the three write to
disjoint directories and none reads another's output.
- The zip step writes INVENTORY_ZIP before running the zip script, so a
missing zip falls under the upload's existing
`if-no-files-found: ignore` instead of failing the step a second time.
The budget overrun itself came from the docs README replay pulled into
the round through partners, which 91022ad37a removes.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The chunk jobs read `🧱 Chunk N` and every priority row carried its own
trailing ⭐, so a priority sweep repeated the star down the whole job
list while the chunk that held those rows looked like any other.
- The four chunk jobs are named `0️⃣ Batch` to `3️⃣ Batch`, with ` ⭐`
when the batch holds priority rows, and their discover job reads
`🔍 Batch N ⭐` through the existing `marker` input.
- call-validation counts those batches once per run
(scripts/github/resolve/priority_batches.sh, which runs
`cli.meta.ci.matrix --priority-chunks`) and exposes the count as the
`starred` output. chunks.plan cuts the priority chunks from the
priority rows alone, so offset and the regular line cannot move them;
the orchestrator hands validation the same sweep and chunk size it
hands the chunk jobs.
- The row label drops its own star. report_failures and the job-name
parsers still accept a starred title from an older run.
- The glossary names the unit `batch` (`#️⃣`) instead of `chunk` (`📦`),
`ci plan` prints the batch column as the keycap the job name leads
with, and the name lint accepts a leading keycap emoji.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_urls_reachable read http.https://github.com/.extraheader as a link and failed with HTTP 404. The header is set through GIT_CONFIG_* for the one git push only, which talks to github.com alone, so the unscoped http.extraheader reaches the same requests.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Local lint runs hand the container the files changed since the merge-base with INFINITO_LINT_DIFF_BASE; a changed tool config or shared lint script, CI, or a missing base fall back to the full run. autoformat records a content stamp after a clean ansible-lint --fix, so lint-ansible skips the second pass on an unchanged tree. shellcheck and sqlfluff check in parallel.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
bf25eb2fc7 removed the sync-main job because the fork had no token that
may push .github/workflows/**: GITHUB_TOKEN cannot be granted the
workflows permission, and the bot App is installed on infinito-nexus/core
only. Since then a fork's main only moves when its owner syncs it by
hand, and the fork's main has fallen 368 commits behind.
The job is back with a token of its own: CI_SYNC_MAIN_TOKEN, a
fine-grained personal access token limited to the fork, with Contents
and Workflows write. main_from_repository.sh resolves the source through
baseline_remote.sh, so the normalization and the off switch exist once:
in infinito-nexus/core itself and with CI_SYNC_MAIN_SOURCE_REPOSITORY
disabled the job ends green without reading the secret. Everywhere else
a missing secret fails the job before any fetch or push and names the
page that explains how to create it. The token reaches git as an HTTP
header through GIT_CONFIG_* variables, never through argv or
.git/config; checkout runs without persisted credentials so its own
header does not compete.
The job runs beside run-policy, not before it: CI keeps fetching its
diff baseline from the source repository read-only, so a failed sync
turns the run red without holding CI back.
fix/ci-mirror-token tried the bot App instead (019fb572a7, 756b180f1d).
That needs the App installed on every fork and granted Workflows write,
which secrets.md rules out for the App, so the token here is a
per-fork PAT.
Checked: the script skips as infinito-nexus/core and with the sync
disabled, and fails before fetching as a fork without the secret.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
When the pulled tree and its device copy would not fit, the drill printed du -sm per machine/repo, which left most of the ~16 GB of Magento#0 and Penpot#0 (run 40131272595) unaccounted for. It now prints the 25 largest directories down to four levels below the backup root, so the next refusal names the volume directories behind the size.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Resolves 68 conflicted files. The governing rule was our structure with their
network semantics: the tor boolean becomes a network mode (clearnet | tor |
multi) throughout the variant layer, while this branch's architecture and mesh
axes, its extracted modules and its i18n content stay.
Where the two sides covered different ground, both are kept rather than one
picked:
- ca_injected answers a ca_client on the deployment's TLS mode OR on an SSO
issuer that resolves self-signed. 3a3e9ed2c9 records why: an issuer is not
an object store, so keeping only one side drops the other's coverage.
- get_merged_domains keeps the loud refusal when svc-net-tor is deployed
without a provisioned node, now gated on their tor_serves_host; their
node_onion_of returns an empty string there, which would have served every
onion-exclusive app on clearnet in silence.
- sso_cookie_domains stays the single source for oauth2-proxy's cookie scope.
Their per-network branch omitted the issuer domain on the onion arm, which
breaks the callback leg.
- The clearnet CSP batch carries --clearnet-timeout again; their batch dict
hardcoded zero while the role still passes the flag.
- The dashboard keeps its i18n content dir, image, version and design
resolution; their side reduced the logo sources to the bare service
provider values.
Their side wins where it is measurably better: the onion mirror rewrite is
gated per vhost rather than per app, the Playwright onion decision reads the
pass's own domain instead of the role's canonical one, and the matomo
reinjection and dashboard config render per network sibling.
Eight files were broken by the merge without ever carrying a conflict marker,
because git marks only the lines both sides touched:
- utils/cache/domains.py lost the onionize block that sat between two hunks
- roles/sys-front-inj-all/templates/body_filter.lua.j2 called it
- web-app-bluesky, keycloak sso_proxy: the get_entity_name filter was renamed
- trigger/__main__.py, playwright_coverage.py and its test: the deleted tor
module
- cli/administration/deploy/ci/timings.py read Label.tor
- plugins/filter/csp_filters.py called _aligned_url with four of six arguments
- tests/unit/python/utils/cache/test_domains.py referenced a renamed class
- tests/.../sys-ctl-hlth-csp/files/test_script.py mocked a function main no
longer calls
Verified: the variant layer (155 tests, 65 subtests), ca_injected (18),
the CI layer (175), the domains cache including the restored guard (32),
retrigger inputs (9), timings and the CSP health script (53). make
quality-high clears autoformat; its remaining findings are tracked separately.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Conflicts resolved:
- Dockerfile: keep HEAD; filesystem.sh already runs at line 37 and pin_epel()
arrived inside centos_mirror.sh, so main's hunk would have run it twice.
- peertube 02_fix-application-schema.yml: take main's task name, its
expect_rows: true (matching the first query in the same file) and its when:
without default(); keep HEAD's changed_when: true, which the notify:
compose-up depends on.
- peertube check-application/defaults.sql: take main's information_schema
query; HEAD's ::regclass cast raises instead of returning no rows when the
table is absent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The wizard bypass left ERPNext without a Company, so no app could create quotes, orders or invoices. The deploy now creates the Company from the service provider's name and country, with the currency from ERPNext's country data and an abbreviation that defaults to the name's initials, plus a fiscal year, five tax accounts, a cash account and the default company. It never rewrites an existing Company and fails loudly on a country ERPNext does not know. The README generator marks roles that read the service provider country, and the CI replay fills the placeholder.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
`shfmt.ensure()` returned as soon as any `shfmt` was on PATH and tried
the distribution package before the release pinned in
roles/sys-lint/meta/services.yml. The Debian dev container therefore
formats with its packaged 3.8.0 (/usr/bin/shfmt), Fedora with its packaged
3.7.0, the CentOS one with the pinned v3.14.1 (/usr/local/bin/shfmt), and
they disagree on
scripts/tests/deploy/swarm/utils/unmount/nfs_mounts.sh: every gate in a
CentOS stack moved the `then` of the heredoc `if` below the terminator.
ensure() now follows ruff's installer: it reads `shfmt --version`, keeps
the binary only when it reports the pinned release, and otherwise installs
the release binary into /usr/local/bin. It fails when PATH still resolves
another version afterwards instead of formatting with it. The system
package path is gone.
nfs_mounts.sh is committed in the pinned release's format (byte-identical
to the v3.14.1 output from the CentOS container), so the first gate on any
distro leaves the tree clean.
The install-lint stamp now also depends on roles/sys-lint/meta/services.yml,
the pin file of actionlint, hadolint and shfmt, so a bumped pin reaches
environments whose stamp predates it; until now only pyproject.toml and
scripts/install/lint.sh invalidated it.
Fixes#1155
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
compose/01_minimal.sh disabled every variable service of the dashboard,
asset included, and ran both matrix rounds; round 1 of web-app-dashboard
pins the asset provider off as well. Without web-svc-asset the dashboard
renders its logo and favicon from the public
https://file.infinito.nexus/assets, so the workspace test depended on
that host: while it does not answer, Playwright's no-console-noise spec
fails on `net::ERR_CONNECTION_REFUSED` for the logo (main run
39684179395, ubuntu and centos).
The test now keeps the asset provider out of the disable set and pins
round 0, so the dashboard serves its branding from the deployment's own
file service. Round 1 stays covered by the regular dashboard matrix.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
scripts/install/filesystem.sh retried `dnf install -y zfs` three times
against the same metadata during the Fedora image build of main run
39684179395 (XMPP#0): every attempt listed pcp-*-7.2.2-1.fc44, every
mirror answered 404, "Failed to download packages" after 3 attempts.
Each retried dnf install now runs with --refresh, so every attempt
resolves against freshly fetched metadata instead of repeating the one
that lists packages no mirror serves anymore.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The package cache creates every Nexus proxy with
strictContentTypeValidation. A raw proxy has no type contract, and Nexus
cannot type a GPG keybox: on a fresh cache it answers the Node.js release
keyring (raw.githubusercontent.com/nodejs/release-keys/.../pubring.kbx)
with 404 "Content type could not be determined". The Node.js installer
from #1108 fetches that keyring, so on main (run 39668792538) Workspace
debian:13 fails in `make install-lint`, and Workspace ubuntu, centos and
manjaro fail in `test-external`, whose URL check reaches the same file
through the cache.
The four raw proxies (raw-githubusercontent, raw-codeload-github,
raw-packagist, raw-alpine) now share the storage block with validation
switched off. Every typed proxy (apt, pypi, npm, helm, rubygems, go, yum)
keeps strict validation. The installer's integrity check does not depend
on Nexus: it verifies the SHASUMS file with gpgv and the tarball with
SHA-256.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
diagnostics.sh runs under a 900-second deadline that the sweep deadline
can shorten further. The package manager section came last, after the NFS,
ganesha backtrace and controller probes, so a cut-off run would have lost
exactly the log this collection exists for. It now runs first.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Six Fedora jobs of main runs 39494396920 and 39590304501 (Postmarks#0,
OpenBao#1, Jenkins#1, Magento#0, Matrix#2, OpenResty#0) died in dnf with
`Failed to download packages`. Ansible's dnf5 module reports nothing
more: `failures: []`. Its file logger writes the download errors, with
mirror, URL and status, at debug level to /var/log/dnf5.log, but no
rescue bundle collects that file, so the cause of five of them stays
unknown.
The swarm stack diagnostics now append the last 3000 lines of
/var/log/dnf5.log and, for dnf4 hosts, /var/log/dnf.librepo.log of every
node. A node without them records the `tail` error and the capture goes
on.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A selection token may pin the mesh state as &vpn or &direct, but check_pins
had no arm for it. pinned_vpn_states intersects the pin with the states the
mode allows and only swarm meshes, so a token like @compose+clearnet&vpn left
an empty state list: the row produced no job at all and the run reported green
for a combination that never deployed. That is the outcome _reject exists to
prevent, and every sibling axis already refuses an impossible pin.
The new arm asks whether any candidate mode keeps a state rather than whether
the pin is reachable, so a held vpn axis still overrides the pin instead of
being refused. That behaviour has its own test and is unchanged.
An earlier check was removed with the compose-mesh revert of 2026-10-16
because it rested on meshable() and smallest_variant(), which that revert
deleted. This one reuses pinned_vpn_states, applies to the swarm-only axis the
revert restored, and covers both mesh states instead of only &vpn.
pin_vpn and vpn_mode now reach check_pins from all three callers. The run's
vpn axis reaches the validation job through call-orchestrator and
call-validation, which passed every other axis but this one. The priority
procedure and its skill state all eight axes a token carries, not six:
architecture and mesh were pinnable and, for architecture, already validated,
yet neither was documented.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
`security-apparmor-teardown` ran `aa-teardown`, which unloads every
profile including dockerd's in-memory `docker-default`. dockerd only loads
that profile when it starts, so every container start afterwards that
requests it fails until dockerd restarts. Since #1067 runs the teardown on
the GitHub runners, BuildKit failed each RUN of the swarm node image three
times (run 39332343490, 39 of 40 swarm jobs):
runc run failed: unable to start container process: error during
container init: unable to apply apparmor profile: apparmor failed to
apply profile: write .../attr/apparmor/exec: no such file or directory
The teardown now removes each loaded profile by name through securityfs
and skips `docker-default` and child profiles, which leave with their
parent. Dev hosts gain the same: no dockerd restart is needed after
`make environment-bootstrap`.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
