fetch() leaked subprocess.TimeoutExpired, which ended the builder thread while it kept the builder lock, so no replica built again. A clone killed at its timeout also left a HEAD that never resolves, and the HEAD-file check ran remote update on it forever. The loop and refs() now catch SubprocessError, fetch() re-clones when HEAD does not resolve, and the clone lands in repo.git.partial and replaces the mirror only after it succeeds.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
On real GitHub Actions INFINITO_BUILD passed through from default.env as 1, so every deploy job built the infinito image locally (~5 min) and then pulled and ran the published ci image anyway (pull_policy always). An env handler now defaults INFINITO_BUILD to 0 there; act and jobs that set it themselves keep building.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
run_builder fetched the git mirror before it claimed any build, so the deployed site, which copies the snapshot baked into the image and reads no mirror, waited for the first git clone --mirror (about 60-150 s on CI, 967 MiB with 1113 refs/pull/* heads). The builder now claims and submits DEPLOYED inside the pool before the first fetch; latest and the tags still wait for it, so none builds from an outdated head. The README no longer claims the fetch waits for a running build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_used_keys_declared reads every INFINITO_* literal in code as an environment key, so the marker string copied from vars/main.yml failed the lint without a registry entry. The marker only lands in the append's header comment, which the test does not inspect.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
taiga-back's image settings build STATIC_URL from TAIGA_URL, the canonical clearnet domain, so the Django admin on the onion vhost linked base.css and its scripts to the clearnet host and the onion CSP blocked them (header background rgba(0, 0, 0, 0) in the design spec). STATIC_URL is now the same-origin path below FORCE_SCRIPT_NAME. The Jinja static() global of the e-mail engine keeps the canonical absolute URL, because a mail client resolves no same-origin path.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Pins of every role applied at every round index, so a role the round did not deploy could pull a partner into a heavier variant (web-app-discourse#1 measured 35.46 GB against the 32 GB reservation cap). resolve_round now starts every role on its clamped index and recomputes the include and the pins in force until the assignment settles, raising VariantPinError on a cycle. The planner resolves each round once (39.1 s to 16.0 s for all rounds), affected judges every (role, index) with the round the planner deploys for it, and the resource budget lint measures the pin-aware rounds. The stale web-app-matomo pin above the primary-only docs entry is dropped, and test_variant_pin_fixed_point.py guards every planned round against unsettled pins and pins above a disabled entry. docs/contributing/artefact/files/role/variants.md explains the settle loop with diagrams.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The server answered 202 for every /deployed/<code>/ path until the deployed Sphinx build finished (about 80 min with -j1), so test-languages.js:61 outwaited its poll budget on attempt one and passed only on retry. The untranslated decision now reads the snapshot's meta/languages.yml and docs catalogs, the same files the build writes into the language index, so the 404 set is identical before and after the build.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The dedicated deploy launched whatever ansible-playbook PATH found. A stale pkgmgr venv without Pillow crashed the galaxyserver deploy after 4h23m at the first branding task. The runner now verifies every distribution of the pyproject deploy extra in the running interpreter before make setup, exits naming the missing ones with the install command, and runs the ansible-playbook next to that interpreter.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A user key that no role declares, such as an inventory leftover, gets no username, firstname, lastname, uid or gid derived. openldap_user_entries then crashed with a bare KeyError 'username'. The filter now raises an AnsibleFilterError that names the user and the missing fields.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI run 41147980556 Nextcloud#0 failed both conductor design specs on a
multi-network node. e0179a0893 handed every conductor the other
network's conductor in LOGOUT_SIBLINGS. With that list set,
universal-logout v1.5.0 hides #iframe-warning on the stand-alone page
("'#iframe-warning' matches nothing visible"), and v1.5.1 fetches the
sibling after its sweep, which the conductor's per-network CSP blocks
("Connecting to 'https://logout.main.infinito.test/' violates ...
connect-src 'self' *.<onion>"). Requirement 037 limits each vhost's CSP
to its own network, so the browser hand-off could never complete in
either direction; its only effects were the two red specs.
Drop the LOGOUT_SIBLINGS wiring and its README bullet, so the compose
template and README match their state before e0179a0893. Each conductor
still logs out of every domain of its own network; ending the sessions
of the other network belongs on the server side, not in the browser.
The test now asserts that no conductor on a single-network, tor-only or
multi-network node is handed a conductor of another network.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A production redeploy of echoserver hung at the swap-forge task and the
host stopped answering SSH. swap-forge v1.0.1 recreates the swapfile
whenever its size differs from the requested MiB by any amount, starting
with swapoff on the live file. The request is the host's MemTotal, which
drifts by a few MiB across kernels, so a routine deploy paged the whole
swap back in, charged it to the containers' memory cgroups and starved
sshd until the OOM killer took swapoff.
Read the active swap files with swapon before swap-forge runs and skip
it when an active file is within 5 % of the requested size, reporting
the kept size against the request. The sizing formula is unchanged, so
no host is forced into a one-time recreate; a deliberate resize beyond
the tolerance still reaches swap-forge.
The unit test resolves the real vars through Ansible's Templar for MiB
drift both ways, no swap, zram and partitions, a halved size, an exact
pin, a lowercase unit and an escaped file name, and shows that the exact
MiB comparison swap-forge uses misses the drift.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Fork CI run 41147980556 pgAdmin#0 failed in sys-ctl-hlth-runtime with
"stalwart_webmail: PHP 8.4.26 is below the security floor 8.4.27".
The deploy ran the fork's GHCR mirror of roundcube/roundcubemail:1.7.5-
apache, copied on 10-19 with PHP 8.4.26, while upstream re-pushed the
same tag on 10-27 with PHP 8.4.27. Branch CI runs the sync with
--only-missing, which skipped the tag because it existed ("destination
exists, skipping") without comparing digests, and the full refresh only
runs on the fork's default branch, which still pins 1.7.4.
- utils/docker/registry.py: manifest_digest() reads the
Docker-Content-Digest of a reference with a HEAD request, which does
not count against Docker Hub's pull allowance.
- GHCRProvider.is_stale() reports a mirrored tag as stale only when both
registries answer with a digest and the two differ.
- Under --only-missing an existing tag is re-copied when it is stale; a
failed re-copy is reported on stderr with the image and keeps the
existing copy, so one refresh cannot block every deploy.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared proxy's nginx.conf derived worker_connections from the
openresty container's pids_limit. pids_limit caps processes and threads,
worker_connections caps simultaneous sockets per worker (client and
upstream), so with pids_limit 256 and one worker the whole host's proxy
held about 256 sockets, roughly 128 proxied requests in flight across
every vhost. Raising pids_limit to buy connections would loosen the
fork-bomb guard.
svc-prx-openresty now declares services.openresty.worker_connections
(4096 per worker, the common nginx production range, enough for several
chunk-heavy apps such as OpenWebUI at ~150-170 requests per page load).
WEBSERVER_PROXY_WORKER_CONNECTIONS reads it through lookup('config'),
without a default, so a missing key fails loudly. pids_limit stays 256.
WEBSERVER_PROXY_WORKER_RLIMIT_NOFILE is derived as 2x worker_connections
(8192), one descriptor for each client and upstream socket of a proxied
request, and nginx.conf renders it as worker_rlimit_nofile. The openresty
container pins its nofile ulimit (soft and hard) to the same value:
workers call setrlimit as root without CAP_SYS_RESOURCE, so they can only
lower the hard limit, and the inherited hard limit depends on the host
daemon (Docker >= 25 defaults to 1024:524288, DiD nests whatever the
outer container has). An explicit ulimit makes the rlimit succeed on
every host instead of logging a setrlimit alert.
The unit test pins worker_connections 4096, worker_rlimit_nofile 8192 and
the container ulimit, and fails when worker_connections follows
pids_limit again.
Live proof (compose deploy of web-svc-html, both plays failed=0): the
deployed nginx.conf renders worker_rlimit_nofile 8192 and
worker_connections 4096; openresty's master, worker and cache manager
run with "Max open files 8192 8192"; the openresty log carries no
setrlimit or "worker_connections are not enough" line.
Closes#1300
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_sso.py, test_sso_oidc_plugin.py and test_sso_proxy_consumers.py
evict utils, utils.cache, utils.cache.applications and the roles
packages from sys.modules to import their plugin against fresh modules,
and never put the originals back. Ansible's plugin loader keeps the
applications lookup it loaded earlier in sys.modules, so that lookup
stayed bound to the evicted utils.cache.applications, whose
_MERGED_APPLICATIONS_CACHE utils.cache._reset_cache_for_tests() no
longer reaches.
On main this turned test_webserver_worker_limits red whenever one xdist
worker ran a test that loads the applications lookup through Ansible,
then one of these sso tests, then the worker-limits test: the stale
cache served the cpus 0.5 payload of test_a_sub_cpu_proxy_still_runs_one_worker,
so the later methods rendered worker_processes 1 instead of 4 (main run
41139196554, Workspace centos and fedora, second make test on gw1).
_load_module now runs inside mock.patch.dict(sys.modules): the plugin
keeps the fresh modules it was built against, and sys.modules returns to
its prior state afterwards. This is the change store-i18n carries since
ec0fde6561, ported unchanged so both lines stay identical.
Evidence:
- Before: pytest -n 0 test_tor_ports.py test_sso_oidc_plugin.py
test_webserver_worker_limits.py -> "3 failed, 33 passed", the CI
failure set ('1' != '4' twice, worker_processes 1 vs 4).
- After: test_tor_ports.py, the four sso test files and
test_webserver_worker_limits.py -> "68 passed".
(cherry picked from commit ec0fde6561)
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
CI run 40979252521 Weblate#1 (swarm) never converged: every task after
the first died in collectstatic with "PermissionError: [Errno 13]
Permission denied: '/app/cache/static/weblate_fonts'". The tmpfs at
/app/cache/static sits inside the shared weblate_cache volume; the first
task's runc creates the mountpoint as root 0755, and runc gives a tmpfs
on an existing mountpoint that directory's mode. 99a7d28067 declared
mode 1777, which compose honours, but docker stack deploy keeps only the
tmpfs size (docker/cli 29.9.2 convert/volume.go), so swarm still got a
root 0755 tmpfs.
- utils/storage/nfs.py: nested_tmpfs_mountpoints() derives every tmpfs
mountpoint whose innermost enclosing volume of the same service is
NFS-backed, with its declared mode; a nested tmpfs without a mode, or
one mountpoint declared with two modes, raises.
- plugins/lookup/nfs_tmpfs_mountpoints.py feeds it the rendered compose
volumes, so compose_volumes stays the source of which volumes live on
NFS.
- sys-svc-compose nfs_prep.yml creates those mountpoints on the NFS
server with the declared mode before every stack deploy, which also
repairs a mountpoint an earlier deploy left at 0755.
- web-app-weblate: the cache volume declares nfs uid 1000, gid 0,
mode 0770; a pre-created mountpoint makes the volume non-empty, so
Docker no longer copies the image's /app/cache root in.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
main's admin allowlist test renders sys-svc-proxy's location/html.conf.j2,
which calls the entity_name Jinja filter. The test stubbed it as
get_entity_name, the name store-i18n retired when it registered the
filter as entity_name (plugins/filter/get/entity_name.py), so all eleven
cases failed with "No filter named 'entity_name'" after the main merge.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Cursor#0 (CI run 40979252521, debian host) installed the pinned cursor_3.27.6 deb on pass 1; its postinst registers Cursor's apt repo, so pass 2's dist-upgrade moved cursor to 3.27.22, and the plan's ansible.builtin.apt deb=<3.27.6> then failed with 'A later version is already installed': the apt module skips only an equal version and otherwise fails or, with allow_downgrade, downgrades. package_install now asks the target before such a call: dpkg-deb --field reads Package and Version from the checksum-verified deb, dpkg-query reads the fully installed version, and dpkg --compare-versions decides; at or above the pin the call is skipped with changed false, so the deploy-time upgrade may move vendor packages forward and the pin never pulls them back. Any query that cannot be read falls through to the apt install, which decides loudly. The dnf path already skips an older local rpm, so both families now behave alike.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Main CI run 40976399687, socialhome#0 (swarm, tor, centos): the CLI tests
of svc-net-tor (onion:25 "socks CONNECT failed with reply 0x01") and
web-app-stalwart (probe_mail relay: ConnectionRefusedError on port 25)
failed. dockerd on the manager had aborted the whole ingress program of
the mail stack:
Failed to add ingress: failed to program ingress ports:
failed to bind host port 0.0.0.0:993/tcp: address already in use
/proc/net/tcp named the squatter: 192.168.244.10:993 -> :2049, the
node's own NFS client. The mount carries no noresvport, so the kernel
drew a source port from the 665-1023 reserved window and landed on
imaps. None of 25/465/993/995/4190/8095 got a DNAT rule or a
docker-proxy listener while every other published port did, and
stalwart still reported healthy because its healthcheck runs inside the
container. 993 and 995 are the only published ports inside that window,
so this is luck of the draw on any NFS-backed swarm host that runs the
mail role.
mount_opts now emits noresvport. insecure on the kernel-flavor export is
the other half: kernel nfsd defaults to secure and answers a mount from
an unprivileged source port with EACCES. It reaches the kernel flavor
only; ganesha already accepts unprivileged clients.
What it costs: the next deploy of every existing NFS client sees a
pending opts change and remounts the shared state; a host-side job
holding that path at that instant fails loudly. A plain remount may keep
a running mount on its old reserved port until the next real mount.
Port of f46c4d0bab from feature/store-i18n, applied identically so the
later store-i18n -> main merge stays conflict-free.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared pacman cache was the first writable CacheDir on every arch runner and lab node. Our arch image sets DisableSandbox, so libalpm downloads straight into that directory (util.c _alpm_download_dir_setup only makes a per-process temp dir with the sandbox on), and the XferCommand writes <pkg>.part there with curl -C -o %o; two nodes fetching the same package appended to the same .part file, and the second rename failed or left a corrupt package in the shared cache.
pacman.sh now gives each node its own download directory <shared>/pacman/node-XXXXXX/ (created once with mktemp, reused from pacman.conf on later runs) as the CacheDir between the read-only host layer and <shared>/pacman/, and writes a PostTransaction hook to pacman.d/hooks/infinito-shared-cache.hook that renames every finished package and signature into <shared>/pacman/; .part files stay for the next resume. Both directories sit in one bind mount, so each publish is an atomic rename(2). pacman.sh fails when pacman.conf sets HookDir, since pacman would then skip the hook silently. Requirement 065 and cache.md describe the three layers.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The svc-cache-package CLI probe GETs every declared route path through the frontend and fails on 502/503/504. registry.ollama.ai declared no paths, so the probe asked for /, which upstream now answers with 503; Nexus relays an upstream 5xx as its own 503, so package#0 (CI run 40979252521) failed with 'registry.ollama.ai/ on port 443 answered 503'. The host now declares /v2/, the registry API root every ollama pull goes through (manifests and blobs); upstream answers it with 404, which Nexus relays as 404 and the probe accepts. On a TLS host the frontend still proxies the whole host through location /, because upstreams.conf.j2 reads routes only for plain-HTTP listeners: the rendered upstreams.conf is byte-identical before and after.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_compose_siblings.py called yaml.safe_load on the rendered compose file, which TestNoDirectYamlCalls rejects in favour of the utils.cache.yaml helpers (quality-high run 6).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
test_s3_public_url.py built roles/web-app-wordpress/meta/services.yml with a pathlib chain, which TestRoleFileSpot rejects in favour of the utils.roles.mapping constant (quality-high run 6).
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Parallel compose slots, swarm lab clusters and GUI lanes each kept their own pip, npm and pacman downloads, and a failed swarm run left its cluster (about 25 GB of node docker data plus the rescue bundle in the act container) on the root filesystem until someone ran make swarm-down. INFINITO_CACHE_SHARED_HOST_PATH (default <DIR_CACHE>/shared) is now mounted read-write at the same path into every compose slot and every swarm lab node, with PIP_CACHE_DIR and npm_config_cache below it; empty, the renders stay byte-identical to before. On a host whose os-release ID is exactly arch running an arch stack, INFINITO_CACHE_HOST_PACMAN_PATH mounts the host pacman cache read-only and scripts/docker/cache/pacman.sh makes it the first CacheDir with <shared>/pacman/ second, so new downloads land in the shared folder and the host cache stays untouched; on Manjaro and every other host pacman uses the shared folder alone.
make swarm-zombie without keep=true now copies the act job's rescue diagnostics and Playwright artefacts to the host and then releases the lab nodes, the lab network, the NFS export volume and the act outer container, on success and on failure; when the copy fails the nodes still go, the act container stays and the run exits naming make swarm-down. make roundtrip passes keep to every swarm step, make clean-cache removes the shared folder, and the disc-space cleanup skips pacman cache directories it cannot write. apt and dnf keep their per-container caches behind the package-cache proxy, and no docker data root is shared between daemons. Requirement 065.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
WordPress put objstore.public_url, the clearnet api.seaweedfs.s3 host, into S3_UPLOADS_BUCKET_URL, so the onion pass of WordPress#0 (CI run 40979252521) loaded the site logo from clearnet and the leak spec failed; the /objstore rebase only fired on onion-only consumers and never on a multi-network node. lookup('objstore', app, 'relative_url') now returns /_infinito/objstore/<bucket> for a shared consumer with a public bucket and an API domain, and locations.conf.j2 serves that path on every vhost of the consumer, clearnet and onion alike, by proxying over the loopback to the provider's own vhost.
The route passes GET and HEAD only, answers 404 for a path ending in / (no ListObjects), drops the query string (no S3 subresources or presigned queries) and the Authorization and Cookie headers, so only the public bucket's anonymous read applies, and adds nosniff plus a sandbox CSP; private and embedded buckets get no route. Only web-app-wordpress switches to relative_url: mastodon, pixelfed, baserow, bookwyrm, snipe-it, listmonk, mediawiki, shopware and peertube need absolute URLs for federation, mail or server-side rendering, and decidim and funkwhale sign URLs, which the route deliberately refuses. The prefix moves from /objstore to /_infinito/objstore.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
A node serving several networks runs one universal-logout conductor per network, and a page served over HTTPS cannot fetch the HTTP onion hosts, so one conductor cannot clear the other network's sessions. compose.yml.j2 now gives every conductor the url.origin of the other networks' conductors in LOGOUT_SIBLINGS (empty on a single-network node, which disables the hand-off), and the Keycloak logout panel that frames the conductor follows its handoff report to the next conductor once the sweep is done, accepting only an http or https URL.
The pinned image v1.4.2 ignores LOGOUT_SIBLINGS and never reports handoff, so behaviour changes only with the bump to the universal-logout release that carries the reachability probe. test_compose_siblings.py renders single- and multi-network nodes; logout-panel.test.js covers the handoff branch and rejects non-web URLs.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
compose_volumes merged extra_volumes first and then skipped every meta/volumes.yml entry whose key was already present, so the meta nfs value was lost with it. Nextcloud injects talk_recording_tmp through nextcloud_extra_volumes without an nfs key, and 5728502a19 marked that volume nfs: false. On swarm with the NFS backend the stack therefore bound it to /var/lib/infinito/nextcloud_talk_recording_tmp, while nfs_prep.yml renders the same lookup without extras, saw nfs: false and never created the subdir; every node rejected the task with 'no such file or directory'. The meta nfs value now carries over onto the injected or derived spec unless that spec sets nfs itself, so both renders agree and the scratch volume stays node-local.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
