The path stood verbatim twice in one command line, once for the redirect and once for the chmod. A change to one of them would have written the file to A and set permissions on B, silently.
It now derives from NEXTCLOUD_DOCKER_WORK_DIRECTORY the same way NEXTCLOUD_INTERNAL_OCC_COMMAND already does, so the loader follows the work directory if that ever moves. Rendered value is byte-identical to the literal it replaces.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
NEXTCLOUD_DOCKER_EXEC resolved the container id through a subshell evaluated on the manager. When the manager holds no running replica, resolve_container_id.sh falls back to a cluster-wide lookup with no node filter and returns the id of a task on another node, which the local daemon does not know: 'No such container', sixty times, then the wait in 02_config.yml fails the deploy.
08_manager_ops.yml now resolves the hosting node and its node-local container id once via resolve_host_cid.yml and delegates every include there with apply:. Redefining NEXTCLOUD_DOCKER_EXEC in terms of NEXTCLOUD_LOCAL_CID lets all seventeen consumers inherit the fix without touching a task line. It re-resolves after the handler flush and after the upgrade, the two points where a rolling restart can move the task. Compose degenerates to delegate-to-self with the previous container name.
Four call sites built their exec inline because they need -u root or -e flags, and their subshell would have run on the worker where resolve-container-id exits 64 for want of a manager. Without converting them, delegation would have made things worse rather than better.
This is the outermost of four stacked defects and not the reason the job was red: the manager was empty because the object store had no free volume slots, which left a half-installed instance holding the init lock. Those layers are fixed separately. This one is a genuine cross-node bug that the rest of the repo already solved with the same pattern.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
taiga-events fell back to the image default 'taiga-events-rabbitmq' because the role set no host. That name exists only under compose, where compose.yml.j2 emits container_name via lookup('compose_only', ...); swarm drops the key, so only 'events-rabbitmq' resolves. The container logged 'getaddrinfo ENOTFOUND taiga-events-rabbitmq' and exited 1 on every start, and swarm restarted it every 15-20 s.
Each restart churns veth and bridge ports in the manager's network namespace, which is the namespace Chromium runs in: 02_run_one.yml passes --network host in swarm mode. Chromium's NetworkChangeNotifier then aborts in-flight requests, and when that lands on Taiga's bootstrap XHRs the SPA drops into its generic error view and the route heading never renders. The gateway logged 499 with UpstreamTime 0.010 on the same request the browser recorded as net::ERR_NETWORK_CHANGED - the client hung up, the backend was fine.
Variant-independent: the variable is absent unconditionally and container_name is dropped for every swarm variant. What varies is exposure. Within web-app-taiga 2 alone the same suite ran 19.3s, 17.8s, 88.3s, 17.3s, 17.3s, 87.0s, 18.7s, 212.6s - five clean, three hit. Jobs 0 and 1 invoked it twice each and got away with it.
events-rabbitmq resolves in both modes: compose registers the service name as a network alias, swarm publishes it as the stack DNS name.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
svc-bkp-volume-2-local passes --volumes-no-backup-required, which 3.3.0 carries, but the generation is only free of unix sockets and fifos from 3.4.0, where the volume copy runs with --no-D. Without the bump the deploy keeps installing a baudolo that writes socket inodes onto the ganesha export and the remote pull keeps exiting 23.
This is the only pin in the repository. It must not reach a deploy before 3.4.0 is on PyPI: sys-pip-install would fail to resolve the version and abort the role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Upstream's entrypoint takes the init lock unconditionally: 'flock -n 9 || { echo "Another process is initializing Nextcloud. Waiting..."; flock 9; }'. Against this ganesha export a non-blocking NFSv4 LOCK works and a blocking one returns EIO, so every replica that lost the race printed the message and then died on the blocking call. The container logs show exactly that pair, and the tasks were gone about one second after start - far too fast for the soft mount's timeo=50,retrans=3 to have elapsed.
The version gate did not prevent it: it serialises initialisation, not lock contention, and slots 2 and 3 passed it before reaching the entrypoint. Slots other than 1 now spin on a non-blocking flock until the lock clears, so upstream's own 'flock -n 9' succeeds on its first try and the blocking branch is never entered. Same primitive friendica-init-gate.sh already relies on over the same mount.
This is the layer that emptied the manager: with mgr-01 and wrk-01 crash-looping out, swarm rescheduled all three replicas onto wrk-02, leaving no local task for the occ wait to exec into.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role passed no -volume.max, so the stock entrypoint injects -volume.max=0, which means 'derive the ceiling from free disk space'. That derivation charges every non-full volume its full volumeSizeLimit: the master's default grow batch creates seven 1 GB volumes on the first write, reserving ~7.0 GB, and the manager's docker data root had 6.8 GB free. unclaimedSpaces goes non-positive, maxVolumeCount collapses to volumeCount, and the node reports zero free slots.
Those seven volumes belong to the default collection, so they can never serve collection 'nextcloud'. The master says it plainly - 'topo failed to pick 1 from 0 node candidates', 'create 7 volume, created 0' - and every S3 PUT returns 500 while seven empty volumes sit on disk. The volume server never left the topology: RestartCount 0, one unbroken log stream across the window.
That 500 is what starts the web-app-nextcloud 0 chain: occ maintenance:install fails eleven times, leaves a half-installed instance, and slot 1 then holds nextcloud-init-sync.lock for four minutes while every other replica dies on a blocking flock. Reproduced independently in variant 5 with collection 'peertube'; both compose jobs stayed green.
-volume.max makes isArgPassed true, so the entrypoint stops injecting the autosizing value. Slots become a fixed budget rather than a function of disk space; volumes stay sparse and unpreallocated, and -volume.minFreeSpacePercent still guards a genuinely full disk.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
nfs(5) lists both 'nolock' and 'local_lock=' under "Options for NFS versions 2 and 3 only". NFSv4 carries locking in the protocol and has no NLM sideband to switch off, so the kernel accepts the token and reports local_lock=none. mount_opts had the condition inverted: the branch that can work sat on v3, the inert one on v4 - and svc-storage-nfs-server pins nfs_version: 4, so the effective locking configuration was always the dead branch.
The artifact proves it landed nowhere: the mount module ran with local_lock=flock in opts on every node, every other token from that same string is present in /proc/mounts (soft, timeo=50, retrans=3, _netdev), and only local_lock reads 'none'.
The false premise had spread. docker-entrypoint-suitecrm.sh justified its mkdir mutex with 'the swarm NFS mount forces local_lock=flock, so flock(2) never crosses nodes'; that comment is removed. The mkdir mutex stays correct for a different reason: against this ganesha export a non-blocking flock works while a blocking one returns EIO.
Nothing about the effective mount changes - the option was already being discarded. On an existing host the first run rewrites fstab and remounts /var/lib/infinito once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
07_admin_identities.yml registered 'occ ldap:search <admin>' with async/poll, and ASYNC_POLL is 0 whenever ASYNC_ENABLED is true. The register then holds an async job handle with no stdout, and thirteen lines later the set_fact parses exactly that stdout for the '(identity)' pattern. The list therefore collapsed to [NEXTCLOUD_ADMINISTRATOR_USERNAME] on every swarm run, the LDAP-mapped administrator_<n> never entered the admin group, and failed_when could not complain because it is keyed on the same global flag.
The symptom is a 403 on every admin-scoped route while user routes return 200. The Playwright trace of one session carries data-user=administrator_2981 and _oc_isadmin=false on BOTH - and _oc_isadmin comes straight from IGroupManager::isAdmin(), so it states the group membership rather than the route's verdict. The compose lane stays green only because it runs a synchronous pass first, where the same task does have stdout and adds administrator_3447; its later async pass degenerates identically.
Three tasks become synchronous: the identity resolution, the group:adduser loop that writes the membership, and the LDAP cache warm in addons/user_ldap.yml whose task name promises the users are present. Measured cost 2.533 s + 0.730 s against a 2:30:13 job. The 'not ASYNC_ENABLED and' guards stay: they hang on the global flag, not on the task being async, so removing async does not arm them.
The condition moves into NEXTCLOUD_USER_LDAP_ENABLED and the two tasks sharing it into one block.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
infinito-http-ca-trust is required: true with no enabled: expression, so a required addon with no unmet bridge is announced as enabled in every variant. Its spec then calls wpAdminLoginViaOidc, the only admin login helper WordPress has here, and variant 1 turns SSO off: the OIDC plugin is not installed, wp-login.php never redirects off-host, and the poll for /wp-admin burns its full 60 s on the native form. Three attempts, identical received URL each time.
The sibling test-admin-oidc-login.js:10 already carries skipUnlessServiceEnabled('sso') and skipped correctly. Adding the same gate costs no coverage: variant 0 runs with sso enabled: true, so the addon stays exercised where its login path exists.
Not expressible as a bridge. bridges answer 'is the partner in this Playwright run' - the wordpress job's TEST_E2E_PLAYWRIGHT_APPS was ['web-app-wordpress'] alone, so a bridge on sso would gate the spec off in variant 0 as well. skipUnlessServiceEnabled reads SSO_SERVICE_ENABLED from the rendered environment, which is the question actually being asked.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
_any_bridge_partner_deployed matched a bridge against deployed role ids by equality or '-<bridge>' suffix. That reaches a partner whose entity carries the name (matrix -> web-app-matrix) but never one registered under a service key: 'sso' is provided by web-app-keycloak and 'ldap' by svc-db-openldap, neither of which ends in the bridge name. Those five declarations therefore evaluated to false in every run, so their addon flags were false everywhere and skipUnlessAddonEnabled skipped the specs in every variant - silently, because a gate that never opens looks exactly like a variant that does not apply.
The lookup now resolves a bridge through build_service_registry_from_applications first and keeps the entity-suffix rule as the fallback, so both spellings work. Existing bridge tests are unaffected: their applications stub yields an empty registry and takes the old path.
The new lint asserts that every bridges: value names a service some meta/services.yml declares. It fails on a typo or an invented name while staying agnostic about which run deploys the service, which is the lookup's question at runtime. Verified both ways: no findings against the real service set, thirty against an empty one.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dnsmasq 2.93 with bind-dynamic enumerates IPv6 before IPv4 (network.c:836-875). A veth that disappears between the RTM_GETADDR snapshot and the SIOCGIFFLAGS ioctl makes that pass return 0, which skips the IPv4 pass entirely while the interface-record collector still runs and releases every listener not marked found. The v6 addresses already dumped survive; the whole IPv4 set does not. Recovery would come from the next netlink address event, so the loss becomes permanent when the churn stops on a release.
That is what killed web-app-matrix 0: the manager kept only [::1]:53 and eight fe80 link-locals while ip -4 addr still showed all four addresses, so ghcr.io stopped resolving and the image pull failed ten times. Both workers, same image and same config, held the complete set on the same fds. daemon.json points container DNS at 172.17.0.1, so every container on that node lost resolution at the same instant.
Reproduced on the same dnsmasq version: 889 full IPv4 release/recreate cycles under a veth storm, with ::1 never released once; the same storm without bind-dynamic produced no listener events at all. Without the directive dnsmasq binds one wildcard pair and reads the arrival interface from IP_PKTINFO, and wildcard listeners carry iface == NULL, which the collector skips outright.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The routine flowchart marked a terminator-driven remote-2-local start on bkp-01, failing for lack of the ssh trust the drill installs later. Both halves are wrong: the lab deploys the app stack and the backup host with --skip-backup, so MODE_BACKUP is false, svc-bkp-remote-2-local never sets system_service_force_flush_final, and the unit never enters system_service_run_list.
The drill is the only caller that starts the pull. The unit journal of the failed run holds exactly one invocation, at 20:58:30 - one line after 01_ssh_trust.sh installed the key, and that first rsync moved 5.8 GB. A terminator-started pull would have run at ~20:55:4x, before the key existed.
The box also loses its trigger colouring, since nothing fires there.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Exclusion was a property of the image: --images-no-backup-required resolves through volume_is_fully_ignored, which skips a volume only when every container using it is ignored. A container holding a derived tree beside state that must be kept could not express the exclusion at all.
matrix_mdad_docker is that case - the data root of a nested docker daemon, sharing one container with matrix_mdad_matrix and matrix_mdad_state, which carry the actual Matrix data. Copying it is not merely wasteful: overlay2 encodes a deletion both as a character-device whiteout and as a trusted.overlay.* xattr, and no rsync in the chain carries -X, so the copy cannot be restored faithfully. The whiteouts also cannot be written onto an overlayfs destination at all, which is what kept the web-app-matrix DR drill red for 58 minutes a round.
meta/volumes.yml gains a 'backup:' key, validated like 'nfs:'. The backup_volume lookup resolves every 'backup: false' entry to its pinned docker name and feeds baudolo's --volumes-no-backup-required. A lint requires the decision to be written down for any volume mounted at /var/lib/docker or /var/lib/containers, so the next docker-in-docker role cannot repeat this silently.
Requires backup-docker-to-local 3.3.0, which carries the flag.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Add basic_auth to the MCP auth schema so Jenkins can declare its
HTTP-Basic endpoint through the contract instead of around it.
Introduce the client-side renderers: mcp_authorization builds the
Authorization header per scheme, mcp_tool_server_connections builds the
Open WebUI tool-server entries and drops servers whose scheme it cannot
present, and mcp_renderable_servers filters a server list, since Jinja
select() needs a test and the repo ships no test plugins.
Replace the inline MCP_DISCOVERED_SERVERS Jinja with a mcp_servers
lookup, and carry auth and username per entry so basic_auth consumers can
build their credential.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The runner now ships the execution modes as one JSON extra-var, so the old assertIn on MODE_CLEANUP=true no longer matches. Locate the last -e instead, parse its argument and compare the whole mapping: that catches a missing or extra mode, which the substring check could not, and it no longer passes when the text happens to appear inside an unrelated argument.
Drop the narration comments the file carried; the assertions state the same thing. Carry the autoformat wrap on the AUR retry test.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The deploy CLI shipped every mode as -e KEY=value, which ansible-playbook parses as a string at the highest precedence. MODE_DEBUG=false therefore arrived as the non-empty, and so truthy, string false. Every when: MODE_X | bool was unaffected because the filter maps it back, but four sites read a flag bare and silently inverted: openldap wrote LDAP attributes with state exact on every deploy instead of only under MODE_CLEANUP, SYS_SERVICE_DEFAULT_STATE was restarted where it should have been omit, SYS_SERVICE_RUNNER_STATE was never started, and mailu always logged at DEBUG.
Send the modes as one JSON extra-var, which keeps the declared type, and give those four sites the | bool they were missing so the meaning of a flag no longer depends on how its value reached Ansible. A lint pins the rule: a MODE_* read in a truthiness position carries the filter or a nocheck naming a genuine string comparison.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A module that dies on an unhandled exception carries no msg: the AnsiballZ wrapper returns only failed and exception, whose event holds the sole account of what went wrong. The aggregation defaulted to the literal string package install failed and discarded it, so an AUR fetch that threw in run 30713666732 reported nothing at all and took a full investigation to place. Walk msg, exception.event.msg, exception, module_stderr, stderr instead, and tag every result with the module that produced it. This applies to apt, dnf, pacman and every other module the plugin drives, not only to the AUR.
The aur module fetches the RPC record and the snapshot tarball through bare open_url calls with no error handling, so one blip kills a deploy; distribution repositories need none of this because apt, dnf and pacman retry inside their own transport. Give the calls that reach an endpoint of their own a single shared policy. Only the AUR path is declared today; copr, ppa and build carry it against the day they are.
Split the result along its responsibilities so no file passes 250 lines: calls.py holds the vocabulary, sources.py one builder per acquisition path, plan.py only the dispatcher, outcome.py the reporting and facts.py the fact shapes. The action plugin is left with executing modules. Tests follow the code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
losetup -f prints the device with a trailing (lost) marker when its node is missing, and both claim sites fed the whole string to mknod. The recovery path that materialises /dev/loop8 and beyond therefore never worked, which is why the pool appeared to stop at the eight preallocated nodes even though max_loop is 0. Strip the marker before the node is built.
Nothing released the data-root loop a node claims, so each distro round leaked four of them: across 140 job logs leg 1 always applies 5 and declines 0, leg 2 applies 3 and declines 2, leg 3 declines 4. The declining nodes fall back to overlayfs, which disables the btrfs snapshot mode the script exists to exercise, and the DR drill then finds no device at all.
Detach right after the mount instead. The mount holds the opener, so the kernel arms autoclear rather than detaching, and the loop plus its sparse image are reclaimed when the node container takes its mount namespace down. Verified in a privileged container: the mount survives the detach, AUTOCLEAR reads 1, the filesystem stays writable, btrfs subvolume create still works, and the loop is released on umount.
Drop the teardown block added earlier: it filtered on (lost), which marks a missing device node rather than a missing backing file, so it selected exactly the devices losetup -d cannot detach, ran unprivileged, and printed success unconditionally. Restore the (deleted) filter in the drill. Require mountpoint up front so a missing tool declines instead of killing the node.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
pgAdmin validates PGADMIN_DEFAULT_EMAIL against the RFC-6761 special-use list and exits 1 when the domain ends in .localhost, .test or .invalid. Under restart: unless-stopped that becomes a crash loop, which is what the health check reported for 25 minutes in run 30713666732: administrator@infinito.localhost does not appear to be a valid email address.
Derive the permitted domain from the admin address itself and hand it to PGADMIN_CONFIG_ALLOW_SPECIAL_EMAIL_DOMAINS. An ordinary TLD in that list is a no-op, so a production domain is unaffected. Both the address and its TLD move into vars as constants, so the config lookup happens once.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An untyped inspect searches every docker object kind, and a role's overlay network carries the same entity name as its main container, so the call can resolve the network and report a missing State key instead of naming the container. That is how the openresty assert died on every swarm job of run 30709070050.
Extend container-exec-resolver to cover inspect, stripping raw markers and quoted spans first so the flag walker survives Go template format strings. Add container-inspect-type, scoped to roles/ because the collision follows the entity naming convention and the outer act namespace carries no role networks. Sweep 40 call sites; suppress the delegate_to-resolved CIDs and the compose co-location probes with a stated reason.
Move the five container rules into tests/lint/ansible/shell/container/ and drop the container_ prefix from their filenames. Register both markers in the suppression docs, where the resolver marker was missing entirely.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The readiness probe passed the bare service name to container inspect. In swarm that name does not belong to a container, because compose_only suppresses container_name there, but it does belong to the overlay network, which get_entity_name names identically. Docker resolved the network and failed with a missing State key on every swarm job once MODE_ASSERT armed the include.
Route the probe through OPENRESTY_CONTAINER_ADDRESS like the validation task two lines below, switch to shell so the resolver subshell is evaluated, and pin --type container so a name shared with a network can no longer select the wrong object.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The swarm drill loads only this file, so MODE_DEBUG fell back to the group_vars default and the ganesha LOG block from 8f5670765 would never have rendered where the wedge happens. Turning it on also enables MODE_ASSERT and SYS_TIMER_ALL_ENABLED, which is a wider behaviour change than the diff shows.
ASYNC_ENABLED is set explicitly because it otherwise derives from 'not MODE_DEBUG': debug would have serialised every round, on a drill whose time budget two jobs of run 30699990912 already exhausted.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The role runs ganesha at -N NIV_EVENT, and roughly eighteen init steps between the CLIENT commits and the listener log below that level, so the silence in run 30699990912 marked the edge of visibility rather than the fault. Raise INIT and DISPATCH under MODE_DEBUG: nfs_Init_svc(), which binds 2049, logs under COMPONENT_DISPATCH, so INIT alone would have missed exactly the step in question.
Add a userspace backtrace beside the kernel stack dump, which only ever says futex_do_wait. eu-stack names the function holding the lock; elfutils goes into the lab image rather than the role, since a production NFS server should not carry a debugger.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
grep exits 1 when it matches nothing, and under set -euo pipefail the command substitution killed the script before the '[ -n VERDICTS ] || exit 0' line beneath it could express that empty is fine -- which the file header already promises. Three jobs of run 30699990912 were red for this alone: web-app-matrix 2, web-app-penpot 0 and the nested act drill in test-workspace, all with their deploys intact.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every node runs docker_dataroot.sh, which attaches a loop for its data root, and nothing ever detached them. Run 30699990912 shows the consequence: all eight kernel loop devices bound to docker-fs.img in two inode groups, i.e. two rounds' worth, leaving none for the DR drill's LUKS image.
Release the orphaned ones after the nodes are gone -- only then is a loop unambiguously ours to reclaim, rather than a concurrent run's. Match (lost) as well as (deleted): a backing file inside a removed container's overlay becomes unfindable rather than merely unlinked, which is why the existing cleanup in 02_luks_device.sh never fired.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The early exit at already-prepared never touched pool properties, so a pool created before 9c0e1efdc would keep acltype=off and setfacl would keep failing on a reused runner. Guard on the pool existing under its current name: e816fc1c9 made that name per-host, so a host carrying an old infinito_ci falls through untouched rather than aborting under set -e.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The entrypoint initialises on a server that listens on the unix socket only, so pg_isready without -h reports healthy while the database is still being built -- visible in run 30699990912 as socket-only at 14:07:37.425 against socket+TCP at 14:07:42.174. Probe 127.0.0.1 instead, which the bootstrap server does not serve.
pg_isready keeps using PQping, so the check stays immune to pg_hba changes as a psql probe would not be. The compose healthcheck gains the start_period it never had, so the stricter probe cannot eat into its 60s retry budget on a cold start.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
MariaDB's entrypoint initialises on a temporary server, shuts it down and starts the real one -- the same two-phase startup that killed the postgres sync in run 30699990912. The probes in 00_core.yml pass against the first server, and these two writes then ran with no retries at all in the swarm path.
A blank until is correct here, unlike the postgres case: both SQL files use IF NOT EXISTS, so a repeat is a no-op rather than an error.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The postgres entrypoint runs a bootstrap server on the unix socket only, then shuts it down and starts the real one on socket plus TCP. Every task here connected over the socket, so both readiness waits passed on their first attempt against the bootstrap server, and the etherpad check -- the only task in the file without retries -- died in the 2.6s shutdown window of run 30699990912.
Gate readiness on TCP, which the bootstrap server does not serve and which pg_isready probes without needing auth, and guard the two remaining tasks. The create gets a scoped retry rather than a blank one: CREATE DATABASE has no IF NOT EXISTS in postgres, as 'database greenlight already exists' in the same log shows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ueberDB's pg driver opens the named database but never creates it, and the role created it from ansible after the stack was already running. etherpad therefore crash-looped on every cold deploy -- seven FATAL: database "etherpad" does not exist between 14:07:47 and 14:08:48 in run 30699990912. Hook it into /docker-entrypoint-initdb.d instead, where upstream already creates bbb_graphql, hasura_app and greenlight, eight seconds before etherpad first connects.
Mounted 0555 on purpose: the postgres entrypoint sources a non-executable initdb hook into its own shell rather than running it, which would leak set -eu into the entrypoint. A separate hook rather than an entry in POSTGRES_MULTIPLE_DATABASES, so a database upstream adds later is not silently dropped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
compose names the swarm-test volume after its project, so every cluster
left a <cluster>_nfs-export behind that no teardown path touched. At
roughly 5GB each they filled the host disk after ~30 runs, which showed
up as unexplained playwright timeouts once writes started stalling.
Remove it per cluster in teardown, per worktree subnet in lab-reclaim,
and host-wide in swarm-clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The name introduced in ebbf9d944 was 85 characters; tests/lint/repository/test_name_emoji_and_length.py caps task names at 80. It still carries the filesystem that decided the outcome, which was the point of templating it in.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible runs inside the infinito container for compose deploys, so the lookup('env', ...) that 1101433ea put into the swarm inventory resolved to the empty string there instead of the manager IP -- a silent path divergence rather than a visible error. Declare the key in the environment block, which is what tests/lint/filesystem/env/test_inventory_envs_forwarded_in_compose.py requires of every inventory env lookup.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ZFS pools live in the kernel's namespace, which the five lab node containers share, and compose/swarm/playbook.yml runs this script on hosts: all. Each created the same 'infinito_ci', so the first would win and the other four would die on 'pool already exists' under set -e. Only resolve.sh dropping zfs for arch/centos/fedora nodes has kept that out of sight; a dispatch limited to debian or ubuntu exposes it.
The mountpoint needs no suffix: mount namespaces are not shared, only the pool name is global.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
OpenZFS defaults acltype=off on Linux, so setfacl on anything under the data root returns 'Operation not supported' -- which killed three compose jobs of run 30695207248 the moment the filesystem matrix drew zfs. Set it at pool creation; -O properties are inherited by the docker dataset, so no second edit is needed. xattr=sa is the companion OpenZFS recommends alongside posixacl and it lowers the space cost.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The block was gated on a denylist of exactly one filesystem, '^nfs'. A zpool defaults to acltype=off and rejects setfacl with the same 'Operation not supported', so three compose jobs of run 30695207248 died the moment the filesystem matrix drew zfs.
Attempt the ACL and gate on its return code. That also covers the next backing store, and stops skipping a zpool that does carry acltype=posixacl. The findmnt task stays as the record, so a skipped block still names the filesystem that caused it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Mailu 2024.06's token endpoint models AuthorizedIP as a list of CIDRs and declares its expect without validate=True; the POST handler assigns token_new.ip from data['AuthorizedIP'] and never from data['ip']. The 'ip' key this role sent was therefore discarded, and every token it created has always been unrestricted.
The value behind it was a forbidden substantive default on networks.internet.ip4, introduced as drive-by collateral in 087175a3c, a commit about the token comment. Removing both is behaviour-preserving and restores the original intent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
networks.internet.ip4 claimed 127.0.0.1 while the lab's dnsmasq maps the whole domain to INFINITO_SWARM_MGR_IP (compose/swarm/playbook.yml:30). Two sources disagreed about one fact, and the DiD DNS check was the first thing that ever compared them. Read the same env SPOT from default.env so both agree by construction.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check asserted the A record equals networks.internet.ip4, which answers a different question: under which public address the deployment answers. The two coincide in compose and diverge in the swarm lab, where dnsmasq maps the domain to the manager, so job 91358007229 failed on three healthy nodes that had resolved the name correctly.
Assert that an address came back instead. The SERVFAIL guard is unchanged, and the role no longer reads networks.internet.ip4 at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
make quality wrapped the conditional NODE_IMAGE expression introduced in 823c6f4ee; the rendered value is unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill died with cryptsetup's 'loop device with autoclear flag is required', which names neither the resource nor the reason. Claim a loop for the image and release it again first, creating the /dev node when the container's tmpfs carries none, so cryptsetup finds what it needs and a genuine shortage stops the drill with a message that says so.
The cryptsetup calls themselves are unchanged and keep owning their own autoclear device, so nothing lingers once the mapper closes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The DR drill's cryptsetup step died claiming a loop device and the artifact carried nothing to say why: no losetup -a, no /dev listing, and the backup host appears in no other collector loop. Record both views, what the kernel has attached and which nodes the container's /dev carries, so an exhausted pool and a missing device node are told apart at a glance.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The check starts a container with the advertised resolver, which is exactly the path a pre-deploy build takes, but MODE_DEBUG gated it off in the swarm drill: that path loads only inventories/development/swarm.yml, which never sets it. Run 30688981182 skipped the check at 07:31 and lost the openldap_custom build at 07:44 on the one node whose dnsmasq held no UDP socket for the advertised address.
Pass NODE_IMAGE empty in swarm mode instead: node:26-alpine is 63.3 of the 65.5 MB pulled per node, and its getaddrinfo assertion is a second opinion on a SERVFAIL check the busybox half already performs. NODE_IMAGE stays mandatory to set, so omitting it remains an error while empty is a stated decision.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The parenthetical claimed the capture names the address and errno of a failed bind. dnsmasq logs that at startup only, and a startup bind failure is fatal, so the half-bound state seen in the artifact cannot originate there. The dynamic bind path is measurably silent on success and untested on failure. The neighbouring markers in this collector carry no description either.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Ansible's run_once inspects the first host and generalises to the whole play, so a cluster where a single node's dnsmasq never bound the advertised resolver still reads as healthy. The property is per node, and forks=25 covers them concurrently. Also require a non-empty resolver, since 'container run --dns ""' cannot work.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The socket table already shows which listening address is missing, but nothing recorded why. A build that cannot resolve dies on the node whose dnsmasq never bound the advertised resolver, and the journal is the only place naming the address and errno. Captured whole rather than tailed, because a failed bind is logged at startup and the neighbouring captures keep only the tail.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The template replaces daemon.json wholesale, so the data root the swarm lab puts on the chosen filesystem was dropped on every managed node. The rescue artifact of run 30688981182 shows nfs-server keeping it while mgr-01, wrk-01 and wrk-02 lost it, leaving those nodes on the container overlay instead of the prepared loop filesystem. Read the file back before rendering and re-emit the value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RUNNER_POOL and NODE_POOL were assertions. With a distro list of just debian
and ubuntu the node pool offered zfs, because the table records that those
distros ship the zfs userland - but the module comes from the runner's kernel,
which carries none. One run in three would then have drawn zfs and either
declined silently, testing ext4 while the summary claimed zfs, or hard-failed on
all five nodes with nothing actually broken.
Ask the kernel instead: /proc/filesystems for what is loaded, modinfo for what
could be. Both answer unprivileged, which resolve.sh needs since it runs without
sudo. The modinfo half carries weight rather than being belt-and-braces - xfs is
absent from /proc/filesystems on this host while modinfo finds the module, so
without it the pool would shrink to whatever happens to be loaded.
The distro table stays above it for the other question: which userland a
container can install. Two layers because they are two things, and only the
kernel one cannot be fixed by a package.
Every exclusion now names its reason on stderr and distinguishes a missing tool
from a missing capability; a filesystem used to drop out of the pool in silence.
For the full five-distro sweep the pool is unchanged: ext4 btrfs, measured
before and after.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An opencloud snapshot carried 174 entries, and every line repeated the whole
path - the deepest ran about 150 characters, most of it the same prefix as the
line above. Print the basename at its depth instead, with a glyph per kind:
folder, json, journal or log, plain text.
The separator is swapped for \001 across the sort. Without it a directory and
its children come apart: '/' outranks '.', so a sibling file sorts between the
two and reads as though it lived inside. The swap also makes the order
independent of the locale, which was deciding it before.
The existing cut-short test looked for 'logs/container.log', which no longer
appears, so it now checks the basename; a fourth case pins the structure rather
than the wording - a child is indented deeper than its folder and siblings sit
level.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The distro table pinned centos to ext4, and since the node pool is the
intersection across every distro in a run, the full five-distro sweep always
landed on ext4. btrfs and zfs are what engage the backup snapshot mode, so that
path was never exercised in swarm.
The premise held for a centos system and not for a centos container: Red Hat
dropped btrfs from its own kernel, but a node container runs the host kernel,
and btrfs-progs is merely absent from the base repos - EPEL carries it. Measured
in a centos:stream9 container on this host: mkfs.btrfs, mount, subvolume create,
snapshot -r, and inode 256 at the subvolume root, which is what baudolo checks.
install_packages is the load-bearing part. docker_dataroot.sh ships its own
installer and runs before the ansible package layer, so meta/packages.yml never
reaches it; without the epel fallback there, opening the pool would have let a
centos node be handed btrfs it cannot deliver, decline silently, and leave the
cluster running four btrfs nodes and one ext4 - the very inconsistency the
intersection exists to prevent.
Verified by running the script itself, not extracted parts: it reports
requested=btrfs effective=btrfs on centos.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
