The shared proxy's nginx.conf derived worker_connections from the
openresty container's pids_limit. pids_limit caps processes and threads,
worker_connections caps simultaneous sockets per worker (client and
upstream), so with pids_limit 256 and one worker the whole host's proxy
held about 256 sockets, roughly 128 proxied requests in flight across
every vhost. Raising pids_limit to buy connections would loosen the
fork-bomb guard.
svc-prx-openresty now declares services.openresty.worker_connections
(4096 per worker, the common nginx production range, enough for several
chunk-heavy apps such as OpenWebUI at ~150-170 requests per page load).
WEBSERVER_PROXY_WORKER_CONNECTIONS reads it through lookup('config'),
without a default, so a missing key fails loudly. pids_limit stays 256.
WEBSERVER_PROXY_WORKER_RLIMIT_NOFILE is derived as 2x worker_connections
(8192), one descriptor for each client and upstream socket of a proxied
request, and nginx.conf renders it as worker_rlimit_nofile. The openresty
container pins its nofile ulimit (soft and hard) to the same value:
workers call setrlimit as root without CAP_SYS_RESOURCE, so they can only
lower the hard limit, and the inherited hard limit depends on the host
daemon (Docker >= 25 defaults to 1024:524288, DiD nests whatever the
outer container has). An explicit ulimit makes the rlimit succeed on
every host instead of logging a setrlimit alert.
The unit test pins worker_connections 4096, worker_rlimit_nofile 8192 and
the container ulimit, and fails when worker_connections follows
pids_limit again.
Live proof (compose deploy of web-svc-html, both plays failed=0): the
deployed nginx.conf renders worker_rlimit_nofile 8192 and
worker_connections 4096; openresty's master, worker and cache manager
run with "Max open files 8192 8192"; the openresty log carries no
setrlimit or "worker_connections are not enough" line.
Closes#1300
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The shared OpenResty nginx.conf took worker_processes and worker_connections
from WEBSERVER_WORKER_*, which are the min() of svc-prx-openresty's limits and
the current play's application_id. The file renders once, in the scope of
whichever role loads the proxy first, so that role's limits capped every vhost:
on feature/store-i18n web-svc-mirror (pids_limit 64) loaded it first, every
vhost got worker_connections 64, and openresty raised 555 "64
worker_connections are not enough while connecting to upstream" alerts while a
single openwebui page fires ~150-170 chunk requests (CI run 40979252521,
homeassistant#1). Main carries the same wiring.
Add WEBSERVER_PROXY_WORKER_PROCESSES and WEBSERVER_PROXY_WORKER_CONNECTIONS,
computed from svc-prx-openresty's own openresty service only (same
resource_filter fallbacks, at least one worker), and render nginx.conf from
them. WEBSERVER_WORKER_* keep pairing the proxy with the play for app-internal
nginx such as Nextcloud's. The values can only stay equal or rise.
Found by the i18n session's triage of homeassistant#1.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answering "which deployed containers run a vulnerable PHP?" after
CVE-2026-109618/109619 took a manual inventory of registry metadata, and
nothing noticed a hardening ini that an image never loads.
- New role sys-ctl-hlth-runtime, routed through sys-svc-compose next to
sys-ctl-hlth-container. A daily timer runs `php` in every running
container and fails, alerting through sys-ctl-alm-compose, when PHP is
below the floor of its line or on a line older than every floor (7.4,
8.1). Every service that mounts php_hardening in meta/volumes.yml fails
while expose_php or session.upload_progress.enabled is on; other PHP
containers of such a stack are listed as info. Compose and swarm
containers are matched by their project/stack and service labels. The
check is not run at deploy time, so a finding never fails a deploy.
- The floors (8.2.35, 8.3.36, 8.4.27, 8.5.12) live once, in
HEALTH_RUNTIME_FLOORS of the role's vars/main.yml.
- `make security-runtimes` lists the PHP of every pinned role image
without pulling: Docker Hub tags through the Hub API, which no pull rate
limit meters, other registries through the OCI config blob (new
registry.config_blob, which the healthcheck probe now reuses).
PHP_VERSION, distro PHP packages and a PHP_BASE build argument count. It
exits 1 on an image below its floor or one the registry did not answer.
Today it flags nfrastack/fusiondirectory:1.5 (line 7.4) and
shopware/docker-base:8.3 (8.3.25).
The role TODO.md keeps the follow-ups: further runtimes, the split web
SAPI of web-app-lam, a scheduled CI run of the report and the CVE scan.
Refs #1127
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The interim CVE-2026-109618 mitigation lived in the shared hardening.ini,
so every PHP role ran without a memory limit until the last image was
patched, and no role's service_memory_mb share applied.
Registry metadata of the images pinned on main shows that 16 of the 20
consumers already ship PHP 8.2.35, 8.3.36, 8.4.27 or 8.5.12 (or the
Alpine/Debian build of it). Four predate the fix:
markoshust/magento-php:8.4-fpm (8.4.21), ghcr.io/heviat/webmail:2024.06
(the Mailu OIDC flavor, built 2025-04-04), phpldapadmin/phpldapadmin:2.3.11
(built 2026-05-15) and ghcr.io/shopware/docker-base:8.3 (8.3.25).
- Move memory_limit = -1 into its own content-addressed ini
(FILE_PHP_MEMORY_UNLIMITED_INI) and mount it only in magento, mailu,
phpldapadmin and shopware.
- hardening.ini keeps expose_php, session.upload_progress.enabled and
fatal_error_backtraces as permanent hardening.
- Joomla's php-upload.ini loads as conf.d/uploads.ini again: no -1 has
to be outranked anymore.
- The root TODO names the four images that still need the override.
Refs #1126
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
#1115 raised HSTS to one year but set no X-Content-Type-Options,
Referrer-Policy, Permissions-Policy or HSTS includeSubDomains/preload,
because none of them is safe globally: nosniff lets ORB block no-cors
responses typed text/html, includeSubDomains on the apex covers hosts
the platform does not serve, and a server-level add_header is dropped
in every location that sets its own.
WEBSERVER_SECURITY_HEADERS (group_vars/all/04_webserver.yml) now holds
the platform values, all opt-in: nosniff false, no Referrer-Policy, no
Permissions-Policy, HSTS max-age one year without includeSubDomains or
preload. A role overrides any key under security_headers in its
meta/server.yml, an inventory under
applications.<role>.server.security_headers.
The new security_headers filter merges both, rejects unknown keys,
wrong types, unknown Referrer-Policy tokens, non-printable header values
and preload without includeSubDomains or below a one-year max-age, and
composes the HSTS value.
- nosniff, Referrer-Policy and Permissions-Policy are set in the Lua
header filter of content_security_policy.conf.j2, next to CSP, so a
location's own add_header cannot drop them. Referrer-Policy is only
set when the upstream sends none; Permissions-Policy only on HTML
documents.
- HSTS stays the single server-level add_header in nginx_ssl_header.j2,
now with the per-app value. A second emitter in Lua would duplicate
the header. It stays inside the TLS branch, which onion vhosts never
enter, so an onion vhost gets no HSTS even when its app opts in.
No role opts in yet, so rendered vhosts are unchanged.
Closes#1123
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Proxy:
- server_tokens off, X-Powered-By stripped from upstream responses,
HSTS max-age raised to one year.
- No global X-Content-Type-Options: under nosniff, ORB (step 3.4)
blocks no-cors cross-origin responses typed text/html, e.g. tracking
beacons answered with an empty 204.
PHP (CVE-2026-109618, CVE-2026-109619):
- New role sys-svc-compose-php installs a shared ini (expose_php Off,
session.upload_progress.enabled 0, fatal_error_backtraces 0 for
PHP 8.5, interim memory_limit -1) and a PHP-FPM [www] drop-in
(pm.max_requests = PHP_FPM_MAX_REQUESTS) once per stack host,
before 04_files renders any compose file.
- The host file names carry a sha1 prefix of the controller-side
source, so a content change alters every PHP app's compose.yml and
recreates it, also on remote deploys where compose_volumes cannot
read the stack host file and hashes the path instead.
- 20 PHP roles mount the ini as a compose/swarm config into their
image's scan dir; moodle, nextcloud and shopware also mount the FPM
drop-in. Magento keeps its own pool: its php-fpm.conf has no
include= and is rendered without notify.
- Joomla's php-upload.ini now loads as zzz-joomla-uploads.ini so its
own memory_limit wins: the media API compares Content-Length with
toBytes(memory_limit) and rejects every upload at -1.
TODOs record the memory_limit revert condition and accepted
trade-offs, Akaunting on PHP 8.1, FusionDirectory on PHP 7.4 and the
public Keycloak admin console.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
In swarm mode `docker stack deploy` ignores the host IP of a short port
mapping, so every `127.0.0.1:<port>:<port>` that container_ports renders for
a `ports.local` entry is published by the ingress routing mesh on every
node, via docker-proxy on 0.0.0.0 and [::] plus DNAT in nat PREROUTING.
Databases, LDAP and application backends answered from outside the cluster.
svc-swarm-node now loads an nftables table through svc-net-firewall. Its
prerouting filter chain (priority mangle, ahead of Docker's dstnat) drops
packets addressed to the node for any ingress-published port that no
application declares under `ports.public`. Loopback, docker0,
docker_gwbridge, the user bridges and the swarm nodes' own addresses stay
admitted. Host-side provisioning over 127.0.0.1, containers that reach a
host port through host-gateway, and node-to-node consumers such as the
registry-cache mirror keep working.
Both sets are read live on the manager (`docker service inspect` for
`.Endpoint.Ports`, `docker node inspect` for `.Status.Addr`), so ephemeral
ingress ports and nodes outside `--limit` are covered. The table is loaded
on a node before it joins, reloaded on every node after the join, and
reloaded after each stack deploy by the sys-svc-compose swarm handler.
Refs #1089
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Swarm resolves exactly as before: the meshed arm puts every node and the
backup host into the group, the direct arm none, and advertise_addr stays ''
there.
WIREGUARD_ENABLED followed the deployment mode alone. A swarm's backup host
is deployed in its own play, which resolves to compose mode, so the role was
included there and skipped: the hub listed it as a data peer and no tunnel
ever answered for it.
The layer now runs in swarm, as before, and on any host that owns a mesh
address the CLI wrote for it, read through the mesh_address lookup. A
compose host without a mesh stays untouched.
Proven on the local swarm lab (svc-dns-unbound, development layout): the
backup host brought up data at 10.100.1.12, passed the handshake and MTU
gate against the hub, the hub saw both data peers handshake, the backup host
reached a worker on the swarm mesh through the manager, and the drill and
both passes stayed green.
The advertise address read meshes.swarm, so a deployment that named its
meshes differently got '' and swarm bound to the underlay the mesh exists to
keep it off. It now reads the manager's primary mesh.
The guarantees were asserted against the declared swarm and data pair,
so they proved two meshes work rather than that the planner works. A
deployment declares its own, and the review asks for any number of them with
the nodes arranged however the client wants.
The same invariants now run over generated topologies: one to eight meshes,
three arrangements each, hosts dealt out so some belong to several meshes and
some to none. 2247 subtests, four seconds, and the planner needed no change to
pass them.
One conflict, in the generated role overview table of README.md.
Both sides had edited the same block. Upstream dropped Redis from the pihole
row; this branch still listed it. Upstream also listed svc-net-wireguard-core,
-plain and -firewalled, which this branch removed in e706170a0b as superseded
by the unified svc-net-wireguard role.
Resolved by taking upstream's block and then running `make readme-index`, so
the table is generated from the roles that actually exist rather than
hand-merged: Redis is gone from pihole, the three legacy WireGuard roles stay
gone, and svc-net-wireguard is listed once. `make readme-index check=true`
reports the table up to date.
Git kept the role deletions silently and correctly - upstream has not touched
those three directories since the merge base, so there was no modify/delete
conflict to resolve.
shared_buffers, work_mem and maintenance_work_mem were a quarter, half
and a twentieth of the host's RAM, while the Postgres container is
limited to 6g. On a 64 GB host that asked for 15997 MB of shared
buffers inside it. They are now the same shares of the container's
mem_limit through lookup('service_memory_mb'): 1536 MB, 3072 MB divided
by max_connections, and 307 MB.
The three variables move from group_vars/all/17_resource.yml into the
role's vars. They no longer describe the host, and only the role's
compose template reads them. Their *_MB helpers and POSTGRES_TOTAL_RAM_MB
had no other reader and are gone. An inventory can no longer override
the three values; the knob is services.postgres.mem_limit.
Discourse's db_shared_buffers was a literal 4096MB, all of the 4g the
role declares. It becomes a quarter of that limit, 1024MB. It only
applies when Discourse runs its bundled database.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
That commit added test_bcrypt_hash.py with an Exception comment and left a
trailing blank line in group_vars/all/11_oidc.yml. Both go.
What the comment said still holds: the test hashes with the raw bcrypt package
because Ansible's password_hash('bcrypt') routes through passlib, whose bcrypt
self-test can misreport a password well under 72 bytes as too long when
passlib and the installed bcrypt disagree on the version.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Add single-node Wazuh stack (manager, indexer, dashboard) as a new
Ansible role, integrated with the platform's Keycloak realm via native
OIDC and secondary LDAP authorization, with three-tier RBAC.
- Pins wazuh/wazuh-{manager,indexer,dashboard}:4.14.6 and
wazuh-certs-generator:0.0.4 (one-shot cert bootstrap job)
- No external database: wazuh.indexer (OpenSearch) is self-contained
- Auth: native OIDC against Keycloak, working end-to-end after two
infra fixes (JVM cacerts trust for the indexer's outbound calls to
Keycloak, front-proxy vhost wiring)
- LDAP wired as a secondary, authentication-independent authorization
backend (group-based backend_roles enrichment only)
- RBAC: three roles (administrator, security-analyst, readonly-auditor)
via the platform's standard meta/rbac.yml + rbac_group_path pipeline
- Documents a genuine upstream Wazuh bug: wz-home as the default
landing route throws an uncaught client exception for any
non-all_access role; default route moved to OpenSearch Dashboards'
own /app/home
Addressed two full rounds of review feedback (Kevin, Alejandro):
WEB_PROTOCOL usage, service-account naming, comment cleanup, shared
Keycloak-admin-helper extraction, a real TOCTOU race in RBAC test
setup, a Mermaid auth-flow diagram, meta/services.yml placement for
the certs-generator, CA-trust-path SPOT usage, CSP tightening
(including a genuine worker-src: blob: requirement found via live
testing), and a loud (non-silent) dev-bucket fallback warning.
Rebased onto main after a 3-month gap; brought the role in line with
~30 platform-convention changes introduced in that window (swarm
compose-template rules, Tor/onion contract, Playwright timeout/goto
wrappers, file layout, task numbering, comment policy, and more).
Live-deploy verification (9 iterations) found and fixed 6 further
real bugs invisible to lint/unit tests alone:
- credentials.* / meta/schema.yml -> secrets.credentials.* /
meta/secrets.yml (another platform-drift casualty)
- notify: "compose up" -> "compose-up" (day-one handler-name typo)
- local.subnet collision with web-app-mailu's /24
- missing primary: false alongside tor.exclusive: false, which let
the onion domain shadow the clearnet vhost entirely
- a second front-proxy include for the onion domain, later reverted
(see below)
- shared sys-stk-front-proxy/sys-front-tls TLS-lookup keying bug
(app-id-keyed vs. domain-keyed), which silently applied TLS to
Wazuh's onion vhost
The last two are related: enabling genuine dual-stack (clearnet +
onion) reachability for tor.exclusive: false surfaced a real bug in
two shared platform roles. That fix was verified safe against all 103
apps currently in the repo, but is deliberately NOT included here -
shared-role changes deserve their own PR and review, not to ride in
on a single app's feature branch. Wazuh's second front-proxy include
was reverted accordingly; the resulting onion-reachability gap is
documented in the README as a known, platform-scoped limitation.
make test: 5508 passed, 3 failed (all three trace to untracked local
scratch files - Dockerfile.bak, scripts/lab-*.sh - not present in any
CI checkout).
One conflict, in the generated role table of README.md. Both sides changed
the same rows: upstream swapped WordPress's Mailu dependency for Stalwart,
while this branch replaced the three svc-net-wireguard-{core,plain,
firewalled} rows with the single svc-net-wireguard it deploys.
Resolved by regenerating rather than by hand, since make readme-index owns
that table and gates it with --check. The regenerated table carries both
sides and the check reports it up to date.
README.md was the only file both sides touched, and nothing in the tree
still names the three replaced roles.
Formatter output only: the parsed YAML and the Python AST of every touched file equal their previous versions.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
One conflict, in the MCP exclusion list of requirement 035. Both sides
edited the same line: main added svc-net-tor-smtp, this branch replaced the
three svc-net-wireguard-{core,plain,firewalled} entries with the single
svc-net-wireguard it deploys. Resolved to the union of both — 24 entries —
which is what the merged tree actually holds.
Everything else merged clean and was checked for overlap rather than
assumed: no reference anywhere still names the three replaced roles;
matrix.py kept main's PRIMARY_APPS alongside the reordered round, and the
bootstrap pass does not need it, since only one template reads it and
group_vars defaults it empty; and main's ansible.cfg addition sits in
[defaults], so the ssh_args the mesh tool derives its own connection
settings from are untouched.
GitLab's migration on its dedicated postgres sidecar failed with "ERROR: out of shared memory" and the hint to raise max_locks_per_transaction. The default of 64 locks per transaction cannot cover a schema that locks thousands of tables and indexes in one transaction: a reproduction on postgres:18 failed at table 7453 with the default and held 20000 tables at 256.
The value sits next to the other postgres tuning constants in group_vars/all/17_resource.yml and reaches both the central svc-db-postgres command and the per-app sidecar. The sidecar's new command keeps the image entrypoint and only adds the flag. The shared lock table grows to about 256 x max_connections entries, roughly 30 MB at the 400-connection cap.
Ported from 7690d2ca6d on feature/store-i18n, which kept the constant in the resource lookup that main no longer has.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A consumer that admits mesh traffic needs every range a member can arrive
from, so a deployment carving its meshes from more than one pool extends this
rather than each consumer.
Brings in 225 commits, including the merged web-app-openbao work and the
LMStudio loopback fix.
Seven conflicts:
- The legacy svc-net-wireguard-core/plain/firewalled roles this branch deleted
in e706170a0 were modified upstream. The deletion stands, which is the whole
premise of the branch.
- The MCP audit takes upstream's list plus svc-net-firewall, with the three
legacy wireguard entries collapsed into the single svc-net-wireguard: 23.
- web-app-openbao/README.md takes main's reviewed version wholesale.
- matrix.py takes both sides: upstream's app_id argument to _reset_credentials
and this branch's _write_mesh call after it.
The merge pushed matrix.py to 511 lines against the 500 cap, so the step
runner moved to utils/tests/swarm/run.py, which is self-contained and leaves
matrix.py at 427. Its test moved with it.
c6e8990e25 kept the daemon's auth key off the shared volume and made the red
job green, but it left multi-replica operation as broken as it found it: three
daemons still shared one /root/.lmstudio, where LM Studio rewrites
extensions/plugins with rm -rf at every start, writes settings.json in place and
resets it to defaults on a torn read, and re-copies a 102 MB bin/lms while peers
execute it.
Splitting the tree by access pattern fixes that. /root/.lmstudio is node-local,
because it is per-daemon state the daemon rewrites in place. models/ stays
shared, because a blob is written once and read forever.
That leaves one question, and it is temporal rather than mechanical: the daemon
discovers models by an unconditional scan at startup and by an inotify watcher,
and inotify never sees writes made by another NFS client. So a model seeded
after a daemon started is invisible to it, whichever tool wrote the file. The
previous design answered that with `container service update --force`, which
restarts the whole inference backend svc-ai-litellm points at, on every deploy.
The seed now runs before the stack does. 00_core.yml places the blob on the NFS
export ahead of the compose include that deploys the stack, so every daemon
finds a complete tree at its own startup scan and nothing has to be restarted.
Measured on a three-node swarm with NFS:
Fetch the models into the shared store 136.14s one 531 MB download
Pre-pull on mgr-01 0.69s
Pre-pull on wrk-01 0.50s
Pre-pull on wrk-02 0.49s
Half a second is not a download. All three daemons recognised the pre-placed
model, so the cluster now stores and transfers it once instead of three times.
The nested mount across two devices raises no EXDEV, and the per-node disk guard
correctly skipped itself four times because the pre-seed covers that check.
Vendor coupling is gone from the role. `repo` and `url` are declared fields
rather than pieces cut out of a HuggingFace URL with a regex; a model on any
host needs only those two. `lms get` still knows only HuggingFace, so `source`
stays for it, and a non-HF model simply leaves it unset: the blob is seeded and
the startup scan picks it up. The external check follows, comparing against the
HuggingFace API only for entries whose url actually points there.
The guard arithmetic was wrong and is now derived rather than guessed. Peak
usage is the missing bytes plus the largest single model, not twice the declared
total, so the read-back moved ahead of the check and subtracts what the store
already holds. Before, the requirement grew after every successful seed. Staging
moved beside the tree instead of inside it, so an aborted transfer cannot leave
orphans where the indexer scans.
A deployment test ships with it and states its findings rather than its exit
code: it fails on `authPacket` by naming the shared key, on a missing or
mismatching blob by naming path and digest, and on a model that is on disk but
absent from /v1/models by saying the startup scan did not happen.
Gone with the restart: import.sh and its container-cp suppression, seed.yml,
rescan.yml, and the disk-copy factor of three that only existed because the
import staged through the container.
Residual: compose is untested here. There the pre-seed is off, the node-local
guard is the one that runs, and `lms get` is the only fetch, so a non-HF model
is not loadable in that mode. Seeding node-local volumes is the follow-up that
closes it.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A wedged LM Studio daemon cost CI job 105558224377 three hours: three kills at
the global 60-minute `CONTAINER_EXEC_TIMEOUT`, for a 531 MB model that a healthy
node pulls in 27 seconds. Nothing in the path was sized against the thing being
fetched, and nothing checked whether it could land at all.
Size. Each model now declares `bytes`, taken from the value the deploy already
printed as "OK (531068224 bytes)", and the pre-pull derives its own timeout per
model: max(preload_floor_seconds, bytes / preload_min_bytes_per_second). The
floor of 600s protects a small model from a brief stall; the rate of 1 MB/s sits
twenty times below the 21.98 MB/s a healthy pull reached and seven thousand
times above the 142 B/s the wedged one managed, so it separates the two without
guessing. Evaluated against ansible-core's filters: 531 MB and 10 MB both land
on the floor, 4 GB gives 4000s.
Disk. The seed measures free space with `df -P -B1` before fetching and refuses
with a message naming both figures in GiB, the factor and the file to edit. The
factor is three because the node cache, the staging copy inside the container
and the models volume all exist at once during the import; that is measured, not
padding.
Fallback. A model may list `mirrors`, extra bases tried after the source. The
mirror task is a no-op once the source delivered, because get_url skips a dest
whose checksum already matches: measured at 145.24s for the first pass against
0.64s for the second. The list ships empty, because the obvious candidate is
not one: hf-mirror.com answers 206 but resolves to us.aws.cdn.hf.co, so it is a
front end for HuggingFace and adds no redundancy against a HuggingFace outage.
The mechanism is here; an independent copy is the operator's to name.
The external check grew with the declaration: `bytes` is compared against the
size the API publishes, and every declared mirror must still serve the pinned
file, or it is redundancy on paper only.
01_manager_ops.yml's rescue rationale claimed that restarting the container
clears a wedged transfer. Job 105558224377 shows the restart returning rc 0
without clearing it, so the claim now says what was measured instead.
Residual: the mirror path has never run in a deploy, since no mirror is
declared. Its loop expression is verified against ansible-core's subelements
over four shapes, including the zero-iteration case when nothing is mirrored.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
swarm.manager.advertise_addr already existed as an empty hook. Resolving it
from the mesh the cross-host writer put in the inventory is what actually moves
swarm's control, gossip and overlay ports onto the tunnel; left empty, swarm
binds the underlay and the tunnel carries nothing it was built for.
Guarded on the mesh being present rather than assumed, so a compose deploy and
any swarm inventory written before the mesh pass still resolve to the previous
empty default.
One SPOT the mesh writer reads and Ansible resolves. 10.100.0.0/16 is chosen to
clear every range the platform already allocates: Docker holds 10.208.0.0/12,
nested Docker 192.168.67.0/24 and 192.168.68.0/22, the swarm lab
192.168.244.0/24, and per-role subnets sit in the lower 192.168.0.0/16. An
overlap surfaces as unroutable peers long after the deploy reports green.
The data mesh selects its backup spoke by the svc-bkp- prefix rather than by
name, mirroring IS_BACKUP_HOST, so a new backup role joins without editing this
declaration.
Two MTU values are pinned rather than discovered. The overlay stacks VXLAN
inside WireGuard: 1500 on the underlay leaves 1420 after the WireGuard header
and 1370 after VXLAN, and a path that exceeds it drops large payloads only,
which reads as flaky CI rather than as a misconfiguration.
The preload is the only artifact this repository pulls raw from the public
internet. Everything else, including the LM Studio image itself, arrives
through the mirror. `lms get` gets exactly one attempt under async, cannot
resume, and wedges its own daemon when a transfer dies, so one dropped socket
costs a whole job. XWiki#2 of run 35152776835 died 28 seconds into the
transfer at 3 MB/s and rising, with 3600 seconds of budget left.
The model is now fetched on the hosting node with get_url_retry, which gives
13 attempts and verifies a pinned sha256, and handed to the container with
`lms import`. Both steps are best effort: when the warm-up fails, `lms get`
runs exactly as before, so the change adds a faster path without removing the
old one. The pattern is web-app-mediawiki's extension install, down to the
`container cp` in the script.
Verified against the binary the role actually deploys, extracted from
lmstudio/llmster-preview:cpu and built 2025-08-28, not against upstream HEAD:
`lms import` takes `<file-path>`, `-y, --yes` and `--user-repo <user/repo>`;
the destination is join(modelsFolderPath, user, repo, basename(path)) under
join(lmstudioHome, "models"); the image ships no settings.json, so
downloadsFolder cannot move that target. The already-present branch of
`lms get` carries no process.exit, so it returns 0 and the task's existing
`until: rc == 0` accepts it.
The pin carries file and sha256 next to the source URL, and the digest is
quoted because an all-digit hash would otherwise parse as an integer and fail
the shape check for the wrong reason. tests/external guards the pin against
what HuggingFace publishes, comparing digests through the API without
transferring the file, and reports a network error as unverified rather than
failing a good pin on a firewalled runner.
Two limits worth stating. `lms get` still queries HuggingFace before it
reaches the download step, so this removes 531 MB of exposure rather than the
network dependency. And the tolerated substring 'already downloaded' in the
task's until, failed_when and reap list matches no output this version emits,
which says "You already have this model."; that branch is dead today and is
left untouched here because the return code carries the contract.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The catalogue held three models and the development inventory carried a second, CI-only list that both AI roles pulled in through their variant 0. That made the model a variant concern and split one fact across a group_var, an inventory file and two meta files.
AI_LOCAL_MODELS now holds the single entry the CI list held, qwen2.5 0.5b instruct at 531 MB in Q8_0, which lms get is already proven to fetch in this stack. AI_LOCAL_MODELS_CI and both variant overrides are gone, so variant 0 states only container_backup and the memory limits.
The dropped entries have no consumer: llama3, mistral and nomic appear solely in test fixtures that stub the lookups, and nothing requests an embedding model, the only embed matches being the filters that exclude one when picking the chat model. LITELLM_CHAT_MODEL takes the first non-embed alias, so it resolves and LITELLM_USABLE stays true.
Note this changes production, not only CI: a deployment no longer pulls a 7B model or an embedding model. It also shrinks the pre-pull that burned 18796s in job 104622514450 from gigabytes to 531 MB.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The rule named roles/<role>/config/main.yml, which no role has: f65f4d60fb migrated the last one, web-app-pihole, to meta/services.yml and did not carry the README with it. A reader following the sentence found nowhere to put the value.
It now names the two real homes and the property that decides between them. meta/services.yml is overridable, because the inventory's applications block is deep-merged on top of it and lookup('config', ...) reads the result; vars/main.yml is not, because role vars outrank inventory variables. The clause about a value only one role reads today is the case that just went wrong: group_vars/all/19_tor.yml looked harmless for exactly that reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
group_vars/all/README.md forbids role-specific configuration in that directory, and 19_tor.yml held five variables no role but svc-net-tor ever read. They now live where the README points: the values an operator may override in meta/services.yml, where the inventory's applications block is deep-merged on top, and the derived ones in vars/main.yml.
The exception is the egress virtual range, which moves the other way into 08_networks.yml. It is not a property of the role but a claim on shared address space, and 0f6728a4c1 records what a collision costs: tor's upstream default 10.192.0.0/10 swallowed the docker pool at 10.208.0.0/12, and every connection to a container address came back as 'errno 104 Connection reset by peer'. The two numbers that must not overlap now sit one line apart.
The onion's HTTP forward becomes a declaration rather than a derivation. lookup('tor_ports') walks group_names and collects what those roles declare under ports.onion; port 80 belongs to svc-prx-openresty, which a node acquires through sys-stk-front-proxy and which therefore appears in no dependency list an inventory carries. A node provisioned with --include svc-net-tor published an onion forwarding SSH and nothing else, leaving every app it serves unreachable - measured as ERR_SOCKS_CONNECTION_FAILED in the CI instructions replay. The new role-local tor_extra_ports lookup composes the flagged forwards with the derived ones and collapses a port named twice, since torrc writes one HiddenServicePort line per entry.
Not moved: tor_ports stays in plugins/lookup because tor_reachable imports from it and three database roles use that; and the flags sit outside ports.onion because every key there mirrors a port the role declares under ports.local, which neither sshd nor the front proxy is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
dnsmasq binds the docker bridge address with the clearnet upstreams configured and the node's own .onion records loaded, and no rule covered port 53. Any source that could route a packet to that address got an open resolver plus the split-DNS records. c89ca42c46 claimed local-service closed this; abb6f37297 showed the keyword never took effect.
The guard now carries two named sets instead of one plus a special case. 'containers' is the docker pool list, for tor's own transparent and DNS ports. 'internal' is NETWORK_INTERNAL_CIDRS, for the resolver and the SOCKS port. Every guarded entry names its set, which folds the hand-written SOCKS rule and its manual append into the same list the reverse-path sets are built from.
The resolver takes the wider set deliberately: 'internal' is the list torrc.j2 already spends on SocksPolicy accept. The narrow set carries only 10.208.0.0/12 and 10.7.7.0/24 out of 10.0.0.0/8, which leaves out swarm's ingress network, every overlay docker allocates without an ipam block, and a node whose own address sits elsewhere in 10/8. A query bound to such an address was measured timing out against the narrow set. Guarding the resolver more strictly than the proxy beside it would refuse a name lookup to a source that is handed a full SOCKS proxy six lines earlier.
NETWORK_DNS_PORT lives next to the other resolver constants rather than in meta/services.yml: ports.local describes host-bound container ports, where 53 collides with svc-dns-unbound and falls outside every PORT_BANDS band.
Measured on the rendered artifact: the table loads and reloads with a byte-identical listing, every compose source of port 53 resolves inside 'internal' and every source of 9040/9053 inside 'containers', and a mistyped set name, an empty CIDR list or a missing admitted key each fail loudly at validate time. Port 53 newly passes a strict reverse-path check, so an asymmetrically routed client loses name resolution.
test_fragments built its guard list from a fixture of three entries while the role shipped six, so it validated the template against a port list nobody deploys. It now takes the shape from the declaration and fills only the numbers.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The generic unit template could express neither an ordering dependency nor an Install section, so a role needing either shipped its own templates/systemctl.service.j2 and re-answered the daemon reload, the enable and the flush behaviour by hand.
Add After, Before, Wants, RemainAfterExit and a WantedBy driven Install section, all inert at their defaults, and route the unit path, its owner, become and the systemctl probes through a system_service_scope so a per-login unit lands under the user's own systemd directory.
00_core is gated on the system scope rather than only its prune and lockdown steps: it sets run_once_sys_service, so a user-scope caller running first would otherwise skip the lockdown for every system-scope role behind it in the same play.
Rendering the previous template against the same inputs drops no directive and adds none; the only difference is the blank line an empty directive used to leave behind.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The private blocks were written out wherever something needed to say "this is us": NETWORK_TRUSTED_PROXY_CIDRS listed all three, Tor's SocksPolicy repeated them, unbound's access-control repeated them again. Four copies of a fact, and nothing tying them together, so narrowing one narrows nothing else.
Worse, the block role networks are carved from was never declared at all. The allocator behind 'infinito contributing network address suggest' derives its umbrella from the subnets already in use, so the tree defined the rule instead of the rule defining the tree. Anything consuming "our own address space" had to guess it, and a guess that is one role out of date silently excludes that role. Measured: 106 of 107 role subnets sit in 192.168.0.0/16 and one does not, so a firewall range derived from the pool by eye would have cut BigBlueButton off.
The blocks are now named once and everything else is derived, NETWORK_TRUSTED_PROXY_CIDRS included, which renders byte-identically to before for its four consumers.
BigBlueButton stays outside because its upstream pins both the block and its gateway, which meta/services.yml repeats as upstream_host 10.7.7.1. The exception list therefore carries the role id rather than the CIDR: the block is already declared in that role's meta/networks.yml, and repeating it here would be the same duplication one layer up. The role_subnets lookup resolves the ids and raises on a role that declares no subnet, because a silently dropped entry would narrow whatever firewall or trust list consumes the result.
A lint keeps the declaration honest in both directions: every role subnet must sit in the declared space, and an exception whose block the pool already admits is rejected so it cannot outlive its reason.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.
The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.
svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.
Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.
The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.
Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
API is a plain mapping, so an inventory that configures three providers replaces the whole declaration from group_vars/all/18_api.yml instead of merging into it. inventories/development/default.yml declares openai, anthropic and openrouter, so github, google and jira did not exist in CI at all and lookup('api', 'github.client_id') raised.
A raising lookup inside a value the applications renderer is templating does not fail the render: _templar_render_best_effort returns the string untouched and, for a value carrying a non-env lookup(, refuses the fallback as well. The raw template text reached plugin_value.enabled | bool, which coerced it to False and emitted the deprecation warning that fails the deploy gate. Run 34893363518, job 104153194109 lost web-app-nextcloud to exactly that, for three integrations at once. The configvalue entries would have stored template text as the client id.
utils/api.py merges the scope's override over the declaration, so an override narrows values and never the provider set, and holds the enable rule: every credential a provider declares must carry a value. The api lookup reads through it; the new api_enabled lookup gates on it and returns a real boolean instead of a string a filter has to coerce.
The same class sat in group_vars/all/16_ai.yml and roles/svc-ai-litellm/meta/secrets.yml as bare API.<provider>.<key> dereferences; both now go through the lookups.
Two lints keep it shut. test_addon_enabled_renders_to_bool renders the whole applications map against a scope with API blanked, the most hostile override an inventory may legally write, and requires every addon gate to come out a boolean. test_api_accessed_via_lookup forbids a bare API dereference in Jinja under roles/, group_vars/ and inventories/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
