runtime: nvidia on its own is a no-op. The NVIDIA container runtime hook reads NVIDIA_VISIBLE_DEVICES off the environment of the container and injects nothing when it is unset, and that variable appeared nowhere in the repository. The engine images build on python slim with CUDA wheels rather than on an nvidia/cuda base, so they carry no such default in their image environment either; a role built on an nvidia/cuda base would have worked by accident. The deployed nllb-engine had been crash-looping on "Found no NVIDIA driver on your system" for 229 restarts.
Measured against the same image on the same nested daemon: --runtime=nvidia alone reports no devices, while adding NVIDIA_VISIBLE_DEVICES and NVIDIA_DRIVER_CAPABILITIES yields the device nodes, a working nvidia-smi and torch reporting 2.9.1+cu128 True 1. After the deploy the live container answers True 1 NVIDIA GeForce RTX 3080 Ti Laptop GPU, with the restart count back to 0 and health reported healthy.
Devices are named explicitly rather than with all. 00_core/gpu.yml asks the driver for memory.total and uuid, sorts by memory descending and publishes the uuid list, so a framework that grabs device zero gets the largest card. With a single card it degenerates to that card. NVIDIA_DRIVER_CAPABILITIES lives in group_vars rather than in the role vars because role vars are out of scope while a different role renders the template, which an earlier attempt proved by failing on an undefined name while rendering svc-ai-nllb-engine; SYS_SVC_CONTAINER_GPU_DEVICES crosses that boundary because it is a host-scoped set_fact.
meta/packages.yml already declared the toolkit for Archlinux, Debian and RedHat, the latter two carrying a note that sys-svc-container configures the nvidia.github.io repository before installing, yet only the apt path was ever built and the include was gated to Debian and Ubuntu. The declaration and the tasks contradicted each other. nvidia.yml now dispatches per family the way the role already does for Docker: an apt source for Debian, a repo file for the RedHat family, and nothing for Arch, where the package sits in extra. The install itself goes through package_install, and the runtime registration in daemon_config.yml was already presence-based rather than distro-gated, so it needed no change.
The discovery tasks live in their own file because a when-guarded block is capped at three direct children, and splitting the guard across two siblings would have tripped the repeated-when rule instead. Nothing renders on a host without a card: the gpu lookup and the set_fact hang on the same register, so the template emits the block exactly when the fact exists.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
- Declare `__all__` in every package `__init__.py` whose `PROJECT_ROOT`
is imported by its modules (163 packages), so the constant counts as
exported; list every public name where the package defines more.
- Empty the 15 package `__init__.py` files whose `PROJECT_ROOT` no module
imports.
- Declare the public API of `utils.github.variant.axes` in `__all__`,
re-exports included, instead of silencing them with `noqa: F401`; the
list matches the one `feature/store-i18n` already carries.
- Use `import unittest.mock as mock` in `test_applications.py`, as the
rest of the suite does.
- Spell `_NON_PRINTABLE` in `utils/yaml_bootstrap.py` as the equivalent
BMP-only positive class; it matches exactly the same 2111 code points
as the old negated class across U+0000..U+10FFFF.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
coreutils timeout without --foreground calls setpgid, so the command it
wraps leaves the caller's process group. A wrapped command that then
reads the controlling terminal takes SIGTTIN and stops forever instead
of being killed at the deadline, which inverts the purpose of the
wrapper.
That is the cause behind d31b49e30d: container exec wraps docker in
this prefix, an ansible.builtin.script task gives the remote shell a
pty, and docker exec -i stops on its first read. Three jobs of run
39186571513 idled 1h37m to 2h22m before the outer budget killed them.
The scripts there close their own stdin; this closes the class.
utils/diagnostics/container.py gets the flag too. docker events reads
no stdin and was never at risk, but leaving one non-compliant
construction would have forced an exemption on the new rule's first
day.
The rule itself cannot be a text search. The prefix comes from a
function and the -i from the caller's argv, so timeout and the command
that reads stdin never meet in one line of source: a grep over the
tree scores zero hits both before and after the defect. The lint reads
the parsed tree instead and requires --foreground in any list literal
that starts with timeout. Checked against the pre-fix line, it reports
it.
Cost, deliberately accepted: --foreground signals the direct child
only, not a process tree. For docker exec the direct child is the CLI,
which is what has to die to release the task, but a wrapped command
that spawns children now leaves them where the group kill would have
taken them. covers the case where that
trade goes the other way.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A local deploy stopped four times in two days in "Ensure Docker apt key
and sources are consistent" (#849). apt_repo.sh downloads the Docker GPG
key on every run, and when download.docker.com refused the connection,
curl's four attempts were over within about six seconds and the script
returned 1, although the key from an earlier run was installed and is
all the following apt operations need.
install_key now fails only when no key is installed. With a key in
place, a refused download prints a warning and keeps it. A downloaded
key that differs still replaces the installed one, and the 0644 mode is
enforced in every case. The keyring path became an argument:
ensure-key-and-sanitize passes the keyring it already hands to
sanitize_sources, so the role's call still resolves to
/etc/apt/keyrings/docker.asc, and a test can point the script at a
temporary keyring.
Verified: tests/unit/python/roles/sys-svc-container/files/test_apt_repo.py
stubs curl and sudo. A refused download keeps the installed key and ends
with changed=0, a refused download without a key exits 1, and a new key
replaces the installed one. A copy of the script whose keep branch always
errors, the old behaviour, fails the first test. Two local deploys ran
the new script with rc 0 and an empty stderr and left the installed key
untouched. shellcheck is clean.
Not verified live: the refused download itself, which needs
download.docker.com to refuse a connection during a deploy. A host
without a key still needs the download and still fails when it is
refused.
Closes#849
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
make autoformat reflowed a resource_filter call, a generator expression and five test files it touched along the way. The new architecture test also carried the two findings the first gate run stopped on: three RUF012, where a TestCase class attribute needs ClassVar, and two PERF401, where an append loop becomes list.extend over a generator.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Collapsing the three GPU call sites into one lookup moved the whole decision behind lookup('gpu', ...), but both consumers' tests still stubbed lookup with a constant. One test per file went red; three more kept passing for the wrong reason, because the stub's {} is falsy and every negative case was therefore true no matter what the template did.
The stubs now branch on the lookup name and record what they were asked, so each file guards what it still owns: the answer reaches the compose key or the role variable, and the lookup is asked for this service. The declared-flag-times-device matrix stays with the lookup's own tests, which is also where the boolean() over bool() rationale now lives, checked rather than written beside the call it protects.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container cap had to be an absolute number, and docker refuses one above
the host's core count, so "use this machine" could only be written as a
number that breaks on a smaller machine. cpus now also takes a percentage,
read against the host's raw count: 100% is every CPU and lands exactly on the
ceiling docker accepts, 50% is half of it.
The share is read against that raw count rather than against the fair-share
default, because the two CPUs the resource lookup reserves exist to bound
what a service gets when it declares nothing. A service writing a percentage
overrides that default on purpose, and subtracting the reserve a second time
would count it twice and leave no way to name the whole machine.
A bare cpus: 0 is refused from here on. Docker reads a zero as uncapped
rather than as none, so the value most likely to be typed by mistake was also
the one that removed the limit; the error names the fix. No role declared
one: all 315 cpus values in the tree are plain numbers, the smallest 0.01.
Two lookups carry the resolution so that no caller repeats it or has to know
what a percentage is read against. lookup('cpus', app, service) replaces six
copies of lookup('config', ...) piped through the resolver; those six size
worker pools and thread counts from their own cpus, and left alone a
percentage would have given libretranslate a single thread and handed the
agent broker an uncapped container. lookup('webserver_resource', key)
replaces the two blocks that took the smaller of the proxy's value and the
app's, which were the same eight lines written twice for cpus and for
pids_limit.
resolve_cpus is no longer registered as a jinja filter. Nothing referenced it
from a template once the lookups existed, and keeping it would have left two
public ways to resolve one value.
Measured: 96 tests across the filter, both lookups and the container render
templates, including that a percentage resolves before the smaller of the two
is chosen, and that pids_limit stays an integer because nginx renders it
straight into worker_connections.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
DEPLOYMENT_MODE was compared against a quoted literal 219 times across 110
files. IS_COMPOSE_MODE and IS_SWARM_MODE say the same thing once, and
test_deployment_mode_booleans keeps it that way: only the file that defines
them may still compare.
Both are real booleans under this ansible, measured before the swap because
the alternative is silent: a variable that templates to the string "False" is
truthy in Jinja, which is why IS_STACK_HOST is written `| bool` at every use.
These need no filter.
What had to learn the names:
* utils/annotations/task_gate.py, which several lints ask whether a task is
compose-only gated;
* four compose-template lints that recognise their gate by matching the
expression;
* the fixtures that stand in for the play's variables. They derive the
booleans from the mode now instead of carrying them, or a test that
overrides DEPLOYMENT_MODE keeps the old pair and asserts the wrong
branch.
Each replacement was checked rather than trusted: the 206 conditions that
changed in place were evaluated old against new through the Templar under
compose, swarm and host, with no disagreement, and no changed line turned out
to be anything but a condition swap. `!= 'swarm'` became `not IS_SWARM_MODE`
rather than the shorter IS_COMPOSE_MODE, because only the literal translation
stays correct should the mode ever be host.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
resource.yml.j2 renders `runtime: nvidia` for any service that declares
`gpu: true`, gated a second time on the host carrying /dev/nvidia0 so a
CPU-only host never gets a runtime its daemon cannot resolve. The device
fact is probed in daemon_config.yml next to the kata and gVisor shims, and
daemon.json.j2 lists nvidia alongside them. nvidia stays out of
SANDBOX_RUNTIME: it passes a device through, it does not isolate.
nvidia.yml drops rival apt sources for libnvidia-container before writing
its own. The other order leaves two sources with different Signed-By keys
for one URL, which makes apt refuse the entire source list.
test_gpu_runtime.py covers both polarities and the missing device fact; the
latter caught `x.stat.exists | default(false)`, which raises on the
undefined object before the filter runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
RFC 6761 reserves .test for testing and .example for documentation, and the
project's own domain was doing both jobs under the documentation name. It is
what default.env hands every local deploy and what the test fixtures assert
against, so it is a testing name.
Rename it wherever it stands for the running machine: the environment
defaults, the development inventory, the workspace and trust scripts, the test
suites, and the four documents that describe real behaviour - the DNS mapping
to loopback, the browser CA trust, WSL2 resolution. Those documents would have
started lying the moment default.env changed.
The sandbox allowlist moves with them, since egress is matched per pattern and
nothing resolves the old name any more, and the sandbox document that mirrors
that list follows it.
Two places keep .example on purpose: the link-style sample in the
documentation guide, which is documentation about documentation, and every
generic fixture name elsewhere in the suite - a.example, foo.example and their
kin. Those are a larger sweep of their own and are not touched here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
update_config never set failure_action, so swarmkit's default applied: pause,
at a max_failure_ratio of zero. One failed task stops the rollout for good, and
nothing in the deploy resumes it.
That contradicts the restart_policy three lines below. The policy replaces the
task and the service ends up healthy, while the update status stays latched and
service_converged.sh reads that status before it looks at any task, so the
deploy reports a failure over a service that is running. The rescue artifact
shows the sequence: the old task shut down, its replacement tripped over a
containerd object runsc could not reap, and the third task was Running while
the job died on "update latched".
Four swarm jobs failed that way, all in the update pass, never in the first.
The convergence check is untouched and stays as strict as it was: a genuinely
broken spec fails every task and still trips "tasks not running yet".
What this does give up: the latch was also the only thing catching a service
that keeps crash-looping. At ratio zero any failure latched, so a task dying
and restarting forever was guaranteed to be reported. Now the rollout
completes, and the check samples task state at one moment - a task on its
fiftieth restart still reads as Running, and wait_converged's `until` needs a
single lucky sample. Restoring that detection needs the check to look at task
churn, which the service ps history already carries and the script prints only
on failure today. Not in this commit.
max_failure_ratio cannot substitute here: at replicas 1, the common case in
this repository, one failure is always the whole service.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
834 commits from the Tor onion-service branch against 299 of ours, from merge
base 7b646bdece: 127 conflicted files, 3506 files changed.
The two sides are largely orthogonal. Theirs adds svc-net-tor and a `tor` service
across the role tree, plus two sweeping refactors: shipped scripts sorted by
language (files/ruby/, files/python/, files/shell/, scripts/lint/*.sh) and the
unit suite moved under tests/unit/python/. Ours is requirement 036, the native AI
gateway, plus the MCP role work. Most conflicts were both sides appending to the
same service map, resolved as unions.
Resolutions that were not unions:
Credentials moved namespace. Theirs renamed meta/schema.yml to meta/secrets.yml
and re-addressed credentials as secrets.credentials.<name>. Git carried our
litellm_api_key blocks through the rename, but every consumer we added still read
'credentials.<name>', which no longer resolves. All 45 call sites are migrated.
plugins/filter/container_volumes.py is deleted by theirs and relocated to
plugins/lookup/container_volumes.py, where it carries the same tmpfs defect our
f85996b43 had just fixed: a hard-coded {"size": ...} that silently drops a per
mount `mode`. The fix is re-applied in the relocated file, as that commit's merge
note anticipated.
compose.yml keeps only INFINITO_PLAYWRIGHT_REPORTS_BASE_DIR. Theirs also forwards
INFINITO_PLAYWRIGHT_STAGE_BASE_DIR; that line is deliberately dropped, per the
analysis recorded in 45804e9ad -- STAGE has a drift-proof role-vars fallback,
while forwarding it promotes a setdefault-pinnable .env value above that SPOT.
tasks/stages/01_constructor.yml takes their include, ./tasks/utils/setup/group.yml,
because their refactor removed tasks/groups/*-roles.yml, leaving our include
target nonexistent. Our task name is kept, since svc-virt is in the loop.
Makefile swarm-zombie is a union: their validate plus teardown, then our
lab_subnet cleanup, which their side dropped while leaving lab_subnet.sh as the
only orphaned reference in the file.
01_bootstrap.sh keeps our deterministic --ip pinning, which distinguishes an
already-attached node from a real failure, over their `|| true`.
Their language-sorted layout orphaned references our side still held: zammad's
two scripts/apply/*.rb, jenkins' java-with-ca-trust.sh, and the notify topics
their handlers renamed to hyphenated form. All corrected.
roles/sys-svc-compose/files/shell/swarm/stack_ready.sh gains our latched-update
block: paused, rollback_paused, rollback_started and rollback_completed exit 2
rather than polling a state that cannot resolve without another deploy.
config_from_run is unioned to (title, logged, jobs); the production caller in
trigger/__main__.py now passes jobs, without which our suite suppression would
have been inert.
READMEs are regenerated rather than hand-merged: 15 LiteLLM rows and 91 Tor rows
now derive from the merged metadata, which is itself evidence the meta merge is
coherent.
Verified: 5161 unit tests pass. The single failure, test_query.py TestSortSpec,
also fails on fork/feature/svc-net-tor at b51d2f9919, measured in a baseline
worktree -- so this merge introduces no new unit-test failure. Two modules are
excluded from that run because `baudolo`, a dependency their branch introduces,
is not installed on this host.
Not verified: no deploy has run against the merged tree, and make quality has not
been run end to end here.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
must_exist accepted any existing path, so a directory that dockerd auto-created where a bind source was missing passed the guard. The wrapper was then mounted over the container path and set as --entrypoint, and every image that has an ENTRYPOINT died with rc 126 'is a directory: permission denied' — seven identical retries, because the directory persists. Requiring a regular file routes that state into the existing 'CA injection disabled' fallback, which the same suite already exercises green when the wrapper is simply absent.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every script a role ships now lives under `files/<language>/`, and the unit
suite is split the same way: `tests/unit/python/` beside new sibling trees for
the other languages. `tests/lint/filesystem/test_role_files_layout.py` keeps it
that way; directories whose shape is dictated from outside carry a `.nocheck`
naming the constraint (Docker build contexts, WordPress mu-plugins, the Joomla
extension, Odoo addons, Playwright suites, the `files/test/test.sh` harness
that `test-e2e-cli` resolves generically).
This cannot be applied in halves. Moving a file and repointing what names it
are the same change: 14 references resolve only at deploy time, so the tree is
green either before or after, never between.
Those 14 were found by static resolution over every reference class, not by the
test suite -- a full `make quality-high` was green while all of them were
broken:
* 7 `src:` values built from a role variable holding a bare filename
(svc-bkp-remote-2-local, decidim, espocrm, friendica, joomla, pixelfed x2).
The variable doubles as the destination name inside the container, so the
language segment belongs at the call site, not in the variable.
* `sys-service/tasks/04_base.yml` resolved every consumer's systemd script
through `first_found(..., errors='strict')` over `files/script.{sh,py}`.
18 roles ship exactly that and have no template fallback: the play would
have died on each.
* `sys-front-inj-javascript` fell back to `<role>/files/javascript.js`; the
interface now reads `files/javascript/javascript.js`.
* `sys-front-inj-logout/templates/head_sub.j2` built the cache-busting mtime
path through `path_absolute(... 'files', ...)`.
* 4 `ansible.builtin.script:` invocations naming a bare command
(pull.py, prune.sh, set_postgres_superuser_password.sh, ganesha-serve.sh).
* `dev-python/files/shell/install.sh` pointed at a sibling role's script.
Depth-dependent paths inside the moved trees were bumped accordingly: 59
`parents[N]` computations, 12 relative climbs, and the per-role recover CLI,
whose path `cli/administration/recover/recoverers.py` still assembled from the
pre-move location.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
