Addresses three CodeRabbit findings on PR #1265.
1. LAM hardening invisible to the probe (thread 4413764929): the runtime
check probes PHP with `php -r`, the CLI SAPI. web-app-lam mounted the
hardening ini only into /etc/php/8.4/apache2/conf.d, which the Debian
CLI does not scan, so the probe read unhardened CLI settings and the
check failed for LAM. The php_hardening entry now also mounts the ini
into /etc/php/8.4/cli/conf.d. No other role mounts it into a Debian
per-SAPI directory; the rest use the shared scan dir of the official
image or of Alpine. The TODO entry now names the remaining gap: the
probe reads only the CLI copy.
2. Explanatory comment above HEALTH_RUNTIME_FLOORS (thread 4413764949):
removed. The README already states the end-of-life rule and now also
defines a floor as the lowest patched release of each line.
3. `apk --no-cache add` missed (thread 4415018662): PACKAGE_INSTALL_PATTERN
required `add` right after `apk`, so `RUN apk --no-cache add php7`
passed as an image without PHP. The apk branch now accepts options
before `add`, like the apt branch. New tests cover the option form and
guard that `apk` in a path or a comment is no package install.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Three CodeRabbit findings on PR #1265:
- script.py listed the running containers with `container ps` and
`container inspect` without a timeout, so a hung engine kept the daily
unit active forever and blocked the next timer run. Both calls now use
EXEC_TIMEOUT_SECONDS; main turns a timeout or engine error into a
one-line failure and exit 1 instead of a traceback.
- runtime.py's PHP_PACKAGE_PATTERN requires a digit after "php", so an
image installing an unversioned interpreter package (`php-cli`) and
setting no PHP_VERSION was reported as PHP-free. The interpreter
packages php, php-cli, php-fpm, php-cgi, php-phpdbg and
libapache2-mod-php are now recorded as PhpRuntime.packages and reported
with an unknown patch level; extension and library packages
(php-mysql, php-pear, php-common, phpmyadmin) and paths do not count.
- php_finding returned early for a patched PHP_VERSION, so an end-of-life
package line installed next to it (e.g. php7) was never reported. The
version only returns a finding now; otherwise the lines are judged too.
Every finding has unit tests that fail on the previous code.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Two CI failures on PR #1265 (run 40834164415):
- Chunk 0 / SuiteCRM#2 (swarm): the post-load flush included
sys-ctl-hlth-runtime on every play host, the NFS server too, which never
gets sys-svc-container and therefore has no `container` CLI. The unit
died with FileNotFoundError: 'container'. The role entry now runs only
where a container runtime exists: in compose mode, or on swarm nodes
('svc-swarm-node' in group_names), the same role-entry host scoping that
sys-ctl-hlth-csp and sys-ctl-hlth-webserver use with IS_STACK_HOST.
Workers stay covered because PHP replicas can run there.
- Workspace (ubuntu): tests/external/repository/test_urls_reachable.py
probed the literal "https://hub.docker.com/v2" from
utils/docker/image/runtime.py:20 and got HTTP 404. The constant is gone;
the endpoint is written as one f-string at the call site, the way
utils/update/docker.py:104 and utils/update/source.py do, which the URL
test skips as a template. With the scheme now visible in the literal,
ruff reported the noqa: S310 on urllib.request.Request as unused.
Both test files that imported unittest twice (CodeQL) now use one
`from unittest import ...` form.
The run_once queue leak that spreads the stack host's post-load queue to
every host is fixed separately on the updater line (70ed8a2e98); when it
meets main, sys-ctl-hlth-runtime moves into sys-svc-container's queue.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Answering "which deployed containers run a vulnerable PHP?" after
CVE-2026-109618/109619 took a manual inventory of registry metadata, and
nothing noticed a hardening ini that an image never loads.
- New role sys-ctl-hlth-runtime, routed through sys-svc-compose next to
sys-ctl-hlth-container. A daily timer runs `php` in every running
container and fails, alerting through sys-ctl-alm-compose, when PHP is
below the floor of its line or on a line older than every floor (7.4,
8.1). Every service that mounts php_hardening in meta/volumes.yml fails
while expose_php or session.upload_progress.enabled is on; other PHP
containers of such a stack are listed as info. Compose and swarm
containers are matched by their project/stack and service labels. The
check is not run at deploy time, so a finding never fails a deploy.
- The floors (8.2.35, 8.3.36, 8.4.27, 8.5.12) live once, in
HEALTH_RUNTIME_FLOORS of the role's vars/main.yml.
- `make security-runtimes` lists the PHP of every pinned role image
without pulling: Docker Hub tags through the Hub API, which no pull rate
limit meters, other registries through the OCI config blob (new
registry.config_blob, which the healthcheck probe now reuses).
PHP_VERSION, distro PHP packages and a PHP_BASE build argument count. It
exits 1 on an image below its floor or one the registry did not answer.
Today it flags nfrastack/fusiondirectory:1.5 (line 7.4) and
shopware/docker-base:8.3 (8.3.25).
The role TODO.md keeps the follow-ups: further runtimes, the split web
SAPI of web-app-lam, a scheduled CI run of the report and the CVE scan.
Refs #1127
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The scheduled mirror job has failed every day since 2026-09-25 on the same two
images: quay.io/minio/minio and quay.io/minio/mc answer `unauthorized` to
anonymous pulls ("2/197 failed", run 38352569503). Both belong to
web-app-minio, an eol role outside the lifecycle envelope that default.env
tests, so no CI deploy pulls them.
iter_role_images now skips every role that is_maintained() reports as
unmaintained, the check the version updaters already apply to the same list.
On the tree the discovery yields 187 images instead of 192; the five it drops
belong to the four eol roles web-app-confluence, web-app-jira, web-app-minio
and web-app-phpldapadmin. The mirror sync, resolver and wait read this list,
and their descriptions and mirror.md now say so.
The roles directory is resolved first: is_maintained reads a relative path as
a role name and would keep every role.
475ab9b386 on feature/store-i18n takes the same approach with a lifecycle
reader of its own; main already has is_maintained for it.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
The GHCR mirror covered role images only. The images the runner pulls on
its own went straight to Docker Hub: coredns from compose.yml, the cache
stack in compose/cache.override.yml and alpine:latest, the init image of
scripts/docker/cache/package-frontend/certs.sh. With the anonymous
githubactions quota spent, the guide step of Postmarks#0 (main run
37954203718) and Shopware#2 (#872) died on "toomanyrequests" while pulling
alpine:latest, after coredns had hit the same limit.
iter_runner_images (utils/docker/image/discovery.py) finds those images,
and the mirror sync and the fork-PR wait now cover them next to the role
images. On a real GitHub runner compose_file_args layers a generated
build/compose/runner-mirror.override.yml that points only the services of
the layered compose files at their mirror, and the certificate generator
gets the mirrored init image. Local runs and act are unchanged, and the
coredns literal stays in compose.yml for Dependabot. test-dns now waits for
images-mirror-missing like the deploy and workspace jobs, since it brings up
the stack too.
Verified with docker compose config on a simulated GitHub runner: the
service set is unchanged and only the runner images point at
ghcr.io/infinito-nexus/core/mirror/docker.io/..., and an existing role mirror
under that scheme pulls anonymously.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A pin outside semver was given its own 'docker-ref' marker on the theory
that every class needs one. That holds for a digest and not for a tag. A
digest pin declares something about the registry, that bytes move under a
standing tag, which no other choice says. A tag outside semver declares
that upstream publishes nothing orderable, which is exactly what
'docker-version' already means and what minio and the act runner already
carry on their version lines.
Keeping both would have put '# nocheck: docker-version, docker-ref' on
every such line: two keys, two checks, one sentence said twice. 'docker-ref'
is dropped instead. It lived in this one mapping, was never entered in the
suppression catalog, and no check ever read it.
The semver entry goes with it, because a semver pin never needs a
declaration. The test that asserted three distinct rules is replaced by two
that pin the separation where it is real: a digest marker differs from a
tag marker, and a semver pin has no marker at all.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A digest pin was emitted as `name:sha256:...`, which no registry accepts:
image_source('ghcr.io/.../csp-checker', 'sha256:d559...') returned
'ghcr.io/.../csp-checker:3.0.0'. The separator depends on the pin class, so
the class has to be decided before the reference is built.
utils/docker/image/pin.py names the three classes in preference order
(semver, then digest, then a general ref that no registry can order) and
maps each to the nocheck marker that declares it. Both discovery entry
points now end in pull_reference(), so '@' versus ':' is decided in one
place instead of at every call site.
The semver pin lint points a digest at 'docker-digest' instead of
'docker-version', so the two fallbacks stay distinguishable in the
suppression catalog. It stays warn-only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
start_interval is the cadence while start_period runs, and only while it runs.
Once the window closes the probe falls back to interval, so a service that
becomes ready just after it pays a whole tick anyway and the cadence never
reached it. Six of the seven flavors were shaped that way, Curl worst: a thirty
second window against a one minute interval.
Measured across 151 converge waits in four job logs, 63 sit exactly on a tick
boundary, 36 in the thirty second step and 27 in the sixty second one.
Attributed by application, web-app-seaweedfs is the largest at 1501s over
sixteen waits with a median of 57.7s: it probes with curl and fell straight into
the gap. web-app-zammad probes with connect, whose window is fifteen minutes,
and its 154s median is real boot time with nothing to recover.
So the window moves to ten minutes and the steady state interval to two, and the
flavor overrides that sat below them go: Curl, Wget and Connect lose interval
1m, Http its twenty second window, Nc its ten. Connect keeps fifteen minutes.
Neither number costs anything while a container works. start_period delays
nothing, it only stops failures counting, and interval no longer touches
convergence at all since that hangs on the window alone; it is steady state
load, which drops from 120 probes an hour per service to 30.
Forty six role files follow. Twenty seven declarations go that repeat the new
default or undercut it, an undercut window being the defect itself. Eight
windows longer than ten minutes keep their value and now say why. The literal
blocks and the three Dockerfiles this repository builds spell every timing out,
because outside the lookup an omission takes docker's value and not the
platform's.
A fourth lint rule keeps the shape: start_period must outlast one interval,
checked against the effective values, so raising interval without the window
cannot quietly bring the cliff back.
Two faults in the lint surfaced while building it and are fixed here. It
compared timings as strings, so 600s and 10m read as different values, and it
matched the healthcheck key exactly, so a line carrying a trailing nocheck was
skipped whole, which is how the two swarm-init partials went unexamined.
One regression, bounded. A container that starts and never turns healthy is
neither rejected nor errored, so stack_ready.sh only calls it fatal after three
killed tasks, and each cycle now spans the longer window. That reaches twenty
minutes rather than six, capped there by swarm_stack_converge_retries at 600
with a two second delay, and it lands only on runs that were failing anyway.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Probe carried the five platform timings as literals on its class body. They are
the answer to a question asked outside the class as well, by anything that has
to decide whether a declared value adds something or merely repeats what the
service would have inherited.
Name them, and let Probe read the names. TIMING_DEFAULTS collects the five so a
caller can walk them without restating the list and drifting from it.
No behaviour changes: every value is the one that stood there, and the class
attributes still resolve to the same strings and the same integer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The emitted block carried test, interval, timeout, retries and start_period,
but never start_interval. Docker therefore fires the probe only every interval
for the whole start_period, so a service that answers after three seconds is
still reported healthy at the next tick: 30s for the base flavor, a full minute
for the curl ones (probes.py:119,131,215). The swarm converge wait sits on that
signal, so every stack pays the quantisation of its slowest service.
Emit start_interval, defaulting to 5s and overridable per service like every
other timing. This changes when the same probe runs, never what it tests:
test, timeout, retries and start_period are untouched, so a service that never
becomes healthy still fails after exactly the same grace and the same attempts.
Supported by the engine that runs it: the failing job reports Docker 28.0.4 and
Compose v2.38.2 (runtime.txt), and start_interval has existed since Docker 25.0.
The saving is reasoned, not measured. Expected padding is about half an
interval per service, so roughly 500-1000s across 17 stacks and two passes,
but only for services that are genuinely ready well before their interval. How
many those are is what a run has to answer.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CodeQL reports cmd and sort_spec as possibly uninitialised after the except branch that calls parser.error, because it does not model the argparse error method as NoReturn. It is: usage is printed and the process exits with status 2. An explicit return 2 on that branch states the exit status the call already produces, so the analyser and a reader see the same path.
The retry loop in the registry client counted two attempts in a variable the body never read; the underscore prefix was the convention for that, but CodeQL only honours the bare name.
No path changes behaviour; the three alerts stop pointing at correct code so the scanner stays readable for the ones that are not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
3529 invalid comments in 528 files, grandfathered in while the linter
only saw the working-tree diff. Removal ran through the linter's own
collectors, so detection and deletion cannot drift: a comment whose
opener line is blank before it and whose closer line is blank after it
loses all its lines, anything else loses only the comment region, and a
line left holding a bare delimiter goes with it.
Five files keep their comments behind a nocheck marker instead. Four are
upstream configs whose inline docs are upstream's own (mautrix, systemd
journald, nextcloud-oidc-login, MediaWiki). In littlejs the
commented-out entries ARE the deactivation mechanism for an example
tile, so deleting them would delete configuration.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A container without a healthcheck counts as healthy the moment its process starts. That is how a service which came back empty passes a restore drill: the drill asks the host's health service, and the host says yes.
The image side is read from the registry rather than by pulling: the manifest names the config blob, the blob carries the HEALTHCHECK. The answer is three-valued on purpose. A mapping means the image ships a probe, an empty mapping means it definitely ships none, and None means the question could not be answered. Only the empty mapping is evidence; reading None as absent turns one network hiccup into a hundred false findings.
Probes address the GHCR mirror first and fall back upstream only for images the mirror does not carry, and a second test requires mirror plus cache to answer more often than upstream, so the fallback cannot quietly become the rule and spend the anonymous rate limit again.
Detection spans the four places a probe can come from: the services.yml declaration, a literal block in a template, a partial pulled in by include or lookup('template'), and a HEALTHCHECK in a Dockerfile the role builds its own image from. Without the last three the sweep reported nextcloud, every redis sidecar and openldap as gaps they are not.
The opposite direction gets its own lint: a declared healthcheck that no template renders is dead configuration. Fixing it surfaced that the shared resolver silently dropped every service_name call site, because it accepted only quoted literals and a role var resolves to a bare key. And the literal-protocol lint could never be suppressed on a wrapped value, since autoformat moves the marker to the value's last line while the offence is reported on its first.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Moodle renders the same msmtprc as WordPress and mounts it into two services,
but nothing ever proved a message left the container: its healthcheck opens the
FPM port and nothing else. Moodle 0 of run 31658241728 passed green while mail
could not have worked -- the defect 2da285067 fixed for WordPress was invisible
here, and would have stayed invisible until a user reset a password.
WordPress proves delivery with the msmtp_curl flavor: the probe sends one mail
and writes a marker, and a task after convergence waits for that marker. Moodle
cannot use it, because curl cannot judge a port that answers FastCGI. Rather
than add a second msmtp_* class per liveness shape, the declaration now composes:
healthcheck:
flavor:
- msmtp
- connect
port_key: fpm
msmtp stops being a flavor and becomes a prefix -- shell that runs on the way
past and never decides liveness. connect joins the probes as the one that only
asks whether the port accepts a TCP connection. A declaration names exactly one
probe and any number of prefixes, as a string or a list; msmtp_curl stays as an
alias for [msmtp, curl], and the lint that validates declarations now shares
resolve_flavors with the lookup instead of keeping its own idea of a flavor.
The module became five responsibilities in one file, so it is a package:
probes.py answers which probes exist, prefixes.py what runs before them,
compose.py how a declaration becomes a rendered block. The curl helpers sit with
the probes and mail_branch with the prefix -- each next to its only caller,
rather than together in a module header.
Two defects in this work were found by measuring, not reading:
- the timing merge took the maximum across prefixes only, which cut Moodle's
start period from 15m to 120s and would have marked a heavy PHP app dead
while it was still installing. A prefix may now raise a timing and never
lower one, pinned by test.
- composing moved msmtp_curl's interval from 30s to 1m, because the composed
class derives from Curl where MsmtpCurl derived from Probe. That is the
interval every other curl probe uses, but it was not intended; it now has
its own test rather than riding along unseen.
Regression measured across the whole repo rather than argued: every declared
healthcheck was rendered with the pre-refactor code and with this one. 58 render
byte-identically, 1 is Moodle's intentionally changed declaration, and 2 differ
-- friendica and wordpress, in that interval alone, with the probe command
itself unchanged.
Coverage was checked in both directions. No test was removed: the four deleted
lines are an import, the registry literal, one class name in a url test, and the
helper, which now goes through compose() instead of reaching into PROBES -- a
tightening. New code had two gaps, both closed: connect was the only probe whose
emitted command no test pinned, and compose() raised KeyError for an unknown
name unobserved.
Not verified: no deploy has run any of these probes. The composed class is built
dynamically, so what it emits is proven for all 59 services while docker
accepting it is not.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every probe in this module targets 127.0.0.1 inside the container, so a request
a proxy picks up is wrong by definition. The Wget probe has said so since it was
written -- `wget --spider --proxy=off` -- while the four curl invocations said
nothing, and a container whose environment carries http_proxy would have sent
its own liveness check through it.
That environment is not hypothetical: 2d6a5fd18 puts http_proxy and https_proxy
into EspoCRM's env_file so the OIDC back-channel can reach an onion issuer. It
also sets no_proxy=localhost,127.0.0.1, so that role's probe is covered by hand.
The next role to route a container through a proxy and forget that line would
have had a healthy web app reported dead.
The flags now come from one place. CURL_NO_PROXY plus curl_argv() build the exec
form, and curl_shell() is the same argv run through shlex.quote. The two
spellings differ in exactly one character -- the exec form needs `*` bare
because docker runs no shell, the CMD-SHELL form needs it quoted or the shell
globs it -- and that difference was previously written out by hand at three
sites plus a fourth in a different shape. Deriving one from the other is what
keeps them from drifting; the Host-header branch moved along for the same
reason, since it too existed twice in two spellings.
The http fallback keeps its wget half untouched on purpose. That flavor exists
for images where it is unknown which tool ships, and busybox wget does not
accept --proxy=off; adding it there would have made the first branch fail on an
unknown option for exactly the images the fallback is for.
Verified by execution at each step:
- adding the flag broke five tests, which is every call site plus the hostname
variant -- the suite pins the argv and the shell string literally, so that
was the completeness check
- the refactor then passed those same 31 tests with no further edit, and the
four rendered probes were diffed against the pre-refactor output character
for character
- one of those tests executes the CMD-SHELL probe against stubbed msmtp and
curl, so the quoting survives a real shell rather than just a string compare
- the option is accepted where it will run: joomla:6.1.2-php8.3-apache, one of
the images configured for this flavor, ships curl 8.14.1 and answers
`curl -f --noproxy "*"` with exit 7 (refused, nothing listening) rather than
exit 2 (unknown option)
Not verified: the other images behind this flavor. --noproxy has existed since
curl 7.19 in 2008, but only one image was measured, and no probe built this way
has yet reported healthy in a live container.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
75 commits: the socialhome role reimplemented against the project it actually
deploys, native-login specs for funkwhale and shopware, a window.open IdP
handoff and a settings-link fallback in the shared persona helpers, a swarm
converge budget that gives up on a terminal task, and a long tail of role
fixes.
Eleven conflicts. The shared persona helpers (keycloak.js, logout.js) take the
incoming structure — they gained real behaviour this branch never had — with
this branch's resolveTimeout convention re-applied over it and
confirmKeycloakLogoutIfPrompted re-attached at every return point; without it
the Keycloak SSO session outlives the logout. playwright.config.js keeps this
branch's version: both sides set actionTimeout, but ours routes through an env
var and 15s is too short behind an onion. socialhome's spec takes the incoming
content — 2dbd6eb10 found the role described the wrong upstream project
entirely — and keeps gotoOnion and expectHstsWhenTls, since demanding HSTS over
a plaintext onion is a false failure. Both READMEs were regenerated rather than
merged by hand.
Three collisions arrived with no conflict marker at all, because each side
changed a different place in a different file:
- group_vars/all/18_swarm.yml declared swarm_stack_converge_retries twice.
Same value, so nothing breaks today; editing the documented one would have
silently changed nothing.
- svc-net-tor and web-app-socialhome both claimed 192.168.74.0/24. Socialhome,
the newer role, moves to 192.168.75.0/24.
- web-app-socialhome/meta/schema.yml arrived as a new file, so ef5ea7cb5's
rename never saw it. Its credential would not have resolved and the role
would have deployed with an unusable administrator account.
The incoming specs also needed this branch's conventions: raw timeouts wrapped,
four page.goto calls routed through gotoOnion, a request.get given an
onion-aware timeout, and 21 roles marked playwright-service-flag over the tor
block this branch added to them.
make quality green on the merge result.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The msmtp_curl probe joined its mail branch to the curl probe with '&&', so a
relay that refuses the startup message short-circuited the whole command: curl
never ran, the container stayed unhealthy forever, and the swarm converge gate
burned all 150 retries before aborting the deploy. Friendica 0 and Fediwall 0
of run 31340965955 died that way, reporting only 'not converged:
friendica_friendica' with the real cause buried in a rescue artifact.
That contradicted the probe's own docstring, which promises an unrelated SMTP
outage cannot flap the container into unhealthy -- it only delivered that for
email_enabled=False. sys-ctl-hlth-msmtp/files/script.sh:23 has always tolerated
exactly these exit codes; the container probe was the outlier.
Liveness is now curl's verdict alone. The '&&' before touch stays, so a refused
mail leaves no marker and the next probe retries it rather than turning the
smoke test one-shot.
Detection does not disappear with the coupling: web-app-friendica now asserts
the delivery marker after convergence and fails with the relay's reason instead
of a timeout three layers away. sys-ctl-hlth-msmtp is a systemd timer and the
Mailu Playwright roundtrip submits through Roundcube, so neither covers a
container relaying over port 25 -- this assertion is what does.
The marker path moves into utils.docker.healthcheck.MAIL_MARKER and reaches
Ansible through lookup('container_mail_marker'), so the probe and the assertion
cannot drift apart.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
IncompleteRead subclasses HTTPException, not OSError, so it slipped past the transport-error handlers the fetchers already had and propagated out of the thread pool, turning a warn-only image-version test into a hard failure.
The affected image is now skipped rather than checked, which the run does not surface: the test's unchecked list keys on registry support, not on fetch success.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Seven include templates under sys-svc-container decided how a container was
probed, and they read their inputs from template-local `{% set %}` scope.
That scope leaks: a value set for one service silently reached every service
below it, which is how taiga's front and gateway ended up probing a port
declared for a different service, and how bluesky's frontend probed a PDS
endpoint.
Probes now live in utils/docker/healthcheck.py, one class per flavor behind a
PROBES registry, and each service declares its own:
healthcheck:
flavor: curl
path: health/ready
start_period: 20m
`port_key` picks another entry under ports.internal for the services that
expose their probe elsewhere (keycloak on management, nextcloud talk on
signaling). Without a flavor the entry must carry an explicit test argv, so a
missing declaration fails loudly instead of rendering a probe against nothing.
Rendering HEAD's templates against the new output confirms 49 of 56 call
sites are byte-identical; the remaining seven are covered by an Ansible run.
Three probes change on purpose: erpnext's backend gains a check it never had,
bluesky's frontend drops the leaked PDS path, and msmtp_curl moves from a
hardcoded localhost:80 to the service's own port on 127.0.0.1 like every
other flavor.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
In a linked worktree `.git` is a file naming that worktree's gitdir, not a
directory, so reading `PROJECT_ROOT/.git/config` raised NotADirectoryError.
`_git_config` catches OSError and returns an empty mapping, which left
`_owner_and_repository` without a remote and `mirror_image` returning None
for every image.
The consequence was silent: every probe addressed the upstream registry
directly, so all image work done from a worktree bypassed the GHCR mirror
and hit Docker Hub instead. The rate limits and handshake timeouts that
showed up as flaky `test-external` runs came from there.
`_config_path` follows the indirection: a `.git` directory keeps its own
config, a `.git` file is read for its gitdir and that dir's `commondir`
points at the shared config. Reads go through utils.cache.files, which the
uncached-read lint requires under utils/.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The min_storage gate measures every declared pin on each run, and every measurement was an anonymous manifest request to Docker Hub, which meters those per IP. A full run exhausted the pull budget, so the following local deploy could no longer pull at all and the gate itself stopped measuring: 96 of 165 images answered indeterminately and only warned.
Read the GHCR mirror first and fall back to the upstream registry only for images it does not carry. The copy is byte-identical, so the measured size is unchanged. Persist measured sizes under the temporary directory; store positive results only, because a cached indeterminate answer would be remembered as fact and the gate would stop measuring for good.
Lift the mirror path formula out of the provider into utils so the writer and the reader share one definition, resolving the namespace from GITHUB_REPOSITORY, then remote.pushDefault, then origin. pushDefault has to win: origin may point at an upstream that hosts no mirror. Parse .git/config by hand because configparser rejects the duplicate keys git writes and the test container ships no git binary.
Measured over all 165 declared pins: no requests to Docker Hub, 277 to the mirror, 9 to registries it does not carry, and 164 of 165 images measured.
Leave manifest_exists on the upstream registry. It answers whether the declared pin is still pullable from where the deploy pulls it, which a mirrored copy cannot attest; it takes the cache only.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
An over-declared min_storage makes a host look too small: it feeds the storage guard, the runner packer and the sufficient-storage filter, so a deploy that would have fit prunes a cache it did not need to lose. Nothing checked the declared number against reality.
The new external test measures what a pull of the service's own image:version actually transfers - config blob plus every layer blob, resolved for linux/amd64 - and fails when min_storage exceeds it by more than 1GB. A service whose data volume genuinely dominates keeps its value by carrying '# nocheck: min-storage-headroom' together with a 'Reason:', the same two-marker shape meta/volumes.yml already uses for nfs: false.
Sizes come from the repo's own registry client rather than 'docker buildx imagetools': make test-external runs inside the compose container, which has no docker socket. manifest_transfer_size reuses the existing bearer-challenge flow, keeps the tri-state contract of manifest_exists (an indeterminate registry warns instead of failing), and resolves a multi-platform index to its amd64 manifest - the case a raw jq over '.layers' would miss entirely. Distinct image refs are measured concurrently through resolve_max_fetch_workers().
A min_storage that parses out of the YAML but cannot be located in the file text fails the test, so a declaration can never silently lose its marker resolution.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The separate CI job is gone; hadolint now runs as lint-dockerfile inside the parallel make lint fan-out through the standard wrapper, with the binary installed by the lint toolchain installer like every other tool. The SARIF gate script had no consumer left and is removed. Scope stays the root Dockerfile at the warning threshold, matching the old job.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Move files that share a delimiter-bounded name prefix within a folder (threshold 2) into a subfolder named after the prefix, dropping the prefix from the filename. Covers plugins/filter (docker, generate, has, merge, get/all), env handler subpackages, utils/roles/entity, utils/tests/swarm/write, role SQL/scripts/templates, keycloak/mailu task-includes, shared tasks/utils/load, and CI scripts. Update every import, query_file/script/include_tasks/src/lookup target, compose bind mount, and python -m path to the new locations.
Extend the prefix-grouped-files lint to threshold 2 with dir-level .nocheck and file-head nocheck opt-outs, and skip groups whose shortest member's stem equals the shared prefix. Keep flat only where a loader forbids nesting: WordPress mu-plugins and the docs/requirements namespace. Ansible filter discovery is recursive, so nested filters stay Jinja-resolvable.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The nfs SPOT loaders pulled PyYAML into the .env generation, which bootstraps fresh hosts BEFORE python dependencies exist: the import crashed cli.meta.env, load.sh produced an empty environment and make install-system-python died on the required-var guard (environment/manjaro and pkgmgr/centos CI jobs). The loaders now line-parse the services.yml SPOTs via the stdlib-backed cached reader and the mapping path constant. The client's protocol version field becomes nfs_version, since services.yml reserves version for image tags (schema lint), and the image discovery casts versions defensively.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Add registry.gitlab.com to the mirrorable registries so discovery
yields the GitLab CNG images to the version updater and mirror sync.
Seed the generic v2 tag scan with the OCI last cursor for v-prefixed
pins and raise the page size to 1000: CNG repos carry tens of
thousands of commit-sha tags that sort before the release tags, so an
unseeded scan exhausted the page cap without reaching any semver tag.
Pin every gitlab service version in meta/services.yml and drop the
GITLAB_VERSION template override so services.yml is the single source
for per-service versions.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Add a generic Docker Registry v2 client (WWW-Authenticate bearer-token
challenge) so tag listing and reachability work for any registry (Docker
Hub, GHCR, MCR, Quay, GitLab/opencode), and route the version updater
through it so quay.io and opencode.de images auto-update like the rest.
Add an external test that hard-fails when a pinned image:version tag is
unreachable (404) and warns on incomplete image/version pairs.
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
OpenTalk's images (web-frontend, controller, recorder) live on
registry.opencode.de, which was absent from both _MIRRORABLE_REGISTRIES and
_ALL_REGISTRY_PREFIXES in utils/docker/image/discovery.py. So is_mirrorable_image
returned False, iter_role_images skipped them, the GHCR mirror never copied
them, and the deploy pulled direct from registry.opencode.de -> TLS handshake
timeout (deploy failed).
Add the registry to both sets so the single iter_role_images gate enrolls the
3 images: the mirror-sync copies them to GHCR and the resolver rewrites the
deploy to pull from the GHCR mirror. The prefix entry strips the host so the
mirror destination is well-formed (no doubled registry). Add a regression test.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- Add MCR_REGISTRY constant, is_mcr / mcr_repo / fetch_mcr_tags helpers
mirroring the Docker Hub and GHCR pattern. Tags are fetched via the
Docker Registry V2 API with RFC 5988 Link-header pagination (anonymous
access; MCR is a public read mirror). Wire MCR into both the external
test_image_versions warning surface and the find_outdated_updates
orchestrator in utils.update.docker.
- Bump mcr.microsoft.com/playwright in test-e2e-playwright from
v1.59.1-noble to v1.60.0-noble — the update was surfaced by the new
MCR detection path.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Folds the parallel Docker-image and git-ref version-update plumbing
into a single subsystem so the auto-update workflow, the external
freshness tests, and the unit tests all share one module layout.
utils/
update/
base.py # semver primitives shared by both backends
docker.py # OCI registry lookups + version: line rewriter
repository.py # git ls-remote walker + ref: line rewriter
cli/contributing/update/
docker/ # python -m cli.contributing.update.docker
repository/ # python -m cli.contributing.update.repository
scripts/github/update/
image_versions.sh
repository_refs.sh
Tests mirror the same shape under tests/{unit,external}/update/{docker,repository}/.
Also widens the Docker bump scope to all roles (previously gated on
`web-*`) so non-web roles like sys-svc-container get bumped too;
pins Bookwyrm's python image at 3.11-bookworm via
`# nocheck: docker-version` with a TODO until upstream supports
Python >= 3.12.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Modules imported by tests run inside the same pytest process as the
tests themselves. Raw ``path.read_text(...)`` calls in those modules
bypass the cache the tests just populated. ``roles/`` stays excluded
(deployed scripts have no ``utils/`` on PYTHONPATH); ``utils/cache/``
stays excluded (cache implementation itself).
Five conversions to ``utils.cache.files.read_text``:
- ``utils/handler/yaml.py`` (YAML loader, read-only)
- ``utils/annotations/summarize.py`` (CI log scanner, with
``UnicodeDecodeError`` fallback replacing ``errors="replace"``)
- ``cli/meta/roles/applications/resolution/combined/__main__.py``
(single-shot CLI JSON input)
- ``cli/core/help.py`` (read_folder_title + read_folder_description;
same README is read many times across CLI help iteration)
Eight ``# nocheck: cache-read`` markers, each annotated inline:
- ``utils/claude/settings_sort.py`` (read then rewrite sorted)
- ``utils/docker/version_updater.py`` (read then rewrite config)
- ``cli/administration/deploy/development/network.py`` (tests mock
``Path.read_text`` directly; caching would bypass the mock)
- ``cli/administration/deploy/development/log_utils.py`` (append flow)
- ``cli/contributing/changelog/archive/changelog_md.py``,
``__main__.py``, ``package_mirror.py`` (file mutated in same flow)
Two callers refactored to variable extraction so ruff's 88-char wrap
keeps the ``# nocheck`` marker on the ``read_text(...)`` AST span line
(``cli/.../network.py`` and ``package_mirror.py:105``).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Consolidates image declarations under a single schema: every role's
`<svc>.{image,version}` lives in `meta/services.yml` alongside compose
service entries. The `image` lookup now reads from disk per role,
`iter_role_images` collapses to a single source, and the mirror chain
(resolver + apply_mirror_overrides) drops the dead `images:` branch.
`test-e2e-playwright` resolves `TEST_E2E_PLAYWRIGHT_IMAGE` directly via
`lookup('image', 'playwright', 'ref')` in vars/main.yml; the runtime
set_fact + override-empty-string dance is gone.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Strip every `req <NNN>`, `req-<NNN>`, `Requirement <NNN>`,
`requirement-<NNN>`, and `docs/requirements/<NNN>-...md` mention from
non-Markdown source files. Comments anchored to requirement numbers
rot when requirements are renumbered or rewritten and conflate
present-state behaviour (which the comment should describe) with the
audit trail of how the code got there (which belongs in commit
messages, PR descriptions, and the requirement file itself).
* Document the rule in `docs/contributing/documentation.md` under
"Comments". Code MUST describe the *why* in present-state language;
contract enforcement lives in tests, not folklore.
* Clean 254 source files across `roles/`, `tests/`, `plugins/`,
`cli/`, `scripts/`, `utils/`, `group_vars/`, and the repo-root
`eslint.config.js`. Pure comment / docstring / log-message edits;
no behavioural change.
* Add `tests/lint/repository/test_no_req_references_in_code.py` to
enforce the rule going forward across `.js`, `.py`, `.yml`,
`.yaml`, `.j2`, and `.sh` files. The legacy
`test_no_stale_requirement_refs.py` is exempt as the only file
whose purpose is to detect these very tokens.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
TC (flake8-type-checking): typing-only imports moved into
``if TYPE_CHECKING:`` blocks across 8 source files via auto-fix
(117 hits). Reduces runtime import surface and rules out circular
imports for type-only deps. Project-wide ``from __future__ import
annotations`` (FA, already enabled) keeps annotations as strings,
so the moved imports never need to load at runtime.
N (pep8-naming): 32 hits handled.
Renames (semantic + lint):
- ``isSmallerThenMaximumBackupSize`` -> ``is_larger_than_maximum_backup_size``
(snake_case + fixes both the typo "Then -> Than" and the
semantic mismatch: the function returns ``current > maximum``)
- ``isDirectoryDeletable`` / ``deleteVersion`` /
``getAmountOfIteration`` / ``deleteIteration`` -> snake_case
- 8 test functions with ALL-CAPS env-var fragments
(``test_..._DOMAIN_PRIMARY_...`` etc.) -> snake_case
- Mock alias ``VH`` -> ``vh`` in test_host_vars.py
Exception class renames (``N818`` Error suffix):
- ``TimeLimitExceededException`` -> ``TimeLimitExceededError``
- ``AttemptException`` -> ``AttemptError``
- ``PathNotFound`` -> ``PathNotFoundError`` (propagated across
4 callers under tests/integration/lookups/config/)
- ``_ExitJson`` / ``_FailJson`` -> ``*_Error``
Refactor:
- ``METADATA = {...}`` local constant in two functions of
``plugins/filter/invokable_paths.py`` lifted to module-level
``_METADATA_KEYS``.
- ``from typing import TYPE_CHECKING`` in
``tasks/utils/migrate_meta_layout/__main__.py`` moved above
the ``sys.path.insert`` hack to satisfy E402 (collateral from
the TC auto-fix that landed it after the path mutation).
Per-file ignores in pyproject.toml for the legitimate cases the
rule cannot accept:
- ``pull-specific-host.py`` -> ``N999`` (Ansible-invoked role-level
script, hyphenated filename pinned by the surrounding stack).
- 4 matrix-deploy files -> ``N806`` for the ``include_R`` /
``raw_include_R`` round-index notation; the uppercase ``R``
here mirrors the formula in the docstring rather than naming
a constant.
- ``plugins/filter/value_generator.py`` -> ``N806`` for the
lazy-loaded class binding ``ValueGenerator =
_value_generator_cls()``, which retains the imported class
identifier per PEP 8.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Add tests/lint/repository/test_role_file_spot.py: AST-walks every .py
file under the production trees and tests/, flags any reference to a
role file (standalone literal, f-string segment, pathlib `/` chain,
`+` concat chain, os.path.join / Path.joinpath / posixpath.join /
ntpath.join call) whose value or joined trailing operands match a
ROLE_FILE_* constant in utils.roles.mapping. The flag set is
auto-discovered from the module so adding a new constant extends
coverage without further edits. ROLE_FILE_README is exempt as too
generic. Per-line `# nocheck: role-file-spot` (same-or-above) is the
documented opt-out, catalog row added in
docs/contributing/actions/testing/suppression.md.
Migrate 139 .py files (377 occurrences) to use the constants:
- 135 files via direct constant import (`from utils.roles.mapping
import ROLE_FILE_*`), with the symbol used in pathlib chains, in
os.path.join / Path.joinpath calls, in f-string interpolation, and
in glob patterns.
- 4 files via `# nocheck: role-file-spot` where the literal lives in
argparse help text or in an allowlist tuple meant to declare the
literal path.
Adjust utils/roles/dependency_resolver.py to use lazy-formatting
positional args in the logger.exception call (G003-clean) instead of
string concatenation against the constant.
Drop a redundant `if cond: return True; return False` guard in the
new test for `return cond` (SIM103-clean).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Hard-coded self-path references rot the moment the file moves: a header
comment ``# utils/foo/bar.py`` at the top of a module, a docstring
mentioning ``utils/foo/bar.py``, or a docs entry linking
``utils/foo/bar.py`` from inside ``utils/foo/bar.py`` itself all silently
desync from reality after the next ``git mv``.
The new lint walks every project file, computes its POSIX-style relative
path from PROJECT_ROOT, and reports any line whose text contains that
path as a substring. Files at the repo root (no ``/`` in their relative
path) and setuptools-generated ``*.egg-info/`` artefacts are exempt.
Per-line opt-out via ``# nocheck: self-path-reference`` on the offending
line, registered in the suppression catalog.
Existing offenders cleaned up:
- 72 redundant ``# <self-path>`` header lines removed across roles,
cli, plugins, scripts, tests, and utils.
- 4 prometheus templates that told consumers to "update
<self-template-path>" rewritten to drop the redundant self-reference.
- 8 docstring/comment self-references in cli, plugins, tests, utils,
and the dashboard oidc.js.j2 header rewritten to describe the layout
without naming the file's own path.
- 17 legitimate usage examples (shell scripts, test data lists, one
template self-include doc-block) marked with ``nocheck:
self-path-reference``.
- ``container.sh`` heredoc usage strings switched to ``${0}`` so the
marker stays out of the user-visible help output.
- ``healthcheck/http.yml.j2`` doc-block rewritten to prose because
``{# ... #}`` Jinja comments cannot host an inner ``{# nocheck #}``
(the inner ``#}`` closes the outer comment).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Tighten the utils namespace so each subpackage owns one concern:
- utils/applications/* moves under utils/roles/applications/* (helpers
that operate on the runtime applications dict and meta/services.yml).
- utils/service_registry.py and utils/database_service.py move under
utils/roles/applications/services/ as registry.py and database.py.
- utils/entity_name_utils.py becomes utils/roles/entity_name.py.
- utils/invokable.py, utils/valid_deploy_id.py, and the existing
utils/roles/resource_validation.py group under utils/roles/validation/
as invokable.py, deploy_id.py, and resources.py.
- utils/templating.py and utils/jinja_strict.py split into
utils/templating/ansible.py (Templar + TrustedAsTemplate, lookup-env
shortcut) and utils/templating/jinja.py (vanilla Jinja2 Environment,
multi-pass strict render).
Every caller in utils, cli, plugins, roles/<role>/{filter,lookup}_plugins,
tests, and the docs that link to the moved files is updated to the new
import paths. Test files mirror the new source layout
(tests/unit/utils/{roles,templating}/...).
New READMEs at utils/cleanup/, utils/roles/, utils/roles/applications/services/,
utils/roles/validation/, and utils/templating/ follow the
documentation.md and readme_md.md guidelines.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds ``S`` (flake8-bandit) to the strict ruleset. ``PTH`` was scoped
out: it surfaced 1000+ violations with no auto-fix support (most PTH
sub-rules are warning-only because each ``str`` ↔ ``Path`` conversion
is context-sensitive). Reserved for its own dedicated commit.
Config tuning
=============
Global ``ignore`` additions with rationale:
- ``S603`` (subprocess call without ``shell=False``) and ``S607`` (start
process with partial path) are over-eager for a project whose CLI
invokes ``make`` / ``git`` / ``docker`` / ``ansible-playbook`` from
PATH using list-form args. The list-form already prevents shell
injection (Python's ``subprocess`` does not parse argv[0] as a shell
command), and resolving every binary's absolute path in the source
would couple the code to the host filesystem layout. The genuinely
dangerous shell-injection patterns (``shell=True``) are still caught
by S602 / S605.
New ``[tool.ruff.lint.per-file-ignores]`` for ``tests/**``:
- ``S101`` — ``assert`` is the unittest idiom.
- ``S102`` — ``exec`` is used to load script-under-test by file path.
- ``S105`` / ``S106`` — hardcoded credentials in fixtures / dummy creds.
- ``S108`` — ``/tmp`` paths in tempdir fixtures.
- ``S110`` / ``S112`` — ``try``/``except``/``pass``/``continue`` in cleanup helpers.
Hand-fixes (38 across 23 files)
================================
S101 × 3
Type-narrowing asserts (``cli/deploy/development/proc.py`` —
``p.stdout is not None`` after ``stdout=PIPE``) and an upstream-Odoo
invariant assert in ``roles/web-app-odoo/.../res_users.py``.
S105 × 1
``cli/core/app.py`` argv parser comparing ``token == "--log"`` —
``token`` is a CLI argv element, not a credential. Inline noqa.
S106 × 1
``utils/cache/users.py`` Jinja template string ``"{{ 42 |
strong_password }}"`` rendered to a real password at runtime; not a
literal credential. Inline noqa.
S108 × 6
Container-internal paths bind-mounted from the host:
- ``/tmp/infinito/ca/root-ca.crt`` and ``/tmp/infinito/bin/with-ca-trust.sh``
in ``roles/sys-svc-compose-ca/files/compose_ca.py`` and
``roles/sys-svc-container/files/container.py`` (CA-injection
contract between role and wrapper).
- ``/tmp/cache_nginx_general/`` and ``/tmp/cache_nginx_image/`` in
``plugins/lookup/nginx.py`` (well-known nginx cache dirs).
S110 × 1
``roles/web-app-bigbluebutton/filter_plugins/compose_mods.py``: filter
intentionally swallows YAML parse errors and falls back to the
unmodified text. Inline noqa.
S112 × 5
Filter plugins / inventory builders iterating over role config files
with skip-on-malformed semantics. The pattern is intentional
throughout (CLI tools / Ansible plugins shouldn't crash on a single
bad role file). Per-line noqa with rationale.
S324 × 2
``utils/cache/base.py`` uses md5 as a fast non-cryptographic
fingerprint for cache keying; ``utils/manager/value_generator.py``
uses sha1 when the caller selects ``algorithm == "sha1"`` for legacy
app compatibility. Both gained ``usedforsecurity=False`` (Python 3.9+
hashlib API) which silences the rule structurally; comments document
the non-security purpose.
S310 × 8
``urlopen`` calls to fixed HTTPS registries (``api.github.com``,
``hub.docker.com``, ``ghcr.io``). URL prefix is a literal; only the
``namespace`` / ``image`` segment varies and comes from the project's
own ``meta/services.yml``, never user input. No ``file://`` scheme
exposure. Inline noqa per call site.
S602 / S605 / S606 × 8
Role-deployed scripts that legitimately use ``shell=True`` /
``os.system`` for pipes/``&&``, or ``os.execvp`` in list-form (safe by
construction — no shell parses argv). Each call site gained an inline
noqa with a per-call rationale identifying where the command string
is sourced (Ansible-templated config, host-trusted, never untrusted
external input).
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files formatted
- 1621 unit tests → all pass
- Integration suite: 44 baseline failures (unchanged)
23 files changed, +97 / −38. Combined with the prior commits the
branch satisfies 21 ruff rule families end-to-end. ``PTH``
(modernise ``os.path`` to ``pathlib``) is the remaining big-impact
candidate, deferred to its own session.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds six more rule families to the strict ruleset:
- ``T10`` — committed ``breakpoint()`` / ``pdb`` calls
- ``PGH`` — pygrep-hooks (blanket ``# type: ignore`` without code)
- ``EXE`` — shebang/executable-bit consistency
- ``RSE`` — ``raise X()`` style cleanups
- ``ICN`` — standard import aliases
- ``DTZ`` — tz-naive ``datetime`` bugs
Fixes (99 violations cleared)
=============================
EXE001 (68): every file with a Python shebang now has the executable
bit set. Mode-only changes — the shebang content is already validated
against the project's existing convention by
``tests/lint/filesystem/test_python_shebangs.py``
(``#\!/usr/bin/env python3`` for portable scripts,
``#\!/usr/bin/python`` for Ansible custom modules in ``library/``).
PGH003 (17): removed blanket ``# type: ignore`` markers. The project
doesn't run mypy/pyright in CI, so these were decoration only.
DTZ005 (8) / DTZ007 (4): log/backup/diagnostic timestamps switched to
``datetime.now(tz=timezone.utc)`` so cross-host log correlation
doesn't depend on each host's local TZ. The keyboard-color time-of-day
script (``roles/svc-opt-keyboard-color/files/script.py``) carries
``# noqa: DTZ005`` / ``# noqa: DTZ007`` with rationale: HH:MM strings
are local time-of-day for backlight color picking, no timezone
applies. The OpenSSL date parser in ``utils/cert_utils.py`` anchors
the parsed naive datetime to UTC explicitly via
``.replace(tzinfo=UTC)`` (openssl always emits the literal "GMT").
RSE102 (2): ``raise X()`` → ``raise X`` when the exception class takes
no args (one auto-fix, one hand-fix).
Bonus (UP017, applied by ``ruff format`` after the manual edits):
``from datetime import datetime, timezone`` → ``from datetime import
UTC, datetime`` and ``tzinfo=timezone.utc`` → ``tzinfo=UTC`` (Python
3.11+ shorthand).
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files formatted
- 1621 unit tests → all pass
- Integration suite: 44 baseline failures, 1 skipped (unchanged)
86 files changed, +55 / −37 content lines, plus 68 mode-only changes
(``chmod +x``). Combined with the prior strict-rule commits the branch
satisfies 17 ruff rule families end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Extends ``[tool.ruff.lint].select`` with six additional rule families:
``RUF`` (ruff-specific), ``SIM`` (flake8-simplify), ``C4``
(flake8-comprehensions), ``PERF`` (perflint), ``RET`` (flake8-return),
``PIE`` (flake8-pie). Adds ``RUF001/002/003`` to ``ignore`` because
those exist for homoglyph-attack detection, not English typography —
the project uses em/en-dashes and ``ℹ`` as a UI-hint glyph in CLI help
output deliberately.
Auto-fixed (~225, mechanical & behavior-preserving)
====================================================
- RUF005: ``a + [b]`` → ``[*a, b]``
- C408: ``dict(k=v)`` → ``{"k": v}``
- C420: ``dict.fromkeys(seq)`` shape simplifications
- SIM103/SIM110/SIM118: truthy-return / dict-key / dict-method
idiom simplifications
- RET504/RET505: drop unneeded final assigns / ``else`` after
``return``
- PIE810: combine ``str.startswith(a) or .startswith(b)``
into one call with a tuple
- RUF010: ``f"{str(x)}"`` → ``f"{x\!s}"``
- RUF100: removed unused ``# noqa`` directives that
became stale (mostly old ``# noqa: E402``
markers that ruff's smart E402 implementation
no longer flags after ``sys.path`` shims)
- RUF015 / RUF013 / RUF022 / RUF046 / RUF059: misc small wins
- I001: re-sorted import blocks where the new
imports above shifted them
Hand-fixed (~140, scripted where mechanical)
============================================
PERF401 (42): scripted AST rewrite of
``for x in iter: target.append(expr)`` (with optional ``if`` guard)
→ ``target.extend(expr for x in iter [if cond])``.
``list.extend(genexp)`` consumes the generator immediately, so the
rewrite is semantically equivalent to the original eager loop.
SIM117 (41): scripted AST combiner for nested ``with`` blocks where
the inner is the only statement of the outer (38 sites). The remaining
3 multi-line ``patch.object(...)`` chains were combined by hand using
the parenthesised ``with (\\n A,\\n B,\\n):`` form.
SIM102 (14): scripted AST combiner for nested ``if`` (10 sites). The
remaining 4 carry an inline ``# noqa: SIM102`` with rationale —
combining would either change semantics (the outer body has siblings
beyond the inner ``if``, e.g. ``cli/encrypt/inventory/__main__.py``
where ``encrypted_value = …`` runs regardless of ``ask_confirmation``)
or hurt readability (long literal whitelists).
RUF012 (9): annotated mutable class attributes with ``ClassVar[…]``
and added ``from typing import ClassVar``.
SIM105 (6): ``try/except: pass`` → ``with contextlib.suppress(…):``
with the necessary ``import contextlib`` added. Verified on
``utils/templating.py`` (3 occurrences in a ``finally`` block) —
cleanup behaviour is identical.
SIM115 (4): wrapped ``tempfile.NamedTemporaryFile(..., delete=False)``
call patterns in a ``with`` block; ``delete=False`` keeps the file
on disk after the block, so callers' ``return f.name`` still works.
PERF402, PERF403, B007, SIM108, SIM103: 5 misc fixes (loop→list-copy,
loop→dict-comprehension, unused-loop-var rename, ternary, return
expression).
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- 1621 unit tests → all pass
- Integration suite: 44 baseline failures, 1 skipped (unchanged)
- Sample-audited PERF401 rewrites in ``resolver.py``, ``inventory.py``,
``cache/files.py``, ``ressources/__main__.py``; SIM117 in
``test_main.py``; SIM105 in ``utils/templating.py``; SIM102 noqa
rationales in ``cli/encrypt/inventory/__main__.py``.
182 files changed, +993 / −960. With this and the prior strict-rule
commit the branch satisfies 11 ruff rule families end-to-end.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Adds ``[tool.ruff.lint]`` with ``select = ["E", "F", "I", "B", "UP"]``
and ``target-version = "py311"``. ``E501`` is ignored because
``ruff format`` already reflows what it can; the remaining long lines
are unsplittable string literals / URLs.
Auto-fixed (2753 issues, behavior-preserving):
- UP006/UP035: ``List[X]`` → ``list[X]``, ``Dict[X]`` → ``dict[X]``,
``from typing import …`` redirected where superseded by builtins.
- UP045: ``Optional[X]`` → ``X | None``.
- UP010/UP009/UP015: dropped Python-2-era pragmas (``from __future__
import absolute_import, division, print_function``,
``# -*- coding: utf-8 -*-``), redundant ``open(..., "r")`` mode.
- UP004: dropped implicit ``class X(object):``.
- UP022/UP008/UP032: stdlib idiom modernization (``capture_output``,
super-call simplification, etc.).
- B905: ``zip(...)`` → ``zip(..., strict=False)`` (preserves prior
behavior; switching to ``strict=True`` is a separate semantic
choice not made here).
- I001: import-block ordering.
Hand-fixed (66 issues):
- B904 × 54: appended ``from <bound>`` (when the enclosing ``except``
binds a name) or ``from None`` (when it doesn't) so chained
exceptions stay distinguishable from errors-during-exception-
handling. Applied via a small AST-aware script that walked back to
the enclosing ``except`` clause and matched the binding.
- B017 × 4: narrowed broad ``assertRaises(Exception)`` to the actual
``AnsibleError`` / ``AnsibleFilterError`` raised by the code under
test.
- B023 × 2: bound captured loop variables via default args
(``def _record(name, *, _path_real=path_real, ...)``) so each
closure gets its own snapshot.
- B007 × 2: unused loop control variables renamed to ``_`` /
rewritten to ``.values()``.
- UP007 × 2: ``Union[X, Y]`` → ``X | Y`` (plus dropping the now-unused
``from typing import Union``).
- B018 × 1: Odoo addon manifests are bare dict expressions at module
level by design — suppressed with an inline ``# noqa: B018`` and
rationale comment.
- UP031 × 1: printf-style format → f-string.
The 65 ``# noqa: …`` markers introduced by the auto-fixer all carry
real ruff codes (``B018``, ``E402``, ``F401``, etc.), so
``test_noqa_only_ruff_codes`` stays green.
Verification
============
- ``ruff check .`` → All checks passed
- ``ruff format --check .`` → 894 files already formatted
- ``tests.lint.repository.{test_project_root_import,
test_no_direct_yaml_calls, test_noqa_only_ruff_codes}`` and
``tests.unit.utils.annotations.test_suppress`` → 20 tests OK
- 1621 unit tests → all pass
- Integration suite: 44 failures (pre-existing baseline, none from
this change), 1 skipped.
Going forward, the new rules block regressions on E (style), F
(pyflakes), I (import order), B (likely-bug patterns), and UP
(modernization to py311) for any new code.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Reason: ruff parses every `# noqa: <code>` marker as a flake8 directive
and warns on every non-flake8 rule key. Every catalog rule in
`docs/contributing/actions/testing/suppression.md` is project-specific
(kebab-case), so `# noqa:` markers fired the warning on every parse.
`# nocheck:` is the project synonym ruff ignores.
Scope:
- 41 `roles/*/meta/services.yml` + 1 task + 1 .cfg.j2 template: pure
noqa→nocheck swap for catalog rules. Mixed flake8 markers
(`# noqa: E402, X`) preserved verbatim.
- 5 production .py + 2 production scripts: same swap.
- All catalog-rule lints' docstrings updated; redundant "or # nocheck:"
parentheticals collapsed to a single nocheck form.
- `utils/annotations/suppress.py`: docstring rewritten to spell out
the new convention. Parser still accepts both keywords as synonyms.
- `docs/contributing/actions/testing/suppression.md`: convention
rewritten to mandate `nocheck:`; missing `domain-spot` and
`literal-protocol-lookup` rows added; "enforced by" pointer added.
New guard:
- `tests/lint/repository/test_noqa_only_ruff_codes.py`: meta-lint
failing any `# noqa: <code>` whose codes don't match `[A-Z]+\d+`
(real ruff/flake8 shape). Allowlists the impl, its unit test, the
catalog page, the direct-yaml lint, and itself.
- `tests/lint/repository/test_no_direct_yaml_calls.py`: hardened to
reject the `noqa:` form via inlined regex (no longer routed through
`utils.annotations.suppress`).
Tangential cleanups carried in this same commit:
- `utils/cache/yaml.py`, `utils/handler/yaml.py`: 6 `direct-yaml`
markers swapped.
- `utils/cache/base.py`: `from . import PROJECT_ROOT, ROLES_DIR` moved
up to the import block (E402).
- `cli/create/role/__init__.py`: reordered so PROJECT_ROOT is defined
before `__main__` is imported (also fixed a latent circular import).
- PROJECT_ROOT migration tail: 2 new `__init__.py` modules + 10 test
files swapped from `def repo_root()` to `from . import PROJECT_ROOT`.
- ~20 ruff auto-fix removals of unused `from pathlib import Path`.
Verification:
- `ruff check .` clean.
- 23 suppression-related tests pass.
- The new lint detects all three regression shapes (project rule alone,
mixed with real ruff code, lowercase-only).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
Introduce a single grammar for per-line, per-block, and per-file
suppression markers across the test suite, replacing seven ad-hoc
syntaxes that had drifted apart.
Helper
utils/annotations/suppress.py exposes is_suppressed_at,
is_suppressed_in_head, is_suppressed_anywhere, line_has_rule, and
suppressed_line_numbers. Markers may use either `noqa` or `nocheck`
(case-insensitive synonyms), live inside `#`, `//`, `{# ... #}`, or
`<\!-- ... -->` comments, and combine multiple kebab-case rules
comma-separated. 17 unit tests cover the grammar in isolation.
Refactored consumers
- tests/external/repository/test_urls_reachable.py (rule: url)
- tests/external/docker/test_image_versions.py via
utils/docker/version_updater.py (rule: docker-version)
- tests/lint/repository/test_no_direct_yaml_calls.py (rule:
direct-yaml)
- tests/lint/ansible/test_service_shared_consistency.py (rule:
shared)
- tests/lint/ansible/roles/test_web_app_email_integration.py (rule:
email)
- tests/lint/repository/test_python_file_size.py (rule: file-size)
- tests/lint/ansible/test_run_once_tags.py +
tests/integration/roles/run_once/test_schema.py (rules: run-once,
run-once-suffix)
Marker migration
12 role tasks/main.yml files: `# run_once_<key>: deactivated`
-> `# nocheck: run-once`. roles/web-app-suitecrm/meta/services.yml
drops a malformed inline `noqa: intentional legacy pin` from a TODO
comment so the new parser does not pick it up as a stray rule.
Caching wins
Test files that previously loaded YAML / read files via raw
yaml.safe_load + Path.read_text now route through utils.cache.yaml
(load_yaml_any / load_yaml_str) and utils.cache.files.read_text:
- test_service_shared_consistency.py
- test_web_app_email_integration.py
- tests/integration/roles/run_once/test_schema.py
- utils/docker/version_updater.py
This both speeds up repeated reads and stays consistent with the
direct-yaml lint policy.
Documentation
New SPOT page at docs/contributing/actions/testing/suppression.md
with grammar, position semantics, the catalog of rules, examples,
and an "adding a new rule" checklist. lint.md and external.md drop
their per-rule tables and link to the new page. All touched docs
conform to docs/contributing/documentation.md (no em-dash thought
breaks, no SPOT self-labels, RFC-2119 keywords used consistently).
Settings sort tooling
scripts/lint/sort_claude_settings.sh wraps utils/claude/settings_sort.py
and is wired into scripts/lint/autoformat.sh. The tooling fixes the
ASCII-ascending requirement enforced by
tests/lint/repository/test_claude_settings_sorted.py; the test's
failure message now points operators at the script. .claude/settings.json
is committed in sorted form.
New lint guard
==============
tests/lint/repository/test_no_direct_yaml_calls.py AST-walks the
production tree (utils/, cli/, plugins/, filter_plugins/,
lookup_plugins/, roles/, scripts/, tasks/) and flags any direct
`yaml.safe_load` / `yaml.safe_dump` / `yaml.load` / `yaml.dump` /
`yaml.safe_load_all` / `yaml.dump_all` call. Aliased imports are
tracked: `import yaml as Y` ⇒ `Y.safe_load(...)` is flagged;
`from yaml import safe_load` ⇒ bare `safe_load(...)` is flagged.
Per-line opt-out: `# noqa: direct-yaml` on the call's physical line
(or any line spanned by a multi-line call). No central exemption
list — every legitimate exception lives next to its call with a
human-readable rationale in the surrounding comment.
Cache module additions
======================
* `dump_yaml_str(data)` — symmetric to `safe_dump(data)` with no file
path. Used for stdout output and filter return values where the
cache cannot help but the API needs to stay consistent.
* `load_yaml_str(text)` — symmetric to `safe_load(text)`. Used when
YAML comes from an HTTP body, a mutated buffer, or a mock-patched
source whose content does not correspond to a stable on-disk file.
Migrated production call sites (33 → 0 direct-yaml violations)
==============================================================
* utils/{entity_name_utils, valid_deploy_id, invokable, service_registry,
domains/list, applications/{config, in_group_deps},
docker/image/discovery, meta/{port_bands, scan},
roles/{dependency_resolver, meta_lookup, resource_validation}}
* cli/{build/{graph, inventory/full/command}, create/inventory/{host_vars,
mirror_overrides}, fix/vars_main_files, meta/{applications/in_group_deps,
callorder}, mirror/resolver, validate/inventory}
* plugins/{filter/{compose_volumes, get_all_application_ids, get_role,
invokable_paths, role_path_by_app_id}, lookup/applications_current_play}
* roles/{svc-db-postgres/filter_plugins/split_postgres_connections,
web-app-bigbluebutton/filter_plugins/compose_mods,
web-app-dashboard/lookup_plugins/docker_cards}
Behaviour adjustments
=====================
* utils/roles/dependency_resolver — `safe_load_all` (multi-doc) →
single-doc `load_yaml_any` wrapped in a list. Ansible task files are
single-doc by convention; the downstream multi-doc loop stays intact
for forward compatibility.
* roles/sys-svc-compose-ca/files/compose_ca.py — KEEPS direct
yaml.safe_load / yaml.safe_dump (annotated with noqa). The script is
copied to the target host via `ansible.builtin.copy` and the target
has no `utils/` package on PYTHONPATH.
noqa-annotated direct-yaml call sites (with rationale)
======================================================
* utils/cache/yaml.py — IS the cache; would recurse.
* utils/handler/yaml.py — vault-aware Loader subclass that wraps
\!vault scalars in VaultScalar; cache-by-path collapses the wrapping.
* cli/encrypt/inventory/__main__.py — vault-aware Dumper subclass that
emits \!vault literal blocks; the cached dump_yaml writes plain YAML
which is the wrong output here.
* roles/sys-svc-compose-ca/files/compose_ca.py — runtime-deployed.
* tasks/utils/migrate_meta_layout.py — once-off migration script.
Validation
==========
* Unit tests: 1573 passed.
* Lint: 56 passed (incl. new test_no_direct_yaml_calls).
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
