Commit Graph
593 Commits
Author SHA1 Message Date
kevinveenbirkenbach f51acba004 Merge remote-tracking branch 'origin/main' into fix/sys-ctl-hlth-runtime/vulnerability-engines/1324 2026-11-03 22:49:44 +01:00
kevinveenbirkenbach d2ac15cf0f Merge remote-tracking branch 'origin/main' into fix/sys-ctl-hlth-runtime/vulnerability-engines/1324 2026-11-03 22:07:55 +01:00
kevinveenbirkenbach 715820d6c1 Merge remote-tracking branch 'origin/main' into fix/ci/pin-dev-runtime-images 2026-11-03 22:07:52 +01:00
kevinveenbirkenbachandClaude Opus 5.5 81ef94e111 style(dev-python): fold the env filter onto one line after the rename
DEV_PYTHON_REQUIRES is shorter than the old name, so ruff format joins the
dict comprehension in test_install.py onto one line, and the CI format
check would report the drift.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 21:47:46 +01:00
kevinveenbirkenbachandClaude Opus 5.5 26d8a01f09 Merge origin/main into fix/sys-ctl-hlth-runtime/vulnerability-engines/1324
main moved buildkit to v0.34.2 and ollama to 0.40.14. Both image_tag
suppressions named the old tags, so they no longer apply and the
suppression lint rejects them; they are dropped, as the docker updater
does on a bump. Whether the new tags still carry CRITICAL findings shows
in the next scan.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 19:55:35 +01:00
kevinveenbirkenbachandClaude Opus 5.5 0ed50ae0a5 feat(sys-os): declare the workspace runtime images where the updater finds them
The runtime image pins lived in meta/distros.yml, which neither the docker
updater, the image mirror nor the vulnerability scan reads, so a pin would
only ever move by hand. The new role sys-os carries them in its
meta/services.yml under the distro id; utils/distros.py renders the
workspace matrix from there with the stdlib block reader, and
meta/distros.yml keeps the distro attributes only.

A new release now arrives as an updater pull request. follow: latest holds
debian, ubuntu and fedora to the release their latest tag points at, by
Docker Hub digest: Ubuntu moves between LTS releases only (26.10 shares the
digest of rolling, not latest) and Fedora skips rawhide (46 shares the
digest of rawhide, latest is 45). Arch stays rolling; CentOS stream tags
carry no number the updater can order.

dockerhub_repo and is_dockerhub move to utils/docker/image/ref.py beside
split_registry_and_name, so utils/update/follow.py can use them without an
import cycle and utils/update/docker.py stays under the size cap.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 19:55:32 +01:00
kevinveenbirkenbachandClaude Opus 5.5 0bdb14cc1d fix(dev-python): name the requires-python hand-over outside the .env key space
#1347 passed the supported Python range to the remote install.sh as
INFINITO_PYTHON_REQUIRES. Every INFINITO_* name is a key of the generated
.env, so test_every_referenced_infinito_key_is_registered fails on main
for this one, which no registry entry can carry: its value is derived
from pyproject.toml's requires-python, the single source, and a default
in default.env would be a second copy of the range. The task environment
and the script now call it DEV_PYTHON_REQUIRES; behaviour is unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 19:55:27 +01:00
kevinveenbirkenbachandClaude Opus 5.5 81e2be2204 fix(dev-python): hand requires-python to the remote script with a valid backreference
76adaeb9e8 wrote the regex_search group reference as '\1' inside the folded
task scalar. Jinja unescapes string literals, so the filter received \x01,
rejected it as an unknown argument and failed the dev-python task on every
remote host. The literal is now '\\1', and the unit test renders the task's
environment expression the way Ansible does instead of only checking that
the variable is named.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:55:22 +01:00
kevinveenbirkenbachandClaude Opus 5.5 76adaeb9e8 fix(dev-python): take the supported Python range from requires-python
bf9c5be173 capped the bootstrap at 3.14 with constants inside install.sh, a
second copy of the range pyproject.toml already declares. requires-python is
now the single source (>=3.11,<3.15): install.sh derives its floor and cap
from it, in the repository or, on a remote host where the script runs alone,
from INFINITO_PYTHON_REQUIRES that the dev-python role reads from the same
file; any other form fails loudly. The Dockerfile copies pyproject.toml
before the bootstrap, and pip now refuses 3.15 for the package as well.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:42:34 +01:00
kevinveenbirkenbachandClaude Opus 5.5 20381538dc fix(security): report truncated OSV downloads and state what a warn verdict means
A truncated chunked OSV download raises http.client.IncompleteRead, an
HTTPException rather than an OSError, so it skipped the .part cleanup and
the error line; the handler now catches both. The runtime check's warn
summary no longer claims the mirror scan tracks every warning, since host
findings and images outside the mirror never reach it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:06:57 +01:00
kevinveenbirkenbachandClaude Opus 5.5 bf9c5be173 fix(dev-python): cap the bootstrap Python at the newest minor ansible-core supports
Fedora 45 defaults to Python 3.15. ansible-core 2.21, which the unpinned
`ansible` extra still resolves to (ansible 14.5.0 pins ~=2.21.5), calls
dataclasses._is_type, which 3.15 removed, so the Workspace (fedora:latest)
job died before the first playbook (run 41491308729).

install.sh now picks and installs the highest Python between
PYTHON_MIN_MINOR and PYTHON_MAX_MINOR (3.11 to 3.14) on apt, rpm and
pick_python_bin, and fails loudly when only newer interpreters exist. On
Fedora 45 it installs python3.14. Raise PYTHON_MAX_MINOR once ansible ships
ansible-core 2.22.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 15:06:41 +01:00
kevinveenbirkenbachandClaude Opus 5.5 43bef02d2c feat(security): keep suppressed CVEs listed in the per-image vulnerability issues
The make test-security report now holds every finding at the threshold, a
suppressed one with its suppressed_until day, and the scan fails only on the
unsuppressed rows. The per-image issue keeps listing a suppressed CVE with
that day, unsuppressed and CRITICAL rows first, so the issue stays open
while a suppression runs instead of closing until it expires. The job
summary counts the failing findings and names the suppressed total apart.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:42:10 +01:00
kevinveenbirkenbachandClaude Opus 5.5 65863a2b20 feat(sys-ctl-hlth-runtime): suppress vulnerability findings per image tag
A suppression entry under <service>.security.suppressions now names either
one CVE (id) or one image tag (image_tag), each with until. An image_tag
entry accepts every finding of the service's image while the owner runs that
tag; each owner of a shared digest is judged by its own tag (Config.Image at
runtime, the pinned version in the mirror scan), so a container on any other
tag stays unsuppressed. Host suppressions are the id entries of the engine
services; an image_tag entry there covers the engine's own image.

The cve lint requires image_tag to equal the service's version, and the
Docker image updater drops a service's image_tag entries with their markers
when it moves the service to another version, so a tag suppression never
outlives its tag.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-03 12:42:09 +01:00
kevinveenbirkenbach c9c7acbb61 Merge remote-tracking branch 'origin/main' into fix/sys-ctl-hlth-runtime/vulnerability-engines/1324
# Conflicts:
#	.github/workflows/call-test-deploy.yml
2026-11-03 07:39:48 +00:00
kevinveenbirkenbachandClaude Opus 5.5 87bc1be60f perf(ci): throttle only Docker Hub copies in the daily mirror and raise the limit
The daily mirror waited about 70 s before each of its 195 copies (50 images
per hour), roughly 3.8 h of pauses, and logged no single rate-limit error.
37 of the copies come from ghcr.io, registry.gitlab.com, quay.io,
registry.opencode.de and mcr.microsoft.com, which the Docker Hub pull limit
does not cover, and --only-missing also waited before images it then skipped.
The throttle now waits only before an actual copy from docker.io, and the
schedule allows 150 Docker Hub copies per hour (24 s apart).

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 17:48:09 +01:00
kevinveenbirkenbachandClaude Opus 5.5 1e1d37d679 feat(sys-ctl-hlth-runtime): map severities to fail or warn and publish every run for Prometheus, Loki and Wazuh
The scan threshold becomes a severity-to-verdict mapping in
roles/sys-ctl-hlth-runtime/meta/services.yml, CRITICAL: fail and HIGH: warn,
for production and CI alike. A finding mapped to fail fails the service and
the deploy, a warn prints a warning line and passes, a suppressed one prints
an info line. Engine errors, missing databases and PHP hardening violations
still fail. The old list form is rejected with a config error. The daily
mirror scan stays strict: every unsuppressed fail or warn finding is a row,
so it fails while an image is red.

Every run publishes one list of judged findings four ways:
- report.json of the latest run (summary + records) under
  DIR_HEALTH_RUNTIME_REPORT, written atomically;
- findings.jsonl under DIR_HEALTH_RUNTIME_LOG, one record per finding,
  appended per run and rotated weekly by a logrotate drop-in;
- one JSON line per target plus a summary line on stdout for the journal;
- infinito_hlth_runtime.prom in DIR_METRICS_TEXTFILE when web-app-prometheus
  is deployed: findings per target, kind, image, severity, scanner and
  verdict (no CVE label), last run time and success, scanner errors.

Each record carries run_id, timestamp, host, environment, target, kind,
image, scanner, id, severity, package, installed_version, fixed_version,
suppressed, suppressed_until and verdict. The paths are SPOT variables in
group_vars/all/05_paths.yml, the script takes them as arguments.

Requirement 070 and the role README describe the mapping, the outputs and
the per-image vulnerability issue of the previous commit.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 17:48:09 +01:00
kevinveenbirkenbachandClaude Opus 5.5 4fa63dd1fb feat(security): keep one vulnerability issue per image and comment its unresolved CVEs
The dry-run mirror scan (run 41286420589) reported 10,179 CVE/image pairs
across 135 red digests; one issue per pair would flood the tracker. The
sync now keeps exactly one open issue per pinned image:

- The body lists every CVE row (id, severity, engine, role, service,
  package, installed, fixed), CRITICAL first, and stops below 60,000
  characters with the number of rows left out and a pointer to the
  vulnerability-findings artifact. On the dry-run data: 135 issues, 8
  capped, the largest body 59,872 characters.
- A further open issue with the same title closes as a duplicate of the
  oldest one.
- When an image's CVE table changed, the issue gets a "Still unresolved"
  comment with the table and an updated body; an unchanged table leaves the
  issue alone, so a daily run does not add 135 identical comments.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 17:48:08 +01:00
kevinveenbirkenbachandClaude Opus 5.5 fcf98028fb test(security): let FakeGh hang a verb instead of subclassing it in the test
CodeQL (py/call-to-non-callable, alerts 1073 and 1074 on PR #1326) could not
see the __call__ of the HangingEdit class defined inside the test method and
flagged both gh() calls in sync(). FakeGh now takes the verbs to hang as a
parameter, so the test double stays one module-level class.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 13:10:42 +01:00
kevinveenbirkenbachandClaude Opus 5.5 090e3c115d fix(security): keep OSV downloads atomic, bound the issue sync and lint every suppression entry
CodeRabbit review of PR #1326:

- fetch_databases downloads each OSV archive to all.zip.part and replaces
  all.zip only after the transfer completes. An OSError removes the partial
  file and comes back as an `osv <ecosystem>: <error>` line instead of a
  traceback.
- run_gh bounds every gh call with a 120 s timeout. A failed or hung create,
  edit or close becomes a `failed <action>: <error>` line while the other
  issues and the close pass still run, and the command exits 1.
- The suppression lint checks every list item of a suppressions block, so an
  entry that starts with `until:` still needs its `# nocheck: cve <reason>`
  marker.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:45:24 +01:00
kevinveenbirkenbachandClaude Opus 5.5 2d7bf740b0 fix(ci): put containerd, Docker and swap on the runner's empty spare disk
The small GitHub runner shape (AWS, 72G root) ran out of disk in swarm and compose rows (#1329: WordPress#2 of run 41261531965, Nextcloud#6 and #7 of run 41262286962) while an empty instance-store disk nvme1n1 sat unused. mount_spare_disk.sh formats and mounts it at /mnt, but only on a GitHub-hosted runner outside act, and only a whole unpartitioned disk without any signature, mount or swap use of at least 75G; every decline leaves a notice. relocate_docker_dataroot.sh then bind-mounts /var/lib/containerd from the spare disk and moves the Docker data-root to /mnt/docker-host (not /mnt/docker, which is the nested daemon's volume that dedicated.sh wipes). It refuses before stopping anything when images, volumes, build cache, foreign containerd namespaces or shims exist or daemon.json is unreadable, and rolls back the bind mount and daemon.json on any later failure or cancel. Swap goes on / once containerd lives on /mnt, dump_state.sh logs disk size, model and free space per storage root, and the swarm watchdog samples / and /var/lib/containerd every 10 s.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 12:31:02 +01:00
kevinveenbirkenbachandClaude Opus 5.5 3531b95de0 fix(sys-ctl-hlth-runtime): give the scanners a writable temp dir and keep the workspace databases outside the checkout
PR #1326 CI failed in two places:

- The vulnerability-db job stopped on `unable to create listing temp file:
  open /tmp/grype-db-listing…: permission denied`. The repository runs the
  engines as `docker run --user <uid>:<gid>`, and the Grype and Syft images
  set no User and WorkingDir /tmp, which only root may write. The dry-run
  mirror scan failed the same way: 156 Syft errors and the Grype database.
  Every engine now gets TMPDIR pointing at its writable /work mount.
- Every workspace job died in `make system-purge`: `git clean -fdX` could
  not remove build/vulnerability-db, because the job bound that directory
  as its own mount inside the mounted checkout. On GitHub the workspace job
  now keeps its databases under ${RUNNER_TEMP}, outside the checkout; act
  keeps the shared host directory.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 08:37:10 +01:00
kevinveenbirkenbachandClaude Opus 5.5 ca6faac318 fix(ci): share the vulnerability databases with act runs and local deploys
Under act the deploy job pinned an engine but skipped the cache restores, so
every local swarm-* run stopped on a missing Grype database. The swarm
matrix also ran inside the act job container, where the shared host
directory did not exist, and the workspace job would have kept a copy of
about 5 GB in every worktree.

- scripts/tests/deploy/act/workflow.sh requires
  INFINITO_VULNERABILITY_DB_HOST_PATH, creates it on the host and binds it
  into the job container at the same path, because that container drives
  the host daemon, which resolves every stack and swarm node bind on the
  host.
- call-test-deploy.yml pins engine and database path in one step that runs
  only outside act; under act the deploy draws an engine and refreshes the
  shared directory. call-test-workspace.yml takes the shared path under act.
- The refresh lock moves into the shared directory, so host deploys and act
  runs serialize on one file, and a root refresh hands its files to the
  directory owner. hand_over skips paths a mocked download never created,
  which failed four TestRefresh cases when the gate ran as root.
- A missing INFINITO_VULNERABILITY_DB_HOST_PATH stops compose up with a
  "run 'make dotenv'" hint instead of a KeyError.
- Rationale paragraphs in the database and compose docstrings are dropped.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 02:44:59 +01:00
kevinveenbirkenbachandClaude Opus 5.5 5e5e61d255 feat(ci): scan the mirrored images daily and track their CVEs as issues
The daily mirror workflow gains a scan job after the sync. It runs
make test-security against the GHCR mirror, writes the failing findings per
image and per CVE into the job summary and uploads the report as the
vulnerability-findings artifact.

- One issue per CVE and image, labelled vulnerability, is opened, updated
  while the finding stays and closed once no engine reports it. Issues are
  only closed when the scan reported no error, so an image that failed to
  scan keeps its issues.
- workflow_dispatch and workflow_call take an issues switch; false is a dry
  run that only produces the summary and the artifact. Scheduled runs always
  sync the issues.
- Only the scan job holds issues: write.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 02:44:59 +01:00
kevinveenbirkenbachandClaude Opus 5.5 043e001669 feat(ci): rotate the vulnerability engine per row and hand its databases over
Each CI deploy row now draws one vulnerability engine (trivy, grype or osv)
as a fifth rotation axis, shown in the job name next to distro and
filesystem (🔬 axis, 🔺 trivy, 🍇 grype, 🐙 osv). The deploy-time scan of
sys-ctl-hlth-runtime runs that engine behind Syft and the Grype runtime pass.

- A vulnerability-db job keeps one Actions cache per database and day and
  downloads only the missing ones. Every deploy job restores Grype plus its
  drawn engine, fails on a cache miss and never downloads a database itself.
- Local stacks share one host directory, INFINITO_VULNERABILITY_DB_HOST_PATH,
  mounted read-only at INFINITO_VULNERABILITY_DB_DIR in compose.yml and in
  every swarm node. A local deploy without a pinned engine draws one and
  refreshes only the databases it reads, at most once a day, under a lock.
- A pinned engine whose database is missing stops the deploy before it
  starts.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 02:44:58 +01:00
kevinveenbirkenbachandClaude Opus 5.5 6b36c8579a fix(sys-ctl-hlth-runtime): scan with Trivy, Grype and OSV instead of PHP floors
The runtime check judged PHP by its upstream version and failed every
container on an end-of-life line. Distribution builds backport fixes without
raising that number, so main run 41154895895 failed Checkmk#0 on
`PHP 8.1.2-1ubuntu2.28 runs the end-of-life line 8.1`, a build Ubuntu lists
as fixed for CVE-2026-109618 and CVE-2026-109619.

- The role now runs Trivy, Grype and OSV-Scanner, with a Syft SBOM per
  target, against the host and the images of the running containers, in
  bounded parallel lanes. Targets, engines and the failing severities are
  lists in meta/services.yml; the default is every target and engine at
  CRITICAL and HIGH with a fix available.
- Grype always scans the binary-classified packages of the SBOM, so a PHP
  built from source in an official php image (8.4.26) still fails on the
  CVEs fixed in 8.4.27, which Trivy and OSV do not see.
- OSV reads an SBOM copy whose distro purls carry the source package name.
- A CVE is suppressed only in the role owning the image, under
  services.<service>.security.suppressions with id, until and a
  `# nocheck: cve <reason>` marker; a lint rejects bare, malformed and
  expired entries.
- The deploy-time run fails on its first finding: sys-service gains
  system_service_run_once and sys-service-terminator skips the retries
  only for those units. The timer runs daily at 00:00 UTC.
- The PHP version floors are removed; the php_hardening check stays.
- make test-security replaces make security-runtimes: Syft builds an SBOM
  of every pinned image straight from the GHCR mirror, cached by digest,
  and the three engines scan it with the role's threshold and suppressions.

Part of #1324

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-01 02:44:58 +01:00
kevinveenbirkenbachandClaude Opus 5.5 8d6040948c fix(sys-svc-webserver-core): match token routes on the decoded path
The access-log map read $request_uri, which keeps percent-encoding and
repeated slashes. A request such as /%70assword/reset/<token>,
/password%2Freset/<token> or //password//reset/<token> missed the
token-route expression, fell through to the plain-path branch and was
logged with the token in plain text, while the backend still decoded it
to its reset handler.

The map now reads $uri: decoded, slash-merged (merge_slashes is not
overridden anywhere) and without the query string, so the existing
expressions match every spelling of a token route. Trade-off: after an
internal redirect the log shows the internal URI; with the generic
vhost's error_page (ws_generic.conf.j2) a 5xx line now carries
/500.html instead of the client's path, with the status unchanged.
Nothing in the repository parses the proxy access log.

The unit test resolves the map against the variable it reads, modelling
$uri as nginx builds it, and adds the three encoded spellings above.
Negative control: with the map back on $request_uri, exactly those three
cases fail (3 failed, 8 passed).

Reported by CodeRabbit on #1322.

Part of #1192

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 12:46:31 +01:00
kevinveenbirkenbach dbcc78e376 Merge remote-tracking branch 'origin/main' into fix/sys-svc-webserver-core/access-log-no-tokens/1192 2026-10-31 10:40:44 +01:00
kevinveenbirkenbachandClaude Opus 5.5 10d2b11cc9 fix(sys-svc-webserver-core): keep URL tokens out of the openresty access log
The front proxy logged the full request line: production had no
access_log, so nginx fell back to `combined` ($request, Referer,
$remote_user), and the debug format logged $request and the Referer.
Query-string tokens (?access_token=, ?key=, ?token=) and token routes
such as /password/reset/{token} and /i/confirm-email/{user}/{random}
ended up in plain text in the container log.

nginx.conf.j2 now declares a map from $request_uri to
$infinito_access_log_path:

- the path is written without the query string;
- the segments after a known token route (password/reset,
  password-reset, reset-password, confirm-email,
  lostpassword/reset/form, email-login, activate-account; matched
  case-insensitively at any depth) are written as [redacted], so the
  request, its status and timings stay visible without the token.

Both modes write an explicit access_log to /dev/stdout:

- production uses the new `redacted` format, laid out like `combined`
  (remote address, time, "method path protocol", status, bytes,
  user agent) with Referer and basic-auth user written as "-", so
  combined-style parsers keep working;
- debug keeps its fields (host, timings, upstream time, lengths,
  X-Forwarded-For, scheme, server name) but logs
  "method path protocol" instead of $request and drops the Referer.

The path comes from $request_uri instead of $uri so internal rewrites
(error_page, try_files) do not replace the path the client requested.
Nothing in the repository parses the proxy access log (Wazuh, the
health checks and the Prometheus log_by_lua metrics do not read it),
so no consumer needs to change. The error log still carries
`request:` and `referrer:` on error-level entries; that is outside
this change.

A unit test renders nginx.conf.j2 in both modes and asserts that every
access_log uses a format declared there, that no format references
$request, $request_uri, $args, $query_string, $is_args, $arg_*,
$http_referer or $remote_user, and, by resolving the map as nginx does,
that query strings are dropped, token routes are redacted and
lookalike paths stay unchanged.

Verified live with web-svc-html in production mode (deploy failed=0, so
openresty accepted the config). A probe request to
/password/reset/<probe>?token=<probe> was logged as
"GET /password/reset/[redacted] HTTP/1.1" 404 62 "-" "curl/8.14.1",
and /plain/page as "GET /plain/page HTTP/1.1". Negative control: the
same request against the previous config logged the probe path token
and the query token in plain text. The debug format with the fix is
covered by the unit test only.

Closes #1192

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 10:40:40 +01:00
kevinveenbirkenbach c37190495b Merge remote-tracking branch 'origin/main' into fix/svc-prx-openresty/worker-connections/1300 2026-10-31 03:00:09 +01:00
kevinveenbirkenbachandClaude Opus 5.5 f52da1c875 fix(svc-prx-openresty): size worker_connections independently of pids_limit
The shared proxy's nginx.conf derived worker_connections from the
openresty container's pids_limit. pids_limit caps processes and threads,
worker_connections caps simultaneous sockets per worker (client and
upstream), so with pids_limit 256 and one worker the whole host's proxy
held about 256 sockets, roughly 128 proxied requests in flight across
every vhost. Raising pids_limit to buy connections would loosen the
fork-bomb guard.

svc-prx-openresty now declares services.openresty.worker_connections
(4096 per worker, the common nginx production range, enough for several
chunk-heavy apps such as OpenWebUI at ~150-170 requests per page load).
WEBSERVER_PROXY_WORKER_CONNECTIONS reads it through lookup('config'),
without a default, so a missing key fails loudly. pids_limit stays 256.

WEBSERVER_PROXY_WORKER_RLIMIT_NOFILE is derived as 2x worker_connections
(8192), one descriptor for each client and upstream socket of a proxied
request, and nginx.conf renders it as worker_rlimit_nofile. The openresty
container pins its nofile ulimit (soft and hard) to the same value:
workers call setrlimit as root without CAP_SYS_RESOURCE, so they can only
lower the hard limit, and the inherited hard limit depends on the host
daemon (Docker >= 25 defaults to 1024:524288, DiD nests whatever the
outer container has). An explicit ulimit makes the rlimit succeed on
every host instead of logging a setrlimit alert.

The unit test pins worker_connections 4096, worker_rlimit_nofile 8192 and
the container ulimit, and fails when worker_connections follows
pids_limit again.

Live proof (compose deploy of web-svc-html, both plays failed=0): the
deployed nginx.conf renders worker_rlimit_nofile 8192 and
worker_connections 4096; openresty's master, worker and cache manager
run with "Max open files 8192 8192"; the openresty log carries no
setrlimit or "worker_connections are not enough" line.

Closes #1300

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 03:00:03 +01:00
kevinveenbirkenbachandClaude Opus 5.5 9aa56b71e9 test(lookup): restore sys.modules after loading the sso plugins fresh
test_sso.py, test_sso_oidc_plugin.py and test_sso_proxy_consumers.py
evict utils, utils.cache, utils.cache.applications and the roles
packages from sys.modules to import their plugin against fresh modules,
and never put the originals back. Ansible's plugin loader keeps the
applications lookup it loaded earlier in sys.modules, so that lookup
stayed bound to the evicted utils.cache.applications, whose
_MERGED_APPLICATIONS_CACHE utils.cache._reset_cache_for_tests() no
longer reaches.

On main this turned test_webserver_worker_limits red whenever one xdist
worker ran a test that loads the applications lookup through Ansible,
then one of these sso tests, then the worker-limits test: the stale
cache served the cpus 0.5 payload of test_a_sub_cpu_proxy_still_runs_one_worker,
so the later methods rendered worker_processes 1 instead of 4 (main run
41139196554, Workspace centos and fedora, second make test on gw1).

_load_module now runs inside mock.patch.dict(sys.modules): the plugin
keeps the fresh modules it was built against, and sys.modules returns to
its prior state afterwards. This is the change store-i18n carries since
ec0fde6561, ported unchanged so both lines stay identical.

Evidence:
- Before: pytest -n 0 test_tor_ports.py test_sso_oidc_plugin.py
  test_webserver_worker_limits.py -> "3 failed, 33 passed", the CI
  failure set ('1' != '4' twice, worker_processes 1 vs 4).
- After: test_tor_ports.py, the four sso test files and
  test_webserver_worker_limits.py -> "68 passed".

(cherry picked from commit ec0fde6561)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-31 02:17:21 +01:00
kevinveenbirkenbach b97f38ca49 Merge pull request #1261 from infinito-nexus/fix/web-app-keycloak/admin-network-allowlist/1121
feat(web-app-keycloak): serve /admin only to declared source networks
2026-10-30 23:15:41 +01:00
kevinveenbirkenbach b0bacd072f Merge remote-tracking branch 'origin/main' into fix/web-app-stalwart/swarm-smtp-listener 2026-10-30 23:02:44 +01:00
kevinveenbirkenbachandClaude Opus 5.5 eb7056dafd fix(storage): keep the NFS client off the ports the platform publishes
Main CI run 40976399687, socialhome#0 (swarm, tor, centos): the CLI tests
of svc-net-tor (onion:25 "socks CONNECT failed with reply 0x01") and
web-app-stalwart (probe_mail relay: ConnectionRefusedError on port 25)
failed. dockerd on the manager had aborted the whole ingress program of
the mail stack:

  Failed to add ingress: failed to program ingress ports:
  failed to bind host port 0.0.0.0:993/tcp: address already in use

/proc/net/tcp named the squatter: 192.168.244.10:993 -> :2049, the
node's own NFS client. The mount carries no noresvport, so the kernel
drew a source port from the 665-1023 reserved window and landed on
imaps. None of 25/465/993/995/4190/8095 got a DNAT rule or a
docker-proxy listener while every other published port did, and
stalwart still reported healthy because its healthcheck runs inside the
container. 993 and 995 are the only published ports inside that window,
so this is luck of the draw on any NFS-backed swarm host that runs the
mail role.

mount_opts now emits noresvport. insecure on the kernel-flavor export is
the other half: kernel nfsd defaults to secure and answers a mount from
an unprivileged source port with EACCES. It reaches the kernel flavor
only; ganesha already accepts unprivileged clients.

What it costs: the next deploy of every existing NFS client sees a
pending opts change and remounts the shared state; a host-side job
holding that path at that instant fails loudly. A plain remount may keep
a running mount on its old reserved port until the next real mount.

Port of f46c4d0bab from feature/store-i18n, applied identically so the
later store-i18n -> main merge stays conflict-free.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 23:02:41 +01:00
kevinveenbirkenbach 8a19b024db Merge pull request #1265 from infinito-nexus/feature/sys-ctl-hlth-runtime/vulnerable-runtimes/1127
feat(sys-ctl-hlth-runtime): alert on EOL or unpatched PHP and missing PHP hardening
2026-10-30 22:45:25 +01:00
kevinveenbirkenbachandClaude Opus 5.5 cb18214760 fix(tests): resolve CodeQL unused PROJECT_ROOT in two test packages
CodeQL py/unused-global-variable flagged two test package inits
(code-scanning alerts #1068 and #1072):

- tests/unit/python/roles/web-app-fider/__init__.py exports PROJECT_ROOT
  to test_tenant_bootstrap_gate.py through `from . import PROJECT_ROOT`
  but declared no __all__. Declare it, as every other test package
  that re-exports PROJECT_ROOT does.
- tests/unit/python/roles/sys-front-tls/__init__.py defined a
  PROJECT_ROOT nothing imports: the only test lives in templates/, whose
  own __init__.py provides it. Leave the role-level package init empty,
  like the other role-level test packages.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 21:10:21 +01:00
kevinveenbirkenbach bad992a537 Merge remote-tracking branch 'origin/main' into feature/sys-ctl-hlth-runtime/vulnerable-runtimes/1127 2026-10-30 18:47:55 +01:00
kevinveenbirkenbach 85febdf3d7 Merge pull request #1297 from infinito-nexus/fix/sys-svc-webserver-core/proxy-worker-limits
fix(sys-svc-webserver-core): size proxy workers from the proxy alone
2026-10-30 17:40:49 +01:00
kevinveenbirkenbachandClaude Opus 5.5 2393e8ddba fix(web-app-keycloak): reject scoped IPv6 admin networks, drop module docstring
Addresses two CodeRabbit findings on PR #1261.

1. Scoped IPv6 (thread 4421596356): ipaddress.ip_network accepts a scope
   ID, so `fe80::1%eth0` resolved to `fe80::1%eth0/128` and the vhost
   would carry `allow fe80::1%eth0/128;`, which nginx refuses to load.
   kc_admin_networks now raises AnsibleFilterError for any entry whose
   network address carries a scope ID. test_admin_allowlist.py covers a
   scoped address and a scoped CIDR; both fail against the previous
   filter.
2. Module docstring (thread 4421596351): removed; it described the
   module instead of documenting parameters or a trip-wire.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 16:52:06 +01:00
kevinveenbirkenbachandClaude Opus 5.5 b894db0a75 fix(svc-bkp-volume-2-local): restore into the resolved volume mountpoint
rsync 3.5.0 (Debian security update 3.5.0+ds1-0+deb13u1) refuses a
destination symlink owned by neither root nor the calling user.
svc-opt-ssd-hdd turns a volume's `_data` into a symlink to /mnt/hdd or
/mnt/ssd. The restore drill empties the volume and runs `compose create`;
docker repopulates it and lchowns the `_data` symlink to the image
directory's owner (e.g. www-data). The restore then failed with
"refusing to follow a symlink owned by an untrusted user" and
"change_dir#1 .../_data/ failed: Too many levels of symbolic links",
the drill left its subjects down, and the dependent checks timed out
(CI run 40979252521, ssd-hdd#0, on feature/store-i18n; recover.py is
identical on main).

VolumeRecovery now resolves the mountpoint before handing it to rsync:
os.path.realpath locally, `ssh <host> realpath -e -- <mountpoint>` for
--docker-host, so rsync writes into the real directory and never meets
the symlink. `--insecure-links` was not used because older rsync builds
on other distributions lack it. Two new unit tests (local symlinked
mountpoint, remote resolution); both fail against the previous code.

Root cause and fix by the i18n session (feature/store-i18n triage of
ssd-hdd#0), ported to main.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 16:51:44 +01:00
kevinveenbirkenbachandClaude Opus 5.5 69a0dfa9b9 fix(sys-svc-webserver-core): size proxy workers from the proxy alone
The shared OpenResty nginx.conf took worker_processes and worker_connections
from WEBSERVER_WORKER_*, which are the min() of svc-prx-openresty's limits and
the current play's application_id. The file renders once, in the scope of
whichever role loads the proxy first, so that role's limits capped every vhost:
on feature/store-i18n web-svc-mirror (pids_limit 64) loaded it first, every
vhost got worker_connections 64, and openresty raised 555 "64
worker_connections are not enough while connecting to upstream" alerts while a
single openwebui page fires ~150-170 chunk requests (CI run 40979252521,
homeassistant#1). Main carries the same wiring.

Add WEBSERVER_PROXY_WORKER_PROCESSES and WEBSERVER_PROXY_WORKER_CONNECTIONS,
computed from svc-prx-openresty's own openresty service only (same
resource_filter fallbacks, at least one worker), and render nginx.conf from
them. WEBSERVER_WORKER_* keep pairing the proxy with the play for app-internal
nginx such as Nextcloud's. The values can only stay equal or rise.

Found by the i18n session's triage of homeassistant#1.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 14:53:54 +01:00
kevinveenbirkenbach 85f47fdbf8 Merge remote-tracking branch 'origin/main' into feature/sys-ctl-hlth-runtime/vulnerable-runtimes/1127 2026-10-30 08:57:26 +01:00
kevinveenbirkenbachandClaude Opus 5.5 284b0873dd fix(sys-ctl-hlth-runtime): harden LAM CLI, catch apk options, drop vars comment
Addresses three CodeRabbit findings on PR #1265.

1. LAM hardening invisible to the probe (thread 4413764929): the runtime
   check probes PHP with `php -r`, the CLI SAPI. web-app-lam mounted the
   hardening ini only into /etc/php/8.4/apache2/conf.d, which the Debian
   CLI does not scan, so the probe read unhardened CLI settings and the
   check failed for LAM. The php_hardening entry now also mounts the ini
   into /etc/php/8.4/cli/conf.d. No other role mounts it into a Debian
   per-SAPI directory; the rest use the shared scan dir of the official
   image or of Alpine. The TODO entry now names the remaining gap: the
   probe reads only the CLI copy.

2. Explanatory comment above HEALTH_RUNTIME_FLOORS (thread 4413764949):
   removed. The README already states the end-of-life rule and now also
   defines a floor as the lowest patched release of each line.

3. `apk --no-cache add` missed (thread 4415018662): PACKAGE_INSTALL_PATTERN
   required `add` right after `apk`, so `RUN apk --no-cache add php7`
   passed as an image without PHP. The apk branch now accepts options
   before `add`, like the apt branch. New tests cover the option form and
   guard that `apk` in a path or a comment is no package install.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 08:57:21 +01:00
kevinveenbirkenbach bbe75cf4ed Merge remote-tracking branch 'origin/main' into fix/web-app-keycloak/admin-network-allowlist/1121 2026-10-30 08:57:15 +01:00
kevinveenbirkenbachandClaude Opus 5.5 b358a8bb00 fix(web-app-keycloak): append the admin allowlist only when networks are declared
tasks/00_core.yml built webserver_extra_configuration by concatenating the
helper iframes, the admin allowlist and the logout panel template lookups
with `~`. With the shipped empty `services.keycloak.admin.allowed_networks`
the allowlist template rendered no text, Ansible's template lookup returned
None for it, and `~` wrote the word "None" into the auth vhost. OpenResty
then refused `unknown directive "Nonelocation"` in front of the logout
panel location and every deploy with Keycloak went red.

The role now resolves the declared networks once in KEYCLOAK_ADMIN_NETWORKS
and the task appends the allowlist template only when that list is not
empty, the same guard the logout panel uses. The template renders the
restricted /admin location unconditionally from KEYCLOAK_ADMIN_NETWORKS.

With a list that did not contain the deploy host, the deploy still went
red: Keycloak answers `/` with a redirect to `/admin/`, so the CSP health
check and the front proxy probe, which request `/` from the host, got 403
and hlth-csp failed. While the allowlist is active, `location = /` now
redirects to the account console of the platform realm instead, and the
README no longer claims the deploy needs no entry for any reason.

test_admin_allowlist.py rendered the template with plain Jinja, which
yields '' for an empty template, so it could not see the None. It now
evaluates the task's own webserver_extra_configuration expression through
Ansible's Templar and real template lookup, which reproduces
"HELPER_IFRAMES\nNoneLOGOUT_PANEL" on the old task, and covers the root
redirect.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-30 08:57:11 +01:00
kevinveenbirkenbach 37cade8df6 Merge pull request #1255 from infinito-nexus/fix/mail/dmarc-rua/1125
feat(mail): publish a DMARC aggregate report address (rua)
2026-10-30 08:19:06 +01:00
kevinveenbirkenbach 87404d62be Merge remote-tracking branch 'origin/main' into feature/sys-ctl-hlth-runtime/vulnerable-runtimes/1127 2026-10-29 23:55:08 +01:00
kevinveenbirkenbach 7404aa4e6a Merge remote-tracking branch 'origin/main' into fix/web-app-fider/tenant-without-sso 2026-10-29 23:55:04 +01:00
kevinveenbirkenbachandClaude Opus 5.5 73b21eef98 fix(web-app-fider): bootstrap the tenant without SSO
00_core.yml included 01_setup_oidc.yml only when services.sso.enabled
was true, but that file carried the whole tenant bootstrap: wait for
HTTP, count tenants, POST /_api/tenants, create the admin user and
activate the tenant. Only its last task configured the Keycloak
provider. With SSO off no tenant was ever created.

Fider's RequireTenant middleware then answers every tenant route,
/api/v1/tags included, with 307 to /signup, and /signup is open to any
visitor, who can claim the fresh installation and become its
administrator. The MCP read fider_list_tags got that 307 in the swarm
SSO-off matrix row.

The tenant bootstrap moves unchanged into tasks/01_tenant.yml, which
00_core.yml includes on the stack host regardless of SSO. The OIDC file
becomes tasks/02_setup_oidc.yml and keeps only the provider upsert,
still gated on services.sso.enabled. The README states which bootstrap
steps run without SSO.

A unit test evaluates the include and task guards for every SSO,
stack-host and deploy-mode state: a fresh stack host posts exactly one
tenant with and without SSO, an existing tenant is never posted again,
the Keycloak provider is written only with SSO, and a non-stack host
runs none of it. The compose matrix row with SSO off stayed green
because nothing there read a tenant route.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-10-29 23:55:01 +01:00
kevinveenbirkenbach 48b571a094 Merge remote-tracking branch 'origin/main' into review/dmarc-rua-1125 2026-10-29 22:24:56 +01:00