mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-09 20:26:33 +00:00
fix/dev-python/python-upper-bound
12
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 5.5
|
e3d2a24bcd |
fix(sys-svc-compose-php): scope memory_limit -1 to unpatched images
The interim CVE-2026-109618 mitigation lived in the shared hardening.ini, so every PHP role ran without a memory limit until the last image was patched, and no role's service_memory_mb share applied. Registry metadata of the images pinned on main shows that 16 of the 20 consumers already ship PHP 8.2.35, 8.3.36, 8.4.27 or 8.5.12 (or the Alpine/Debian build of it). Four predate the fix: markoshust/magento-php:8.4-fpm (8.4.21), ghcr.io/heviat/webmail:2024.06 (the Mailu OIDC flavor, built 2025-04-04), phpldapadmin/phpldapadmin:2.3.11 (built 2026-05-15) and ghcr.io/shopware/docker-base:8.3 (8.3.25). - Move memory_limit = -1 into its own content-addressed ini (FILE_PHP_MEMORY_UNLIMITED_INI) and mount it only in magento, mailu, phpldapadmin and shopware. - hardening.ini keeps expose_php, session.upload_progress.enabled and fatal_error_backtraces as permanent hardening. - Joomla's php-upload.ini loads as conf.d/uploads.ini again: no -1 has to be outranked anymore. - The root TODO names the four images that still need the override. Refs #1126 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5.5
|
d89c74feab |
feat(security): harden PHP services and proxy response headers
Proxy: - server_tokens off, X-Powered-By stripped from upstream responses, HSTS max-age raised to one year. - No global X-Content-Type-Options: under nosniff, ORB (step 3.4) blocks no-cors cross-origin responses typed text/html, e.g. tracking beacons answered with an empty 204. PHP (CVE-2026-109618, CVE-2026-109619): - New role sys-svc-compose-php installs a shared ini (expose_php Off, session.upload_progress.enabled 0, fatal_error_backtraces 0 for PHP 8.5, interim memory_limit -1) and a PHP-FPM [www] drop-in (pm.max_requests = PHP_FPM_MAX_REQUESTS) once per stack host, before 04_files renders any compose file. - The host file names carry a sha1 prefix of the controller-side source, so a content change alters every PHP app's compose.yml and recreates it, also on remote deploys where compose_volumes cannot read the stack host file and hashes the path instead. - 20 PHP roles mount the ini as a compose/swarm config into their image's scan dir; moodle, nextcloud and shopware also mount the FPM drop-in. Magento keeps its own pool: its php-fpm.conf has no include= and is rendered without notify. - Joomla's php-upload.ini now loads as zzz-joomla-uploads.ini so its own memory_limit wins: the media API compares Content-Length with toBytes(memory_limit) and rejects every upload at -1. TODOs record the memory_limit revert condition and accepted trade-offs, Akaunting on PHP 8.1, FusionDirectory on PHP 7.4 and the public Keycloak admin console. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
29995093b0 |
revert(web-app-shopware): unmount the S3 filesystem config again
The mount restored in 574c8954f would have broken every asset URL. plugins/lookup/objstore.py:226 builds public_url as domain/bucket without the per-filesystem root, so public, theme, sitemap and asset all advertise the same URL while their objects live under different roots. shopware.filesystem.asset feeds the administration bundle's script URLs, so the admin would have 404'd on clearnet and over tor alike. Re-mounting requires objstore to expose a root-aware URL and web-app-shopware to whitelist the S3 origin in meta/csp.yml. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
46686d72ad |
fix(web-app-shopware): keep OIDC admins in possession of their rights
AdminOpenAuth clears the admin flag and the ACL roles on every login that no role-assignment rule matched (UserResolver::getUserChangeSet). The Playwright login therefore demoted the provisioned administrator, the admin backend rendered empty, and the next provisioning run answered 403 FRAMEWORK__MISSING_PRIVILEGE_ERROR. The client now carries a rule with userBecomeAdmin, and a separate step re-asserts the flag for accounts an earlier login already demoted, which no console command can do. discoveryDocumentUrl stays empty because a successful discovery assigns the document over the explicit endpoints and moves the back channel onto the .onion host cURL refuses to resolve. system:install --basic-setup provisions admin/shopware with a published password; that account is locked on first install. shopware.yaml is mounted again after being rendered but unmounted since the mounts moved into meta/volumes.yml. files/ is split into php/ and shell/ trees. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
cace98976b |
fix(web-app-shopware): let the OIDC client reach a plaintext identity provider
AdminOpenAuth ships heptacom.admin_open_auth.oidc.http_client with protocols: [https]. An onion carries no TLS -- Tor encrypts the circuit itself -- so guzzle aborted OIDC discovery inside CurlFactory before opening a socket. Discovery never ran, getAuthorizationEndpoint stayed empty, and getLoginUrl returned a bare query string that the browser resolved against the current URL: the admin login button redirected to itself until the test timed out on a blank page. No request ever reached the reverse proxy, and the only trace was a warning in the container-local var/log/dev.log. A project-level config/packages template redefines that one service id. It grants http only when the resolved provider URL is itself plaintext, so a clearnet deployment keeps the https-only guarantee untouched; the condition keys on the actual scheme rather than on the tor flag, because the scheme is what guzzle rejects. Discovery, token, userinfo and jwks all flow through this single client, so one override suffices; base_http_client and saml.http_client are left alone. Deployment and mount mirror framework.yaml. Four independent skeptics attacked the earlier claim that this was a hard third-party constraint and refuted it: the pin is an ordinary Symfony service id, Shopware loads project configuration after plugin bundles, and config/packages/http_discovery.yaml demonstrably compiles its services block into the running container. Both branches of the template condition were rendered through Ansible. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
750476b26a |
feat(lint): give every mount target one spot in meta/volumes.yml
A volume that reaches several services repeated its container path once per
mount. Written out each time it has no single point of truth: a rename that
misses one line still parses, still deploys, and mounts one service somewhere
nobody intended. Nothing fails until that service looks for content which is
not there.
YAML already solves this, and
|
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
e9acbb91a6 |
fix(swarm): make config mounts readable by non-root app users
Swarm mounts type: config files as root:root; a non-root container process (or su'd user) then cannot read a 0440 file. Widen to 0444 (single-tenant containers). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
5477b56e00 |
feat(volumes): file-bind → config so single-file mounts work in swarm
Adds a strict lint that flags every `type: bind` entry in roles/<role>/meta/volumes.yml whose source or any mount target points to a single FILE (path ends in a recognised file extension or a Jinja var with a file-suffix name like *_CONF/*_FILE). Single-file binds are swarm-fragile: the source path is only present on the node where the file was rendered, so swarm rejects the task as soon as it lands on a different node. Fix is `type: config`: the manager loads the file at `stack deploy` time and swarm distributes it through raft to every node. compose-mode honours the same canonical entry (the compose_volumes filter renders it as a top-level `configs:` block; container_volumes emits the per-service reference), so the role stays mode-agnostic. Per-line opt-out: `# nocheck: meta-volume-file-bind-should-be-config` with a short `# Reason:` for legitimate exceptions (sockets, kernel interfaces, runtime-rotated files, multi-MB blobs, host writebacks). This commit migrates 23 file-bind entries across 16 roles to type:config: web-app-flowise, friendica, kix (2), listmonk, mailu, matrix (2), mediawiki, mobilizon, nextcloud, pgadmin, pixelfed, prometheus (4), shopware (2), suitecrm, taiga (2), zammad. All gain a default mode:"0440"; mount-level read_only:true is dropped (configs are read-only by default and the schema forbids the field for type:config). Updates the prometheus role-wiring test to accept both type:bind and type:config for the alert_rules mount probe. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
6e49fe74fb |
feat(volumes): canonical dict-of-dicts meta schema + lookup('volume')
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
5adf3011fe |
refactor(swarm-nfs): generalize swarm+NFS chain across all roles
Migrate the mediawiki-specific swarm patterns into shared sys-svc-compose
helpers so every role inherits swarm + NFS portability:
- meta/volumes.yml: dict schema with mandatory name|path + optional nfs
block (uid/gid/mode); 47 roles migrated, 48 lookup consumers updated to
volumes.<key>.{name,path}
- sys-svc-compose helpers: finalize.yml (flush + post_deploy),
post_deploy.yml (swarm-resolve + DB-healthy-wait), nfs_prep.yml
(pre-create NFS subdirs), swarm/resolve_self.yml (canonical
<ENTITY>_CONTAINER fact), render_replicated_templates.yml +
_render_one_template.yml (template render + swarm-peer replication,
per-entry notify, fileglob support)
- 15 roles migrated to render_replicated_templates.yml helper
(mediawiki, prometheus, kix, friendica, dashboard, nextcloud,
suitecrm, matrix, flowise, openldap)
- 5 new lint tests pin the architecture: volumes schema, compose chain,
volume-constants-are-lookups, template-render-centralization,
meta-no-role-var-refs
- ROLE_FILE_TEMPL_COMPOSE SPOT constant + 7 tests migrated to use it
- Handler test scanner extended to detect nested notify in
vars: role_templates: lists
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
2a023ca6a1 |
chore(lint): autoformat strict-failure semantics + claude.sh rename
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|