mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
fix/dev-python/python-upper-bound
15
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 5.5
|
d89c74feab |
feat(security): harden PHP services and proxy response headers
Proxy: - server_tokens off, X-Powered-By stripped from upstream responses, HSTS max-age raised to one year. - No global X-Content-Type-Options: under nosniff, ORB (step 3.4) blocks no-cors cross-origin responses typed text/html, e.g. tracking beacons answered with an empty 204. PHP (CVE-2026-109618, CVE-2026-109619): - New role sys-svc-compose-php installs a shared ini (expose_php Off, session.upload_progress.enabled 0, fatal_error_backtraces 0 for PHP 8.5, interim memory_limit -1) and a PHP-FPM [www] drop-in (pm.max_requests = PHP_FPM_MAX_REQUESTS) once per stack host, before 04_files renders any compose file. - The host file names carry a sha1 prefix of the controller-side source, so a content change alters every PHP app's compose.yml and recreates it, also on remote deploys where compose_volumes cannot read the stack host file and hashes the path instead. - 20 PHP roles mount the ini as a compose/swarm config into their image's scan dir; moodle, nextcloud and shopware also mount the FPM drop-in. Magento keeps its own pool: its php-fpm.conf has no include= and is rendered without notify. - Joomla's php-upload.ini now loads as zzz-joomla-uploads.ini so its own memory_limit wins: the media API compares Content-Length with toBytes(memory_limit) and rejects every upload at -1. TODOs record the memory_limit revert condition and accepted trade-offs, Akaunting on PHP 8.1, FusionDirectory on PHP 7.4 and the public Keycloak admin console. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
7162b7bf4c |
Merge fork/feature/svc-net-tor into feature/ai-agent-employees
834 commits from the Tor onion-service branch against 299 of ours, from merge base |
||
|
kevinveenbirkenbachandClaude Opus 5
|
43260cdee3 |
fix(nextcloud): size the php-fpm pool for the Context Agent's callback
The context_agent ExApp calls back into the same php-fpm pool it is being proxied from: its UserAuthMiddleware resolves the bearer against /ocs/v2.php/cloud/user while a child is already parked forwarding the MCP request out to the ExApp. Every in-flight MCP request therefore occupies TWO children. Below twice mcp.limits.concurrent_requests the two legs deadlock, and because the ExApp's callback carries no timeout the pool never recovers -- which is why the Context Agent sat unhealthy rather than merely slow. services.nextcloud.performance.php.fpm now sizes the pool explicitly: max_children 16, start_servers 4, min/max spare 4/8. request_terminate_timeout is 900s and must not drop below the proxy's fastcgi_read_timeout; lower kills a child while nginx is still waiting on it, equal means only work nginx has already abandoned gets reclaimed. The readiness path was spending the same scarce workers. /healthz/ready had no location of its own, so it fell through to `location /`, was rewritten to /index.php/healthz/ready and booted a full Nextcloud request on a php-fpm child once per probe interval against a pool of five. It is an exact-match location returning 200 from nginx now, with access_log off, and never reaches PHP. The pool file is rendered through render_replicated_templates.yml so swarm peers receive it too, and mounted from meta/volumes.yml at a path that sorts after zz-docker.conf, which reopens the [www] section last -- mount it earlier and the image's own values win while the deploy looks correct. The context_agent healthcheck moves from a hand-written probe in compose.yml.j2 to the shared container_healthcheck lookup, declared as flavor curl on path heartbeat with start_period 20m. The ExApp installs dependencies on first boot and legitimately outlives a 60s start period, which is the other half of why it reported unhealthy. Not verified: none of this has deployed. max_children 16 is derived from twice concurrent_requests rather than measured against real memory use, and the mount-ordering requirement is read from the image layout rather than confirmed against a merged pool config in a running container -- if it sorts wrong the whole change is inert while appearing correct. request_terminate_timeout 900s also means a genuinely stuck request now holds a child for fifteen minutes. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
750476b26a |
feat(lint): give every mount target one spot in meta/volumes.yml
A volume that reaches several services repeated its container path once per
mount. Written out each time it has no single point of truth: a rename that
misses one line still parses, still deploys, and mounts one service somewhere
nobody intended. Nothing fails until that service looks for content which is
not there.
YAML already solves this, and
|
||
|
kevinveenbirkenbachandClaude Opus 5
|
6ca511a6f4 |
fix(volumes): declare the cache volumes backup-disabled
svc-registry-cache is the twin of the registry volume fixed in
|
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
0b2d081daf |
fix(nextcloud): deliver extra config via a native additional-config file
The infinito config loader was appended to the occ-managed config.php with a shell printf while the image entrypoint was still in its post-install write phase; the entrypoint read config.php before the append and wrote it back after, clobbering the loader, so /var/www/infinito extras (oidc_login_provider_url among them) never reached Nextcloud and /apps/oidc_login/oidc answered 500 in the compose e2e job. Install the loader as config/zzz-infinito.config.php after occ reports installed instead: Nextcloud merges config/*.config.php on every read, occ never rewrites those files, and the entrypoint rsync excludes /config/, so no clobber path remains. A file mount was rejected because a non-empty config/ suppresses the entrypoint's base-config copy on first install. The loader now declares $CONFIG directly (additional config files load after unset($CONFIG)), cron gets the /var/www/infinito bind it silently lacked, and the dead include-path vars and host render go away. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
f78ec52385 |
fix(nextcloud): stop the config copy-up race and the root-owned config bind
The custom Dockerfile ran mkdir -p /var/www/html/config, which broke the
stock image's empty-/var/www/html invariant: Docker then had image content
at the shared nextcloud_data mount point and did a non-idempotent copy-up
os.Mkdir(config) per container. compose up --force-recreate starts
nextcloud and cron concurrently, so both raced that mkdir and the loser
crashed with 'failed to mkdir .../_data/config: file exists'. Reproduced
char-for-char locally (31/60 with the infinito bind, 7/80 without, so the
bind is only an aggravator); removing the image content gives 0/80.
Drop the Dockerfile mkdir so /var/www/html stays empty and the stock
entrypoint (reached via nextcloud-init-gate.sh -> /entrypoint.sh) creates
config/ as www-data with no copy-up. The mkdir only existed to pre-seed
config as www-data because binding infinito at /var/www/html/config/infinito
forced Docker to create config/ root:root; move that bind (and its
NEXTCLOUD_DOCKER_CONF_ADD_PATH loader path) out to /var/www/infinito so
config/ is owned by www-data and config.php stays writable. Keeps type:
bind, so the swarm config-object fix from
|
||
|
kevinveenbirkenbachandClaude Fable 5
|
699008a168 |
fix(web-app-nextcloud): mount the infinito config dir as a bind, not a swarm config
infinito_conf's source is a DIRECTORY (volumes/infinito, rendered from config/*.j2), but a Docker Swarm config object is a single file, so docker stack deploy aborted the whole nextcloud stack with 'read .../infinito: is a directory'. Compose only tolerated it by emulating configs: as bind-mounts. Declare it as type: bind read_only so a directory payload mounts correctly in both modes; the single-file nginx_conf and nextcloud_includes stay config objects. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Fable 5
|
1e50e4adeb |
fix(web-app-nextcloud): replace volumes_from with an explicit ro data mount on the proxy
docker stack deploy rejects volumes_from ('compose file contains
unsupported options'), so every nextcloud swarm variant died before the
stack was even created. Mount the shared data volume on the proxy
read-only via meta/volumes.yml instead; the render pipeline emits
data:/var/www/html/:ro and the proxy nginx only reads the tree
(try_files/static/fastcgi_pass; temp dirs live in the container fs).
Compose behaviour is equivalent, matching upstream fpm+nginx examples.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
9e7f28db20 |
test(lint): enforce swarm config objects are world-readable + widen 18 roles to 0444
Swarm config objects mount root:root; a 0440 mode is unreadable by a non-root container app user and crashloops the service (jenkins JCasC AccessDeniedException, exit 5). docker compose never hits this since there the file is a host bind-mount inheriting host perms. New lint test_config_object_world_readable flags type:config modes lacking o+r (type:secret exempt, per-line # nocheck escape). Widen the 22 offending 0440 configs across 18 roles to 0444: adds read only, cannot break a reader, and fixes latent non-root cases (e.g. synapse, pixelfed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
5477b56e00 |
feat(volumes): file-bind → config so single-file mounts work in swarm
Adds a strict lint that flags every `type: bind` entry in roles/<role>/meta/volumes.yml whose source or any mount target points to a single FILE (path ends in a recognised file extension or a Jinja var with a file-suffix name like *_CONF/*_FILE). Single-file binds are swarm-fragile: the source path is only present on the node where the file was rendered, so swarm rejects the task as soon as it lands on a different node. Fix is `type: config`: the manager loads the file at `stack deploy` time and swarm distributes it through raft to every node. compose-mode honours the same canonical entry (the compose_volumes filter renders it as a top-level `configs:` block; container_volumes emits the per-service reference), so the role stays mode-agnostic. Per-line opt-out: `# nocheck: meta-volume-file-bind-should-be-config` with a short `# Reason:` for legitimate exceptions (sockets, kernel interfaces, runtime-rotated files, multi-MB blobs, host writebacks). This commit migrates 23 file-bind entries across 16 roles to type:config: web-app-flowise, friendica, kix (2), listmonk, mailu, matrix (2), mediawiki, mobilizon, nextcloud, pgadmin, pixelfed, prometheus (4), shopware (2), suitecrm, taiga (2), zammad. All gain a default mode:"0440"; mount-level read_only:true is dropped (configs are read-only by default and the schema forbids the field for type:config). Updates the prometheus role-wiring test to accept both type:bind and type:config for the alert_rules mount probe. Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
6e49fe74fb |
feat(volumes): canonical dict-of-dicts meta schema + lookup('volume')
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
5adf3011fe |
refactor(swarm-nfs): generalize swarm+NFS chain across all roles
Migrate the mediawiki-specific swarm patterns into shared sys-svc-compose
helpers so every role inherits swarm + NFS portability:
- meta/volumes.yml: dict schema with mandatory name|path + optional nfs
block (uid/gid/mode); 47 roles migrated, 48 lookup consumers updated to
volumes.<key>.{name,path}
- sys-svc-compose helpers: finalize.yml (flush + post_deploy),
post_deploy.yml (swarm-resolve + DB-healthy-wait), nfs_prep.yml
(pre-create NFS subdirs), swarm/resolve_self.yml (canonical
<ENTITY>_CONTAINER fact), render_replicated_templates.yml +
_render_one_template.yml (template render + swarm-peer replication,
per-entry notify, fileglob support)
- 15 roles migrated to render_replicated_templates.yml helper
(mediawiki, prometheus, kix, friendica, dashboard, nextcloud,
suitecrm, matrix, flowise, openldap)
- 5 new lint tests pin the architecture: volumes schema, compose chain,
volume-constants-are-lookups, template-render-centralization,
meta-no-role-var-refs
- ROLE_FILE_TEMPL_COMPOSE SPOT constant + 7 tests migrated to use it
- Handler test scanner extended to detect nested notify in
vars: role_templates: lists
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
951f6d36d2 |
chore(lint): unify autoformat/galaxy parallelism into one PARALLEL env var
* scripts/lint/autoformat.sh: rename AUTOFORMAT_PARALLEL to PARALLEL, declare it at file start with default `true`; still env-overridable. Reword the worker comment so the literal `# shellcheck` token no longer appears at start-of-line and trips SC1072/SC1073. * scripts/lint/galaxy.sh: drop the GALAXY_FATAL / GALAXY_NAMESPACE / GALAXY_PARALLEL env vars (over-engineered for a single repo). Failure is always fatal. NAMESPACE is an internal constant. Add the same PARALLEL env var (default `true`, env-overridable); falsy demotes the xargs fan-out to one worker. * docs/contributing/environment/variables.md: collapse the four AUTOFORMAT_*/GALAXY_* rows into one `PARALLEL` row covering both consumers. * docs/contributing/tools/make.md: refer to `PARALLEL=0` everywhere; add the lint-galaxy parallelism note. Plus working-tree drift across roles/, group_vars/, inventories/, and tasks/ that accumulated from the make-test setup phase regenerating generated artefacts during recent test runs. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|