kevinveenbirkenbach/computer-playbook
1
0 Fork 0
mirror of https://github.com/kevinveenbirkenbach/computer-playbook.git synced 2026-11-08 11:46:35 +00:00
Code Issues Packages Projects Releases Wiki Activity
14,682 Commits 7 Branches 78 Tags
fix/dev-python/python-upper-bound
Commit Graph
9 Commits
Author SHA1 Message Date
kevinveenbirkenbachandClaude Opus 5.5 e3d2a24bcd fix(sys-svc-compose-php): scope memory_limit -1 to unpatched images 
The interim CVE-2026-109618 mitigation lived in the shared hardening.ini,
so every PHP role ran without a memory limit until the last image was
patched, and no role's service_memory_mb share applied.

Registry metadata of the images pinned on main shows that 16 of the 20
consumers already ship PHP 8.2.35, 8.3.36, 8.4.27 or 8.5.12 (or the
Alpine/Debian build of it). Four predate the fix:
markoshust/magento-php:8.4-fpm (8.4.21), ghcr.io/heviat/webmail:2024.06
(the Mailu OIDC flavor, built 2025-04-04), phpldapadmin/phpldapadmin:2.3.11
(built 2026-05-15) and ghcr.io/shopware/docker-base:8.3 (8.3.25).

- Move memory_limit = -1 into its own content-addressed ini
  (FILE_PHP_MEMORY_UNLIMITED_INI) and mount it only in magento, mailu,
  phpldapadmin and shopware.
- hardening.ini keeps expose_php, session.upload_progress.enabled and
  fatal_error_backtraces as permanent hardening.
- Joomla's php-upload.ini loads as conf.d/uploads.ini again: no -1 has
  to be outranked anymore.
- The root TODO names the four images that still need the override.

Refs #1126

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
 2026-10-28 22:39:07 +01:00
kevinveenbirkenbachandClaude Opus 5.5 d89c74feab feat(security): harden PHP services and proxy response headers 
Proxy:
- server_tokens off, X-Powered-By stripped from upstream responses,
  HSTS max-age raised to one year.
- No global X-Content-Type-Options: under nosniff, ORB (step 3.4)
  blocks no-cors cross-origin responses typed text/html, e.g. tracking
  beacons answered with an empty 204.

PHP (CVE-2026-109618, CVE-2026-109619):
- New role sys-svc-compose-php installs a shared ini (expose_php Off,
  session.upload_progress.enabled 0, fatal_error_backtraces 0 for
  PHP 8.5, interim memory_limit -1) and a PHP-FPM [www] drop-in
  (pm.max_requests = PHP_FPM_MAX_REQUESTS) once per stack host,
  before 04_files renders any compose file.
- The host file names carry a sha1 prefix of the controller-side
  source, so a content change alters every PHP app's compose.yml and
  recreates it, also on remote deploys where compose_volumes cannot
  read the stack host file and hashes the path instead.
- 20 PHP roles mount the ini as a compose/swarm config into their
  image's scan dir; moodle, nextcloud and shopware also mount the FPM
  drop-in. Magento keeps its own pool: its php-fpm.conf has no
  include= and is rendered without notify.
- Joomla's php-upload.ini now loads as zzz-joomla-uploads.ini so its
  own memory_limit wins: the media API compares Content-Length with
  toBytes(memory_limit) and rejects every upload at -1.

TODOs record the memory_limit revert condition and accepted
trade-offs, Akaunting on PHP 8.1, FusionDirectory on PHP 7.4 and the
public Keycloak admin console.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
 2026-10-21 21:03:38 +02:00
kevinveenbirkenbachandClaude Opus 5 750476b26a feat(lint): give every mount target one spot in meta/volumes.yml 
A volume that reaches several services repeated its container path once per
mount. Written out each time it has no single point of truth: a rename that
misses one line still parses, still deploys, and mounts one service somewhere
nobody intended. Nothing fails until that service looks for content which is
not there.

YAML already solves this, and 62722a6ab used it here first -- web-app-openproject
anchors await_migrations_target and aliases it for the second service. Nineteen
roles now follow, and tests/lint/repository/spot/test_volumes_target_single_spot.py
fails on any target spelled out twice. An alias carries no path, so the rule
cannot see it, which is the point.

The conversion was mechanical and is provable rather than reviewed: every file
was rewritten and then reloaded, and any file whose yaml.safe_load result
differed from the original was discarded instead of kept. Nineteen written,
none discarded. Anchors and aliases are a serialisation form; the loader
resolves them before any consumer sees a document.

validate_volumes_meta over all 81 meta/volumes.yml reports no violation.
make test-lint: 328 passed.

web-app-prometheus carries two unrelated deletions: comments that predate this
change and fell under the comment lint once the file was touched. They explained
a 0444 mode and a compose-only bind; both facts are visible in the entries
themselves.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-14 02:02:29 +02:00
kevinveenbirkenbachandClaude Opus 5 c4a04b73bc fix(magento): get the install through, no Adobe keys needed 
Five independent blockers, none of which the role could get past.

1. Credentials. create-project pulled the metapackage from repo.magento.com,
which needs Adobe Marketplace keys; without them the bootstrap got HTTP 401
after 40 minutes of stack build. meta/schema.yml declared both keys with
validation ^.*$, so an empty value passed inventory validation silently.
mirror.mage-os.org serves the same packages unauthenticated and hosts the
dists itself - verified on the deploy's own network path, HTTP 200 for
project-community-edition-2.4.8.zip. Repository, package and version now
live in meta/services.yml, which also makes a later switch to the Mage-OS
distribution proper a data change rather than a code change.

2. Premature handler flush. The config renders notified 'compose up', so the
handler chain ran before sys-stk-full had rendered compose.yml and the CA
handler died on the missing file. Worse, the render had already created the
instance directory, so the repo's own guard for exactly this
(Fail: instance dir missing at handler flush) no longer tripped.

3. Anonymous volumes. Both markoshust images declare VOLUME /var/www while
the named volume mounted one level deeper at /var/www/html, so docker created
an anonymous volume per container and sys-ctl-hlth-volumes refused to start.
The mount moves to /var/www, where the image declares it; the image ships an
empty html/ that a fresh named volume initialises from.

4. Unescaped expansions. setup:install passed --db-host=\"$MYSQL_HOST\" and
friends, but only the quotes were escaped for the outer shell, not the
dollars. The DiD shell expanded them to empty, Magento fell back to localhost,
took that as a unix socket and reported SQLSTATE[HY000] [2002]. 18 expansions.

5. A non-resumable bootstrap. create-project needs an empty target and cannot
be retried, yet it carried the dependency download - and mirror.mage-os.org
runs nginx with the default http2_max_requests=1000, so composer's ~490
requests on one connection hit GOAWAY mid-transfer at the same package twice
in a row (curl error 18, not the flake it looked like). It is now
create-project --no-install plus a retryable composer install, which resumes
from the cache; measured succeeding in the live container after the abort.
The creates: guards follow: composer.json marks the skeleton, bin/magento the
finished install - it only exists once dependencies are in.

MAGENTO_COMPOSER_SPEC folds package and version because autoformat broke the
create-project line inside a Jinja expression; it rendered correctly by luck.

Not fixed here: the base URL renders with a double slash, and the lifecycle
stays pre-alpha - the role reaches setup:install now but has not yet served
a page.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-06 08:34:52 +02:00
kevinveenbirkenbachandClaude Opus 5 ef81055e94 fix(magento): make nginx and php-fpm actually serve 
The role could not deliver anything. The markoshust nginx image ships a
default vhost built for a different topology: fastcgi points at
unix:/sock/phpfpm.sock, port 8000 only returns 301 to https, the real
server sits on 8443 with a self-signed cert, and it references an upstream
host 'phpfpm' that does not exist here - so nginx aborted at config load
with 'host not found in upstream' and the container never started.
php-fpm listened on that same unix socket, because its
/usr/local/etc/php-fpm.conf carries no include line and php-fpm.d/*.conf
is therefore dead config.

Both are now replaced through type: config mounts. The role never had any
- that absence is why every template it previously rendered was inert.

The vhost stays minimal and includes Magento's own nginx.conf, copied from
nginx.conf.sample by the bootstrap. Its location blocks carry the .php
whitelist, the /static/version rewrites and the /media to get.php fallback
that a hand-written vhost has to reimplement - the previous attempt did
not, and would have executed every .php under pub/.

PHP moves from 8.5 to 8.4: Magento 2.4.8 requires ~8.2.0||~8.3.0||~8.4.0
and the bootstrap passes no platform-requirement override, so
composer create-project failed before nginx ever mattered.

nginx waits for the php socket before starting, for the reason given in
the erpnext frontend commit: the wildcard search domain turns a DNS miss
into a routable wrong answer, and nginx pins upstreams at config load.

Also: the oauth2 origin port named 8080, where nothing listens, and the
healthcheck proved only that some listener existed - it stayed green on a
redirect loop and on 502.

Measured in a rebuild of the role topology against Magento's real
nginx.conf.sample: HTTP 200 on / and /index.php, healthy healthcheck, and
the counter-test - without the gate and with php down, nginx exits 1.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
 2026-08-05 18:34:09 +02:00
kevinveenbirkenbachandClaude Opus 4.7 6e49fe74fb feat(volumes): canonical dict-of-dicts meta schema + lookup('volume') 
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.

What's in this commit (210 paths):

Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
  top-level volumes:/configs:/secrets: section and per-service mount
  blocks from the canonical meta. container_volumes uses the SEMANTIC
  name as per-service mount source so it resolves against compose_volumes'
  top-level dict key (docker_name as source produced 'undefined volume'
  on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
  volume(application_id, name) returns a normalised single entry;
  volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
  _CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
  raw Jinja source: strings never enter the templar walk (root-cause of
  the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
  list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
  both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
  compose-only validate+reload behind a single when-block; swarm uses
  service-update instead (bind-mounts of a FILE pin source inode at
  task-create time, so atomic write-tmp+rename leaves the running task
  on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
  lookup('volume', application_id) | dict2items so NFS subdir
  pre-creation works under the new cache split.

Tests (12)
- New strict lints flagging inline mount sections in compose templates,
  single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
  call sites — all volume-mount surfaces MUST flow through container_volumes
  / compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
  accept both the legacy and the new lookup('volume', app, 'X').name forms
  during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
  nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
  alert_rules.yml bind mount.

Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
  with lookup('container_volumes', application_id, '<service>') |
  indent(N) per service + a single trailing lookup('compose_volumes',
  application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
  legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
  application_id, 'X').Y across 9 roles

Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
 2026-06-14 19:08:46 +02:00
kevinveenbirkenbachandClaude Opus 4.7 5adf3011fe refactor(swarm-nfs): generalize swarm+NFS chain across all roles 
Migrate the mediawiki-specific swarm patterns into shared sys-svc-compose
helpers so every role inherits swarm + NFS portability:

- meta/volumes.yml: dict schema with mandatory name|path + optional nfs
  block (uid/gid/mode); 47 roles migrated, 48 lookup consumers updated to
  volumes.<key>.{name,path}
- sys-svc-compose helpers: finalize.yml (flush + post_deploy),
  post_deploy.yml (swarm-resolve + DB-healthy-wait), nfs_prep.yml
  (pre-create NFS subdirs), swarm/resolve_self.yml (canonical
  <ENTITY>_CONTAINER fact), render_replicated_templates.yml +
  _render_one_template.yml (template render + swarm-peer replication,
  per-entry notify, fileglob support)
- 15 roles migrated to render_replicated_templates.yml helper
  (mediawiki, prometheus, kix, friendica, dashboard, nextcloud,
  suitecrm, matrix, flowise, openldap)
- 5 new lint tests pin the architecture: volumes schema, compose chain,
  volume-constants-are-lookups, template-render-centralization,
  meta-no-role-var-refs
- ROLE_FILE_TEMPL_COMPOSE SPOT constant + 7 tests migrated to use it
- Handler test scanner extended to detect nested notify in
  vars: role_templates: lists

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-06-02 18:57:35 +02:00
kevinveenbirkenbachandClaude Opus 4.7 5f60f959be chore: working-tree drift sync 
Picked up by the make-test setup phase regenerating generated artefacts
during recent test runs.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-05-09 01:54:54 +02:00
kevinveenbirkenbachandClaude Opus 4.7 ed04dad5d5 refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration 
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).

req-008 — role meta layout
  * Move every role's metadata under meta/<topic>.yml with file-root
    convention (no wrapping key matching the basename):
      - schema/main.yml + config/main.yml.credentials → meta/schema.yml
      - users/main.yml                                → meta/users.yml
      - config/main.yml.server                        → meta/server.yml
      - config/main.yml.rbac                          → meta/rbac.yml
      - config/main.yml.compose.services + every other top-level key
        of config/main.yml inlined under <entity>.<key>
                                                      → meta/services.yml
      - config/main.yml.compose.volumes               → meta/volumes.yml
  * Drop the legacy schema/, users/, config/ directories from every role.
  * meta/schema.yml: support nested credential keys, optional `default:`
    field (Jinja literal, not rendered/validated/regenerated), implicit
    `algorithm: plain`.
  * Materialised paths drop the `compose.` prefix:
      compose.services.<e>.<…> → services.<e>.<…>
      compose.volumes.<k>      → volumes.<k>
      <former-top-level>.<…>   → services.<entity_name>.<former-top-level>.<…>
  * Update every Python consumer (utils/cache, utils/manager, utils/applications,
    plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
    and task file to the new paths.
  * No legacy fallback. Repository-wide grep for `compose.services.` /
    `compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
    is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
  * Add tests/lint/repository/test_role_meta_layout.py guarding against the
    old paths returning, and an explicit lint guard that meta/variants.yml
    files do not carry a `compose:` wrapper.

req-009 — per-role networks/ports
  * Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
    dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
    dns_resolver}.
  * Delete group_vars/all/09_ports.yml entirely; ports now live on the
    service entity that exposes them at
    roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
  * Rename + int-cast every former top-level `services.<entity>.port`
    field to `services.<entity>.ports.inter` (single int).
  * Always-category-keyed maps for ports.local / ports.public; relay
    spans use {start, end} integers.
  * Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
    of truth for the per-<scope>.<category> band ranges, consumed by both
    the new CLI helpers and the lint test.
  * New CLI helpers (cli meta ports suggest, cli meta networks suggest)
    walk the live role tree gap-first then increment-fallback, with
    deterministic output and non-zero exit on capacity overflow.
  * cli create role consumes both helpers when scaffolding a new role.
  * Lint extensions: bare-int port shape rejection, host-bound port
    collision check (single ints + relay span integers, inter excluded),
    band-membership check with documented allow-list for the legacy
    web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.

req-010 — run_after / lifecycle migration
  * Move galaxy_info.run_after and galaxy_info.lifecycle out of
    meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
    lifecycle}; <primary_entity> is the value returned by
    get_entity_name(role_name).
  * For multi-entity roles whose primary entity has no compose service
    (web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
    gen roles), create a dedicated metadata-only top-level entry under
    <primary_entity>.
  * Introduce utils/roles/meta_lookup.py with get_role_run_after /
    get_role_lifecycle that resolve <primary_entity> once and read from
    meta/services.yml; degrade gracefully when meta/services.yml is
    absent OR the field is absent; raise MetaServicesShapeError when the
    file exists but is malformed.
  * Update every consumer (utils/service_registry, utils/invokable,
    cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
    lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
    cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
    init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
    use the helper.
  * Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
    values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
    run_after/lifecycle never on a non-primary entity; empty run_after
    lists rejected.

Also folded into this change set
  * services_disabler now writes `services.<svc>.{enabled,shared}` into
    host_vars instead of the old `compose.services.<svc>.{...}` path; the
    discover/find pair walks meta/services.yml and only treats entries
    carrying enabled/shared/provides as provider declarations (so
    metadata-only entries from req-010 do not create spurious provider
    rows).
  * service_registry skips `run_after` deps whose target was filtered
    out via SERVICES_DISABLED instead of failing the whole load.
  * cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
    consume the new applications.<role>.services.<svc>.image shape
    (mirror.md updated to match).
  * utils/domains/list.build_applications_from_roles now reads
    meta/server.yml directly (it had been silently returning {} since
    config/main.yml stopped existing).
  * utils/roles/resource_validation.filter_roles_by_min_storage reads
    meta/services.yml.<entity>.min_storage instead of the legacy
    compose.services path.
  * web-app-bluesky pre-clones the social-app build context BEFORE
    sys-svc-compose so the build context is on disk by the time
    sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
  * 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
    web-app-wordpress, web-app-keycloak) lose their `compose: services:`
    wrapper so deep-merge over meta/services.yml lands on the new
    file-root shape.

Validation
  * Unit tests:        1571 passed.
  * Lint tests:        50 passed.
  * Integration tests: 161 passed (1 skipped).
  * Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
    set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
    discourse,mattermost}.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
 2026-04-27 09:58:26 +02:00