mirror of
https://github.com/kevinveenbirkenbach/computer-playbook.git
synced 2026-11-06 10:59:59 +00:00
fix/dev-python/python-upper-bound
10
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
  kevinveenbirkenbachandClaude Opus 5
|
2389b6e38b |
fix(gitlab): discover the OIDC issuer over the scheme it actually uses
GitLab could not log anyone in through Keycloak on an onion deployment. The sign-in page carried the omniauth flash: Could not authenticate you from OpenIDConnect because Ssl connect returned=1 errno=0 peeraddr=172.30.0.10:443 state=error: tlsv1 unrecognized name (ssl alert number 112), and gitlab-webservice logged OpenIDConnect::Discovery::DiscoveryFailed eleven times. The openid_connect gem rebuilds the well-known URL and never reads the scheme of the issuer. Its config resource also drops the port when it is 80 or 443, so http://auth.onion loses port 80 and URI::HTTPS.build puts 443 back. The hidden service publishes port 80 only, so 443 reaches the SNI-less default listener, which answers ssl_reject_handshake - alert 112, exactly the string in the flash. An initializer sets SWD.url_builder and WebFinger.url_builder to URI::HTTP when the issuer is plaintext. These are the same two lines web-app-decidim already ships for the identical bug, and this repo has one such fix per role for zammad, mastodon, decidim and peertube; gitlab was the gap. The mount copies the smtp_settings.rb initializer pattern onto webservice and sidekiq. The scheme test lives in the ruby file rather than in the render and mount conditions, which both gate on services.sso.enabled alone. Two separately spelled conditions could drift once an inventory sets OIDC_OVERRIDE.URL, and would then either render a file that is never mounted or mount one that was never rendered. Boot safety: omniauth_openid_connect and openid_connect are default-group Gemfile entries without require: false, so Bundler.require loads them before the initializers run, and openid_connect requires swd and webfinger itself. Both writers are public and assign an ivar. GitLab never assigns url_builder anywhere. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 5
|
6e93862dd5 |
refactor(gitlab): ship the config trees as swarm configs, not host binds
|
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
22a391ac56 |
test(lint): require data-loss justification for nfs: false volumes
A node-local volume does not follow a swarm reschedule, so its data is lost on drain. Add a meta/volumes.yml lint that fails when a 'nfs: false' opt-out lacks both '# nocheck: nfs-false-data-loss' and a '# Reason:' arguing why that loss is acceptable. Justify the 10 existing entries (checkmk OMD site; gitlab repositories, logs, runtime state). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Fable 5
|
cb174cb9c3 |
chore(gitlab): drop explanatory comments
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Fable 5
|
4ac854efbd |
fix(gitlab): survive container restarts and silence hlth-volumes
CNG's set-config copies the rendered configs into CONFIG_DIRECTORY on every container start; the first cp creates the destination with the source mode, so 0440/0400/0444 sources left the copy owner-unwritable and the next start's truncating cp died with EACCES (permanent crash-loop after any restart, including the deploy's own recreate). Grant owner-write to exactly the files set-config copies; files bind- mounted read-only at their final destination stay tight. Mount the CNG images' declared VOLUME paths (/var/log*, /var/opt/gitlab) as named node-local volumes so containers stop spawning anonymous volumes that fail the hlth-volumes check. Verified live: HTTP 200, all services healthy across recreate, hlth-volumes Result=success. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.8
|
0bf1e8ef8c |
refactor(gitlab): replace Omnibus with the Cloud Native GitLab composition
Deploy GitLab from the official CNG CE images as six services: webservice (puma) and sidekiq scale to the topology default, workhorse is the sole HTTP entry, gitaly and a one-shot migrations job are pinned singletons, shell runs gitlab-sshd. The Omnibus container, its Dockerfile and the GITLAB_OMNIBUS_* config groups are removed. All rails/workhorse/shell/gitaly configs and shared secrets are pre-rendered by Ansible and mounted read-only; eleven credentials land in meta/schema.yml including an rsa_pem_2048 OIDC signing key (new value_generator algorithm). Postgres, redis and S3 object storage ride the existing lookups; OIDC, SMTP, the public SSH port and the root password are preserved. compose_volumes gains a per-volume nfs:false opt-out so the gitaly repositories volume stays node-local in swarm (upstream forbids git repositories on NFS); nfs_prep skips volumes without driver_opts accordingly. The Nextcloud gitlab integration execs the CNG rails runner. ruby_single_line dies with its only consumer; ruby_dq keeps test coverage. Ports (8181/8080/2222/8075/3808/9122) and service names resolve from single SPOT declarations in meta/services.yml via vars constants; the version pin sits on the webservice image pair so the updater can bump it. Live-validated in compose up to all six services healthy and migrations completed; fresh-install-only semantics for existing Omnibus data are documented. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> |
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
6e49fe74fb |
feat(volumes): canonical dict-of-dicts meta schema + lookup('volume')
Migrate every role's meta/volumes.yml from ad-hoc dict-of-named-volumes
to a canonical dict-of-dicts schema where the YAML key is the semantic
short name and the entry body carries type / name / source / mounts /
nfs / mode / read_only. The optional 'name:' field is the docker volume
name (defaults to the key); the previous 'docker_name:' field is gone.
What's in this commit (210 paths):
Infrastructure (23)
- New filter plugins/filter/{compose,container}_volumes.py — render the
top-level volumes:/configs:/secrets: section and per-service mount
blocks from the canonical meta. container_volumes uses the SEMANTIC
name as per-service mount source so it resolves against compose_volumes'
top-level dict key (docker_name as source produced 'undefined volume'
on swarm).
- New lookup plugins/lookup/{compose,container,volume}_volumes.py.
volume(application_id, name) returns a normalised single entry;
volume(application_id) returns the whole canonical dict.
- utils/cache/applications.py keeps a sibling registry
_CANONICAL_VOLUMES_BY_ROLE outside the merged applications payload so
raw Jinja source: strings never enter the templar walk (root-cause of
the 200x cache-miss + recursive get_merged_applications storm).
- utils/roles/applications/mounts.py — schema validator + normaliser;
list-shape is rejected.
- plugins/lookup/nginx.py: _volume_path_by_name reads canonical directly.
- scripts/tests/deploy/swarm/_context.sh: NFS_VOLUMES extraction handles
both legacy dict and canonical dict shapes.
- roles/svc-prx-openresty/handlers/restart_and_validate.yml: gate
compose-only validate+reload behind a single when-block; swarm uses
service-update instead (bind-mounts of a FILE pin source inode at
task-create time, so atomic write-tmp+rename leaves the running task
on a deleted inode).
- roles/sys-svc-compose/tasks/utils/nfs_prep.yml: switch to
lookup('volume', application_id) | dict2items so NFS subdir
pre-creation works under the new cache split.
Tests (12)
- New strict lints flagging inline mount sections in compose templates,
single-file bind mounts, and legacy lookup('config', ..., 'volumes.X.Y')
call sites — all volume-mount surfaces MUST flow through container_volumes
/ compose_volumes / volume lookups.
- Integration: test_volumes_compose_chain + test_volume_constants_are_lookups
accept both the legacy and the new lookup('volume', app, 'X').name forms
during the migration window.
- Unit tests for the new lookup, filter, mounts helper, plus updated
nginx lookup test.
- Prometheus role-wiring test reads canonical meta for the
alert_rules.yml bind mount.
Role migrations (175)
- 62 meta/volumes.yml converted to dict-of-dicts canonical shape
- 63 templates/compose*.yml.j2 replace inline volumes:/configs:/secrets:
with lookup('container_volumes', application_id, '<service>') |
indent(N) per service + a single trailing lookup('compose_volumes',
application_id) for the top-level block
- 50 vars/main.yml drop dead volume constants and refactor 23 remaining
legacy lookup('config', ..., 'volumes.X.Y') to lookup('volume',
application_id, 'X').Y across 9 roles
Co-Authored-By: Claude Opus 4.7 <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
5adf3011fe |
refactor(swarm-nfs): generalize swarm+NFS chain across all roles
Migrate the mediawiki-specific swarm patterns into shared sys-svc-compose
helpers so every role inherits swarm + NFS portability:
- meta/volumes.yml: dict schema with mandatory name|path + optional nfs
block (uid/gid/mode); 47 roles migrated, 48 lookup consumers updated to
volumes.<key>.{name,path}
- sys-svc-compose helpers: finalize.yml (flush + post_deploy),
post_deploy.yml (swarm-resolve + DB-healthy-wait), nfs_prep.yml
(pre-create NFS subdirs), swarm/resolve_self.yml (canonical
<ENTITY>_CONTAINER fact), render_replicated_templates.yml +
_render_one_template.yml (template render + swarm-peer replication,
per-entry notify, fileglob support)
- 15 roles migrated to render_replicated_templates.yml helper
(mediawiki, prometheus, kix, friendica, dashboard, nextcloud,
suitecrm, matrix, flowise, openldap)
- 5 new lint tests pin the architecture: volumes schema, compose chain,
volume-constants-are-lookups, template-render-centralization,
meta-no-role-var-refs
- ROLE_FILE_TEMPL_COMPOSE SPOT constant + 7 tests migrated to use it
- Handler test scanner extended to detect nested notify in
vars: role_templates: lists
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
2a023ca6a1 |
chore(lint): autoformat strict-failure semantics + claude.sh rename
* scripts/lint/autoformat.sh:
- rename run_sort_claude_settings worker to run_claude (matches the
renamed scripts/lint/claude.sh).
- drop the EXIT trap that wiped status_dir; cleanup happens explicitly
after `wait` returns, so a parallel worker that interrupts the
parent never races a vanished status dir.
- each worker now captures its tool's exit code and writes one of
OK / SKIP / FAIL. Tolerated `expected violations remain`-style exits
(markdownlint-cli2 rc≤1, ansible-lint rc≤2) still count as OK; only
real failures count as FAIL.
- all workers run to completion regardless of individual tool failures
(so the user always sees the full picture); after the summary, the
script exits 1 if any worker reported FAIL.
* scripts/lint/sort_claude_settings.sh -> scripts/lint/claude.sh
(history preserved via git mv); update test_claude_settings_sorted.py
fix-hint accordingly.
Plus working-tree drift across roles/, group_vars/, inventories/,
requirements/, and .pre-commit-config.yaml accumulated from the
make-test setup phase regenerating generated artefacts during recent
test runs.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|
||
|
kevinveenbirkenbachandClaude Opus 4.7
|
ed04dad5d5 |
refactor(meta): land req-008/009/010 role-meta layout/networks/ports/run_after migration
Single atomic landing for three back-to-back role-meta refactor requirements
(see docs/requirements/008-role-meta-layout.md, 009-per-role-networks-and-ports.md,
010-role-meta-runafter-lifecycle-migration.md).
req-008 — role meta layout
* Move every role's metadata under meta/<topic>.yml with file-root
convention (no wrapping key matching the basename):
- schema/main.yml + config/main.yml.credentials → meta/schema.yml
- users/main.yml → meta/users.yml
- config/main.yml.server → meta/server.yml
- config/main.yml.rbac → meta/rbac.yml
- config/main.yml.compose.services + every other top-level key
of config/main.yml inlined under <entity>.<key>
→ meta/services.yml
- config/main.yml.compose.volumes → meta/volumes.yml
* Drop the legacy schema/, users/, config/ directories from every role.
* meta/schema.yml: support nested credential keys, optional `default:`
field (Jinja literal, not rendered/validated/regenerated), implicit
`algorithm: plain`.
* Materialised paths drop the `compose.` prefix:
compose.services.<e>.<…> → services.<e>.<…>
compose.volumes.<k> → volumes.<k>
<former-top-level>.<…> → services.<entity_name>.<former-top-level>.<…>
* Update every Python consumer (utils/cache, utils/manager, utils/applications,
plugins/lookup, plugins/filter, cli/*) plus every Jinja template, vars/*.yml
and task file to the new paths.
* No legacy fallback. Repository-wide grep for `compose.services.` /
`compose.volumes.` / `schema/main.yml` / `users/main.yml` / `config/main.yml`
is empty outside docs/requirements/ and tasks/utils/migrate_meta_layout.py.
* Add tests/lint/repository/test_role_meta_layout.py guarding against the
old paths returning, and an explicit lint guard that meta/variants.yml
files do not carry a `compose:` wrapper.
req-009 — per-role networks/ports
* Move group_vars/all/08_networks.yml.defaults_networks.local.<role>.{subnet,
dns_resolver} into roles/<role>/meta/server.yml.networks.local.{subnet,
dns_resolver}.
* Delete group_vars/all/09_ports.yml entirely; ports now live on the
service entity that exposes them at
roles/<role>/meta/services.yml.<entity>.ports.{inter,local,public}.
* Rename + int-cast every former top-level `services.<entity>.port`
field to `services.<entity>.ports.inter` (single int).
* Always-category-keyed maps for ports.local / ports.public; relay
spans use {start, end} integers.
* Add PORT_BANDS map to group_vars/all/08_networks.yml — single source
of truth for the per-<scope>.<category> band ranges, consumed by both
the new CLI helpers and the lint test.
* New CLI helpers (cli meta ports suggest, cli meta networks suggest)
walk the live role tree gap-first then increment-fallback, with
deterministic output and non-zero exit on capacity overflow.
* cli create role consumes both helpers when scaffolding a new role.
* Lint extensions: bare-int port shape rejection, host-bound port
collision check (single ints + relay span integers, inter excluded),
band-membership check with documented allow-list for the legacy
web-app-bigbluebutton.bigbluebutton.local.http: 48087 entry.
req-010 — run_after / lifecycle migration
* Move galaxy_info.run_after and galaxy_info.lifecycle out of
meta/main.yml into meta/services.yml.<primary_entity>.{run_after,
lifecycle}; <primary_entity> is the value returned by
get_entity_name(role_name).
* For multi-entity roles whose primary entity has no compose service
(web-app-bluesky → bluesky, web-app-matrix → matrix, desk/dev/drv/
gen roles), create a dedicated metadata-only top-level entry under
<primary_entity>.
* Introduce utils/roles/meta_lookup.py with get_role_run_after /
get_role_lifecycle that resolve <primary_entity> once and read from
meta/services.yml; degrade gracefully when meta/services.yml is
absent OR the field is absent; raise MetaServicesShapeError when the
file exists but is malformed.
* Update every consumer (utils/service_registry, utils/invokable,
cli/meta/applications/resolution/run_after/__main__, cli/meta/roles/
lifecycle_filter/__main__, cli/meta/applications/resolution/combined,
cli/build/{tree,graph,role_include}, cli/deploy/development/{common,
init,deps,deploy}, plugins/filter/canonical_domains_map, etc.) to
use the helper.
* Lint guards: no run_after/lifecycle in meta/main.yml; lifecycle
values restricted to {planned,pre-alpha,alpha,beta,stable,deprecated};
run_after/lifecycle never on a non-primary entity; empty run_after
lists rejected.
Also folded into this change set
* services_disabler now writes `services.<svc>.{enabled,shared}` into
host_vars instead of the old `compose.services.<svc>.{...}` path; the
discover/find pair walks meta/services.yml and only treats entries
carrying enabled/shared/provides as provider declarations (so
metadata-only entries from req-010 do not create spurious provider
rows).
* service_registry skips `run_after` deps whose target was filtered
out via SERVICES_DISABLED instead of failing the whole load.
* cli/mirror/resolver and cli/create/inventory/mirror_overrides emit/
consume the new applications.<role>.services.<svc>.image shape
(mirror.md updated to match).
* utils/domains/list.build_applications_from_roles now reads
meta/server.yml directly (it had been silently returning {} since
config/main.yml stopped existing).
* utils/roles/resource_validation.filter_roles_by_min_storage reads
meta/services.yml.<entity>.min_storage instead of the legacy
compose.services path.
* web-app-bluesky pre-clones the social-app build context BEFORE
sys-svc-compose so the build context is on disk by the time
sys-stk-front-proxy/tasks/main.yml flushes the docker build handler.
* 4 meta/variants.yml files (svc-ai-ollama, web-app-phpmyadmin,
web-app-wordpress, web-app-keycloak) lose their `compose: services:`
wrapper so deep-merge over meta/services.yml lands on the new
file-root shape.
Validation
* Unit tests: 1571 passed.
* Lint tests: 50 passed.
* Integration tests: 161 passed (1 skipped).
* Full-cycle fresh-purged-apps deploy succeeded for the 8-app validation
set: web-app-{nextcloud,wordpress,yourls,bigbluebutton,bluesky,gitea,
discourse,mattermost}.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
|