Commit Graph
684 Commits
Author SHA1 Message Date
kevinveenbirkenbach efaa9b1c49 Merge remote-tracking branch 'origin/main' into feature/ai-agent-employees
# Conflicts:
#	roles/web-app-flowise/meta/services.yml
2026-08-10 00:14:48 +02:00
kevinveenbirkenbach abc5cd5754 fix(coredns): render the Corefile readable to the container user
mkstemp creates its file 0600 and Path.replace carries that mode into the target, so every freshly rendered Corefile was owner-only while coredns runs as nonroot. The container exits with 'open /Corefile: permission denied', name resolution dies for the whole stack, and deploys then fail at whatever they try to fetch first.

Long-running stacks hid it: coredns reads its config once at startup, so a container created before a re-render keeps working. Only new worktree stacks broke.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
(cherry picked from commit efc281e327)
2026-08-09 13:30:57 +02:00
kevinveenbirkenbachandClaude Opus 5 4e168286cc fix(ci-trigger): read the priority input from the job log
GitHub cuts display_title at 512 UTF-8 bytes and closes it with a
literal '...'. The run-name template renders priority second-to-last
and whitelist last, and on the --failed path the whitelist is empty, so
the cut always lands inside the priority list: 21 of 69 roles survived
the source run's title, the 21st a half-token.

untriggered_priority parsed that title and passed undecodable names
through verbatim, so the fragment reached the dispatched priority
input, became the orchestrator's whitelist, and hit the complexity
filter's lexer, whose word class has no rule for CJK. Sequencing died
with argparse exit 2 and took the run with it. The retrigger that is
supposed to repair a red run was breaking its successor.

Read the priority from the ##[group] Inputs block a called job's log
records instead: it carries all 69 roles. Depth-1 jobs only -- a deeper
one echoes its own workflow's inputs, which have no priority.

Drop the decode passthrough now that the source is lossless. The loud
failure is only affordable once a whole value exists; before, it would
have aborted every large retrigger.

dispatched_priority cross-checks against the title: it still proves a
priority line existed even when truncation mangled it, so an empty log
read is a broken reader -- expired logs, a moved block -- and not a run
without priority. Without that check the carry-over would vanish
silently, which is what the rest of this change exists to prevent.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 19:33:20 +02:00
kevinveenbirkenbachandClaude Opus 5 f3826589f2 fix(compose): serialize concurrent CoreDNS renders
Parallel `make test` targets each auto-ran `compose-up`, which rendered
into a fixed Corefile.tmp; the first replace() stole the file from the
others and surfaced as "envsubst did not produce an output file".
mkstemp per process plus a flock'd re-check in the test wrapper so only
one target brings the stack up.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-08 10:29:26 +02:00
kevinveenbirkenbachandClaude Opus 5 71888d86af refactor(ci): prefix workflows by trigger
The workflow directory mixed entry points, reusable stages and scheduled jobs into one flat list sorted by topic, so the first question about any file - what starts it - was the one the name did not answer. Every workflow now carries one of three prefixes, derived from its on: block: cron- for a schedule trigger, call- for one only reachable through workflow_call, entry- for everything a repository event or a human starts. Scheduled workflows additionally carry an alarm clock as the first character of their name.

Two lints keep it that way. One derives the expected prefix from the triggers and compares it against the file name; it flagged all 17 offenders before the rename and none after. The other resolves every .github/workflows/<file> string in the repository against the directory, which is what a rename silently breaks in Markdown links, Python constants and shell API URLs alike.

Renaming pulled 69 pre-existing comments into the comment lint's changed-file scope. The vendored CodeQL template got the nocheck marker its scorecard sibling already carried, two workflow headers moved above name: where they count as file headers, and the remaining banners were deleted.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-07 17:05:51 +02:00
kevinveenbirkenbach eccb502197 Merge branch 'feature/swarm-nfs' into feature/ai-agent-employees
One conflict, in the category catalogue: this branch added a virt category for
the sandboxed container runtimes, the other did not.

Keeping our side alone would have closed the conflict and broken the build. The
same merge introduces a test requiring every category to declare a hanzi label,
unique among its siblings, and virt predates that rule and carries none. It now
declares 沙箱, checked against its ten siblings for a collision.
2026-08-06 21:49:56 +02:00
kevinveenbirkenbach 37e74ddd63 fix(inventory): generate user passwords the exec chain does not mangle
A user password reaches its application through Ansible, a shell, a container
exec -e and finally the runtime's own environment reader. The generator seeded
it from the same alphabet as the become password, punctuation included, and
that does not survive the chain intact.

Three roles had already worked around it by declaring a separate alphanumeric
credential for an identity that their meta/users.yml already declares, and
saying so in the description. Removing those without fixing the generator would
have reintroduced exactly the bug they were built against, so the generator is
fixed first and all three now read the user's own password. At length 64 the
smaller alphabet costs nothing worth measuring, and the become password keeps
the old one because it travels a different path.

Zammad additionally used the bot's password for its wizard-bypass admin, one
secret for two users. Each has its own declaration and now its own value.

A test asserts the generated password is alphanumeric, so a future change to
the alphabet fails there rather than inside a Ruby ENV.fetch in a container.
2026-08-06 21:08:21 +02:00
kevinveenbirkenbachandClaude Opus 5 58e18872ce refactor(ci): tighten the display name to one separator
The category labels now run together into a single block instead of being
separated from each other, the glyph sits flush against that block, and the
only `·` left is the one between the categories and the role's own fragment:
`🐳网络应用·Nextcloud` where it used to read `🐳 网络·应用·Nextcloud`.

The deploy-job parser drops the whitespace requirement behind the glyph in
the same move. Without it no job name from a fresh run matches any more, and
the retrigger would find zero roles without reporting an error.

The name still never holds a space, so the heading keeps collapsing its own
whitespace to `·` and the space-separated whitelist and priority inputs stay
parseable with more than one role. Empty parts are dropped when joining, so a
role with no category at all (pkgmgr, test-e2e-*) does not gain a leading
separator.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 20:09:53 +02:00
kevinveenbirkenbachandClaude Opus 5 add00234c9 feat(ci): name deploy jobs after the role, not its id
CI deploy jobs were titled with the raw role id, so a run page read as a
column of dash-joined prefixes. They now carry a display name: the category
path as hanzi labels, then the role's README heading or the part of the id
the categories do not already say, whichever is shorter.

utils/roles/display.py owns both directions. Encoding happens once, at
discovery time, as a `label` key on every matrix entry; the workflows render
`<glyph> ${{ matrix.label }}` and keep the glyph as a literal because the
retrigger reads the deploy mode off it.

Decoding is what makes the retrigger keep working. It accepts a display name
with or without its variant shard and a bare role id unchanged, so job names
from runs that predate this still parse and a hand-typed dispatch never has
to spell hanzi. The parser therefore stopped matching an id-shaped tail and
asks the codec instead; a job the codec does not recognise is skipped rather
than counted as a role.

The role selections go out as display names too, so the run title reads like
the jobs it spawns. Four consumers decode them back, each the single place
the value is actually used: the whitelist resolver before it checks the role
directory, the discovery query on its env, and the sequencing and plan CLIs
on their arguments.

Every fragment is joined by `·`, including before the title, and the heading
loses its whitespace and any emoji variation selector. A display name holding
a space would make the space-separated whitelist and priority inputs
unparseable with more than one role.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 20:09:53 +02:00
kevinveenbirkenbach f4398e1c58 feat(inventory): pin a password on every user the roles declare
A user definition that declares no password falls back to
{{ 42 | strong_password }}. That is a template, not a value: it is re-rendered
at every use site, so the task that creates an account and the task that later
authenticates as it read two different secrets. The account exists and the
login fails. Inventory creation now decides the value once, after the roles are
resolved and next to the role credentials that were already generated there.
The scope follows the deployment: only roles resolved into this inventory
contribute users, an existing password is never overwritten, and a second run
writes nothing.

Building it duplicated what the package already carried three to seven times
over: an ensure-map helper defined three times, the same fifteen-line
round-trip load repeated seven times, and the vault encrypt-and-reparse dance
twice. They now live in one module, the round-trip counterpart to the existing
read-only yaml_io. host_vars drops from 213 to 130 lines, and its two override
entry points share the merge they used to copy. Behaviour is unchanged, and
that is measured rather than asserted: the committed versions and the new ones
were driven against the same host_vars and produced byte-identical output for
all six writers, comment placement and vault block layout included.

The adapter probes move into their own folder because a shared file-name prefix
reads as a folder in disguise, and the lint that enforces it scans the tracked
file list, so the repository was failing on it. A test that was defined at
module level instead of inside its class, and therefore never ran, is indented
back in.
2026-08-06 19:54:19 +02:00
kevinveenbirkenbachandClaude Opus 5 3883d73c43 feat(ci): retrigger reproduces the source run
The REST API answers inputs: null for a finished workflow_dispatch, so
the run name is the only record of what a manual run was dispatched
with. It carried one value, distros; everything else silently fell back
to the workflow defaults, and a failure that only reproduces on swarm
with btrfs came back green from a run that deployed all three modes on
whatever filesystem it drew.

Render every input as its own <emoji><value> segment instead, omitted
when it holds its default, so the title carries exactly what deviates
and an absent input means 'leave it on the default'. run_name derives
those segments from the workflow rather than knowing them, so the format
cannot drift away from the parser without a failing test.

distros no longer defaults to debian -- empty draws one at random, the
way untagged pushes already do. That leaves nothing in the title, so the
effective list is recovered from the discover job names, matched by
content against the meta/distros.yml SPOT rather than by job-name shape.

Priority roles that fell behind the budget cut in every mode hold no job
at all, so failed_roles cannot see them: they are neither green nor red,
they never ran. Carry them along or they never deploy.

--run now resolves before the selection, so it reproduces a run whatever
picks the roles, not only with --failed.

tests/utils/ci_*.py move into tests/utils/ci/ -- two files sharing a
delimiter-bounded prefix are a folder in disguise, and the structure
lint says so.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:47:40 +02:00
kevinveenbirkenbachandClaude Opus 5 636701ac26 feat(ci): serialise a big deploy line's modes
GitHub cancels a job queued for 24h. A line starting swarm, compose and
host at once queues its whole matrix in one go, so with more roles than
runner slots the tail waits past that cut and dies unrun -- 54 of 241
jobs in run 30937913632.

Deploy the modes one after the other instead, heaviest first, which
restarts the queue clock per mode. cli.meta.ci.sequencing counts a
line's matrix jobs on the row basis the discover steps use and
serialises above INFINITO_CI_SEQUENTIAL_THRESHOLD; below it the modes
still start together, so a small diff pays no wall-clock for this.

A needs: edge is static, so both layouts exist as jobs and the decision
skips one. Their conditions are structurally asserted, not evaluated:
ten near-identical if expressions invite a copied sentinel or a missing
chain edge, and both failures are silent at runtime.

mode_fail_fast stops a serial line at its first failed mode, on by
default.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-06 10:47:40 +02:00
kevinveenbirkenbach 255ee13f60 Merge branch 'feature/swarm-nfs' into feature/ai-agent-employees 2026-08-03 04:21:37 +02:00
kevinveenbirkenbachandClaude Opus 5 92615e2176 fix(deploy): pass the execution modes as JSON so a false mode is false
The deploy CLI shipped every mode as -e KEY=value, which ansible-playbook parses as a string at the highest precedence. MODE_DEBUG=false therefore arrived as the non-empty, and so truthy, string false. Every when: MODE_X | bool was unaffected because the filter maps it back, but four sites read a flag bare and silently inverted: openldap wrote LDAP attributes with state exact on every deploy instead of only under MODE_CLEANUP, SYS_SERVICE_DEFAULT_STATE was restarted where it should have been omit, SYS_SERVICE_RUNNER_STATE was never started, and mailu always logged at DEBUG.

Send the modes as one JSON extra-var, which keeps the declared type, and give those four sites the | bool they were missing so the meaning of a flag no longer depends on how its value reached Ansible. A lint pins the rule: a MODE_* read in a truthiness position carries the filter or a nocheck naming a genuine string comparison.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-02 00:07:36 +02:00
kevinveenbirkenbachandClaude Opus 5 6c816e893e refactor(github): group the variant modules and drop the blind-except
variant_bundle_limits.py next to variant_bundles.py formed the prefix group
'variant', which the structure lint reads as a folder in disguise. It stayed
green locally because that lint asks git ls-files: the new file was untracked
through every quality run and only became visible once committed. CI saw it
immediately - the sole failure of run 30675284663.

Both move to utils/github/variant/ as bundles.py and limits.py, and the test
mirrors along.

The BLE001 per-file ignore is removed rather than repointed. parse_size raises
humanfriendly.InvalidSize for a malformed string and TypeError only for a
non-string, and the value reaching it is always str - so the catch-all was
never needed. Measured, not assumed.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-01 02:46:36 +02:00
kevinveenbirkenbachandClaude Opus 5 0eb04b2239 feat(ci): retrigger failed roles on the distros the source run swept
The retrigger sent only whitelist and priority, so entry-manual.yml applied its
own distros default of 'debian'. A run that swept five distros came back as one,
and a failure that only reproduces on centos returned green with nothing fixed.

The dispatch inputs are not recoverable from the API - it answers inputs: null
for a finished workflow_dispatch - so they are read back from the run title.
The title format is not copied here: run_name.py parses the workflow's own
run-name declaration and derives the literals framing an input, and the tests
build their titles through the same helper instead of spelling one out.

Known gap: a run from another entry point carries an unrelated title and yields
no distros, leaving the retrigger on the workflow default. Job step names carry
the list for every entry point and arrive in the same gh call, so that is where
to look when it matters.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-08-01 02:13:31 +02:00
kevinveenbirkenbach b121ed1f8c Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/ai-agent-employees
# Conflicts:
#	tests/unit/utils/storage/test_constrained.py
#	utils/storage/constrained.py
2026-07-31 00:20:16 +02:00
kevinveenbirkenbachandClaude Opus 5 435ff42219 fix(storage): measure the docker root from the caller's namespace
docker info reports DockerRootDir as the DAEMON sees it, and
host_storage_constrained fed that string straight into shutil.disk_usage,
which statvfs()es it in the CALLING process's mount namespace. The two
coincide only when the caller is the daemon's host. In the nested act
container of the workspace jobs they do not, so os.statvfs('/var/lib/docker')
raised FileNotFoundError, matrix.py died, and because 00_one.sh invokes it
bare under set -euo pipefail the whole swarm drill went down with it. All
five workspace jobs of run 30560683608 failed this way, byte-identically,
across five base images and five nested distros.

docker_root_free_bytes() now takes the vantage point as a required keyword:
it measures the daemon's data root when that path resolves here, and the
caller's stated local path when it does not. matrix.py passes "/", which is
the same vantage it already trusts for its ENOSPC abort 270 lines earlier;
overlayfs passes statfs through to the upper filesystem, so a container of a
daemon measures that daemon's data-root filesystem through its own rootfs.

Nothing is caught and nothing is guessed. An unreachable daemon still raises
RuntimeError out of docker_data_root(), which is the guard this module exists
for, and local_vantage carries no default because only the caller knows where
it is running.

The compose-side twin loses its || true and its hardcoded /var/lib/docker
substitution, so an unresolvable root no longer measures a guessed filesystem.
Its compose.exec locality was already correct and is unchanged.

Verified against the pre-fix module under the CI condition: with the data root
stubbed to a path absent from this namespace, the old code raises
FileNotFoundError and the new code measures the local vantage; with the daemon
stubbed unreachable, the new code still raises RuntimeError.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 23:59:09 +02:00
kevinveenbirkenbach b8df33e065 Merge branch 'feature/swarm-nfs' into feature/ai-agent-employees 2026-07-30 17:25:30 +02:00
kevinveenbirkenbachandClaude Opus 5 f4a713d827 perf(registry): probe the image mirror before Docker Hub and cache sizes
The min_storage gate measures every declared pin on each run, and every measurement was an anonymous manifest request to Docker Hub, which meters those per IP. A full run exhausted the pull budget, so the following local deploy could no longer pull at all and the gate itself stopped measuring: 96 of 165 images answered indeterminately and only warned.

Read the GHCR mirror first and fall back to the upstream registry only for images it does not carry. The copy is byte-identical, so the measured size is unchanged. Persist measured sizes under the temporary directory; store positive results only, because a cached indeterminate answer would be remembered as fact and the gate would stop measuring for good.

Lift the mirror path formula out of the provider into utils so the writer and the reader share one definition, resolving the namespace from GITHUB_REPOSITORY, then remote.pushDefault, then origin. pushDefault has to win: origin may point at an upstream that hosts no mirror. Parse .git/config by hand because configparser rejects the duplicate keys git writes and the test container ships no git binary.

Measured over all 165 declared pins: no requests to Docker Hub, 277 to the mirror, 9 to registries it does not carry, and 164 of 165 images measured.

Leave manifest_exists on the upstream registry. It answers whether the declared pin is still pullable from where the deploy pulls it, which a mirrored copy cannot attest; it takes the cache only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 17:22:29 +02:00
kevinveenbirkenbach f39073c95c Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/ai-agent-employees
# Conflicts:
#	roles/web-app-flowise/meta/services.yml
2026-07-30 15:39:29 +02:00
kevinveenbirkenbachandClaude Opus 5 c93f2583ef feat(storage): derive STORAGE_CONSTRAINED from the declared min_storage sum
The flag was decided by a fixed free-space threshold (100 GiB by default), which fired on roomy hosts and stayed silent on a 145 GiB CI runner that a single deploy round could fill. It now compares free space on Docker's data root against what the deploy declares it needs: the summed min_storage of the selected applications and their transitive dependencies.

The need is variant-aware. utils/storage/constrained.py overlays each app's meta/variants.yml entry before walking the service registry, so a round that trims services down is measured at its own size instead of the base configuration's.

Both entry paths pass their per-round variant map: cli/administration/deploy/development/init.py decides inside the round loop (compose and host share this via ci/one.sh -> dedicated.sh), and utils/tests/swarm/matrix.py feeds the decision into the swarm host_vars payload. --threshold-gib is gone; --force-storage-constrained still overrides.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-30 15:08:51 +02:00
kevinveenbirkenbach cf6e7f75e9 Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/ai-agent-employees 2026-07-29 23:29:03 +02:00
kevinveenbirkenbachandClaude Opus 5 3257376fce refactor(ci): resolve the dev-runtime matrix through a named module
The workspace workflow asked the distro SPOT two questions through embedded python one-liners: the matrix image list, and the distro behind a matrix entry. Give both a name -- distro_of_dev_runtime_image() as the counterpart to dev_runtime_image(), and a cli.meta.ci.dev_runtime module shaped like its image_ref neighbour. The empty-SPOT guard that protected the matrix from silently skipping every job moves along with it. No caller yet; the workflow switches over in the next commit.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 22:47:30 +02:00
kevinveenbirkenbach d7551061bf Merge remote-tracking branch 'fork/feature/swarm-nfs' into feature/ai-agent-employees
# Conflicts:
#	scripts/tests/deploy/swarm/routine/01_bootstrap.sh
#	scripts/tests/deploy/swarm/utils/clean/teardown.sh
2026-07-29 15:59:54 +02:00
kevinveenbirkenbachandClaude Opus 5 f482b979bd refactor(ci): rename env/guide jobs, drop roles_only
test-environment.yml and test-guide.yml become test-workspace.yml and
test-instructions.yml. Orchestrator job ids, the distros-SPOT matrix
lint, the act example, the diagnostics slug and the rescue artefact
names follow.

Replace the roles_only input with per-job workspace/instructions
tri-state inputs (auto|true|false). 'auto' derives the former
roles_only skip straight from whitelist: both jobs run on a global
scope (whitelist empty or __ALL__) and are skipped on a role-scoped
run, which keeps the PR behaviour identical because the diff resolver
never emits an empty whitelist. entry-manual exposes both as choice
dropdowns defaulting to false, so a manual run opts in explicitly.
The now redundant roles_only output is dropped from
pr_affected_roles.sh and subset_roles.py.

Branch-protection required-status-check names change along with the
job renames.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-29 12:55:04 +02:00
kevinveenbirkenbach df502c4acb Merge branch 'feature/swarm-nfs' into feature/ai-agent-employees 2026-07-29 00:19:06 +02:00
kevinveenbirkenbachandClaude Opus 5 5666616bbd fix(ci): wipe the inner Docker root as root
The nested dockerd writes /mnt/docker as root, so shutil.rmtree from the runner user silently removed nothing and the previous distro's containers survived into the next distro run. They restarted under unless-stopped against a fresh container rootfs, failed with exit 127, and sys-ctl-rpr-container-soft exited 2 because their compose working_dir no longer existed, which failed the guide deploy on the second distro.

Escalate the wipe the same way dedicated.sh does and hard-fail when the root survives, so a silent no-op can no longer leak state across distros.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 23:53:43 +02:00
kevinveenbirkenbach dd72b7222d Merge branch 'feature/swarm-nfs' into feature/ai-agent-employees 2026-07-28 21:09:23 +02:00
kevinveenbirkenbachandClaude Opus 5 dce60df815 feat(ci): replay the guide on every distro
guide_select picked the role, the mode and the distro in one draw, so a run proved the README instructions worked on exactly one randomly chosen distribution. The distro list was also a literal in the workflow, which meant the manual-CI distros input never reached the guide at all.

The distro draw is removed. guide_select still picks the role and derives the mode, while the shared loop from 1bbe7a0eb drives the distro dimension, and ci-orchestrator forwards its distros input.

The four mode-specific steps collapse into guide/one.sh, which the loop runs once per distro: it renders the pkgmgr base image for the current distro, then either execs host_deploy.sh or brings the compose server up and replays the README block against it.

one.sh releases the compose server in an EXIT trap. Without it the second distro would start against the stack the first one left behind. Rescue diagnostics gain a distro segment for the same reason the swarm collectors did.

The job timeout goes from 90 to 345 minutes because up to five rounds now have to fit; the loop's own budget still decides how many actually run.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 21:06:47 +02:00
kevinveenbirkenbach fa937423c8 Merge branch 'feature/swarm-nfs' into feature/ai-agent-employees 2026-07-28 17:05:24 +02:00
kevinveenbirkenbachandClaude Opus 5 8aba198aec feat(worktree): isolated per-branch checkouts sharing one cache stack
Parallel checkouts collided on the compose subnet, host ports and container
names, so a quality run in a worktree died at compose-up. Slot each checkout
via custom.env and layer two overrides on top: the shared cache network in
place of a second cache stack, and the shared .git directory without which
git aborts inside the container.

Both overlays gate on the resource they need, not on the slot number, so a
stray INFINITO_INSTANCE cannot make compose demand a value nobody set.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 16:15:59 +02:00
kevinveenbirkenbachandClaude Opus 5 ebd7d2e72b refactor(meta): centralize category/distro SPOTs
categories.yml moves from roles/ to the repository-root meta/, joining packages.yml. utils/roles/categories.py owns its location plus the tree-walking helpers that were stranded in entity/name.py, so no caller spells the path or duplicates the lookup.

meta/distros.yml becomes the SPOT for every distribution: os_family, Galaxy platform name, dev-environment base image, and the pkgmgr/pkgmgr-virgin/environment reference templates. It replaces seven scattered copies (DISTRO_FAMILY, VALID_DISTROS, _DISTROS, _CANONICAL_PLATFORMS, two shell arrays, a Makefile literal). INFINITO_DISTROS is no longer a static default but generated by utils/env/handlers/infinito/distros.py; the shell consumers read it through the env layer.

Drops dead CLI surface found along the way: --categories-file and --roles-dir on the provision CLI plus three unused parameters of generate_dynamic_inventory, and the positional roles_file of the invokable categories command.

tests/lint/repository/test_distros_spot.py pins the copies that cannot call the SPOT (workflow literals, docs table) and forbids composing a pkgmgr reference outside it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
2026-07-28 16:15:56 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0ac7872e56 style(deploy): apply autoformat to the deploy round handler
Import ordering and one comprehension reflowed by make autoformat. No
behaviour change; kept out of the orphan-prune commit so that diff stays
readable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 11:29:47 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f62d432873 fix(deploy): apply the disable orphan-prune on the deploy side too
init.py prunes the include set twice: it drops the roles named by disable
and then re-walks the dependency graph so providers that only existed to
serve a disabled role are dropped as well. deploy/cli.py filtered only
the named roles, so its deploy-id set stayed a superset of what the
inventory actually contains.

With disable=matomo,dashboard,prometheus,email,css the init side pruned
svc-db-redis, svc-db-mariadb, web-svc-cdn, web-svc-file, web-svc-asset
and web-svc-simpleicons; the deploy side still passed them to
validate_application_ids, which aborted the run with 'not present in
inventory' before any role deployed.

Both sides now call prune_orphans_after_disable with the same
round-merged services map, so the sets match. Document the two-sided
contract in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-27 11:29:47 +02:00
kevinveenbirkenbach 87dce12024 Merge branch 'feature/swarm-nfs' into feature/ai-agent-employees 2026-07-27 04:45:47 +02:00
kevinveenbirkenbachandClaude Fable 5 eacaa4a53b fix(ci): honor tests.yml skip in guide role pool
guide_select built its random pool from invokable + lifecycle only, so
the Guide job could pick roles whose meta/tests.yml declares every
deploy mode untestable (svc-storage-nfs-client, which hard-asserts a
reachable NFS server). Filter the pool by the role's declared skip for
its guide mode; a malformed tests.yml now fails selection loudly
instead of being silently ignored.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
2026-07-27 04:42:31 +02:00
kevinveenbirkenbach a3ef5c835d Merge branch 'feature/swarm-nfs' into feature/ai-agent-employees 2026-07-25 22:18:09 +02:00
kevinveenbirkenbachandClaude Opus 4.8 c9bbc33654 style(deploy): apply autoformat to the deploy round handler
Import ordering and one comprehension reflowed by make autoformat. No
behaviour change; kept out of the orphan-prune commit so that diff stays
readable.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:26:12 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5cf0b1c543 feat(readme): enforce consistent Credits attribution
A person credited with a link in one place and a bare name in another
reads as two different people and drops the reader's path to them.
sys-svc-engine showed it in two adjacent lines: the same name once bare,
once linked.

The new credits-attribution lint requires a name linked in any Credits
section to carry that link everywhere, and rejects one name pointing at
two URLs. It found 13 offenders, including a second contributor whose
attribution had drifted in test-e2e-cli.

The generator produced the defect on every run: its context carried
application_author but never application_author_url, so the template
always took the unlinked branch. Existing READMEs kept their correct
Credits because the generator only fills missing sections, which is why
239 roles were right and 12 wrong.

The name to URL map is derived rather than declared: galaxy_info.company
names the project owner, not the role author (web-app-mattermost is
authored by Alejandro Roman Ibanez but lists Kevin Veen-Birkenbach as
company), so only the Credits lines themselves carry the mapping.
Generator and lint share that derivation and cannot drift apart, and a
new contributor needs one linked credit rather than a code change.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 20:18:06 +02:00
kevinveenbirkenbachandClaude Opus 4.8 5d59d72afe fix(deploy): apply the disable orphan-prune on the deploy side too
init.py prunes the include set twice: it drops the roles named by disable
and then re-walks the dependency graph so providers that only existed to
serve a disabled role are dropped as well. deploy/cli.py filtered only
the named roles, so its deploy-id set stayed a superset of what the
inventory actually contains.

With disable=matomo,dashboard,prometheus,email,css the init side pruned
svc-db-redis, svc-db-mariadb, web-svc-cdn, web-svc-file, web-svc-asset
and web-svc-simpleicons; the deploy side still passed them to
validate_application_ids, which aborted the run with 'not present in
inventory' before any role deployed.

Both sides now call prune_orphans_after_disable with the same
round-merged services map, so the sets match. Document the two-sided
contract in the README.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-25 18:13:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 68c0ffa84e feat(ci-trigger): add --strict to retrigger only hard failures, plus i8cifo/i8pufo
The --failed selection treats anything not green (cancelled, timed-out, still-running) as a failure, so retriggering a partially-running or aborted run re-fires jobs that never actually failed. Add --failed --strict to select only roles with a hard failure conclusion, excluding cancelled/aborted and running. New aliases i8cifo (strict base, mirrors i8cifail) and i8pufo (sign-push then strict retrigger, mirrors i8pufail); i8pufail is unchanged. Verified against a live run: --failed matched 48 roles (8 failed + 40 still-running), --strict matched exactly the 8 hard failures.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-25 12:44:03 +02:00
kevinveenbirkenbachandClaude Opus 4.8 f4fa8adf16 feat(deploy): prune orphaned transitive deps on disable
disable= now drops providers left with no surviving parent, not only
the named provider roles. Compose (init) and swarm (derive_includes)
share one reachability cut: BFS from the surviving primaries over
dependency + service edges with disabled roles as cut-nodes. No-disable
deploys stay byte-identical.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-24 20:05:59 +02:00
kevinveenbirkenbachandClaude Opus 4.8 d9a53a0545 chore(lint): pin ruff to 0.16.0 and resolve the rules that version enforces
The ruff toolchain was unpinned, so host venvs, the lint container and CI
installed different releases over time. That drift is why lint-python only
failed on CI: locally ruff 0.15.x preview-gated ISC004/FURB192 and (falsely)
fired S310 on urllib.request.Request, while CI's 0.16.0 has ISC004/FURB192
stable and drops the S310-on-Request false positive.

Pin ruff==0.16.0 in pyproject.toml's dev extras (Dependabot keeps it current)
and have utils.install.lint.ruff read that pin and reinstall whenever the
found version drifts, so every environment resolves the same ruff. Then fix
what 0.16.0 flags: parenthesise the implicit string concatenations (ISC004),
use min() over sorted()[0] (FURB192), drop the now-unused S310 noqa on the
Request() builders (RUF100), and delete the narration comments that the
comment lint surfaces in the touched files.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 23:08:26 +02:00
kevinveenbirkenbachandClaude Opus 4.8 373795d84e style: apply autoformat wrapping to two long literals
Formatter output from make quality: the git ls-tree argument list in
complexity.model and the xwiki job-name dict in the report_failures test
wrapped across lines. No semantic change.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 0771c55103 refactor(ci): route new-role and failure-report reads through the cached reader
report_failures.decisive_excerpt and new_in_branch._declares_application_id
now read via utils.cache.files.read_text instead of Path.read_text, and the
new_in_branch test constructs its vars path from ROLE_FILE_VARS_MAIN. Clears
the uncached-read and role-file-spot lint gates; behaviour is unchanged
(same UTF-8 decode, same exceptions, same resolved path).

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 8b58b67121 feat(ci): open or append a per-role issue when a role deploy fails on main
On a main run, report-main-failures parses the failed deploy jobs, downloads
each role's rescue-diagnostics artifact, and opens a 'CI failure: <role>' issue
(label ci-failure) carrying the run link, the artifact names, and the decisive
rescue excerpt inline. An existing open issue for the role gets a comment
instead. Mode emojis are resolved through the symbol glossary, not hardcoded.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 a3f4c1869c refactor(ci): sort clones last in the swarm matrix too
The compose and host discovery sorts already push clones (redundant
same-service-set variants) behind the budget cut so one representative
per dna cluster stays covered. Extend the same 'asc clone' prefix to the
swarm mode so its per-variant matrix drops redundant clone variants first
when the job budget is tight.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6b71d3e8bb feat(ci): deploy branch-new roles first on the push priority line
A push resolves the application roles that exist under roles/ locally but
not at origin/main and passes them as the orchestrator priority line, so
brand-new roles deploy and gate ahead of the regular matrix. The resolver
is stdlib-only (git ls-tree plus an application_id line scan) so it runs
on a bare runner without installing the resolver stack, and yields an
empty set when the baseline ref cannot be resolved.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:32 +02:00
kevinveenbirkenbachandClaude Opus 4.8 6651450894 feat(complexity): mark whether each role exists on origin/main
Add an in_main column (🌳) to the role complexity matrix: it resolves
the role directories present under roles/ at origin/main once and flags
every row true/false, so a role that is new on the current branch stands
out. When the ref cannot be resolved (a non-git checkout or an unfetched
main) the marker falls back to present for every role rather than
falsely reporting them new.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
2026-07-23 20:01:32 +02:00