The image ships entrypoint=null and cmd=["/mattermost/bin/mattermost"]. Invoked bare, this version prints its help and exits 1, so the container restart-loops with no error of its own in the log and the mmctl healthcheck never runs.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The applications: host_vars block is the config namespace keyed by role id, so it also carries system services (sys-ctl-hlth-csp, sys-svc-container). Those declare system_service_id and match no application role, and the resolver died on them instead of reporting their absence.
role_name now raises RoleNotFoundError and exits 3, distinct from argparse's usage error (2) and from a genuine failure (1). run_subprocess grew ok_returncodes so a caller can accept an expected non-zero status while every other failure still surfaces, and resolve_role_path returns None for that code.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The negative control probed tools/list, which gitea answers with 404 "Invalid session ID" when no session exists - indistinguishable from a real refusal, while the same server serves every caller that runs initialize first. An open endpoint therefore passed as guarded. initialize is the only method that needs no session, so it reaches the auth layer.
A 4xx now refuses on its own regardless of the body; previously a clear HTTP refusal without a JSON-RPC body was classed unreachable and kept the caller retrying. 5xx stays retryable.
The unauthenticated handshake no longer leaks its session id into the authenticated pass.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CONFIG_INPUTS lost modes, sequencing and mode_fail_fast when the deploy
modes collapsed into one chunked sweep, but two mirrors of it did not
follow.
The --run help text enumerates what a retrigger carries over, so it was
telling the operator about three inputs the dispatch no longer has. The
job-log fixture described a run recording mode_fail_fast, a state the
dispatch can no longer produce, which left the log half of
config_from_run asserting on an input nobody writes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A workflow_dispatch input declared type: number reaches the calling
expression as a string. Handing that to a workflow_call input declared
number is a type mismatch, and GitHub rejects the job before it is
created -- no failed job, no log, just a run that stops after
pick-distro.
Run 31863925900 died exactly there. The push entry on the same sha
(31863720155) built all 33 jobs including the orchestrator, so the
callee chain was never the problem: only the manual entry passes
resume_from_chunk, and it was the sole type-risky input among the four
it adds.
Both sides become string with default "0". The comparisons are
unaffected -- GitHub casts a string operand to a number, so
'inputs.resume_from_chunk <= 1' in the chunk gates and '!= 0' in the
run-name keep their meaning. No type: number input remains in any
workflow.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The ruff bump in 6320da703 changed how the formatter picks quote style
inside nested strings and where it wraps a generator argument. Both
files were written before that landed and this is the first autoformat
pass since, so make quality rewrote them.
Formatting only; no assertion, message or rule changes.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Replaces the deploy-line sequencing section, which documented the
three-mode split, the serial/parallel decision and mode_fail_fast --
the only place any of that was written down.
Records what a run now does instead: why chunks exist at all (the queue
clock), how the chunk size and count are derived, what priority means
now that a named role is deployed in every combination it can take
rather than sampled, and how the mode and tor rotations are decoupled
for the regular rows.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
report_failures matched job names against a raw role id
((?:web-app|web-svc|svc|sys)-[a-z0-9-]+), but job titles have carried
display names for a while, so the regex matched nothing and every failed
deploy on main went unreported. runs.py had the same shape and stopped
at the first mode glyph, which swallowed a caller path like
'z / Host / sys-front-proxy' into the role name.
Both now go through axes.parse_label, kept next to the function that
builds the label so the two cannot drift, and resolve the middle through
the display-name codec. The parser anchors on the LAST mode glyph so a
reusable-workflow caller path carrying one of its own cannot capture the
role.
The onion state is part of what identifies a job, because a priority row
runs the same mode and variant twice; artifact_name takes it so the
reporter looks for the artifact the run actually uploaded.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
call-test-deploy-{compose,host,swarm,single-node}.yml differed only in
which mode they pinned and which artifact prefix they wrote; compose and
host already ran the identical drill. They become one call-test-deploy.yml
whose matrix rows carry the mode, so the swarm-specific steps are guarded
on it and everything else is shared.
The orchestrator loses its eight deploy jobs and the sequencing job for
three serial chunk blocks. GitHub Actions cannot generate a variable
number of jobs, so the blocks are written out and INFINITO_CI_MAX_CHUNKS
tells the budget how many exist; blocks whose slice is empty skip
themselves. chunk_gate stops the chain at the first red chunk,
resume_from_chunk re-enters after a fix instead of re-running the green
chunks, and modes collapses to a single mode choice.
apps.sh now emits matrix entries for one chunk rather than a token list
for one mode, and no longer sorts them: the discovery order is what the
chunker slices on. split_runner_apps.sh reads its runner quota from
INFINITO_CI_CONCURRENCY instead of carrying a second copy of the 20.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
One discovery query per run replaces one per mode, so the row basis, the
ranking and the budget cut all speak the same role#variant unit. The
result is deployed in serial chunk blocks rather than three parallel
mode matrices.
Chunk size is derived, not pinned: GitHub cancels a job queued for
INFINITO_CI_QUEUE_HOURS, so with INFINITO_CI_CONCURRENCY runners free a
chunk holds concurrency * floor(queue_hours / the deploy job's
timeout-minutes) rows. Assuming every job burns its full timeout is
deliberately pessimistic -- no job can outlast it -- so the drain
estimate can only overshoot.
The run job cap bounds the sweep as a whole; rows beyond it roll into
the next sweep, whose regular line starts at an offset so consecutive
sweeps walk the whole list instead of re-testing the same head. Priority
rows lead every sweep, never move with that offset, and force a chunk
boundary at the seam so a chunk is either all priority or all regular.
sequencing.py goes with the mode lines it ordered: chunking restarts the
queue clock at a controlled size, which is what serialising the modes
was approximating with three buckets of arbitrary size.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant bundling packed several variants onto one runner and made the
job count depend on storage sums. Every variant now gets its own matrix
row instead, and the two things that used to be fixed per workflow --
which deploy mode a row runs in, and whether it sits behind the node
onion -- become axes assigned per row.
Both axes are a deterministic rotation over the row's position and the
sweep number, never random, so a red job reproduces by re-running the
same sweep. Tor turns on sweep//2 so it does not flip in lockstep with
the mode: a row walks all four mode/tor combinations over four sweeps
instead of only two.
A role offers at most two modes in practice, since swarm requires its
own stack and host requires the absence of one.
Drops with the bundler: limits.py, the variant_bundle env handlers,
INFINITO_VARIANT_BUNDLE_SIZE/_MAX_STORAGE, the per-role
meta/tests.yml variant_bundle_size key and its only user, and the
ComplexityRow bundles/jobs fields with the --deploy-mode and --max-jobs
flags that consumed them.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The repository env lint requires ascending keys in every env file, and
test.env.j2 listed TOR_SOCKS ahead of TOR_DNSMASQ_CONF, so make quality
stopped there. Order carries no meaning in a dotenv template: Jinja
resolves the values from the role vars, not from preceding lines.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Job 94849355639 of run 31823845576 failed on exactly one name: the bare apex
infinito.example, HTTP 000, "Could not resolve host", while
espo.crm.infinito.example answered 200 in the same second through the same
resolver. The probe printed dns= and stopped there, so the log named the
symptom and no artifact carried the cause. There is no dnsmasq log
(log-queries is off, the journal has zero dnsmasq lines), no CoreDNS log in
the compose artifact set, and no query output from the host at failure time.
Reproducing the resolver stack locally did not close it either. CoreDNS on the
CI Corefile, dnsmasq on the drop-in pulled byte-identical from that job's own
--diff at 20:58:35, and the full dnsmasq -> 127.0.0.11 -> CoreDNS chain with a
live forward path all answer the apex with 172.30.0.10. The configuration is
exonerated; what is missing is the transaction itself.
So the failure branch now prints the active resolver, any /etc/hosts match,
what the local dnsmasq answers, and what each upstream answers. The last two
are the split that matters: they separate "dnsmasq dropped it" from "upstream
dropped it", which is the fork the bench could not settle. Upstreams are read
from the drop-in directory rather than one file, so a shared dnsmasq carrying
foreign drop-ins is covered, and matching only bare-IP server= lines skips
server=/onion/127.0.0.1#9053.
TOR_DNSMASQ_CONF arrives through test.env instead of a hardcoded
/etc/dnsmasq.d, using the same :? idiom 8d7fff4d2 gave the other four values,
so the path keeps its single source in vars/main.yml. dig is opportunistic:
without bind-tools the resolver and hosts lines still land.
Verified: bash -n and shellcheck clean, the failure path exercised end to end
against a CI-shaped drop-in with 127.0.0.11 extracted and the /onion/ line
correctly skipped, test_comments_valid passed.
Two things this does not close. dig is probably absent on the Arch target that
failed, since the bind9-dnsutils lines in that job come from a Debian build
stage, not the target; adding bind-tools to the role would fix that and is a
separate concern. And the trigger is now rare, because pinning tor true across
the espocrm variants removed the mixed round that put the clearnet apex into
the vhost list at all. This is insurance, not a plan that pays out next run.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Three rounds of run 31823845576 died at 356 minutes against the 355-minute cap
in call-test-deploy-swarm.yml: variants 0, 2 and 3, with 214.9, 214.0 and 202.5
minutes in the sync pass alone and no async pass reached. The role-runtimes
artifacts put the boundary between 183 sync (variant 4, 303 total, survived) and
202.5 (cancelled), so a round has to come in under roughly 185.
Each of the three carried more than it could finish. Variant 0 is the baseline
and had bigbluebutton on top of every tightly-coupled service; variant 2 paired
openwebui with flowise; variant 3 carried openproject, mattermost and matrix.
Splitting them leaves one partner each, and the freed partners fill three new
rounds.
bigbluebutton is the lever for variant 0 because f1dff50fb already prepared it:
that commit dropped its bond to 0.5 and noted "bigbluebutton stays in variant 0
for now, by explicit request; its bond change only makes the move possible
later." Everything else in variant 0 is bond >= 1 and would need redeclaring
first.
The xwiki split 6fcbffe08 merged away comes back here without costing a variant
of its own: xwiki takes flowise along, which variant 2 had just given up. sso and
tor stay on everywhere except variant 1, the minimal round, and variant 8, which
pairs the clearnet-only PeerTube.
Verified: tests/integration/roles/meta/variants plus the role-meta lints, 39
passed. Every variant resolves between 18.26GB and 27.77GB reservation against
the 32GB cap. The planner confirms the split is clean - round 2 pulls
web-app-openwebui without web-app-flowise, round 9 pulls flowise and xwiki
without openwebui, and neither role references the other in meta/services.yml.
Costs three more jobs in compose and in swarm, measured 9 -> 12 each. The slot
pool is fixed, so they displace three lower-priority rows per mode rather than
growing the run. host is unaffected: test_host is false for this role because it
ships a stack.
Not measured, and the reason to expect less than the arithmetic suggests: the
round closures sit at 21-24 roles before and after, because the shared base -
keycloak, mailu, matomo, dashboard, prometheus, the database engines, tor -
dominates the partner. Whether dropping one partner moves 214 sync under 185 is
open, and every measurement here predates 8f98e1648, which put the onion stack
into every round.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
72fd9c870 required the onion in every variant but exempted variant 1, on the
argument that the minimal round is the one place the stack runs without it. That
argument does not hold: the onion is part of the deployment under test, not a
feature bolted onto the full shape. A round that skips it is not a smaller
deployment, it is a different one.
So the exemption is gone and 95 variant-1 entries pin the onion on. Two
exclusions remain, both read from data rather than a comment: a role with no tor
block has no onion to pin, and a role whose meta/services.yml pins tor.enabled to
a literal false has already declared it has none.
That put the rule in conflict with test_coverage, which requires both polarities
of every dynamic flag: with tor on everywhere, the false side has no variant left
to live in. 94 roles take the exemption that contract already provides for
exactly this - "flags whose group-derived value is a hard invariant that variants
must never override", the same one 4e6ef2542 used for the seaweedfs frontend.
web-app-nextcloud needs none: its variant 8 pairs the clearnet-only PeerTube and
keeps the false polarity.
Measured, because the declaration is not what CI runs. assign_tor gives the onion
to every second capable entry with the run number as the parity switch, so more
capable variants do not mean proportionally more onion jobs: across five roles
and 21 entries the count goes 8 -> 10 on an even run and 9 on an odd one. The
clearnet path also survives - checkmk reports v1=false on run 0 and v1=true on
run 1, so both states are still covered across two consecutive runs. What the
94 markers give up is a declarative guarantee in variants.yml, not the coverage
itself.
The real cost is weight, not job count. web-app-nextcloud variant 1 goes from
3.09GB reservation / 7.27GB limit to 18.26GB / 43.01GB once svc-net-tor and its
closure join, so the cheap round is no longer cheap. Every variant stays under
the 32/64GB budget, and the same effect applies to the other 94 unmeasured.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles/meta/
variants plus the role-meta lints 39 passed. The rule was checked in both
directions - it reported the variant-1 offenders before the flips and none after.
Not measured: what the heavier rounds do to wall-clock, against swarm jobs capped
at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
7a70d27b7 made a variant that switches SSO on also switch the onion on, because
that pairing is where the platform breaks in ways clearnet never shows. The same
argument reaches further than SSO: a Secure cookie is dropped, an https identity
provider refuses to embed, and crypto.subtle is undefined for any feature on a
non-secure origin, not only for a login. A variant that never runs the onion is
not testing the deployment users get.
So the rule now covers every variant of a role that declares a tor block.
Variant 1 stays exempt - it is the minimal, partner-free round every role
declares and the one place the stack is exercised without the onion at all - and
a role whose meta/services.yml pins tor.enabled to a literal false is skipped
outright, because that declaration already says the role has no onion and needs
no comment repeating it.
That found 137 variants. 101 were variant 1. Of the rest, 28 are the LDAP-only
auth-isolation rounds, which gain real coverage from running LDAP over the onion
and are flipped on here. Four cannot: svc-net-tor is the provider itself, and
nextcloud's variant 8 pairs the clearnet-only PeerTube. Both carry a marker
naming the reason; opentalk and coturn need none, since their services.yml pins
the literal false the rule now reads.
Flipping them exposed a contract that contradicted itself. test_auth_isolation
and test_non_baseline_explicit_disables demanded `shared: false` beside
`enabled: false`, while variants-static-override flagged that same line as a
static override - which is why all 133 of them carried a nocheck marker. The
demand only makes sense where `shared` is dynamic: a literal base value cannot
leak through deep-merge, so restating it protects nothing. Both contracts now
ask for it only in that case, and the 133 lines are gone.
The result is a stricter rule with fewer lines: 189 insertions against 193
deletions.
Verified: tests/lint 336 passed / 307 subtests, tests/integration/roles 107
passed / 610 subtests. The rule is checked in both directions - it reported 137
offenders before the resolution and none after.
Not measured: 28 additional onion rounds cost matrix wall-clock, against swarm
jobs already capped at 355 minutes and one that has hit that ceiling before.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Every variants.yml header states the rule the bond feeds: variant 0 carries each
tightly-coupled (bond>=1, same-host) service, while loosely-coupled (bond<1,
separate-host) partners are excluded from the closure and distributed across
later variants to keep every host under budget. tor was declared bond: 1 in all
101 consumers, so it had to sit in every variant-0 closure.
It does not belong there. The onion is a separate-host provider reached over the
node's hidden service, the same shape as the bridged partners that already carry
0.5, and f1dff50fb established both the reasoning and the precedent when it
dropped xwiki and bigbluebutton from 1 to 0.5: bridged partners with their own
role and their own spec, whose bond declaration was the outlier rather than
their placement.
This also unblocks the variant-level tor contract. Requiring a non-baseline
variant to pin the onion on while tor still counted as tightly coupled would
have pulled variant 0's closure requirements along with it.
Verified: tests/lint 336 passed / 307 subtests and tests/integration/roles 107
passed / 610 subtests, including test_resource_budget over every variant's
deduplicated footprint and the bond-aware test_order closure.
Not measured, and the honest cost: f1dff50fb's own caveat applies unchanged
here. Whether redistributing a service out of the baseline closure buys back
wall-clock is not established; role-runtimes has the per-role seconds and no run
has been made with this shape.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A deployment that keeps the shipped DOMAIN_PRIMARY serves names under
infinito.localhost. Those exist in no zone - coredns only knows the outer
stack's domain - so anything reaching the deployment's own domain from the host
depends on the myhostname NSS module, which maps *.localhost to loopback per RFC
6761. 3ebc4d152 established that and titled it "on every target", but the role
carrying it is reachable from exactly one place, sys-svc-mail, so a deploy
without mail never gets the guarantee.
web-app-bigbluebutton is where that surfaces. Job 94846729503 of run
31823654078 replays the role's README per distro; fedora and arch answered its
Greenlight gate with 200, and debian failed 30 attempts with elapsed: 0 and
"<urlopen error [Errno -2] Name or service not known>". In that distro's window
sys-hostname does not appear at all - the play reaches sys-stk-front-proxy at
18:27:45 and the gate at 18:30:06, both before anything pulls the role in. The
asymmetry is declared, not incidental: meta/packages.yml gives Debian
libnss-myhostname and both other families an empty list, because there the
module ships inside systemd.
The front proxy is the chokepoint every domain-serving role passes through, and
it stands up two domain probes of its own, so the guarantee belongs in front of
it. Including sys-hostname there would have been wrong: half that role is
gated only on DOCKER_IN_CONTAINER, so on a production target it would newly run
ansible.builtin.hostname on every domain-serving deploy and rename hosts whose
inventory entry is an alias. CI cannot show this - it always runs in a
container, where exactly those tasks skip.
So the resolution half moves to sys-svc-nss and sys-hostname includes it,
leaving the mail path unchanged while the front proxy takes only what it needs.
required_by names both callers rather than taking the coverage opt-out
sys-hostname still carries as a TODO.
Verified: tests/lint 335 passed / 307 subtests, tests/integration/roles 107
passed / 610 subtests, make readme-check clean. The role itself is unchanged in
behaviour and was already exercised on all five CI distros in job 94636437319 -
ubuntu and debian install the package, fedora, arch and centos skip it, and the
nsswitch line is a no-op everywhere.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The readiness probe 5bd323999 added for the swarm path dialled the role's public
domain on port 80. wait_for opens a plain TCP socket from the controller, and
once 1d9d355bc gave the domain roles onion variant coverage that target became
a .onion authority, which needs Tor's SOCKS resolver and cannot be reached with
a socket. Variant 6 of run 31823845576 therefore spent the full budget on it -
"Timeout when waiting for lab.git.<onion>:80", elapsed: 600 - while the
preceding resolve_host_cid had already converged, so GitLab itself was up the
whole time.
The repository had already settled this. DOCKER_REACH_HOST
(group_vars/all/00_general.yml:47) is 127.0.0.1, swapped for the Docker bridge
gateway when the controller runs in a container, and every one of the other
eleven wait_for probes under roles/ uses it. The nearest neighbour is the same
task shape - same resolve preamble, same retries and delay - reaching its
partner as `host: DOCKER_REACH_HOST` with `ports.local.http`
(integration_openproject_provision.yml:12-17). This one was written from that
template with the host and port substituted, and the substitution is the defect.
Restoring the idiom keeps the readiness gate 5bd323999 wanted instead of
dropping it, so retries stays at 12.
The lint pins the shape, because the divergence is invisible in review: both
forms read as a plausible readiness probe, and the wrong one only fails where
the domain happens to be unroutable. It parses each wait_for block under roles/,
locates the host key by indentation, and requires DOCKER_REACH_HOST unless the
line carries `# nocheck: wait-for-reach-host` naming what the probe proves - the
opt-out a probe needs when it is verifying the ingress rather than a backend.
Checked against the original line, where it reports
integration_gitlab_provision.yml:14, and against the eleven correct probes,
which it accepts.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Variant 7 pinned sso, tor and peertube together, and since 8ffb1f567 those three
cannot coexist. test_sso_variant_requires_tor forbids sso: true without
tor: true, so tor cannot give way. With tor on, keycloak is published onion-only
- auth.infinito.example does not appear once in the job log - and
align_domain_to_consumer falls through to `return primary_domain` because there
is no clearnet sibling to align to. PeerTube therefore receives
http://auth.<onion>/realms/.../.well-known/openid-configuration, oauth4webapi
rejects it on scheme before any request leaves ("only requests to HTTPS are
allowed", OAUTH_HTTP_REQUEST_FORBIDDEN), the plugin registers without building a
client, and the gate at oidc/enable.yml:105 fails the deploy. Both deploy modes
reproduced it, with different onions, which rules out routing: the SSO
back-channel pin from container_extra_hosts is emitted for peertube and fixes
where a name points, not which scheme a library will accept.
peertube is the only one of the three that can move, so it moves to a variant of
its own with tor off. sso follows tor off, which costs nothing here: the addon
gate is services.peertube.enabled alone and the spec signs in with a direct
password, not through Keycloak. PeerTube keeps its OIDC coverage in its own
variant 0, which is sso-on and clearnet - the topology where the plugin works.
The slot is paid for by folding xwiki into variant 7 rather than adding a tenth
variant. This does not undo f1dff50fb: that commit moved xwiki out of variant 0
because job 94319317296 was cancelled at exactly the 355-minute ceiling, and
variant 0 is untouched here. Nor does it grow a round - variant 7 carried two
partners before (discourse + peertube) and carries two now (discourse + xwiki),
while variant 3 already carries three. Measured: the merged variant 7 resolves to
20.26GB reservation / 48.01GB limit against caps of 32/64, the lightest
non-baseline variant in the file, and the two-partner variant 7 of run
31823845576 spent 101.4 minutes against a 355-minute box.
mariadb and redis stay shared: true in the new variant because
variants-shared-false-once allows exactly one shared: false pin each and variant
1 already holds both.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
67b6072bb tightened this test's assertion from "Skipping distro|budget
exhausted" to the headroom line alone, and in the same change shrank the
command and budget from sleep 3 / 5s to sleep 2 / 3s. That left the branch
reachable only when the first iteration measures exactly 2s.
distros.sh times with `date +%s`, whole seconds, so a measured duration
carries up to a second of boundary rounding on top of bash startup and two
`source scripts/meta/env/load.sh`. Run 31823654078 booked `sleep 2` as 3s,
took the `remaining <= 0` branch instead, and never printed the line the
test greps for:
>>> Duration: distro=debian took 3s (max_seen=3s)
[WARN] Global budget exhausted (remaining=0s). Stopping further distro runs.
It is a race and not a defect in the script: on the same commit manjaro and
centos both passed base/06_test.sh and moved on to 07_console.sh, and only
ubuntu lost it, twice within the one job.
Reaching the branch needs d < B < 2.15*d for a measured first duration d.
At 2 / 3s the window admitted d=2 alone. At 4 / 7s it admits 4, 5 and 6 -
three seconds of slack against the one second CI actually overshot by. The
assertion is unchanged, so the branch is still genuinely exercised rather
than relaxed, and the suite pays two more seconds.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
30% was a guess. Measuring it: across 60 successor distro runs in 22 jobs of run
31756400129, each against the running max of the runs before it, the median
overshoot is 0.991 - a successor is typically slightly faster - and the worst
ever observed is 1.149 (fedora 5541s against a 4822s max_seen).
15% therefore covers every observed case, and nothing above it buys more. It
costs less, too: 30% refuses 23 of 74 start decisions, 15% refuses 18. Five
distro runs that would have fitted, for no additional protection.
The tests read the percentage back out of the log line and check the formula
against it, so they hold at any value.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
6c558e7fe introduced GNOME_EXTENSION_MANAGER_REPO/_REF in both desk-gnome and
desk-gnome-extensions, bound to different lookups - services.gnome.* against
services.gnome-extensions.*. Top-level ALL-CAPS names are project-wide constants
and must resolve to one value, so the same name on two values is what the rule
exists to catch. Prefix each with its role.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
8ffb1f567 pinned the role to clearnet with a literal enabled: false but put the
dynamic-flag nocheck on the shared: line, and the lint reads enabled and shared
separately. The playwright-service-flag nocheck was missing entirely - a service
declared with enabled: needs a <NAME>_SERVICE_ENABLED= line in playwright.env.j2,
which this role cannot have because it never joins the onion topology.
The two READMEs are generated and follow from the same decision: svc-net-tor
leaves PeerTube's dependency graph and the role table.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
508669809 booked a budget SIGKILL as skipped so the job went green. That is a
soft-skip: a kill is indistinguishable from the command hanging, so it hid a
real failure behind a green tick.
Fail on it again, and prevent it upstream instead. A distro now starts only
while the remaining budget covers the slowest run so far plus 30%. Fediwall run
31756400129 shows why the bare comparison was not enough: max_seen=6770s against
remaining=6878s let the distro start, and it took 6942s, 64s past the deadline.
max_seen is the maximum of the durations already observed in the same sweep -
a small sample over distros of genuinely different weight, drawn in randomised
order, so it underestimates systematically.
Removing the soft-skip also removed its diagnosis, so a killed distro now
reports both numbers without claiming a cause: the run only knows the exit code
and how much budget was left when it started.
The suite drops from 85s to 4.1s. The 68s case made a process ignore SIGTERM to
force the -k escalation, which exercised coreutils rather than this script; the
124 path reaches the same branch in a second.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
With AUTH_TYPE=ldap the main firewall gets json_login_ldap and nothing else -
the native branch returns early, so the two are mutually exclusive and no admin
special case survives. The seeded admin keeps its rights but its local password
is never consulted, while the persona kept typing exactly that. Omit the key
when LDAP is on so admin.js falls through to the directory password, matching
the espocrm gate.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The watchdog reads free space every 30s but surfaces it only in the abort
banner, so a job that dies without tripping the floor leaves no disk series at
all - which is exactly the silent-ENOSPC shape the function's own docstring
names. Reuse the reading the guard already takes and put it in the heartbeat,
so the next occurrence is decidable from the job log alone.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
EspoCRM resolves authenticationMethod against metadata files whose keys are
their filenames, and the file is LDAP.json. The value 'Ldap' matched no key,
and the generated fallback class Logins\Ldap does not exist in any casing, so
InjectableFactory raised before any directory I/O - HTTP 500 in 37ms on every
authenticated request of the LDAP-only variant.
Making LDAP resolve exposes a second break: the directory bind now really runs,
while the shared persona prefers ADMIN_NATIVE_PASSWORD, EspoCRM's app-local
secret, over the directory password. Omit that key when LDAP is on so admin.js
falls through to ADMIN_PASSWORD.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eight roles switch from pkgmgr-install to sys-pip-install with global_install: true, so the console scripts land on the system PATH where systemd looks for them.
btrfs-auto-balancer, certreap, certbundle and unit-suffix-remover come from PyPI by name. The three that are not published yet install from git, with repository and ref declared in meta/services.yml and resolved through vars, following the pkgmgr and web-app-taiga precedent. That satisfies the repository-declaration guard and lets the scheduled Update: Versions workflow read and bump the refs.
Drops a dead guard in svc-opt-swapfile that tested run_once_pkgmgr_install, a fact nothing sets any more.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
gitlab leaves the server-side OIDC reachability list. It was there as a candidate for the container_extra_hosts one-liner, but run 31756560361 shows it resolving the issuer host and completing a TCP connect without any pin (peeraddr=172.30.0.10:443); the failure was the openid_connect gem rebuilding the discovery URL over https regardless of the issuer scheme, fixed in 2389b6e38. Left as written the entry sends the next reader down a path this run already closed. mobilizon joins it, having received the lookup in 0e39f3fad after OAuth2.Error Connection refused twenty-four times.
espocrm leaves the baseline exclusions. Its onion variant failed in the shared admin persona, fixed in b97cf8888; its clearnet variant returns HTTP 500 on GET /api/v1/App/user with the cause still open. Neither is out of scope any more.
The environment entry keeps its verdict for the 4h30m job cap and gains the boundary to the distro sweep budget, which is a different clock and did have a code fix in 508669809. The two were described interchangeably enough that I mixed them up myself while reporting.
The section header claimed completeness against a run and commit that are now two runs and twelve commits old.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
PeerTube distributes video over WebRTC and the role proxies its tracker socket on purpose (templates/proxy.conf.j2:48, with a raised read timeout because peers ping every fifteen minutes). Tor carries no UDP and an onion client cannot hold peer connections, so an onion PeerTube is the app without the feature it exists for. What remained was plain HTTP video at Tor throughput, and a login path that only worked through a sed over the vendored dist of a third-party plugin.
That patch is gone with this change, along with PEERTUBE_OIDC_ALLOW_INSECURE_ISSUER, the variable that gated it, and both tor pins in meta/variants.yml. The root TODO entry describing it is removed too - it carried a three-step exit path (upstream MR, fork, delete) that no longer needs walking. The https-on-onion entry keeps its analysis and loses only the sentence that named peertube among the per-consumer workarounds.
The discovery gate stays. Failing the deploy when the plugin registered without building a client is not tor-specific and was the honest half of e64ed1cd9.
tor.enabled becomes a literal false rather than the block being deleted. Two lints define the contract: test_tor_contract requires every domain-serving role to declare the block, and test_sso_variant_requires_tor names a literal false as the first of three cases that may opt out of pinning the onion on with a reason. Deleting the block failed both; this is the language they specify.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
ad9222630 wrapped each distro in timeout so a run that outlives the budget stops instead of drifting into the action timeout, and mapped rc 124 to skipped because stopping at the budget is the documented intent rather than a defect. It verified that timeout returns 124 for its own kill. It did not cover the other one: timeout -k escalates to SIGKILL when the command survives SIGTERM, and then reports 137, which is equally its own kill. An ansible run with live docker children survives SIGTERM routinely, so 137 is the common case rather than the exotic one.
The result was that the governor never fired as documented. Fediwall swarm and Nextcloud swarm 0 in run 31756400129 and 31756560361 both ended on [ERROR] Run failed for distro=arch (rc=137) with budget=20400s remaining=-60s. Fediwalls playbook had already printed PLAYBOOK RECAP, the stack had converged and every node port answered; it was booked as a failure 0.3 seconds later.
The two narrowing conditions carry the discrimination and are unchanged. dur >= remaining - 5 rejects a SIGKILL that arrives at an arbitrary time, such as an OOM kill mid-run, because a budget kill lands at the deadline by construction. passed > 0 still requires an earlier distro to have succeeded.
A unit case in tests/unit/scripts/tests/deploy/test_distros.py drives a distro that traps SIGTERM and sleeps. Without the change it reproduces the CI failure locally, exit 137 and Run failed for distro; with it the sweep exits 0 and the summary marks the distro skipped.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sixteen role specs decided something immediately after an action that had not settled, because locator.isVisible() discards the timeout it is handed. Three shapes: an assertion that could never fire, a login outcome read off the DOM before the app re-rendered, and a branch that selected a locator by whether an iframe had mounted yet.
The ones that could never fire are the worst of them. taiga test-seaweedfs polled for a rejection toast right after saveAction.click(), so a real upload rejection was swallowed and misreported as a SeaweedFS storage failure; friendica ldapauth decided whether to run its wrong-password test at the instant a fresh context finished goto, so a slow render skipped a security check silently. wordpress test-discourse-roundtrip picked titleBox by whether the Gutenberg iframe had mounted and, when it had not, bound the wrong one and then burned the full budget failing with a misleading message.
Every conversion keeps the control flow: the .catch(() => false) fallback stays, so a waitFor that throws still yields the boolean the caller expects. Twenty of the twenty-two waits use base 2000, one 3000, one 5000. resolveTimeout multiplies by five on an onion target, so the common case is ten seconds there. Across thirteen onion traces the largest wait of any kind that ever succeeded was 790 ms, so ten seconds is a twelvefold margin; the bases started higher and were pulled down because a swallowed deadline is a duration in disguise - where the optional element is normally absent, the full timeout is paid on every run, including green ones.
Not exercised: no spec was run. Three of these make a test stricter, so a job that turns red on them is surfacing a real failure that was previously hidden, not regressing.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
