refactor(container_ports): express the remaining published ports through the lookup

The list form could publish a host:container pair and nothing else, so thirteen blocks stayed hand-written: the ones needing a protocol suffix, a port range, the swarm long syntax with mode host, or a container-port-only publish. Each of those duplicated numbers the role already declares in meta/services.yml, which is the drift the lookup exists to remove.

The lookup keeps the list form unchanged for its 93 existing call sites and gains a dict form: transport emits one item per protocol, container 'same' serves the services that cannot remap (STUN/TURN, media relays), mode 'host' emits the long block an edge role needs in swarm so the client address survives the hop, publish 'ephemeral' emits the container port alone, and key names the block for Discourse's launcher file, whose expose: list is what becomes docker run -p there. A declared {start, end} mapping renders as start-end on both sides, so a relay range needs no term of its own.

svc-prx-openresty had no ports block at all and bound 80 and 443 as literals in its template, while torrc.j2 carried a third literal 80 for the hidden service. None of the three referenced the others. openresty now declares the ports, the compose template reads them, and the onion's port 80 comes from ports.onion.http through TOR_ONION_EXTRA_PORTS.

Three blocks stay raw, each with the reason it cannot route through one call: keycloak's sso_proxy maps a published port owned by the consuming role to a container port owned by web-app-keycloak, and matrix's ansible flavor takes element on 8080 and federation on 8048 from the upstream playbook rather than from this repo's ports.internal.

The enforcing lint scanned only files with compose in the name and only double-quoted list items, so nine of the thirteen blocks were invisible to it. It now reads every role template that can carry a ports block, catches unquoted items, protocol suffixes, ranges, flow style and the long syntax, and tracks the block by the indentation of its key: keying on the next line closed the block on anything shaped like a key, which target, published, protocol and mode all are. Under expose: only a real mapping counts, because there the single-segment form documents a container port and publishes nothing.

Verified by rendering all 110 container_ports call sites through the real templar, the real lookup and the real merged applications view.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

group_vars/all/08_networks.yml

@@ -82,6 +82,12 @@ PORT_BANDS: # nocheck: unused-var  # consumed by the port-allocation plugin outs
       start: 9053
       end: 9053
   public:
+    http:
+      start: 80
+      end: 80
+    https:
+      start: 443
+      end: 443
     ssh:
       start: 2201
       end: 2299

plugins/lookup/container_ports.py

@@ -1,33 +1,67 @@
-"""Lookup `container_ports`: build a compose ``ports:`` publish block from
-``[service_name, protocol]`` or ``[service_name, protocol, host_ip]`` terms.
+"""Lookup `container_ports`: build a compose ``ports:`` block from declared ports.
 
-Single SPOT replacing the repeated, verbose port lookups. Each term is a
-two- or three-element list; pass one or more. The optional third element is
-the host IP to bind that port to; omit it to publish on every interface:
+Single SPOT for every published port in the repo. A term is either the short
+list form or the dict form; pass one or more.
+
+List form, unchanged::
 
     {{ lookup('container_ports',
               ['gitea', 'http', DOCKER_BIND_HOST],
               ['gitea', 'ssh']) | indent(4) }}
 
-produces
+produces::
 
     ports:
-      - "<DOCKER_BIND_HOST>:<services.gitea.ports.local.http>:<services.gitea.ports.internal.http>"
+      - "<DOCKER_BIND_HOST>:<services.gitea.ports.local.http>:<...internal.http>"
       - "<services.gitea.ports.public.ssh>:<services.gitea.ports.internal.ssh>"
 
-Per term the published (host) port is ``services.<svc>.ports.local.<proto>``
-when declared, else ``services.<svc>.ports.public.<proto>``; the container
-port is always ``services.<svc>.ports.internal.<proto>``. A third element
-prefixes ``<host_ip>:``; without it the port binds to all interfaces. This
-makes every mapping expressible, including binds to a specific public IP
-(e.g. ``['mailu', 'smtp', MAILU_IP4_PUBLIC]``).
+The published (host) port is ``ports.local.<proto>`` when declared, else
+``ports.public.<proto>``; the container port is ``ports.internal.<proto>``.
 
-``application_id`` is read from the play variables unless given as ``application_id=``.
+Dict form, for the shapes the list form cannot express::
+
+    {{ lookup('container_ports',
+              {'service': 'coturn', 'protocol': 'stun_turn',
+               'transport': ['udp', 'tcp'], 'container': 'same'},
+              {'service': 'coturn', 'protocol': 'relay', 'transport': 'udp',
+               'container': 'same'},
+              {'service': 'openresty', 'protocol': 'http', 'mode': 'host'},
+              {'service': 'funkwhale', 'protocol': 'api',
+               'publish': 'ephemeral'}) | indent(4) }}
+
+Keys:
+
+``service``, ``protocol``
+    Required, as in the list form.
+``ip``
+    Host address to bind to; omit to publish on every interface.
+``transport``
+    ``tcp``, ``udp``, or a list of both. One list item per transport. Omit for
+    compose's own default, which publishes no suffix.
+``container``
+    ``internal`` (default) reads ``ports.internal.<proto>``; ``same`` reuses the
+    published port, for services that cannot remap (STUN/TURN, media relays).
+``mode``
+    ``ingress`` (default) emits the short ``"host:published:container"`` string;
+    ``host`` emits the long block with ``mode: host``, which is what an edge
+    role needs in swarm so the client address survives the hop.
+``publish``
+    ``ephemeral`` publishes the container port on a host port docker picks, for
+    a service addressed only through the compose network.
+
+A declared value that is a ``{start, end}`` mapping renders as ``start-end`` on
+both sides, so a relay range needs no special term.
+
+``application_id`` is read from the play variables unless given as
+``application_id=``. ``key='expose'`` names the block instead of ``ports``, for
+Discourse's launcher file, whose ``expose:`` list is what becomes ``docker run
+-p`` there.
 """
 
 from __future__ import annotations
 
 import contextlib
+from collections.abc import Mapping
 from typing import Any
 
 from ansible.errors import AnsibleError
@@ -37,11 +71,115 @@ from ansible.template import trust_as_template
 
 from utils.roles.applications.config import get
 
+_TERM_KEYS = frozenset(
+    {"service", "protocol", "ip", "transport", "container", "mode", "publish"}
+)
+_TRANSPORTS = ("tcp", "udp")
+_CONTAINER_SOURCES = ("internal", "same")
+_MODES = ("ingress", "host")
+
 
 def _as_str(value: Any) -> str:
     return "" if value is None else str(value).strip()
 
 
+def _rendered(value: Any) -> str:
+    """A declared port value as compose writes it, ranges included."""
+    if isinstance(value, Mapping):
+        start, end = value.get("start"), value.get("end")
+        if start is None or end is None:
+            return ""
+        return f"{_as_str(start)}-{_as_str(end)}"
+    return _as_str(value)
+
+
+def _transports_of(term: Mapping) -> list[str]:
+    declared = term.get("transport")
+    if declared is None:
+        return [""]
+    wanted = [declared] if isinstance(declared, str) else list(declared)
+    for transport in wanted:
+        if transport not in _TRANSPORTS:
+            raise AnsibleError(
+                f"container_ports: transport must be one of {_TRANSPORTS}, "
+                f"got {transport!r}"
+            )
+    return wanted
+
+
+def _normalised(term: Any) -> dict[str, Any]:
+    """One term as a dict, accepting the legacy list form."""
+    if isinstance(term, (list, tuple)):
+        if len(term) not in (2, 3):
+            raise AnsibleError(
+                "container_ports: a list term must be [service_name, protocol] "
+                f"or [service_name, protocol, host_ip], got {term!r}"
+            )
+        term = {
+            "service": term[0],
+            "protocol": term[1],
+            **({"ip": term[2]} if len(term) == 3 else {}),
+        }
+    if not isinstance(term, Mapping):
+        raise AnsibleError(
+            f"container_ports: each term must be a list or a mapping, got {term!r}"
+        )
+    unknown = set(term) - _TERM_KEYS
+    if unknown:
+        raise AnsibleError(
+            f"container_ports: unknown term key(s) {sorted(unknown)}; "
+            f"valid keys are {sorted(_TERM_KEYS)}"
+        )
+    normalised = {
+        "service": _as_str(term.get("service")),
+        "protocol": _as_str(term.get("protocol")),
+        "ip": _as_str(term.get("ip")),
+        "transport": term.get("transport"),
+        "container": _as_str(term.get("container")) or "internal",
+        "mode": _as_str(term.get("mode")) or "ingress",
+        "publish": _as_str(term.get("publish")),
+    }
+    if not normalised["service"] or not normalised["protocol"]:
+        raise AnsibleError("container_ports: service and protocol must be non-empty")
+    if normalised["container"] not in _CONTAINER_SOURCES:
+        raise AnsibleError(
+            f"container_ports: container must be one of {_CONTAINER_SOURCES}, "
+            f"got {normalised['container']!r}"
+        )
+    if normalised["mode"] not in _MODES:
+        raise AnsibleError(
+            f"container_ports: mode must be one of {_MODES}, got {normalised['mode']!r}"
+        )
+    if normalised["publish"] and normalised["publish"] != "ephemeral":
+        raise AnsibleError(
+            "container_ports: publish accepts only 'ephemeral', "
+            f"got {normalised['publish']!r}"
+        )
+    if normalised["mode"] == "host" and normalised["ip"]:
+        raise AnsibleError(
+            "container_ports: mode 'host' binds the node's own interface and "
+            "takes no ip"
+        )
+    return normalised
+
+
+def _short_line(term: Mapping, published: str, container: str, transport: str) -> str:
+    suffix = f"/{transport}" if transport else ""
+    if term["publish"] == "ephemeral":
+        return f'  - "{container}{suffix}"'
+    head = f"{term['ip']}:" if term["ip"] else ""
+    return f'  - "{head}{published}:{container}{suffix}"'
+
+
+def _host_lines(published: str, container: str, transport: str) -> list[str]:
+    return [
+        f"  - target: {container}",
+        f"    published: {published}",
+        f"    protocol: {transport or 'tcp'}",
+        "    mode: host",
+    ]
+
+
 class LookupModule(LookupBase):
     def run(
         self,
@@ -58,22 +196,9 @@ class LookupModule(LookupBase):
         if not terms:
             raise AnsibleError(
                 "lookup('container_ports', [service_name, protocol[, host_ip]], ...) "
-                "expects one or more [service_name, protocol] terms"
+                "expects one or more terms"
             )
-        triples: list[tuple[str, str, str]] = []
-        for term in terms:
-            if not (isinstance(term, (list, tuple)) and len(term) in (2, 3)):
-                raise AnsibleError(
-                    "container_ports: each term must be a [service_name, protocol] "
-                    f"or [service_name, protocol, host_ip] list, got {term!r}"
-                )
-            service, protocol = _as_str(term[0]), _as_str(term[1])
-            host = _as_str(term[2]) if len(term) == 3 else ""
-            if not service or not protocol:
-                raise AnsibleError(
-                    "container_ports: service_name and protocol must be non-empty"
-                )
-            triples.append((service, protocol, host))
+        parsed = [_normalised(term) for term in terms]
 
         templar = getattr(self, "_templar", None)
         variables = variables or getattr(self._templar, "available_variables", {}) or {}
@@ -96,7 +221,7 @@ class LookupModule(LookupBase):
         ).run([], variables=variables)[0]
 
         def _port(service: str, scope: str, protocol: str, *, required: bool) -> str:
-            value = _as_str(
+            value = _rendered(
                 get(
                     applications=applications,
                     application_id=application_id,
@@ -112,14 +237,24 @@ class LookupModule(LookupBase):
                 )
             return value
 
-        lines = ["ports:"]
-        for service, protocol, host in triples:
-            internal = _port(service, "internal", protocol, required=True)
-            published = _port(service, "local", protocol, required=False) or _port(
-                service, "public", protocol, required=True
+        block_key = _as_str(kwargs.get("key")) or "ports"
+        if block_key not in ("ports", "expose"):
+            raise AnsibleError(
+                f"container_ports: key must be 'ports' or 'expose', got {block_key!r}"
             )
-            if host:
-                lines.append(f'  - "{host}:{published}:{internal}"')
+        lines = [f"{block_key}:"]
+        for term in parsed:
+            service, protocol = term["service"], term["protocol"]
+            published = _port(service, "local", protocol, required=False) or _port(
+                service, "public", protocol, required=term["publish"] != "ephemeral"
+            )
+            if term["container"] == "same":
+                container = published
             else:
-                lines.append(f'  - "{published}:{internal}"')
+                container = _port(service, "internal", protocol, required=True)
+            for transport in _transports_of(term):
+                if term["mode"] == "host":
+                    lines.extend(_host_lines(published, container, transport))
+                else:
+                    lines.append(_short_line(term, published, container, transport))
         return ["\n".join(lines)]

roles/svc-dns-unbound/meta/services.yml

@@ -21,6 +21,8 @@ unbound:
   mem_limit: "512m"
   pids_limit: 256
   ports:
+    internal:
+      unbound: 53
     local:
       unbound: 53
     onion:

roles/svc-dns-unbound/templates/compose.yml.j2

@@ -6,9 +6,9 @@
     {{ lookup('container_image', application_id, 'unbound') }}
     {% include 'roles/sys-svc-container/templates/base.yml.j2' %}
     {% if DEPLOYMENT_MODE == 'compose' and not (DOCKER_IN_CONTAINER | bool) %}
-    ports:
-      - "{{ DOCKER_BIND_HOST }}:{{ UNBOUND_PORT }}:53/udp"  # nocheck: compose-ports-must-use-container-ports
-      - "{{ DOCKER_BIND_HOST }}:{{ UNBOUND_PORT }}:53/tcp"  # nocheck: compose-ports-must-use-container-ports
+    {{ lookup('container_ports',
+              {'service': 'unbound', 'protocol': 'unbound', 'ip': DOCKER_BIND_HOST,
+               'transport': ['udp', 'tcp']}) | indent(4) }}
     {% endif %}
     {{ lookup('container_volumes', service_name) | indent(4) }}
     networks:  # nocheck: networks-literal  static-IP resolver on its own engine overlay so consumers can point dns: at it

roles/svc-dns-unbound/vars/main.yml

@@ -4,7 +4,6 @@ entity_name: "{{ application_id | get_entity_name }}"
 
 UNBOUND_CONTAINER: "{{ lookup('config', application_id, 'services.' ~ entity_name ~ '.name') }}"
 UNBOUND_CONTAINER_ADDRESS: "{{ lookup('container_address', application_id, UNBOUND_CONTAINER) }}"
-UNBOUND_PORT: "{{ lookup('config', application_id, 'services.unbound.ports.local.unbound') }}"
 UNBOUND_NETWORK_NAME: "{{ entity_name }}"
 UNBOUND_RESOLVER_IP: "{{ lookup('config', application_id, 'networks.local.dns_resolver') }}"
 

roles/svc-net-tor/templates/compose.yml.j2

@@ -12,11 +12,9 @@
     {{ lookup('compose_only', 'container_name', TOR_CONTAINER) }}
     {{ lookup('container_image', application_id, 'tor') }}
     {% if DEPLOYMENT_MODE == 'swarm' %}
-    ports:
-      - target: {{ TOR_SOCKS_PORT }}
-        published: {{ TOR_SOCKS_PORT }}
-        protocol: tcp
-        mode: host
+    {{ lookup('container_ports',
+              {'service': 'tor', 'protocol': 'socks', 'mode': 'host',
+               'container': 'same'}) | indent(4) }}
     {{ lookup('container_networks') | indent(4) }}
     {% else %}
     network_mode: "host"

roles/svc-net-tor/vars/main.yml

@@ -6,7 +6,6 @@ TOR_IMAGE: "{{ lookup('config', application_id, 'services.tor.image') }}"
 TOR_VERSION: "{{ lookup('config', application_id, 'services.tor.version') }}"
 TOR_CONTAINER: "{{ lookup('config', application_id, 'services.tor.name') }}"
 
-TOR_SOCKS_PORT: "{{ lookup('config', application_id, 'services.tor.ports.local.socks') }}"
 TOR_NODE_ONION: "{{ lookup('config', application_id, 'services.tor.node', '') }}"
 TOR_USE_ENTRY_GUARDS: "{{ lookup('config', application_id, 'services.tor.use_entry_guards') | bool }}"
 TOR_FLAVOR: "{{ lookup('config', application_id, 'services.tor.flavor') }}"

roles/svc-prx-openresty/meta/services.yml

@@ -11,6 +11,17 @@ openresty:
   enabled: true
   shared: true
   name: openresty
+  ports:
+    internal:
+      http: 80
+      https: 443
+    public:
+      http: 80
+      https: 443
+    onion:
+      http: true
+      # nocheck: onion-flag  the hidden service is already the transport encryption, and the onion address has no certificate a TLS listener could present
+      https: false
   healthcheck:
     test: ["CMD", "openresty", "-t", "-q"]
   min_storage: 264MB

roles/svc-prx-openresty/templates/compose.yml.j2

@@ -14,15 +14,9 @@
     {{ lookup('compose_only', 'container_name', OPENRESTY_CONTAINER) }}
     {{ lookup('container_image', application_id, 'openresty') }}
     {% if DEPLOYMENT_MODE == 'swarm' %}
-    ports:
-      - target: 80
-        published: 80
-        protocol: tcp
-        mode: host
-      - target: 443
-        published: 443
-        protocol: tcp
-        mode: host
+    {{ lookup('container_ports',
+              {'service': 'openresty', 'protocol': 'http', 'mode': 'host'},
+              {'service': 'openresty', 'protocol': 'https', 'mode': 'host'}) | indent(4) }}
     {{ lookup('container_extra_hosts', extra_hosts=['host.docker.internal:host-gateway']) | indent(4) }}
     {{ lookup('container_networks') | indent(4) }}
     {% else %}

roles/web-app-bigbluebutton/templates/compose.override.yml.j2

@@ -90,12 +90,12 @@ services:
 {% set service_name = 'coturn' %}
   {{ service_name }}:
     {{ lookup('container_volumes', service_name) | indent(4) }}
-    ports:
-      - "{{ BBB_TURN_PORT }}:{{ BBB_TURN_PORT }}/udp"
-      - "{{ BBB_TURN_PORT }}:{{ BBB_TURN_PORT }}/tcp"
-      - "{{ BBB_STUN_PORT }}:{{ BBB_STUN_PORT }}/udp"
-      - "{{ BBB_STUN_PORT }}:{{ BBB_STUN_PORT }}/tcp"
-      - "{{ BBB_RELAY_PORT_RANGE }}/udp"
+    {{ lookup('container_ports',
+              {'service': 'bigbluebutton', 'protocol': 'stun_turn',
+               'transport': ['udp', 'tcp'], 'container': 'same'},
+              {'service': 'bigbluebutton', 'protocol': 'relay',
+               'transport': 'udp', 'container': 'same',
+               'publish': 'ephemeral'}) | indent(4) }}
     command: >-
       --use-auth-secret
       --static-auth-secret=${TURN_SECRET}

roles/web-app-bigbluebutton/vars/main.yml

@@ -33,7 +33,6 @@ BBB_STUN_PORT: "{{ lookup('config', application_id, 'services.bigbluebutton.port
   'services.coturn.ports.public.stun_turn_tls') }}"
 BBB_RELAY_PORT_START: "{{ lookup('config', application_id, 'services.bigbluebutton.ports.public.relay.start') }}"
 BBB_RELAY_PORT_END: "{{ lookup('config', application_id, 'services.bigbluebutton.ports.public.relay.end') }}"
-BBB_RELAY_PORT_RANGE: "{{ BBB_RELAY_PORT_START }}-{{ BBB_RELAY_PORT_END }}"
 
 BBB_COLLABORA_ENABLED: "{{ not (lookup('config',application_id, 'services.collabora.shared')) }}"
 BBB_COLLABORA_PORT: "{{ lookup('config', 'web-svc-collabora', 'services.collabora.ports.internal.http') }}"

roles/web-app-dashboard/templates/compose.yml.j2

@@ -1,12 +1,10 @@
 {% include 'roles/sys-svc-compose/templates/base.yml.j2' %}
   {% set service_name = 'dashboard' %}
-  {% set container_port = lookup('config', application_id, 'services.' ~ service_name ~ '.ports.internal.http') %}
   {{ service_name }}:
     {% include 'roles/sys-svc-container/templates/deploy.yml.j2' %}
     {{ lookup('container_image', application_id, 'dashboard') }}
     {{ lookup('compose_only', 'container_name', DASHBOARD_CONTAINER) }}
-    ports:
-      - {{ DOCKER_BIND_HOST }}:{{ http_port }}:{{ container_port }}
+    {{ lookup('container_ports', ['dashboard', 'http', DOCKER_BIND_HOST]) | indent(4) }}
     {{ lookup('container_volumes', service_name) | indent(4) }}
     {{ lookup('compose_restart') }}
     {{ lookup('container_networks') | indent(4) }}

roles/web-app-discourse/templates/config.yml.j2

@@ -15,8 +15,7 @@ templates:
 ## which TCP/IP ports should this container expose?
 ## If you want Discourse to share a port with another webserver like Apache or nginx,
 ## see https://meta.discourse.org/t/17247 for details
-expose:
-  - "{{ DOCKER_BIND_HOST }}:{{ lookup('config', application_id, 'services.discourse.ports.local.http') }}:80"   # http
+{{ lookup('container_ports', ['discourse', 'http', DOCKER_BIND_HOST], key='expose') }}
 
 params:
   db_default_text_search_config: "pg_catalog.english"

roles/web-app-funkwhale/templates/compose.yml.j2

@@ -30,8 +30,9 @@
     {{ lookup('container_image', application_id, 'api') }}
     command: gunicorn
     {{ lookup('container_volumes', service_name) | indent(4) }}
-    ports:
-      - "{{ FUNKWHALE_DOCKER_API_PORT }}"
+    {{ lookup('container_ports',
+              {'service': 'api', 'protocol': 'http',
+               'publish': 'ephemeral'}) | indent(4) }}
     {{ lookup('container_depends_on', application_id, extra={
   'migrate': {'condition': 'service_completed_successfully'},
 }) | indent(4) }}

roles/web-app-jitsi/templates/compose.yml.j2

@@ -28,8 +28,9 @@
     {{ lookup('compose_only', 'container_name', JITSI_JVB_CONTAINER) }}
     {% include 'roles/sys-svc-container/templates/base.yml.j2' %}
     {{ lookup('depends_on', {JITSI_PROSODY_SERVICE: 'service_started'}) }}
-    ports:
-      - "{{ JITSI_JVB_MEDIA_PORT }}:{{ JITSI_JVB_MEDIA_PORT }}/udp"  # nocheck: compose-ports-must-use-container-ports
+    {{ lookup('container_ports',
+              {'service': 'jitsi', 'protocol': 'media',
+               'transport': 'udp', 'container': 'same'}) | indent(4) }}
     {{ lookup('container_volumes', service_name) | indent(4) }}
     {{ lookup('container_networks') | indent(4) }}
 

roles/web-app-keycloak/templates/sso_proxy/container.yml.j2

@@ -8,6 +8,7 @@
     labels:
       infinito.sso-proxy.config-sha256: "{{ lookup('template', lookup('path_absolute', 'roles/web-app-keycloak/templates/sso_proxy/oauth2-proxy-keycloak.cfg.j2'), template_vars={'sso_proxy_application_id': application_id}) | hash('sha256') }}"
     ports:
+      # nocheck: compose-ports-must-use-container-ports  the published port belongs to the consuming role's own entity and the container port to web-app-keycloak's sso_proxy, so the mapping spans two application_ids and no single container_ports call can read both
       - {{ DOCKER_BIND_HOST }}:{{ lookup('config', application_id, 'services.' ~ (application_id | get_entity_name) ~ '.ports.local.sso') }}:{{ lookup('config', 'web-app-keycloak', 'services.sso_proxy.ports.internal.http') }}/tcp
     volumes:
       - "{{ lookup('container', application_id, 'directories.volumes') }}{{ lookup('config', 'web-app-keycloak', 'services.sso_proxy.configuration_file') }}:/oauth2-proxy.cfg"

roles/web-app-matrix/templates/flavor/ansible/services.yml.j2

@@ -35,7 +35,8 @@
       - /run
       - /run/lock
       - /tmp
-    ports:
-      - "{{ DOCKER_BIND_HOST }}:{{ MATRIX_SYNAPSE_PORT }}:8008"
+    {{ lookup('container_ports', ['synapse', 'http', DOCKER_BIND_HOST]) | indent(4) }}
+      # nocheck: compose-ports-must-use-container-ports  matrix-docker-ansible-deploy fixes element on 8080 while this repo declares element.ports.internal.http 80; the two disagree and the playbook owns the container
       - "{{ DOCKER_BIND_HOST }}:{{ MATRIX_ELEMENT_PORT }}:8080"
+      # nocheck: compose-ports-must-use-container-ports  the upstream playbook serves federation on 8048 and no ports.internal.federation is declared for it
       - "{{ DOCKER_BIND_HOST }}:{{ MATRIX_FEDERATION_PORT }}:8048"

roles/web-app-matrix/templates/flavor/compose/services.yml.j2

@@ -1,7 +1,6 @@
 {% set service_name = 'synapse' %}
   {{ service_name }}:
 {% include 'roles/sys-svc-container/templates/deploy.yml.j2' %}
-{% set container_port = MATRIX_SYNAPSE_CONTAINER_PORT %}
     {{ lookup('container_image', application_id, 'synapse') }}
     {{ lookup('compose_only', 'container_name', MATRIX_SYNAPSE_NAME) }}
     {{ lookup('compose_restart') }}
@@ -11,8 +10,7 @@
     environment:
       - SYNAPSE_SERVER_NAME={{ MATRIX_SYNAPSE_DOMAIN }}
       - SYNAPSE_REPORT_STATS=no
-    ports:
-      - "{{ DOCKER_BIND_HOST }}:{{lookup('config', 'web-app-matrix', 'services.synapse.ports.local.http')}}:{{ container_port }}"
+    {{ lookup('container_ports', ['synapse', 'http', DOCKER_BIND_HOST]) | indent(4) }}
     {{ lookup('container_extra_hosts') | indent(4) }}
     {{ lookup('container_healthcheck', service_name) | indent(4) }}
     {{ lookup('container_depends_on', application_id) | indent(4) }}
@@ -20,13 +18,11 @@
 {% set service_name = 'element' %}
   {{ service_name }}:
 {% include 'roles/sys-svc-container/templates/deploy.yml.j2' %}
-{% set container_port = lookup('config', application_id, 'services.' ~ service_name ~ '.ports.internal.http') %}
     {{ lookup('container_image', application_id, 'element') }}
     {{ lookup('compose_only', 'container_name', MATRIX_ELEMENT_NAME) }}
     {{ lookup('compose_restart') }}
     {{ lookup('container_volumes', service_name) | indent(4) }}
-    ports:
-      - "{{ DOCKER_BIND_HOST }}:{{ MATRIX_ELEMENT_PORT }}:{{ container_port }}"
+    {{ lookup('container_ports', ['element', 'http', DOCKER_BIND_HOST]) | indent(4) }}
     {{ lookup('container_healthcheck', service_name) | indent(4) }}
     {{ lookup('container_networks') | indent(4) }}
 

roles/web-app-moodle/templates/compose.yml.j2

@@ -40,8 +40,7 @@
     {{ lookup('compose_only', 'container_name', MOODLE_NGINX_CONTAINER) }}
     {{ lookup('container_image', application_id, 'nginx') }}
     {{ lookup('depends_on', {'moodle': 'service_healthy'}) }}
-    ports:
-      - {{ DOCKER_BIND_HOST }}:{{ lookup('config', application_id, 'services.moodle.ports.local.http') }}:{{ MOODLE_NGINX_PORT }}
+    {{ lookup('container_ports', ['moodle', 'http', DOCKER_BIND_HOST]) | indent(4) }}
     {{ lookup('container_volumes', service_name) | indent(4) }}
     {{ lookup('compose_restart') }}
     {{ lookup('container_networks') | indent(4) }}

roles/web-app-nextcloud/templates/compose.yml.j2

@@ -75,11 +75,13 @@
     init: true
     {% if NEXTCLOUD_HPB_NETWORK_MODE == 'bridge' %}
     {% if NEXTCLOUD_HPB_TURN_ONBOARD_ENABLED | bool and NEXTCLOUD_HPB_TURN_ONBOARD_PUBLISH_PORTS | bool %}
-    ports:
-      - {{ NEXTCLOUD_HPB_TURN_ONBOARD_BIND_HOST }}:{{ NEXTCLOUD_HPB_TURN_ONBOARD_PORT }}:{{ NEXTCLOUD_HPB_TURN_ONBOARD_PORT }}/tcp # nocheck: compose-ports-must-use-container-ports
-      - {{ NEXTCLOUD_HPB_TURN_ONBOARD_BIND_HOST }}:{{ NEXTCLOUD_HPB_TURN_ONBOARD_PORT }}:{{ NEXTCLOUD_HPB_TURN_ONBOARD_PORT }}/udp # nocheck: compose-ports-must-use-container-ports
-      - {{ NEXTCLOUD_HPB_TURN_ONBOARD_BIND_HOST }}:{{ NEXTCLOUD_HPB_TURN_ONBOARD_RELAY_PORT_START }}-{{ NEXTCLOUD_HPB_TURN_ONBOARD_RELAY_PORT_END }}:{{ NEXTCLOUD_HPB_TURN_ONBOARD_RELAY_PORT_START }}-{{ NEXTCLOUD_HPB_TURN_ONBOARD_RELAY_PORT_END }}/udp # nocheck: compose-ports-must-use-container-ports
-      - {{ NEXTCLOUD_HPB_TURN_ONBOARD_BIND_HOST }}:{{ NEXTCLOUD_HPB_TURN_ONBOARD_RELAY_PORT_START }}-{{ NEXTCLOUD_HPB_TURN_ONBOARD_RELAY_PORT_END }}:{{ NEXTCLOUD_HPB_TURN_ONBOARD_RELAY_PORT_START }}-{{ NEXTCLOUD_HPB_TURN_ONBOARD_RELAY_PORT_END }}/tcp # nocheck: compose-ports-must-use-container-ports
+    {{ lookup('container_ports',
+              {'service': 'nextcloud', 'protocol': 'stun_turn',
+               'ip': NEXTCLOUD_HPB_TURN_ONBOARD_BIND_HOST,
+               'transport': ['tcp', 'udp'], 'container': 'same'},
+              {'service': 'nextcloud', 'protocol': 'relay',
+               'ip': NEXTCLOUD_HPB_TURN_ONBOARD_BIND_HOST,
+               'transport': ['udp', 'tcp'], 'container': 'same'}) | indent(4) }}
     {% endif %}
     expose:
       - "{{ container_port }}"

roles/web-svc-coturn/templates/compose.yml.j2

@@ -18,13 +18,23 @@
     {{ lookup('compose_only', 'container_name', COTURN_CONTAINER) }}
     network_mode: {{ COTURN_NETWORK_MODE }}
     {% if COTURN_NETWORK_MODE == 'bridge' %}
-    ports:
-      - "{{ COTURN_STUN_TURN_PORT }}:{{ COTURN_STUN_TURN_PORT }}/udp"  # nocheck: compose-ports-must-use-container-ports
-      - "{{ COTURN_STUN_TURN_PORT }}:{{ COTURN_STUN_TURN_PORT }}/tcp"  # nocheck: compose-ports-must-use-container-ports
-      {% if COTURN_TLS_ENABLED| bool %}
-      - "{{ COTURN_STUN_TURN_PORT_TLS }}:{{ COTURN_STUN_TURN_PORT_TLS }}/tcp"  # nocheck: compose-ports-must-use-container-ports
-      {% endif %}
-      - "{{ COTURN_RELAY_PORT_RANGE }}/udp"  # nocheck: compose-ports-must-use-container-ports
+    {% if COTURN_TLS_ENABLED | bool %}
+    {{ lookup('container_ports',
+              {'service': 'coturn', 'protocol': 'stun_turn',
+               'transport': ['udp', 'tcp'], 'container': 'same'},
+              {'service': 'coturn', 'protocol': 'stun_turn_tls',
+               'transport': 'tcp', 'container': 'same'},
+              {'service': 'coturn', 'protocol': 'relay',
+               'transport': 'udp', 'container': 'same',
+               'publish': 'ephemeral'}) | indent(4) }}
+    {% else %}
+    {{ lookup('container_ports',
+              {'service': 'coturn', 'protocol': 'stun_turn',
+               'transport': ['udp', 'tcp'], 'container': 'same'},
+              {'service': 'coturn', 'protocol': 'relay',
+               'transport': 'udp', 'container': 'same',
+               'publish': 'ephemeral'}) | indent(4) }}
+    {% endif %}
     {{ lookup('container_networks') | indent(4) }}
     {% endif %}
     {{ lookup('container_volumes', service_name) | indent(4) }}

roles/web-svc-coturn/vars/main.yml

@@ -11,7 +11,6 @@ COTURN_STUN_TURN_PORT: "{{ lookup('config', application_id, 'services.coturn.por
 COTURN_STUN_TURN_PORT_TLS: "{{ lookup('config', application_id, 'services.coturn.ports.public.stun_turn_tls') }}"
 COTURN_RELAY_PORT_START: "{{ lookup('config', application_id, 'services.coturn.ports.public.relay.start') }}"
 COTURN_RELAY_PORT_END: "{{ lookup('config', application_id, 'services.coturn.ports.public.relay.end') }}"
-COTURN_RELAY_PORT_RANGE: "{{ COTURN_RELAY_PORT_START }}-{{ COTURN_RELAY_PORT_END }}"
 
 COTURN_USER_NAME: "{{ lookup('users', 'coturnconsumer').username }}"
 COTURN_USER_PASSWORD: "{{ lookup('users', 'coturnconsumer').password }}"

tests/lint/ansible/jinja/test_compose_ports_use_container_ports_lookup.py

@@ -7,8 +7,8 @@ declared ``ports.local`` / ``ports.public`` (host) and ``ports.internal``
 (container) values, so the published and container ports stay one source
 of truth:
 
-    {{ lookup('container_ports', ['gitea', 'http'], ['gitea', 'ssh'],
-              ip=DOCKER_BIND_HOST) | indent(4) }}
+    {{ lookup('container_ports', ['gitea', 'http', DOCKER_BIND_HOST],
+              ['gitea', 'ssh']) | indent(4) }}
 
 Hand-writing the mapping
 ``- "{{ ... }}:{{ lookup('config', ..., 'services.<e>.ports.local.<k>') }}:..."``
@@ -37,31 +37,94 @@ from . import PROJECT_ROOT
 
 _RULE = "compose-ports-must-use-container-ports"
 
-_PORT_SEGMENT = re.compile(r"\A(?:\{\{.*\}\}|[\d.]+|\d+/(?:tcp|udp))\Z")
+_PORT_SEGMENT = re.compile(
+    r"\A(?:\{\{.*\}\}|\$\{[^}]*\}|[\d.]+|[\d-]+)(?:/(?:tcp|udp))?\Z",
+)
 
-_LIST_ITEM = re.compile(r'-\s*"([^"]*)"')
+_LIST_ITEM = re.compile(r"""-\s*["']?(.*?)["']?\s*(?:\#.*)?\Z""")
+
+_LONG_SYNTAX_KEY = re.compile(r"\A-?\s*(?:target|published|mode|protocol):")
+
+_BLOCK_KEY = re.compile(r"\A(ports|expose):\s*(.*)\Z")
 
 _PAIR = re.compile(r"\[\s*'([\w-]+)'\s*,\s*'([\w-]+)'\s*\]")
 
 
 def _is_scan_target(rel_path: str) -> bool:
-    if not rel_path.startswith("roles/") or "/templates/" not in rel_path:
-        return False
-    name = Path(rel_path).name
-    return name.endswith(".yml.j2") and "compose" in name
+    """Every role template that can carry a compose ``ports:`` block.
+
+    ``compose.yml.j2`` and its ``compose.<flavor>.yml.j2`` siblings, plus the
+    ``*.yml.j2`` fragments they include: a mapping hand-written in an included
+    file publishes exactly the same port as one written inline.
+    """
+    return (
+        rel_path.startswith("roles/")
+        and "/templates/" in rel_path
+        and rel_path.endswith(".yml.j2")
+    )
 
 
-def _is_raw_port_mapping(line: str) -> bool:
+def _is_raw_port_mapping(line: str, *, mapping_only: bool = False) -> bool:
+    """Whether *line* publishes a port by hand, quoted or not.
+
+    ``mapping_only`` drops the single-segment form. Under ``expose:`` that form
+    names a container port for documentation and publishes nothing, so only a
+    ``host:container`` mapping there is a publication.
+    """
     match = _LIST_ITEM.match(line.strip())
     if match is None:
         return False
-    inner = match.group(1)
-    if "=" in inner:
+    inner = match.group(1).strip()
+    if not inner or "=" in inner:
         return False
-    segments = inner.split(":")
-    return len(segments) >= 2 and all(
-        _PORT_SEGMENT.match(segment.strip()) for segment in segments
-    )
+    segments = [segment.strip() for segment in inner.split(":")]
+    if not all(_PORT_SEGMENT.match(segment) for segment in segments):
+        return False
+    if len(segments) >= 2:
+        return True
+    return not mapping_only and bool(_PORT_SEGMENT.match(inner))
+
+
+def _raw_port_lines(lines: list[str]) -> list[int]:
+    """1-based line numbers that publish a port by hand.
+
+    The block is tracked by the indentation of its ``ports:`` / ``expose:`` key
+    rather than by the next key seen. Keying on the next line closed the block
+    on anything shaped like a key, which the long-syntax members ``target:``,
+    ``published:``, ``protocol:`` and ``mode:`` are, so the block could never
+    end while it was being read; blank, comment and ``{% %}`` lines equally have
+    no say in where a YAML block ends.
+    """
+    found: list[int] = []
+    block_indent: int | None = None
+    mapping_only = False
+    for number, line in enumerate(lines, start=1):
+        stripped = line.strip()
+        if not stripped or stripped.startswith(("#", "{%")):
+            continue
+        indent = len(line) - len(line.rstrip("\n").lstrip())
+        if (
+            block_indent is not None
+            and indent <= block_indent
+            and not stripped.startswith("- ")
+        ):
+            block_indent = None
+        if block_indent is None:
+            opener = _BLOCK_KEY.match(stripped)
+            if opener:
+                block_indent = indent
+                mapping_only = opener.group(1) == "expose"
+                inline = opener.group(2).strip().strip("[]")
+                if inline and _is_raw_port_mapping(
+                    "- " + inline, mapping_only=mapping_only
+                ):
+                    found.append(number)
+            continue
+        if _LONG_SYNTAX_KEY.match(stripped) or _is_raw_port_mapping(
+            line, mapping_only=mapping_only
+        ):
+            found.append(number)
+    return found
 
 
 class TestComposePortsUseContainerPortsLookup(unittest.TestCase):
@@ -75,27 +138,9 @@ class TestComposePortsUseContainerPortsLookup(unittest.TestCase):
             if not _is_scan_target(rel):
                 continue
             lines = content.splitlines()
-            in_ports_block = False
-            for idx, line in enumerate(lines):
-                stripped = line.strip()
-                if not stripped:
-                    continue
-                if stripped.startswith("- "):
-                    if (
-                        in_ports_block
-                        and _is_raw_port_mapping(line)
-                        and not is_suppressed_at(
-                            lines, idx + 1, _RULE, mode="same-or-above"
-                        )
-                    ):
-                        findings.append((rel, idx + 1, stripped))
-                    continue
-                if stripped.startswith("{{"):
-                    in_ports_block = "container_ports" in stripped
-                    continue
-                key = re.match(r"[\w-]+:", stripped)
-                if key:
-                    in_ports_block = key.group(0) == "ports:"
+            for number in _raw_port_lines(lines):
+                if not is_suppressed_at(lines, number, _RULE, mode="same-or-above"):
+                    findings.append((rel, number, lines[number - 1].strip()))
 
         if findings:
             formatted = "\n".join(
@@ -107,8 +152,8 @@ class TestComposePortsUseContainerPortsLookup(unittest.TestCase):
                 "Build the `ports:` block with the container_ports lookup instead, "
                 "so the published (local/public) and container (internal) ports stay "
                 "a single declared source of truth:\n\n"
-                "    {{ lookup('container_ports', ['<svc>', '<proto>'],\n"
-                "              ip=DOCKER_BIND_HOST) | indent(4) }}\n\n"
+                "    {{ lookup('container_ports',\n"
+                "              ['<svc>', '<proto>', DOCKER_BIND_HOST]) | indent(4) }}\n\n"
                 "One ['<svc>', '<proto>'] pair per published port; the lookup reads "
                 "ports.local/public for the host side and ports.internal for the "
                 "container side. Mark with "

tests/unit/python/plugins/lookup/test_container_ports.py

@@ -119,5 +119,133 @@ class TestContainerPortsLookup(unittest.TestCase):
         self.assertEqual(out, ['ports:\n  - "10.0.0.1:8002:3000"'])
 
 
+def _edge_apps():
+    return {
+        "web-svc-coturn": {
+            "services": {
+                "coturn": {
+                    "ports": {
+                        "public": {
+                            "stun_turn": 3481,
+                            "relay": {"start": 20000, "end": 39999},
+                        }
+                    }
+                },
+                "openresty": {
+                    "ports": {
+                        "internal": {"http": 80},
+                        "public": {"http": 80},
+                    }
+                },
+                "api": {"ports": {"internal": {"http": 5000}}},
+            }
+        }
+    }
+
+
+def _run_edge(terms):
+    lookup = LookupModule()
+    lookup._loader = mock.MagicMock()
+    apps = _edge_apps()
+    with patch("plugins.lookup.container_ports.lookup_loader") as loader_mock:
+        loader_mock.get.return_value = mock.MagicMock(run=lambda *_a, **_k: [apps])
+        return lookup.run(terms, variables={"application_id": "web-svc-coturn"})[0]
+
+
+class TestContainerPortsDictTerms(unittest.TestCase):
+    def test_one_list_item_per_transport(self):
+        out = _run_edge(
+            [
+                {
+                    "service": "coturn",
+                    "protocol": "stun_turn",
+                    "transport": ["udp", "tcp"],
+                    "container": "same",
+                }
+            ]
+        )
+        self.assertEqual(out, 'ports:\n  - "3481:3481/udp"\n  - "3481:3481/tcp"')
+
+    def test_a_range_renders_on_both_sides(self):
+        out = _run_edge(
+            [
+                {
+                    "service": "coturn",
+                    "protocol": "relay",
+                    "transport": "udp",
+                    "container": "same",
+                }
+            ]
+        )
+        self.assertEqual(out, 'ports:\n  - "20000-39999:20000-39999/udp"')
+
+    def test_host_mode_emits_the_long_block(self):
+        out = _run_edge([{"service": "openresty", "protocol": "http", "mode": "host"}])
+        self.assertEqual(
+            out,
+            "ports:\n  - target: 80\n    published: 80\n"
+            "    protocol: tcp\n    mode: host",
+        )
+
+    def test_ephemeral_publishes_only_the_container_port(self):
+        out = _run_edge(
+            [{"service": "api", "protocol": "http", "publish": "ephemeral"}]
+        )
+        self.assertEqual(out, 'ports:\n  - "5000"')
+
+    def test_host_mode_rejects_an_ip(self):
+        with self.assertRaises(AnsibleError):
+            _run_edge(
+                [
+                    {
+                        "service": "openresty",
+                        "protocol": "http",
+                        "mode": "host",
+                        "ip": "10.0.0.1",
+                    }
+                ]
+            )
+
+    def test_an_unknown_term_key_is_rejected(self):
+        with self.assertRaises(AnsibleError):
+            _run_edge([{"service": "openresty", "protocol": "http", "protokol": "tcp"}])
+
+    def test_key_names_the_block(self):
+        lookup = LookupModule()
+        lookup._loader = mock.MagicMock()
+        with patch("plugins.lookup.container_ports.lookup_loader") as loader_mock:
+            loader_mock.get.return_value = mock.MagicMock(
+                run=lambda *_a, **_k: [_edge_apps()]
+            )
+            out = lookup.run(
+                [["openresty", "http"]],
+                variables={"application_id": "web-svc-coturn"},
+                key="expose",
+            )
+        self.assertEqual(out, ['expose:\n  - "80:80"'])
+
+    def test_an_unknown_block_key_is_rejected(self):
+        lookup = LookupModule()
+        lookup._loader = mock.MagicMock()
+        with (
+            patch("plugins.lookup.container_ports.lookup_loader") as loader_mock,
+            self.assertRaises(AnsibleError),
+        ):
+            loader_mock.get.return_value = mock.MagicMock(
+                run=lambda *_a, **_k: [_edge_apps()]
+            )
+            lookup.run(
+                [["openresty", "http"]],
+                variables={"application_id": "web-svc-coturn"},
+                key="publish",
+            )
+
+    def test_an_unknown_transport_is_rejected(self):
+        with self.assertRaises(AnsibleError):
+            _run_edge(
+                [{"service": "openresty", "protocol": "http", "transport": "sctp"}]
+            )
+
+
 if __name__ == "__main__":
     unittest.main()