fix(gitlab): pin only the two keys that cannot be rotated, and say what is skipped

An earlier change in this branch marked six GitLab key bases rotatable: false after
one of them, db_key_base, was proven to break a deploy. Six was the shape of
"everything in secrets.yml.j2 that could plausibly be blamed", not the shape of
"what is lost". Measured in the shipped image (gitlab-rails-ce:v19.3.1), four of
them rotate fine:

    A_new_key_only                       => FAIL Decryption   (control: reproduces the rotation)
    B_old_then_new_list                  => s3cret-written-before-rotation
    C_new_only_reads_post_rotation_write => written-after-rotation
    D_salt_rotated                       => FAIL Decryption

B and C prove active_record_encryption_primary_key is list-rotatable and that the
old key can later be dropped; D proves the salt is not. Note the new key goes LAST
-- key_provider.rb:21 is `@keys.last` -- so a prepend would have shipped a broken
rotation.

Unpinned, with the reason each:
  otp_key_base                                gitlab:two_factor:rotate_key:apply exists, with rollback!
  encrypted_settings_key_base                 protects zero bytes here: grep -c encrypted on
                                              gitlab.yml.j2 is 0, ldap and incoming_email are off,
                                              and upstream has rotated_encrypted_settings_key_base
  active_record_encryption_primary_key        list rotation measured above
  active_record_encryption_deterministic_key  one column, a push device token the device re-registers

Still pinned, and the descriptions now name why rather than leaving it to a commit
message: db_key_base (settings.rb raises MultipleDbKeyBaseError on a second key and
upstream ships no re-encrypt task) and active_record_encryption_key_derivation_salt
(scalar PBKDF2 input, no previous-salt slot in Rails). What db_key_base protects is
also worse than first recorded -- grep -rl returns 36 files: CI/CD variable values at
every scope, terraform state, Pages TLS private keys, webhook and integration
credentials. The two tokens that actually crashed were only the ones custom-instance-setup
happened to touch.

The second half matters more than the first. _drop_generated skipped a pinned path
with a bare `continue`, and reset/cli.py printed "Rotated 66 value(s)" with no mention
that six were refused. An operator rotating a disclosed key was told it worked. A tool
that silently declines to rotate a leaked secret and reports success inverts the
property it exists to provide, so every skip now names itself on stderr.

Not addressed here, and it is the real repair: gitlab:doctor:reset_encrypted_tokens in
the db-migrate path would let any instance with undecryptable tokens heal itself,
including after a restore under a different secrets file, which no pin can help. It
changes the migrations container's command and is not measurable without a live GitLab.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

cli/administration/inventory/provision/reset.py

@@ -10,12 +10,14 @@ secret into the application and the database it already created.
 
 from __future__ import annotations
 
+import sys
 from typing import TYPE_CHECKING
 
 from ruamel.yaml.comments import CommentedMap
 
 from cli.administration.inventory.credentials.vault import is_ruamel_vault
 from utils.manager.credential_key import CREDENTIALS_KEY, SECRETS_KEY
+from utils.roles.mapping import ROLE_FILE_META_SECRETS, ROLE_FILE_VARS_MAIN
 
 from .credentials_generator import generate_credentials_for_roles
 from .ruamel_io import dump_document, load_document
@@ -25,11 +27,66 @@ if TYPE_CHECKING:
     from pathlib import Path
 
 
-def _drop_generated(node: CommentedMap) -> int:
+ROTATABLE_KEY = "rotatable"
+
+
+def _pinned_paths(
+    schema: CommentedMap, prefix: tuple[str, ...]
+) -> set[tuple[str, ...]]:
+    """Return the credential paths ``schema`` marks ``rotatable: false``.
+
+    Args:
+        schema: a ``credentials`` branch of a role's ``meta/secrets.yml``.
+        prefix: the path walked so far.
+
+    Returns:
+        The pinned paths, each as a tuple of keys.
+    """
+    pinned: set[tuple[str, ...]] = set()
+    for key, meta in schema.items():
+        if not isinstance(meta, CommentedMap):
+            continue
+        if meta.get(ROTATABLE_KEY) is False:
+            pinned.add((*prefix, key))
+        else:
+            pinned |= _pinned_paths(meta, (*prefix, key))
+    return pinned
+
+
+def _pinned_credentials(roles_dir: Path) -> dict[str, set[tuple[str, ...]]]:
+    """Map application id to the credentials its role refuses to have rotated.
+
+    Args:
+        roles_dir: directory the roles live in.
+
+    Returns:
+        Application id to the set of pinned credential paths.
+    """
+    pinned: dict[str, set[tuple[str, ...]]] = {}
+    for role_dir in sorted(roles_dir.iterdir()):
+        schema = load_document(role_dir / ROLE_FILE_META_SECRETS).get(CREDENTIALS_KEY)
+        if not isinstance(schema, CommentedMap):
+            continue
+        paths = _pinned_paths(schema, ())
+        if not paths:
+            continue
+        application_id = load_document(role_dir / ROLE_FILE_VARS_MAIN).get(
+            "application_id"
+        )
+        if application_id:
+            pinned[str(application_id)] = paths
+    return pinned
+
+
+def _drop_generated(
+    node: CommentedMap, pinned: set[tuple[str, ...]], prefix: tuple[str, ...]
+) -> int:
     """Remove every vault-encrypted leaf below ``node``.
 
     Args:
         node: a ``credentials`` branch of the inventory.
+        pinned: paths the role marked ``rotatable: false``.
+        prefix: the path walked so far.
 
     Returns:
         How many values were removed.
@@ -40,21 +97,33 @@ def _drop_generated(node: CommentedMap) -> int:
     """
     dropped = 0
     for key in list(node.keys()):
+        path = (*prefix, key)
+        if path in pinned:
+            print(
+                f"[WARN] not rotated, pinned as rotatable: false: {'.'.join(path)}",
+                file=sys.stderr,
+            )
+            continue
         value = node.get(key)
         if isinstance(value, CommentedMap):
-            dropped += _drop_generated(value)
+            dropped += _drop_generated(value, pinned, path)
         elif is_ruamel_vault(value):
             del node[key]
             dropped += 1
     return dropped
 
 
-def _drop_app_credentials(document: CommentedMap, exclude: set[str]) -> int:
+def _drop_app_credentials(
+    document: CommentedMap,
+    exclude: set[str],
+    pinned: dict[str, set[tuple[str, ...]]],
+) -> int:
     """Remove the generated credentials of every application in ``document``.
 
     Args:
         document: the host_vars document.
         exclude: application ids to leave untouched.
+        pinned: per-application credential paths that must survive a rotation.
 
     Returns:
         How many values were removed.
@@ -72,7 +141,9 @@ def _drop_app_credentials(document: CommentedMap, exclude: set[str]) -> int:
             continue
         credentials = secrets.get(CREDENTIALS_KEY)
         if isinstance(credentials, CommentedMap):
-            dropped += _drop_generated(credentials)
+            dropped += _drop_generated(
+                credentials, pinned.get(str(application_id), set()), ()
+            )
     return dropped
 
 
@@ -148,7 +219,9 @@ def reset_credentials(
     document = load_document(host_vars_file)
     dropped = 0
     if schema:
-        dropped += _drop_app_credentials(document, exclude)
+        dropped += _drop_app_credentials(
+            document, exclude, _pinned_credentials(roles_dir)
+        )
     if users:
         dropped += _drop_user_passwords(document, roles_dir, application_ids, exclude)
     dump_document(host_vars_file, document)

roles/web-app-gitlab/meta/secrets.yml

@@ -15,9 +15,11 @@ credentials:
     algorithm: random_hex
     validation: "^[a-f0-9]{128}$"
   db_key_base:
-    description: Key base encrypting database columns (CI variables, tokens)
+    description: "Key base encrypting CI/CD variable values, terraform state, Pages TLS private keys, and integration and webhook credentials. settings.rb raises MultipleDbKeyBaseError
+      on a second key and upstream ships no re-encrypt task, so a rotation is one-way."
     algorithm: random_hex
     validation: "^[a-f0-9]{128}$"
+    rotatable: false
   encrypted_settings_key_base:
     description: Key base encrypting settings files
     algorithm: random_hex
@@ -31,9 +33,10 @@ credentials:
     algorithm: random_hex_16
     validation: "^[a-f0-9]{32}$"
   active_record_encryption_key_derivation_salt:
-    description: Key derivation salt for Active Record encryption
+    description: "PBKDF2 salt every Active Record encryption key derives from. It has no list form, so rotating it voids the primary-key rotation list as well."
     algorithm: random_hex_16
     validation: "^[a-f0-9]{32}$"
+    rotatable: false
   gitlab_shell_secret:
     description: Shared token authenticating gitlab-shell and gitaly against the rails API
     algorithm: alphanumeric