Allow Dockerfile image builds with Podman #107

Closed
opened 2026-09-27 00:22:15 -07:00 by tepichord · 1 comment
Owner

The design

A builder capability in milner: podman build + podman push, run in a user-namespaced pod (hostUsers: false), enabled by raising the worker's
user.max_user_namespaces. One step per pipeline may ask for it, and a test says so.

Why this shape and not privileged:

  • It's the same command you run on your laptop (podman build … && podman push …), which is the entire point of A2 — one description, one tool, one
    syntax, locally and in CI.
  • It's the direction the industry actually moved: Kubernetes v1.33 made userns pods enabled-by-default via hostUsers: false; GitLab's docs call rootless
    BuildKit "the replacement for Kaniko" and the no-privileged-containers option; moby ships *.rootless.yaml k8s manifests next to the *.privileged.yaml
    ones.
  • It keeps policy/k8s.rego rule 3 intact. A privileged buildkitd Deployment would need an exemption in your own OPA policy — which is the tell that
    userns is the answer that fits this repo.
  • Root in a userns pod is root inside a namespace that can mount and unshare, and not root on the node. That's exactly what the k8s userns feature was
    built for, and it's the difference between "we have a builder" and "a PR-supplied YAML file can be node root".

Concretely, the step asks for this and nothing more:

      backend_options:
        kubernetes:
          hostUsers: false          # a user namespace; root in here is not root on the node
          securityContext:
            seccompProfile:
              type: Unconfined     # podman's mount/unshare calls; not privilege
      environment:
        BUILDAH_ISOLATION: chroot
        STORAGE_DRIVER: vfs        # no /dev/fuse, no privileged, no overlay-on-overlay

(vfs is the honest choice in a pod: no fuse device, no nested overlay. It's slower and uncached. A PVC for podman's storage is the follow-up if the base
re-pull starts to sting, not part of this change.)

The change set

milner, the capability (new):

┌───────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ File │ Change │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ worker/main.tf │ config_patches gains machine.sysctls: { user.max_user_namespaces: "11255" }. Talos v1.13.9 uses machine.sysctls; the │
│ │ SysctlConfig document is the v1.14+ form. This is a machine config change, so it reboots the single worker — plan it, don't │
│ │ let it happen mid-drill │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ docs/nix.md §15 │ The paragraph "no container builder in the pipeline and no privileged step in it, which is the point: buildah needed a user │
│ │ namespace, these nodes run with user.max_user_namespaces = 0…" is now false in its first half and wrong in its second. │
│ │ Rewrite it: the pipeline has no privileged step; the one step that assembles a Dockerfile takes a user namespace, which is │
│ │ why the worker's sysctl is not 0, and buildah-as-nixbld inside a nix build is still the thing that fails │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ docs/woodpecker-ci.md │ New ### 9n. Building a Dockerfile in a step — the YAML above, the from_secret credential, podman push --tls-verify=true │
│ │ --creds annia-ci:$ZOT_TOKEN, and the digest check. This is also where §9k step 3's hand-run podman build for the │
│ │ secret-extension image gets its automation, and it should say so │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ tests/test_static.py │ The new invariant, in the shape the resources rule already has: walk .woodpecker/*.yaml; no step anywhere may set │
│ │ securityContext.privileged: true; a step setting hostUsers: false or adding capabilities must be named on an explicit │
│ │ allowlist. That converts §15's sentence into something that fails a build when a PR ignores it — and it closes the │
│ │ pre-existing hole (nothing enforces this today; I grepped tests/ for privileged and found nothing) │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ policy/k8s.rego │ Unchanged, deliberately. It keeps denying privileged workloads in rendered layers │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ OPERATIONS.md │ One line in the bring-up order: a fresh cluster gets the sysctl from Terraform; an existing one needs the one-time worker │
│ │ reboot │
└───────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

annia-infra #310, the first consumer:

┌─────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ File │ Change │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ .woodpecker/ci.yaml │ test unchanged; build (PR) = podman build with the block above, push nothing; publish (main) = podman build + podman push, │
│ │ ZOT_TOKEN via the secret extension. No nix build, no skopeo copy docker-archive: │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Dockerfile │ FROM nixos/nix:2.35.2; keep the payload moved to /usr/share/annia so a mounted /workspace can't hide it │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ flake.nix │ Drop packages.annia-image (nothing builds it any more). Keep anniaVersion (still the tag). Keep the skopeo wrapper only if you │
│ │ keep the served-digest verification │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ start-workspace.sh │ Unchanged — podman build -t "pi-dev-container-${NAME}" .. Local and CI are now the same command │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ tests/ │ The same step-security guard as milner's, over annia-infra's own .woodpecker/ │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Prereqs │ The secret-extension rule for ZOT_TOKEN (push events only) stops being a formality and becomes load-bearing │
└─────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘

The story, in that same change — story 20 keeps its first half and loses its second, and a new story states the capability:

│ 20. An app's image is the whole of what it runs. I build it with Nix from the app's own source, so it carries the app and its dependencies and nothing
│ else: no base distribution, and nothing mounted at runtime to keep in step. A change pushes one small layer; the layers it did not change are ones
│ the registry already holds.
│
│ 20b. A repository that needs a Dockerfile can build and publish it in the pipeline. The command is the one I run on my laptop — podman build, podman
│ push — and it runs in a user namespace of its own, so root inside the build is not root on the node and the pipeline still has no privileged step.
│ Exactly one step per pipeline may ask for that namespace, and the suite is what says so.

Fallback, with the trigger

If the PoC below fails — hostUsers: false rejected by the pinned k8s, or podman build in a userns pod not working on Talos — then the fallback is the
industry default: one named step with securityContext.privileged: true, still behind the same allowlist test, so the exposure is one auditable step name
instead of "whatever the PR wrote". I'd take that without further argument, and note in §15 that the privilege is real and why it was accepted.

What has to be proven first (three things, one throwaway step)

  1. Does this cluster's k8s accept hostUsers: false? (kubectl version — userns pods are enabled-by-default from 1.33; Talos v1.13.9 should be past it,
    and Sidero's guide says the sysctl is the only thing Talos needs.)
  2. Does podman build --storage-driver=vfs complete in that pod and does podman push --creds reach zot?
  3. Does the pinned Woodpecker agent understand backend_options.kubernetes.hostUsers? (It's in the 3.17 docs; the chart tag is in
    apps/woodpecker/ocirepository.yaml.)
The design A builder capability in milner: podman build + podman push, run in a user-namespaced pod (hostUsers: false), enabled by raising the worker's user.max_user_namespaces. One step per pipeline may ask for it, and a test says so. Why this shape and not privileged: - It's the same command you run on your laptop (podman build … && podman push …), which is the entire point of A2 — one description, one tool, one syntax, locally and in CI. - It's the direction the industry actually moved: Kubernetes v1.33 made userns pods enabled-by-default via hostUsers: false; GitLab's docs call rootless BuildKit "the replacement for Kaniko" and the no-privileged-containers option; moby ships *.rootless.yaml k8s manifests next to the *.privileged.yaml ones. - It keeps policy/k8s.rego rule 3 intact. A privileged buildkitd Deployment would need an exemption in your own OPA policy — which is the tell that userns is the answer that fits this repo. - Root in a userns pod is root inside a namespace that can mount and unshare, and not root on the node. That's exactly what the k8s userns feature was built for, and it's the difference between "we have a builder" and "a PR-supplied YAML file can be node root". Concretely, the step asks for this and nothing more: ```yaml backend_options: kubernetes: hostUsers: false # a user namespace; root in here is not root on the node securityContext: seccompProfile: type: Unconfined # podman's mount/unshare calls; not privilege environment: BUILDAH_ISOLATION: chroot STORAGE_DRIVER: vfs # no /dev/fuse, no privileged, no overlay-on-overlay ``` (vfs is the honest choice in a pod: no fuse device, no nested overlay. It's slower and uncached. A PVC for podman's storage is the follow-up if the base re-pull starts to sting, not part of this change.) The change set milner, the capability (new): ┌───────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ File │ Change │ ├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ worker/main.tf │ config_patches gains machine.sysctls: { user.max_user_namespaces: "11255" }. Talos v1.13.9 uses machine.sysctls; the │ │ │ SysctlConfig document is the v1.14+ form. This is a machine config change, so it reboots the single worker — plan it, don't │ │ │ let it happen mid-drill │ ├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ docs/nix.md §15 │ The paragraph "no container builder in the pipeline and no privileged step in it, which is the point: buildah needed a user │ │ │ namespace, these nodes run with user.max_user_namespaces = 0…" is now false in its first half and wrong in its second. │ │ │ Rewrite it: the pipeline has no privileged step; the one step that assembles a Dockerfile takes a user namespace, which is │ │ │ why the worker's sysctl is not 0, and buildah-as-nixbld inside a nix build is still the thing that fails │ ├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ docs/woodpecker-ci.md │ New ### 9n. Building a Dockerfile in a step — the YAML above, the from_secret credential, podman push --tls-verify=true │ │ │ --creds annia-ci:$ZOT_TOKEN, and the digest check. This is also where §9k step 3's hand-run podman build for the │ │ │ secret-extension image gets its automation, and it should say so │ ├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ tests/test_static.py │ The new invariant, in the shape the resources rule already has: walk .woodpecker/*.yaml; no step anywhere may set │ │ │ securityContext.privileged: true; a step setting hostUsers: false or adding capabilities must be named on an explicit │ │ │ allowlist. That converts §15's sentence into something that fails a build when a PR ignores it — and it closes the │ │ │ pre-existing hole (nothing enforces this today; I grepped tests/ for privileged and found nothing) │ ├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ policy/k8s.rego │ Unchanged, deliberately. It keeps denying privileged workloads in rendered layers │ ├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ OPERATIONS.md │ One line in the bring-up order: a fresh cluster gets the sysctl from Terraform; an existing one needs the one-time worker │ │ │ reboot │ └───────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ annia-infra #310, the first consumer: ┌─────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐ │ File │ Change │ ├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ .woodpecker/ci.yaml │ test unchanged; build (PR) = podman build with the block above, push nothing; publish (main) = podman build + podman push, │ │ │ ZOT_TOKEN via the secret extension. No nix build, no skopeo copy docker-archive: │ ├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Dockerfile │ FROM nixos/nix:2.35.2; keep the payload moved to /usr/share/annia so a mounted /workspace can't hide it │ ├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ flake.nix │ Drop packages.annia-image (nothing builds it any more). Keep anniaVersion (still the tag). Keep the skopeo wrapper only if you │ │ │ keep the served-digest verification │ ├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ start-workspace.sh │ Unchanged — podman build -t "pi-dev-container-${NAME}" .. Local and CI are now the same command │ ├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ tests/ │ The same step-security guard as milner's, over annia-infra's own .woodpecker/ │ ├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Prereqs │ The secret-extension rule for ZOT_TOKEN (push events only) stops being a formality and becomes load-bearing │ └─────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘ The story, in that same change — story 20 keeps its first half and loses its second, and a new story states the capability: │ 20. An app's image is the whole of what it runs. I build it with Nix from the app's own source, so it carries the app and its dependencies and nothing │ else: no base distribution, and nothing mounted at runtime to keep in step. A change pushes one small layer; the layers it did not change are ones │ the registry already holds. │ │ 20b. A repository that needs a Dockerfile can build and publish it in the pipeline. The command is the one I run on my laptop — podman build, podman │ push — and it runs in a user namespace of its own, so root inside the build is not root on the node and the pipeline still has no privileged step. │ Exactly one step per pipeline may ask for that namespace, and the suite is what says so. Fallback, with the trigger If the PoC below fails — hostUsers: false rejected by the pinned k8s, or podman build in a userns pod not working on Talos — then the fallback is the industry default: one named step with securityContext.privileged: true, still behind the same allowlist test, so the exposure is one auditable step name instead of "whatever the PR wrote". I'd take that without further argument, and note in §15 that the privilege is real and why it was accepted. What has to be proven first (three things, one throwaway step) 1. Does this cluster's k8s accept hostUsers: false? (kubectl version — userns pods are enabled-by-default from 1.33; Talos v1.13.9 should be past it, and Sidero's guide says the sysctl is the only thing Talos needs.) 2. Does podman build --storage-driver=vfs complete in that pod and does podman push --creds reach zot? 3. Does the pinned Woodpecker agent understand backend_options.kubernetes.hostUsers? (It's in the 3.17 docs; the chart tag is in apps/woodpecker/ocirepository.yaml.)
tepichord added this to the MVP project 2026-09-27 00:22:15 -07:00
Author
Owner

Naina:

Overview

Issue #107 adds a builder capability to milner: one step per pipeline may run podman build && podman push in a user-namespaced pod (hostUsers: false), enabled by raising the worker's user.max_user_namespaces. The capability lives in consumers' pipelines (annia-infra #310), never in milner's own .woodpecker/, and a repo-level test makes "one named step" enforceable. Three proof items must pass on the cluster first, with one allowlisted privileged: true step as the stated fallback.

Technical Approach

  1. Prove items 1–2 on a throwaway pod before any file changes — nothing committed:
kubectl run userns-poc -n woodpecker-runtime --image=quay.io/podman/stable:<pin> \
  --overrides='{"spec":{"hostUsers":false,
    "securityContext":{"seccompProfile":{"type":"Unconfined"}}}}' -- sleep 3600
kubectl exec userns-poc -- podman build --storage-driver=vfs -t zot.faun-snapper.ts.net/poc/t:$SHA .
kubectl exec userns-poc -- sh -c 'echo "$ZOT_TOKEN" | podman login -u annia-ci --password-stdin zot.faun-snapper.ts.net'
kubectl exec userns-poc -- podman push --tls-verify=true --digestfile /tmp/d zot.faun-snapper.ts.net/poc/t:$SHA

If buildah fails before STEP 1 with mounting an overlay over build context directory, try buildah < 1.34 (or a build with #6126), then CERN's driver = "overlay" + emptyDir + runAsUser: 1000 shape; otherwise jump to step 8. The build-context overlay failure is confirmed at buildah 1.44.01; the exact version where it starts and the #6126 fix are unverified.

  1. Item 3 is settled by reading, not a step — chart 3.7.3 → app 3.18.02, and hostUsers was added in Woodpecker 3.16.03:
// pipeline/backend/kubernetes/pod.go (v3.18.0)
HostUsers: options.HostUsers,   // PodSpec field, pod level

Verified in v3.18.0: HostUsers: options.HostUsers alongside SecurityContext: podSecurityContext(..., options.HostUsers)4.

  1. Raise the sysctl in worker/main.tf — a fourth config_patches entry5:
yamlencode({
  machine = {
    # A step that builds a Dockerfile takes a user namespace of its own
    # (docs/nix.md §15); Talos defaults user.max_user_namespaces to 0, which
    # is the ENOSPC the gates comment measured (.woodpecker/gates.yaml).
    sysctls = { "user.max_user_namespaces" = "11255" }
  }
}),

Verified: Sidero's usernamespace guide prescribes exactly machine.sysctls: { user.max_user_namespaces: "11255" } for Talos < v1.146; the in-repo measurement of the ENOSPC is at .woodpecker/gates.yaml7.

  1. Check the apply is reboot-free, then write OPERATIONS.md §8 step 4. Talos v1.13 lists .machine.sysctls as applyable immediately unverified and the provider's apply_mode defaults to auto8; if the apply reports a reboot, pin apply_mode = "no_reboot" on talos_machine_configuration_apply.worker9:
4. `terraform -chdir=worker apply`; the first apply also attaches the shared
   data disk. It sets the worker's `user.max_user_namespaces`, which a
   Dockerfile-building step needs (docs/nix.md §15); a fresh cluster gets it
   here, an existing one needs the `no_reboot` apply or the one-time restart.
  1. New invariant in tests/test_static.py — test_a_user_namespaced_step_is_named_on_the_allowlist, glob shape of test_step_pods_take_one_ceiling_from_their_namespace10:
# One named step per pipeline may take a user namespace; nothing may be
# privileged; the backend_options veto is the field that looks right and is not.
NAMESPACED_ALLOWLIST = {"build-image": "annia-infra/.woodpecker/ci.yaml"}  # empty in milner

for path in sorted(pathlib.Path(".woodpecker").glob("*.yaml")):
    for step in _load_yaml(str(path))["steps"]:
        opts = ((step.get("backend_options") or {}).get("kubernetes") or {})
        ...
assert privileged == []
assert {n for n in namespaced if n not in NAMESPACED_ALLOWLIST} == set()

Cross-assert the docs half, in the §9m style:

section = pathlib.Path("docs/woodpecker-ci.md").read_text()
section = section[section.index("### 9n."):]
assert "hostUsers: false" in section and "STORAGE_DRIVER: vfs" in section
  1. Make docs/nix.md §15 true again (lines ~211–220)11. Replace the user.max_user_namespaces = 0 sentence12 with the sysctl and the namespaced step; keep "no privileged step in the pipeline"; keep the nixbld/buildah-inside-a-derivation failure as the reason milner's own images stay derivations.

  2. Append ### 9n. Building a Dockerfile in a step to docs/woodpecker-ci.md (file ends at §9m, line 1467)13 with the issue's YAML and the digest check reused from §15d14. Say it automates §9k step 3's hand-run build.

  3. Reword story 20 and add 20b. — story 20 loses "no Dockerfile, no container builder in the pipeline"15 (it keeps "no base distribution, and nothing mounted at runtime"):

20b. **A Dockerfile builds with the same command I run locally.** A repository
    that needs one builds and publishes it in the pipeline — `podman build`,
    `podman push` — in a user namespace of its own, so root inside the build is
    not root on the node and the pipeline still has no privileged step. Exactly
    one step per pipeline may ask for that namespace, and the suite says so.

Then widen the story test to accept a suffix, keeping 21/22/23 where they are (renumbering would repoint ~20 references — the failure that test exists to prevent)16:

parts = re.split(r"(?m)^(\d+[a-z]?)\. ", stories)
bases = [int(re.match(r"\d+", n).group()) for n in parts[1::2]]
assert bases == sorted(bases)
assert sorted(set(bases)) == list(range(1, len(set(bases)) + 1))
  1. Update the stale prose that the capability falsifies: .woodpecker/gates.yaml lines ~406–43217 (name the sysctl as the missing precondition; drop "runtime limitation"), test_no_step_in_the_pipeline_is_privileged's docstring ("hostUsers: false measured and closed")18, and tests/test_static.py:1742 ("a container builder is what story 20 refuses" now points at 20b)19.

  2. Fallback, explicitly: if proof item 1 or 2 fails and the alternatives don't rescue it, ship one named securityContext.privileged: true step behind the same allowlist test, and say in §15 that the privilege is real and why it was accepted.

Implementation Details

File Change
worker/main.tf One yamlencode patch: machine.sysctls = { user.max_user_namespaces = "11255" }, with the §15 comment
docs/nix.md §15 paragraph rewrite only (lines ~211–220); §15d already carries the push + digest recipe
docs/woodpecker-ci.md Append §9n after §9m; YAML block + podman login/push + served-digest compare
tests/test_static.py New test_a_user_namespaced_step_is_named_on_the_allowlist; docstring fix at 2593; prose fix at 1742
USER_STORIES.md Story 20 reworded; new 20b.
tests/test_docs.py Story split regex and sequence assertion widened for the suffix
OPERATIONS.md One sentence in §8 step 4
.woodpecker/gates.yaml Comment block ~406–432 rewritten
gitops/clusters/milner/infrastructure/zot/README.md "Authentication" still says a PR pipeline "runs privileged"22; reword

Constraints

  • policy/k8s.rego is unchanged, deliberately: rule 3 reads only container.securityContext.privileged20, so hostUsers/seccompProfile pass; the namespace is PSA privileged.
  • capabilities is a Drop-only struct in v3.18.021, so the "step adding capabilities" guard is defensive (a future schema change), not currently reachable — say so in the test.
  • Step commands stay bound by test_step_commands_survive_woodpeckers_substitution (${…} names a Woodpecker variable or :-); no new step lands in milner's pipeline.
  • zot.faun-snapper.ts.net presents a Let's Encrypt cert unverified, so --tls-verify=true is right; annia/** push needs annia-ci + ZOT_TOKEN.

Out-of-scope notes

  • annia-infra #310 (the consumer: .woodpecker/ci.yaml, Dockerfile, flake.nix, the same guard over its own .woodpecker/) is a separate repo and a separate change.
  • A PVC for podman storage (the vfs re-pull cost) is the stated follow-up, not part of this change.
  • AGENTS.md's repo-map "privileged tier is empty, one derivation per image" stays true for milner's own pipeline; read it so it doesn't become false for consumers.

Acceptance Criteria

  • worker/main.tf sets machine.sysctls = { user.max_user_namespaces = "11255" }; terraform -chdir=worker fmt -check and validate pass
  • tests/test_static.py fails a .woodpecker/*.yaml step with privileged: true, fails a hostUsers: false step not on the allowlist, and passes on the current tree
  • docs/nix.md §15 no longer claims the nodes run with user.max_user_namespaces = 0, and names the sysctl and the namespaced step instead
  • docs/woodpecker-ci.md §9n carries the step YAML and the push/digest recipe, and §15 points at it
  • USER_STORIES.md has story 20 reworded and 20b.; tests/test_docs.py::test_user_stories_are_the_spec_and_only_stories passes
  • OPERATIONS.md §8 step 4 states where a fresh cluster gets the sysctl and what an existing one needs
  • Proof items 1–3 have a recorded outcome (PR description), and the fallback is chosen explicitly if 1–2 failed

Testing Plan

  • pytest tests/test_static.py -k "namespac or privileged or substitution or command" — new guard, both stale docstrings, and the substitution rule over .woodpecker/*.yaml
  • pytest tests/test_docs.py::test_user_stories_are_the_spec_and_only_stories — the suffix regex and the consecutive-base assertion
  • terraform -chdir=worker fmt -check -recursive and terraform -chdir=worker validate
  • Manual, on the cluster: kubectl run the userns PoC and run podman build --storage-driver=vfs + podman push + the §15d digest compare; then kubectl get pod userns-poc -o jsonpath='{.spec.hostUsers}'
  • Manual, after the sysctl apply: talosctl -n 10.5.0.20 read /proc/sys/user/max_user_namespaces reports non-zero, and a hostUsers: false pod reaches Running (not the former failed to start noop process for unshare error)
  • Consumer: annia-infra #310 (first pipeline to use the capability)
  • Sidero, User Namespaces — the machine.sysctls recipe for Talos < v1.14
  • Woodpecker PR #6710 — k8s userns support, shipped in 3.16.0 (chart 3.7.3 → app 3.18.0)
  • siderolabs/talos #8554 — the exact ENOSPC this issue fixes
  • containers/buildah #6910 — the overlay-context failure that gates proof item 2

Research Findings

  • machine.sysctls is the documented Talos < v1.14 form and has no per-key allowlist; v1.13 lists .machine.sysctls as applyable immediately, so the provider's default apply_mode = "auto" is expected not to reboot.
  • hostUsers: false is stable and locked at k8s v1.36 (default-on since 1.33); no feature gate or kubelet config is needed, and Talos v1.13's containerd 2.x plus kernel > 6.3 leave the sysctl as the only missing piece.
  • The gates comment's failed to start noop process for unshare: … no space left on device is the classic max_user_namespaces = 0 ENOSPC, not a Talos runtime limit — which is why 34 GB free was never relevant.
  • Woodpecker 3.18.0 wires hostUsers straight to PodSpec.HostUsers and puts seccompProfile on the pod security context, so the issue's YAML serialises as written; runAsUser: 0 is permitted only alongside hostUsers: false.
  • STORAGE_DRIVER=vfs and BUILDAH_ISOLATION=chroot are each valid but not a documented pairing; buildah ≥ ~1.34 can fail on the build-context overlay before STEP 1 regardless of isolation, which is why proof item 2 is a real gate and CERN's overlay-on-emptyDir shape is the first alternative.
  • Zot serves a publicly-trusted Let's Encrypt cert on the tailnet name, and the digest check to reuse is §15d's --digestfile + curl …/manifests/<tag> | sha256sum; annia/** push requires the annia-ci credential.
  • OPA policy/k8s.rego reads neither hostUsers nor seccompProfile, and the step namespace enforces PSA privileged, so no policy or namespace change is needed.

Pre-Mortem

  1. The unprivileged build never works, so the capability is never proven. Buildah >= ~1.34 mounts an overlay over the build context unconditionally and fails in a hostUsers: false pod with no /dev/fuse, regardless of --isolation=chroot or --storage-driver=vfs. Proof item 2 therefore dies before STEP 1 on the pinned quay.io/podman/stable image, and the change ships its privileged fallback instead of the capability it promised.
  2. The PoC is sequenced before the sysctl it depends on, producing a false negative. The plan runs the throwaway hostUsers: false pod before touching worker/main.tf, but the worker still has user.max_user_namespaces = 0, so the pod dies with the same failed to start noop process for unshare ENOSPC the issue exists to fix. That result is then read as proof the sysctl is not the fix, and the work is abandoned or over-scoped.
  3. The story-20b/test change does not land, and the central guard is vacuous in this repo. Widening test_docs.py's split to \d+[a-z]? either still trips the 90-word and one-sequence assertions or weakens the consecutive-numbering invariant the test exists to enforce, so review rejects it. Meanwhile the new allowlist entry names annia-infra/.woodpecker/ci.yaml, which does not exist in milner, so "exactly one namespaced step per pipeline" can never match a step here and the guard proves nothing.
Naina: ## Overview Issue #107 adds a **builder capability** to milner: one step per pipeline may run `podman build && podman push` in a user-namespaced pod (`hostUsers: false`), enabled by raising the worker's `user.max_user_namespaces`. The capability lives in *consumers'* pipelines (annia-infra #310), never in milner's own `.woodpecker/`, and a repo-level test makes "one named step" enforceable. Three proof items must pass on the cluster first, with one allowlisted `privileged: true` step as the stated fallback. ## Technical Approach 1. **Prove items 1–2 on a throwaway pod before any file changes** — nothing committed: ```bash kubectl run userns-poc -n woodpecker-runtime --image=quay.io/podman/stable:<pin> \ --overrides='{"spec":{"hostUsers":false, "securityContext":{"seccompProfile":{"type":"Unconfined"}}}}' -- sleep 3600 kubectl exec userns-poc -- podman build --storage-driver=vfs -t zot.faun-snapper.ts.net/poc/t:$SHA . kubectl exec userns-poc -- sh -c 'echo "$ZOT_TOKEN" | podman login -u annia-ci --password-stdin zot.faun-snapper.ts.net' kubectl exec userns-poc -- podman push --tls-verify=true --digestfile /tmp/d zot.faun-snapper.ts.net/poc/t:$SHA ``` If buildah fails *before STEP 1* with `mounting an overlay over build context directory`, try buildah < 1.34 (or a build with #6126), then CERN's `driver = "overlay"` + `emptyDir` + `runAsUser: 1000` shape; otherwise jump to step 8. The build-context overlay failure is confirmed at buildah 1.44.0<sup>[1](https://github.com/podman-container-tools/buildah/issues/6910)</sup>; the exact version where it starts and the `#6126` fix are _<sup>unverified</sup>_. 2. **Item 3 is settled by reading, not a step** — chart `3.7.3` → app `3.18.0`<sup>[2](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/Chart.yaml)</sup>, and `hostUsers` was added in Woodpecker 3.16.0<sup>[3](https://github.com/woodpecker-ci/woodpecker/pull/6710)</sup>: ```go // pipeline/backend/kubernetes/pod.go (v3.18.0) HostUsers: options.HostUsers, // PodSpec field, pod level ``` _Verified in v3.18.0: `HostUsers: options.HostUsers` alongside `SecurityContext: podSecurityContext(..., options.HostUsers)`<sup>[4](https://github.com/woodpecker-ci/woodpecker/blob/v3.18.0/pipeline/backend/kubernetes/pod.go#L195-L196)</sup>._ 3. **Raise the sysctl in `worker/main.tf`** — a fourth `config_patches` entry<sup>[5](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/worker/main.tf?display=source#L385)</sup>: ```hcl yamlencode({ machine = { # A step that builds a Dockerfile takes a user namespace of its own # (docs/nix.md §15); Talos defaults user.max_user_namespaces to 0, which # is the ENOSPC the gates comment measured (.woodpecker/gates.yaml). sysctls = { "user.max_user_namespaces" = "11255" } } }), ``` > Verified: Sidero's usernamespace guide prescribes exactly `machine.sysctls: { user.max_user_namespaces: "11255" }` for Talos < v1.14<sup>[6](https://docs.siderolabs.com/kubernetes-guides/security/usernamespace)</sup>; the in-repo measurement of the ENOSPC is at `.woodpecker/gates.yaml`<sup>[7](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/.woodpecker/gates.yaml?display=source#L411)</sup>. 4. **Check the apply is reboot-free, then write `OPERATIONS.md` §8 step 4.** Talos v1.13 lists `.machine.sysctls` as applyable immediately _<sup>unverified</sup>_ and the provider's `apply_mode` defaults to `auto`<sup>[8](https://github.com/siderolabs/terraform-provider-talos/blob/v0.12.0/pkg/talos/talos_machine_configuration_apply_resource.go#L109-L117)</sup>; if the apply reports a reboot, pin `apply_mode = "no_reboot"` on `talos_machine_configuration_apply.worker`<sup>[9](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/worker/main.tf?display=source#L381)</sup>: ```markdown 4. `terraform -chdir=worker apply`; the first apply also attaches the shared data disk. It sets the worker's `user.max_user_namespaces`, which a Dockerfile-building step needs (docs/nix.md §15); a fresh cluster gets it here, an existing one needs the `no_reboot` apply or the one-time restart. ``` 5. **New invariant in `tests/test_static.py`** — `test_a_user_namespaced_step_is_named_on_the_allowlist`, glob shape of `test_step_pods_take_one_ceiling_from_their_namespace`<sup>[10](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/tests/test_static.py?display=source#L2336)</sup>: ```python # One named step per pipeline may take a user namespace; nothing may be # privileged; the backend_options veto is the field that looks right and is not. NAMESPACED_ALLOWLIST = {"build-image": "annia-infra/.woodpecker/ci.yaml"} # empty in milner for path in sorted(pathlib.Path(".woodpecker").glob("*.yaml")): for step in _load_yaml(str(path))["steps"]: opts = ((step.get("backend_options") or {}).get("kubernetes") or {}) ... assert privileged == [] assert {n for n in namespaced if n not in NAMESPACED_ALLOWLIST} == set() ``` Cross-assert the docs half, in the §9m style: ```python section = pathlib.Path("docs/woodpecker-ci.md").read_text() section = section[section.index("### 9n."):] assert "hostUsers: false" in section and "STORAGE_DRIVER: vfs" in section ``` 6. **Make `docs/nix.md` §15 true again** (lines ~211–220)<sup>[11](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/docs/nix.md?display=source#L212)</sup>. Replace the `user.max_user_namespaces = 0` sentence<sup>[12](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/docs/nix.md?display=source#L219)</sup> with the sysctl and the namespaced step; keep "no privileged step in the pipeline"; keep the `nixbld`/buildah-inside-a-derivation failure as the reason milner's own images stay derivations. 7. **Append `### 9n. Building a Dockerfile in a step` to `docs/woodpecker-ci.md`** (file ends at §9m, line 1467)<sup>[13](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/docs/woodpecker-ci.md?display=source#L1347)</sup> with the issue's YAML and the digest check reused from §15d<sup>[14](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/docs/nix.md?display=source#L568)</sup>. Say it automates §9k step 3's hand-run build. 8. **Reword story 20 and add `20b.`** — story 20 loses "no Dockerfile, no container builder in the pipeline"<sup>[15](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/USER_STORIES.md?display=source#L103)</sup> (it keeps "no base distribution, and nothing mounted at runtime"): ```markdown 20b. **A Dockerfile builds with the same command I run locally.** A repository that needs one builds and publishes it in the pipeline — `podman build`, `podman push` — in a user namespace of its own, so root inside the build is not root on the node and the pipeline still has no privileged step. Exactly one step per pipeline may ask for that namespace, and the suite says so. ``` Then widen the story test to accept a suffix, keeping 21/22/23 where they are (renumbering would repoint ~20 references — the failure that test exists to prevent)<sup>[16](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/tests/test_docs.py?display=source#L76)</sup>: ```python parts = re.split(r"(?m)^(\d+[a-z]?)\. ", stories) bases = [int(re.match(r"\d+", n).group()) for n in parts[1::2]] assert bases == sorted(bases) assert sorted(set(bases)) == list(range(1, len(set(bases)) + 1)) ``` 9. **Update the stale prose that the capability falsifies**: `.woodpecker/gates.yaml` lines ~406–432<sup>[17](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/.woodpecker/gates.yaml?display=source#L406)</sup> (name the sysctl as the missing precondition; drop "runtime limitation"), `test_no_step_in_the_pipeline_is_privileged`'s docstring ("`hostUsers: false` measured and closed")<sup>[18](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/tests/test_static.py?display=source#L2599)</sup>, and `tests/test_static.py:1742` ("a container builder is what story 20 refuses" now points at 20b)<sup>[19](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/tests/test_static.py?display=source#L1742)</sup>. 10. **Fallback, explicitly**: if proof item 1 or 2 fails and the alternatives don't rescue it, ship one named `securityContext.privileged: true` step behind the same allowlist test, and say in §15 that the privilege is real and why it was accepted. ## Implementation Details | File | Change | |---|---| | `worker/main.tf` | One `yamlencode` patch: `machine.sysctls = { user.max_user_namespaces = "11255" }`, with the §15 comment | | `docs/nix.md` | §15 paragraph rewrite only (lines ~211–220); §15d already carries the push + digest recipe | | `docs/woodpecker-ci.md` | Append §9n after §9m; YAML block + `podman login`/`push` + served-digest compare | | `tests/test_static.py` | New `test_a_user_namespaced_step_is_named_on_the_allowlist`; docstring fix at 2593; prose fix at 1742 | | `USER_STORIES.md` | Story 20 reworded; new `20b.` | | `tests/test_docs.py` | Story split regex and sequence assertion widened for the suffix | | `OPERATIONS.md` | One sentence in §8 step 4 | | `.woodpecker/gates.yaml` | Comment block ~406–432 rewritten | | `gitops/clusters/milner/infrastructure/zot/README.md` | "Authentication" still says a PR pipeline "runs `privileged`"<sup>[22](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/gitops/clusters/milner/infrastructure/zot/README.md?display=source#L213)</sup>; reword | **Constraints** - `policy/k8s.rego` is **unchanged, deliberately**: rule 3 reads only `container.securityContext.privileged`<sup>[20](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/policy/k8s.rego?display=source#L99)</sup>, so `hostUsers`/`seccompProfile` pass; the namespace is PSA `privileged`. - `capabilities` is a Drop-only struct in v3.18.0<sup>[21](https://github.com/woodpecker-ci/woodpecker/blob/v3.18.0/pipeline/backend/kubernetes/backend_options.go#L90)</sup>, so the "step adding capabilities" guard is defensive (a future schema change), not currently reachable — say so in the test. - Step commands stay bound by `test_step_commands_survive_woodpeckers_substitution` (`${…}` names a Woodpecker variable or `:-`); no new step lands in milner's pipeline. - `zot.faun-snapper.ts.net` presents a Let's Encrypt cert _<sup>unverified</sup>_, so `--tls-verify=true` is right; `annia/**` push needs `annia-ci` + `ZOT_TOKEN`. **Out-of-scope notes** - annia-infra #310 (the consumer: `.woodpecker/ci.yaml`, `Dockerfile`, `flake.nix`, the same guard over its own `.woodpecker/`) is a separate repo and a separate change. - A PVC for podman storage (the `vfs` re-pull cost) is the stated follow-up, not part of this change. - `AGENTS.md`'s repo-map "privileged tier is empty, one derivation per image" stays true for milner's own pipeline; read it so it doesn't become false for consumers. ## Acceptance Criteria - [ ] `worker/main.tf` sets `machine.sysctls = { user.max_user_namespaces = "11255" }`; `terraform -chdir=worker fmt -check` and `validate` pass - [ ] `tests/test_static.py` fails a `.woodpecker/*.yaml` step with `privileged: true`, fails a `hostUsers: false` step not on the allowlist, and passes on the current tree - [ ] `docs/nix.md` §15 no longer claims the nodes run with `user.max_user_namespaces = 0`, and names the sysctl and the namespaced step instead - [ ] `docs/woodpecker-ci.md` §9n carries the step YAML and the push/digest recipe, and §15 points at it - [ ] `USER_STORIES.md` has story 20 reworded and `20b.`; `tests/test_docs.py::test_user_stories_are_the_spec_and_only_stories` passes - [ ] `OPERATIONS.md` §8 step 4 states where a fresh cluster gets the sysctl and what an existing one needs - [ ] Proof items 1–3 have a recorded outcome (PR description), and the fallback is chosen explicitly if 1–2 failed ## Testing Plan - `pytest tests/test_static.py -k "namespac or privileged or substitution or command"` — new guard, both stale docstrings, and the substitution rule over `.woodpecker/*.yaml` - `pytest tests/test_docs.py::test_user_stories_are_the_spec_and_only_stories` — the suffix regex and the consecutive-base assertion - `terraform -chdir=worker fmt -check -recursive` and `terraform -chdir=worker validate` - Manual, on the cluster: `kubectl run` the userns PoC and run `podman build --storage-driver=vfs` + `podman push` + the §15d digest compare; then `kubectl get pod userns-poc -o jsonpath='{.spec.hostUsers}'` - Manual, after the sysctl apply: `talosctl -n 10.5.0.20 read /proc/sys/user/max_user_namespaces` reports non-zero, and a `hostUsers: false` pod reaches Running (not the former `failed to start noop process for unshare` error) ## Related Issues/PRs - Consumer: annia-infra #310 (first pipeline to use the capability) - Sidero, [User Namespaces](https://docs.siderolabs.com/kubernetes-guides/security/usernamespace) — the `machine.sysctls` recipe for Talos < v1.14 - Woodpecker PR [#6710](https://github.com/woodpecker-ci/woodpecker/pull/6710) — k8s userns support, shipped in 3.16.0 (chart 3.7.3 → app 3.18.0) - siderolabs/talos [#8554](https://github.com/siderolabs/talos/issues/8554) — the exact ENOSPC this issue fixes - containers/buildah [#6910](https://github.com/podman-container-tools/buildah/issues/6910) — the overlay-context failure that gates proof item 2 ## Research Findings - `machine.sysctls` is the documented Talos < v1.14 form and has no per-key allowlist; v1.13 lists `.machine.sysctls` as applyable immediately, so the provider's default `apply_mode = "auto"` is expected **not** to reboot. - `hostUsers: false` is stable and locked at k8s v1.36 (default-on since 1.33); no feature gate or kubelet config is needed, and Talos v1.13's containerd 2.x plus kernel > 6.3 leave the sysctl as the only missing piece. - The gates comment's `failed to start noop process for unshare: … no space left on device` is the classic `max_user_namespaces = 0` ENOSPC, not a Talos runtime limit — which is why 34 GB free was never relevant. - Woodpecker 3.18.0 wires `hostUsers` straight to `PodSpec.HostUsers` and puts `seccompProfile` on the pod security context, so the issue's YAML serialises as written; `runAsUser: 0` is permitted only alongside `hostUsers: false`. - `STORAGE_DRIVER=vfs` and `BUILDAH_ISOLATION=chroot` are each valid but not a documented pairing; buildah ≥ ~1.34 can fail on the build-context overlay before STEP 1 regardless of isolation, which is why proof item 2 is a real gate and CERN's overlay-on-`emptyDir` shape is the first alternative. - Zot serves a publicly-trusted Let's Encrypt cert on the tailnet name, and the digest check to reuse is §15d's `--digestfile` + `curl …/manifests/<tag> | sha256sum`; `annia/**` push requires the `annia-ci` credential. - OPA `policy/k8s.rego` reads neither `hostUsers` nor `seccompProfile`, and the step namespace enforces PSA `privileged`, so no policy or namespace change is needed. ## Pre-Mortem 1. **The unprivileged build never works, so the capability is never proven.** Buildah >= ~1.34 mounts an overlay over the build context unconditionally and fails in a `hostUsers: false` pod with no `/dev/fuse`, regardless of `--isolation=chroot` or `--storage-driver=vfs`. Proof item 2 therefore dies before STEP 1 on the pinned `quay.io/podman/stable` image, and the change ships its privileged fallback instead of the capability it promised. 2. **The PoC is sequenced before the sysctl it depends on, producing a false negative.** The plan runs the throwaway `hostUsers: false` pod before touching `worker/main.tf`, but the worker still has `user.max_user_namespaces = 0`, so the pod dies with the same `failed to start noop process for unshare` ENOSPC the issue exists to fix. That result is then read as proof the sysctl is not the fix, and the work is abandoned or over-scoped. 3. **The story-20b/test change does not land, and the central guard is vacuous in this repo.** Widening `test_docs.py`'s split to `\d+[a-z]?` either still trips the 90-word and one-sequence assertions or weakens the consecutive-numbering invariant the test exists to enforce, so review rejects it. Meanwhile the new allowlist entry names `annia-infra/.woodpecker/ci.yaml`, which does not exist in milner, so "exactly one namespaced step per pipeline" can never match a step here and the guard proves nothing.
tepichord 2026-09-29 17:27:44 -07:00
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner#107
No description provided.