Allow Dockerfile image builds with Podman #107
Labels
No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
tepichord/milner#107
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
The design
A builder capability in milner: podman build + podman push, run in a user-namespaced pod (hostUsers: false), enabled by raising the worker's
user.max_user_namespaces. One step per pipeline may ask for it, and a test says so.
Why this shape and not privileged:
syntax, locally and in CI.
BuildKit "the replacement for Kaniko" and the no-privileged-containers option; moby ships *.rootless.yaml k8s manifests next to the *.privileged.yaml
ones.
userns is the answer that fits this repo.
built for, and it's the difference between "we have a builder" and "a PR-supplied YAML file can be node root".
Concretely, the step asks for this and nothing more:
(vfs is the honest choice in a pod: no fuse device, no nested overlay. It's slower and uncached. A PVC for podman's storage is the follow-up if the base
re-pull starts to sting, not part of this change.)
The change set
milner, the capability (new):
┌───────────────────────┬──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ File │ Change │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ worker/main.tf │ config_patches gains machine.sysctls: { user.max_user_namespaces: "11255" }. Talos v1.13.9 uses machine.sysctls; the │
│ │ SysctlConfig document is the v1.14+ form. This is a machine config change, so it reboots the single worker — plan it, don't │
│ │ let it happen mid-drill │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ docs/nix.md §15 │ The paragraph "no container builder in the pipeline and no privileged step in it, which is the point: buildah needed a user │
│ │ namespace, these nodes run with user.max_user_namespaces = 0…" is now false in its first half and wrong in its second. │
│ │ Rewrite it: the pipeline has no privileged step; the one step that assembles a Dockerfile takes a user namespace, which is │
│ │ why the worker's sysctl is not 0, and buildah-as-nixbld inside a nix build is still the thing that fails │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ docs/woodpecker-ci.md │ New ### 9n. Building a Dockerfile in a step — the YAML above, the from_secret credential, podman push --tls-verify=true │
│ │ --creds annia-ci:$ZOT_TOKEN, and the digest check. This is also where §9k step 3's hand-run podman build for the │
│ │ secret-extension image gets its automation, and it should say so │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ tests/test_static.py │ The new invariant, in the shape the resources rule already has: walk .woodpecker/*.yaml; no step anywhere may set │
│ │ securityContext.privileged: true; a step setting hostUsers: false or adding capabilities must be named on an explicit │
│ │ allowlist. That converts §15's sentence into something that fails a build when a PR ignores it — and it closes the │
│ │ pre-existing hole (nothing enforces this today; I grepped tests/ for privileged and found nothing) │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ policy/k8s.rego │ Unchanged, deliberately. It keeps denying privileged workloads in rendered layers │
├───────────────────────┼──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ OPERATIONS.md │ One line in the bring-up order: a fresh cluster gets the sysctl from Terraform; an existing one needs the one-time worker │
│ │ reboot │
└───────────────────────┴──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
annia-infra #310, the first consumer:
┌─────────────────────┬────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
│ File │ Change │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ .woodpecker/ci.yaml │ test unchanged; build (PR) = podman build with the block above, push nothing; publish (main) = podman build + podman push, │
│ │ ZOT_TOKEN via the secret extension. No nix build, no skopeo copy docker-archive: │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Dockerfile │ FROM nixos/nix:2.35.2; keep the payload moved to /usr/share/annia so a mounted /workspace can't hide it │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ flake.nix │ Drop packages.annia-image (nothing builds it any more). Keep anniaVersion (still the tag). Keep the skopeo wrapper only if you │
│ │ keep the served-digest verification │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ start-workspace.sh │ Unchanged — podman build -t "pi-dev-container-${NAME}" .. Local and CI are now the same command │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ tests/ │ The same step-security guard as milner's, over annia-infra's own .woodpecker/ │
├─────────────────────┼────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Prereqs │ The secret-extension rule for ZOT_TOKEN (push events only) stops being a formality and becomes load-bearing │
└─────────────────────┴────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
The story, in that same change — story 20 keeps its first half and loses its second, and a new story states the capability:
│ 20. An app's image is the whole of what it runs. I build it with Nix from the app's own source, so it carries the app and its dependencies and nothing
│ else: no base distribution, and nothing mounted at runtime to keep in step. A change pushes one small layer; the layers it did not change are ones
│ the registry already holds.
│
│ 20b. A repository that needs a Dockerfile can build and publish it in the pipeline. The command is the one I run on my laptop — podman build, podman
│ push — and it runs in a user namespace of its own, so root inside the build is not root on the node and the pipeline still has no privileged step.
│ Exactly one step per pipeline may ask for that namespace, and the suite is what says so.
Fallback, with the trigger
If the PoC below fails — hostUsers: false rejected by the pinned k8s, or podman build in a userns pod not working on Talos — then the fallback is the
industry default: one named step with securityContext.privileged: true, still behind the same allowlist test, so the exposure is one auditable step name
instead of "whatever the PR wrote". I'd take that without further argument, and note in §15 that the privilege is real and why it was accepted.
What has to be proven first (three things, one throwaway step)
and Sidero's guide says the sysctl is the only thing Talos needs.)
apps/woodpecker/ocirepository.yaml.)
Naina:
Overview
Issue #107 adds a builder capability to milner: one step per pipeline may run
podman build && podman pushin a user-namespaced pod (hostUsers: false), enabled by raising the worker'suser.max_user_namespaces. The capability lives in consumers' pipelines (annia-infra #310), never in milner's own.woodpecker/, and a repo-level test makes "one named step" enforceable. Three proof items must pass on the cluster first, with one allowlistedprivileged: truestep as the stated fallback.Technical Approach
If buildah fails before STEP 1 with
mounting an overlay over build context directory, try buildah < 1.34 (or a build with #6126), then CERN'sdriver = "overlay"+emptyDir+runAsUser: 1000shape; otherwise jump to step 8. The build-context overlay failure is confirmed at buildah 1.44.01; the exact version where it starts and the#6126fix are unverified.3.7.3→ app3.18.02, andhostUserswas added in Woodpecker 3.16.03:Verified in v3.18.0:
HostUsers: options.HostUsersalongsideSecurityContext: podSecurityContext(..., options.HostUsers)4.worker/main.tf— a fourthconfig_patchesentry5:OPERATIONS.md§8 step 4. Talos v1.13 lists.machine.sysctlsas applyable immediately unverified and the provider'sapply_modedefaults toauto8; if the apply reports a reboot, pinapply_mode = "no_reboot"ontalos_machine_configuration_apply.worker9:tests/test_static.py—test_a_user_namespaced_step_is_named_on_the_allowlist, glob shape oftest_step_pods_take_one_ceiling_from_their_namespace10:Cross-assert the docs half, in the §9m style:
Make
docs/nix.md§15 true again (lines ~211–220)11. Replace theuser.max_user_namespaces = 0sentence12 with the sysctl and the namespaced step; keep "no privileged step in the pipeline"; keep thenixbld/buildah-inside-a-derivation failure as the reason milner's own images stay derivations.Append
### 9n. Building a Dockerfile in a steptodocs/woodpecker-ci.md(file ends at §9m, line 1467)13 with the issue's YAML and the digest check reused from §15d14. Say it automates §9k step 3's hand-run build.Reword story 20 and add
20b.— story 20 loses "no Dockerfile, no container builder in the pipeline"15 (it keeps "no base distribution, and nothing mounted at runtime"):Then widen the story test to accept a suffix, keeping 21/22/23 where they are (renumbering would repoint ~20 references — the failure that test exists to prevent)16:
Update the stale prose that the capability falsifies:
.woodpecker/gates.yamllines ~406–43217 (name the sysctl as the missing precondition; drop "runtime limitation"),test_no_step_in_the_pipeline_is_privileged's docstring ("hostUsers: falsemeasured and closed")18, andtests/test_static.py:1742("a container builder is what story 20 refuses" now points at 20b)19.Fallback, explicitly: if proof item 1 or 2 fails and the alternatives don't rescue it, ship one named
securityContext.privileged: truestep behind the same allowlist test, and say in §15 that the privilege is real and why it was accepted.Implementation Details
worker/main.tfyamlencodepatch:machine.sysctls = { user.max_user_namespaces = "11255" }, with the §15 commentdocs/nix.mddocs/woodpecker-ci.mdpodman login/push+ served-digest comparetests/test_static.pytest_a_user_namespaced_step_is_named_on_the_allowlist; docstring fix at 2593; prose fix at 1742USER_STORIES.md20b.tests/test_docs.pyOPERATIONS.md.woodpecker/gates.yamlgitops/clusters/milner/infrastructure/zot/README.mdprivileged"22; rewordConstraints
policy/k8s.regois unchanged, deliberately: rule 3 reads onlycontainer.securityContext.privileged20, sohostUsers/seccompProfilepass; the namespace is PSAprivileged.capabilitiesis a Drop-only struct in v3.18.021, so the "step adding capabilities" guard is defensive (a future schema change), not currently reachable — say so in the test.test_step_commands_survive_woodpeckers_substitution(${…}names a Woodpecker variable or:-); no new step lands in milner's pipeline.zot.faun-snapper.ts.netpresents a Let's Encrypt cert unverified, so--tls-verify=trueis right;annia/**push needsannia-ci+ZOT_TOKEN.Out-of-scope notes
.woodpecker/ci.yaml,Dockerfile,flake.nix, the same guard over its own.woodpecker/) is a separate repo and a separate change.vfsre-pull cost) is the stated follow-up, not part of this change.AGENTS.md's repo-map "privileged tier is empty, one derivation per image" stays true for milner's own pipeline; read it so it doesn't become false for consumers.Acceptance Criteria
worker/main.tfsetsmachine.sysctls = { user.max_user_namespaces = "11255" };terraform -chdir=worker fmt -checkandvalidatepasstests/test_static.pyfails a.woodpecker/*.yamlstep withprivileged: true, fails ahostUsers: falsestep not on the allowlist, and passes on the current treedocs/nix.md§15 no longer claims the nodes run withuser.max_user_namespaces = 0, and names the sysctl and the namespaced step insteaddocs/woodpecker-ci.md§9n carries the step YAML and the push/digest recipe, and §15 points at itUSER_STORIES.mdhas story 20 reworded and20b.;tests/test_docs.py::test_user_stories_are_the_spec_and_only_storiespassesOPERATIONS.md§8 step 4 states where a fresh cluster gets the sysctl and what an existing one needsTesting Plan
pytest tests/test_static.py -k "namespac or privileged or substitution or command"— new guard, both stale docstrings, and the substitution rule over.woodpecker/*.yamlpytest tests/test_docs.py::test_user_stories_are_the_spec_and_only_stories— the suffix regex and the consecutive-base assertionterraform -chdir=worker fmt -check -recursiveandterraform -chdir=worker validatekubectl runthe userns PoC and runpodman build --storage-driver=vfs+podman push+ the §15d digest compare; thenkubectl get pod userns-poc -o jsonpath='{.spec.hostUsers}'talosctl -n 10.5.0.20 read /proc/sys/user/max_user_namespacesreports non-zero, and ahostUsers: falsepod reaches Running (not the formerfailed to start noop process for unshareerror)Related Issues/PRs
machine.sysctlsrecipe for Talos < v1.14Research Findings
machine.sysctlsis the documented Talos < v1.14 form and has no per-key allowlist; v1.13 lists.machine.sysctlsas applyable immediately, so the provider's defaultapply_mode = "auto"is expected not to reboot.hostUsers: falseis stable and locked at k8s v1.36 (default-on since 1.33); no feature gate or kubelet config is needed, and Talos v1.13's containerd 2.x plus kernel > 6.3 leave the sysctl as the only missing piece.failed to start noop process for unshare: … no space left on deviceis the classicmax_user_namespaces = 0ENOSPC, not a Talos runtime limit — which is why 34 GB free was never relevant.hostUsersstraight toPodSpec.HostUsersand putsseccompProfileon the pod security context, so the issue's YAML serialises as written;runAsUser: 0is permitted only alongsidehostUsers: false.STORAGE_DRIVER=vfsandBUILDAH_ISOLATION=chrootare each valid but not a documented pairing; buildah ≥ ~1.34 can fail on the build-context overlay before STEP 1 regardless of isolation, which is why proof item 2 is a real gate and CERN's overlay-on-emptyDirshape is the first alternative.--digestfile+curl …/manifests/<tag> | sha256sum;annia/**push requires theannia-cicredential.policy/k8s.regoreads neitherhostUsersnorseccompProfile, and the step namespace enforces PSAprivileged, so no policy or namespace change is needed.Pre-Mortem
hostUsers: falsepod with no/dev/fuse, regardless of--isolation=chrootor--storage-driver=vfs. Proof item 2 therefore dies before STEP 1 on the pinnedquay.io/podman/stableimage, and the change ships its privileged fallback instead of the capability it promised.hostUsers: falsepod before touchingworker/main.tf, but the worker still hasuser.max_user_namespaces = 0, so the pod dies with the samefailed to start noop process for unshareENOSPC the issue exists to fix. That result is then read as proof the sysctl is not the fix, and the work is abandoned or over-scoped.test_docs.py's split to\d+[a-z]?either still trips the 90-word and one-sequence assertions or weakens the consecutive-numbering invariant the test exists to enforce, so review rejects it. Meanwhile the new allowlist entry namesannia-infra/.woodpecker/ci.yaml, which does not exist in milner, so "exactly one namespaced step per pipeline" can never match a step here and the guard proves nothing.