Allow Dockerfile image builds with Podman

tepichord commented

2026-09-27 02:08:41 -07:00

Owner

Closes #107

A builder capability for consumer repositories: one named step per pipeline may run podman build / podman push in a pod that owns a user namespace (hostUsers: false), enabled by raising the worker's user.max_user_namespaces to 11255. Milner's own pipeline is unchanged — its images are derivations — and a suite guard makes "one step per pipeline" a count rather than a claim.

What changed

File	Change
`worker/main.tf`	A fourth `config_patches` entry: `machine.sysctls."user.max_user_namespaces" = "11255"` — asserted by `worker/tests/plan.tftest.hcl`
`tests/test_static.py`	`test_a_user_namespaced_step_is_named_on_the_allowlist` (empty here; the shape a consumer fills), plus the prose in the privileged-tier test, the enqueuer-image test and the "no container builder" tool list
`docs/woodpecker-ci.md`	§9n: the step shape, the credential, the served-digest compare, and the measurement below
`docs/nix.md`	§15 says again what is true — a derivation is a choice, not a limitation — and points at §9n
`USER_STORIES.md`, `tests/test_docs.py`	Story 20 reworded, `20b.` added, the story-sequence test widened for a suffix
`OPERATIONS.md`, `.woodpecker/gates.yaml`, `ARCHITECTURE.md`, `gitops/…/zot/README.md`	The prose the capability falsifies
`AGENTS.md`	The `tests/` row records the invariant for whoever reads the map

Proof, on this cluster

Item 1 — the sysctl is live. terraform -chdir=worker apply reported 0 added, 1 changed, 0 destroyed, in place, no reboot — Talos lists .machine.sysctls in CanApplyImmediate, so the plan's worry about a reboot (and its apply_mode remedy) did not apply. A throwaway pod then read /proc/sys/user/max_user_namespaces from the worker: 11255.

Item 2 — the build, the push, and the bytes. Throwaway pod in woodpecker-runtime: hostUsers: false, quay.io/podman/stable:v5.8.7-immutable, STORAGE_DRIVER=vfs, BUILDAH_ISOLATION=chroot, seccompProfile: Unconfined, build context on an emptyDir.

unshare -U true succeeds from inside the pod;
podman build --storage-driver=vfs → Successfully tagged … (podman 5.8.7);
podman push --tls-verify=true → stored sha256:3dcde6e0d00e74f52bebe41c3e81e6bdcd3e71c0fc6c186c139673ed52393b0e;
the manifest the registry serves for that tag hashes to the same digest — MATCH.

The one arrangement the test needed: the context on a volume rather than on the container's own overlayfs, where buildah ≥ 1.34 fails before STEP 1 (containers/buildah#6910). §9n records it, because a context built inside a container needs the same care.

Item 3 — the field reaches the sandbox. The pod reached Running with spec.hostUsers: false. That is exactly what failed before the sysctl: containerd could not create the sandbox at all — failed to start noop process for unshare: … no space left on device with 34 GB free on the node, which is user.max_user_namespaces = 0 and not a full disk.

Not exercised: the credentialed push. The PoC pushed anonymously to preview/poc:userns (preview/** is anonymously writable), which proves build + push + served bytes from a namespaced pod without a Vault read. The annia-ci / ZOT_TOKEN login belongs to annia-infra#310 and depends on milner#110: the secret extension has no rule for that repository yet, so a step asking for ZOT_TOKEN would not even compile. preview/poc:userns is a disposable tag under zot's preview/** retention; deleting it early needs the annia-ci credential.

Fallback

Not taken. If the build had failed, the plan's fallback was one named privileged: true step behind the same allowlist, with §15 saying the privilege is real and why. It was not needed, so no step in any pipeline is privileged and policy/k8s.rego needed no exemption — which is the point of the user namespace.

Closes #107 A builder capability for consumer repositories: one named step per pipeline may run `podman build` / `podman push` in a pod that owns a user namespace (`hostUsers: false`), enabled by raising the worker's `user.max_user_namespaces` to `11255`. Milner's own pipeline is unchanged — its images are derivations — and a suite guard makes "one step per pipeline" a count rather than a claim. ## What changed | File | Change | |---|---| | `worker/main.tf` | A fourth `config_patches` entry: `machine.sysctls."user.max_user_namespaces" = "11255"` — asserted by `worker/tests/plan.tftest.hcl` | | `tests/test_static.py` | `test_a_user_namespaced_step_is_named_on_the_allowlist` (empty here; the shape a consumer fills), plus the prose in the privileged-tier test, the enqueuer-image test and the "no container builder" tool list | | `docs/woodpecker-ci.md` | §9n: the step shape, the credential, the served-digest compare, and the measurement below | | `docs/nix.md` | §15 says again what is true — a derivation is a choice, not a limitation — and points at §9n | | `USER_STORIES.md`, `tests/test_docs.py` | Story 20 reworded, `20b.` added, the story-sequence test widened for a suffix | | `OPERATIONS.md`, `.woodpecker/gates.yaml`, `ARCHITECTURE.md`, `gitops/…/zot/README.md` | The prose the capability falsifies | | `AGENTS.md` | The `tests/` row records the invariant for whoever reads the map | ## Proof, on this cluster **Item 1 — the sysctl is live.** `terraform -chdir=worker apply` reported `0 added, 1 changed, 0 destroyed`, in place, no reboot — Talos lists `.machine.sysctls` in `CanApplyImmediate`, so the plan's worry about a reboot (and its `apply_mode` remedy) did not apply. A throwaway pod then read `/proc/sys/user/max_user_namespaces` from the worker: `11255`. **Item 2 — the build, the push, and the bytes.** Throwaway pod in `woodpecker-runtime`: `hostUsers: false`, `quay.io/podman/stable:v5.8.7-immutable`, `STORAGE_DRIVER=vfs`, `BUILDAH_ISOLATION=chroot`, `seccompProfile: Unconfined`, build context on an `emptyDir`. - `unshare -U true` succeeds from inside the pod; - `podman build --storage-driver=vfs` → `Successfully tagged …` (podman 5.8.7); - `podman push --tls-verify=true` → stored `sha256:3dcde6e0d00e74f52bebe41c3e81e6bdcd3e71c0fc6c186c139673ed52393b0e`; - the manifest the registry **serves** for that tag hashes to the same digest — MATCH. The one arrangement the test needed: the context on a volume rather than on the container's own overlayfs, where buildah ≥ 1.34 fails before STEP 1 (containers/buildah#6910). §9n records it, because a context built *inside* a container needs the same care. **Item 3 — the field reaches the sandbox.** The pod reached `Running` with `spec.hostUsers: false`. That is exactly what failed before the sysctl: containerd could not create the sandbox at all — `failed to start noop process for unshare: … no space left on device` with 34 GB free on the node, which is `user.max_user_namespaces = 0` and not a full disk. **Not exercised: the credentialed push.** The PoC pushed anonymously to `preview/poc:userns` (`preview/**` is anonymously writable), which proves build + push + served bytes from a namespaced pod without a Vault read. The `annia-ci` / `ZOT_TOKEN` login belongs to annia-infra#310 and depends on milner#110: the secret extension has no rule for that repository yet, so a step asking for `ZOT_TOKEN` would not even compile. `preview/poc:userns` is a disposable tag under zot's `preview/**` retention; deleting it early needs the `annia-ci` credential. ## Fallback Not taken. If the build had failed, the plan's fallback was one named `privileged: true` step behind the same allowlist, with §15 saying the privilege is real and why. It was not needed, so no step in any pipeline is privileged and `policy/k8s.rego` needed no exemption — which is the point of the user namespace.

tepichord added 8 commits

2026-09-27 02:08:41 -07:00

feat(worker): raise user.max_user_namespaces so a Dockerfile-building step gets a user namespace 4b55125f41

Talos defaults user.max_user_namespaces to 0, which is the ENOSPC the
gates comment measured; a step that runs podman build in a user-namespaced
pod (hostUsers: false) cannot create its namespace without it. The patch
is appended as the fourth config_patches entry so the vault/data
UserVolumeConfig indices stay put, and both plan.tftest.hcl length
assertions follow to 4 — with a new assertion that the fourth patch
carries the sysctl itself.

Refs: #107

docs(nix): make §15 true again and give Dockerfile builds their own §9n e25366595c

§15 claimed no container builder in the pipeline and nodes running with
user.max_user_namespaces = 0; the worker's sysctl is now 11255, so the
paragraph names the sysctl and the one namespaced Dockerfile-building
step (hostUsers: false) instead, and keeps buildah-as-nixbld as the
reason milner's own images stay derivations. §9n carries the step shape
— the backend_options YAML, login/push on stdin, the digestfile compare
reused from §15d — and automates §9k step 3's hand-run build; §15 and
§15d both point at it.

Refs: #107

test(static): gate namespaced steps behind an explicit allowlist c7e4d595ff

test_a_user_namespaced_step_is_named_on_the_allowlist walks every
.woodpecker/*.yaml and fails a privileged step (field or the
backend_options copy), a step adding capabilities (defensive — the field
is Drop-only in v3.18.0), and a hostUsers: false step whose name is not
on NAMESPACED_ALLOWLIST — empty in milner, so no step of this pipeline
may take a namespace; a consumer names itself by step and file. It
cross-asserts docs/woodpecker-ci.md §9n carries the step's shape.

Two stale claims the capability falsifies are reworded with it: the
privileged test's docstring no longer says the nodes run with
user.max_user_namespaces = 0 or that hostUsers: false was closed (it is
now the sanctioned route), and the enqueuer assertion about "a fetch
plus a container builder" points at story 20b, which now governs
container builders.

Refs: #107

feat(stories): restate story 20 and add 20b for the Dockerfile builder 1612790cae

Story 20 loses "no Dockerfile, no container builder in the pipeline" —
the builder is now a sanctioned capability — and keeps the half it still
owns (no base distribution, nothing mounted at runtime). Story 20b
states the capability: podman build/push in a user namespace of its own,
no privileged step, exactly one namespaced step per pipeline enforced by
the suite. The story test accepts a letter suffix: ids are split on
\d+[a-z]?, ordering is asserted on the bases, and the bases must still
be one consecutive sequence from 1 — all four assertion layers kept, and
21/22/23 unrenumbered.

Refs: #107

docs(operations): state where the userns sysctl comes from in §8 step 4 44865f7c8d

A fresh cluster gets user.max_user_namespaces from the worker apply; an
existing one needs the no_reboot apply or the one-time worker restart
before a Dockerfile-building step has a namespace to take.

Refs: #107

docs: retract the claims the userns sysctl and the allowlist falsify 2d358561c6

The gates comment's buildah measurements are kept as history, but the
sysctl is named as the missing precondition (the ENOSPC was
user.max_user_namespaces = 0, not a runtime limit or a full disk), and
"nothing narrower than privileged" is scoped to capabilities, with the
namespaced route behind the allowlist named. The zot README no longer
says the PR pipeline runs privileged, and ARCHITECTURE.md no longer says
the nodes disable unprivileged user namespaces.

Refs: #107

docs(tests): trim redundant comments in the namespaced-step guards fa01882ace

Refs: #107

test(docs): reject a duplicate story number in the story sequence

ci/woodpecker/pr/gates Pipeline was successful

Details

7b0d4bfa07

The suffix widening split the exact-sequence assertion into a base
sequence; that let a repeated number (two `20.`, or `20b.` twice) pass,
where the old range check rejected it. Restore the uniqueness half so a
story number still names exactly one story.

tepichord added the

pr

reviewing

label

2026-09-27 02:08:41 -07:00

tepichord added this to the MVP project

2026-09-29 16:35:08 -07:00

tepichord added 1 commit

2026-09-29 16:57:20 -07:00

docs(userns): what the live proof settled, and the guard's own builder

ci/woodpecker/pr/gates Pipeline was successful

Details

dbae6b719c

The capability is proven on the cluster, not read out of the source: a
`hostUsers: false` pod reaches Running where at `user.max_user_namespaces = 0` it
could not be created at all, `podman build --storage-driver=vfs` and `podman
push` both work inside it, and the registry serves the digest the push returned.
Four things follow from that.

`docs/nix.md` §15 loses the `nixbld` reason it was carrying: its only evidence
was the sysctl being `0`, which this change removes. It says what is true
instead — the derivation is a choice, one description with nothing to keep in
step — and keeps the history as history.

`OPERATIONS.md` §8 step 4 named an `apply_mode` value the pinned provider does
not have (`0.11.0` offers `staged_if_needing_reboot`, not `no_reboot`) to avoid
a reboot that does not happen: Talos lists `.machine.sysctls` in
`CanApplyImmediate`, and the apply was in place, on a running cluster, in 0s.

The guard that refuses a container builder in this pipeline now names `podman`,
the one §9n prescribes, so it would notice the step this repo would plausibly
add — and §9n says the allowlist belongs to the repository that owns the
pipeline, which is where a consumer's `build-image` entry goes. The `tests/` row
in `AGENTS.md` records the invariant for whoever reads the map.

Measured on `quay.io/podman/stable:v5.8.7-immutable` (podman 5.8.7) in a
throwaway pod in `woodpecker-runtime`, context on an `emptyDir`: `unshare -U
true` succeeds from inside the pod, and the pushed bytes and the served manifest
hash equal. §9n carries the measurement and the one thing that had to be
arranged — a context on a volume rather than on the container's overlayfs,
where buildah ≥ 1.34 fails before STEP 1.

tepichord added 1 commit

2026-09-29 17:05:19 -07:00

docs(nix): "both refused" means this repository's pipelines, not every pipeline

ci/woodpecker/pr/gates Pipeline was successful

Details

ci/woodpecker/pull_request_closed/cleanup Pipeline was successful

Details

2bb08128a4

Read repo-wide the sentence now contradicts what the change adds: a consumer
repository *may* declare one container-builder step (§9n), which is the whole
point of the sysctl. The refusal is this repository's own — its images are
derivations — so the sentence says so and points at where the consumer's step is
named.

tepichord merged commit d30c6bd8a6 into main

2026-09-29 17:27:44 -07:00

tepichord deleted branch feat/107-dockerfile-image-builds-podman

2026-09-29 17:27:44 -07:00

tepichord referenced this pull request from a commit

2026-09-29 17:27:45 -07:00

Merge pull request 'Allow Dockerfile image builds with Podman' (#108) from feat/107-dockerfile-image-builds-podman into main

tepichord removed the

pr

reviewing

label

2026-09-29 17:34:45 -07:00