tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Give the Dockerfile-building step a builder image, and make §9n true again #118

New issue
Closed
opened 2026-10-07 06:43:08 -07:00 by tepichord · 1 comment
tepichord commented 2026-10-07 06:43:08 -07:00
Owner
Copy link

Overview

#107 gave this cluster the ability to build a Dockerfile in a step: a
hostUsers: false pod, machine.sysctls.user.max_user_namespaces = 11255, a
seccompProfile: Unconfined, and a suite that names the one step allowed to ask.
The capability is real. annia-infra #310's first pipeline built its image in that
pod and is green (pipelines #10 and #11 in that repo), and no step anywhere is
privileged — #107's stated fallback, one named privileged: true step, was never
needed and stays unused.

What is not real is the shape #107 published. Its proof item 2 measured
podman build --storage-driver=vfs on quay.io/podman/stable:v5.8.7-immutable —
a distro podman, on a distro filesystem, with setuid helpers, /etc/subuid,
/etc/containers, fuse, and a login session, building a Dockerfile that never
needed a foreign chown. The consumer's command is a different toolchain:
nixpkgs' podman inside nix develop, over the shared nix-store PVC, building a
Dockerfile whose RUN is nix build. The PoC passed for that toolchain and not
for this one, which is why the shape in §9n — BUILDAH_ISOLATION: chroot and
STORAGE_DRIVER: vfs and nothing else — does not work, and why the consumer's
step has four measures around it instead.

This issue does what #107's own Pre-Mortem 1 pointed at: move the measures into
something milner owns and tests, make the documentation say what was measured,
and write down which image path needs which measures so the next consumer is not
re-deriving them from a doc that is wrong.

What is measured today

The consumer's step reaches green with four measures and one landmine. Each is a
fact about the toolchain; only one is a kernel rule.

Measure Why it is there (measured) Fundamental?
unshare --map-users=0,0,$IDS --map-groups=0,0,$IDS --mount --propagation private + podman build --userns=host podman in that pod runs rootless with a single-id nested map (uid_map reads 0 0 1 in every shape, plain and --userns=host). chown to any uid but 0 then fails EINVAL, which is what nix profile install and nix build do (error: changing ownership of "/nix/store": Invalid argument). A wider map through podman's own flags is refused (write to \gid_map`: Operation not permitted). With the wide map, chown` succeeds for 0, 1, 1000, 30000 and 65534, and the whole Dockerfile builds no — toolchain
-v /proc:/proc only the user namespace that owns the pid namespace may mount procfs. The build container reuses the pod's, so a fresh procfs mount is refused (mount \proc` to `proc`: Operation not permitted`, in every shape tested). Binding the pod's own procfs works: the build then sees the same procfs superblock, and buildah leaves a caller-supplied mount alone yes — kernel rule
mkdir -p /run/user/0/libpod/tmp rootless podman keeps a pause process and writes its pid under $XDG_RUNTIME_DIR; a pod has no login session to make /run/user/0. Without the directory every podman call outside the namespace dies (unable to create a new pause process: … pause.pid: No such file or directory). Inside the namespace it is not needed — podman there already holds the id range it would spawn a pause process to get no
STORAGE_DRIVER: vfs the pod's own root is an overlay, so there is no overlay-on-overlay, and a pod has no /dev/fuse no
BUILDAH_ISOLATION: chroot — in #107's YAML and in §9n must be removed, not documented: at the pinned buildah 1.43.2, chroot isolation dies at STEP 2 with error setting supplemental groups list: operation not permitted (containers/buildah#6947 — fixed on buildah main, in neither v1.43.2 nor v1.43.4) a version bug

Two things that table is not. It is not a criticism of hostUsers: false, the
sysctl, PSA or the allowlist — those are right, necessary, and unchanged here.
And it is not "podman in a pod needs workarounds": a busybox-and-echo Dockerfile
builds in that pod under the single-id map with no unshare at all. What pays
these costs is the specific combination — nix build inside podman build
inside a user-namespaced pod — because nix's build sandbox needs ids the
single-id map cannot name.

1. A builder image, not a recipe in every consumer's YAML

The measures are the same for every consumer, and they are pod facts rather than
repository facts, yet today each consumer re-derives them from a doc. Move them
into an image milner owns, publishes, and tests — in the shape §15a already
gives images here: a dockerTools.buildLayeredImage derivation, no base image,
and nothing mounted at runtime for the tool to work.

What it carries

  • podman and its helpers (buildah, crun, conmon, netavark, util-linux,
    mount) at the dev shell's pins, so local and CI remain one tool.
  • nix, at the same pin as the store's seeded nix (docs/storage.md §10g), so
    a Dockerfile whose RUN is nix profile install / nix build works inside
    the image without fetching nix itself, and against the shared store when the
    consumer mounts it. This is what makes the image a builder rather than a
    podman wrapper, and it is why the entrypoint below is load-bearing for nix and
    not only for podman: nix's sandbox chowns to build users, which needs a map
    wider than one id (measured, table above).
  • /etc/containers/policy.json and /etc/containers/registries.conf — podman
    refuses to build without the former (no policy.json file found at any of the above paths).
  • /etc/subuid and /etc/subgid, and a writable runtime dir (the entrypoint may
    equally make both at start).
  • the entrypoint.

What its entrypoint does (the measured sequence, in one command, so no
consumer writes it again)

  1. derive the map width from the pod rather than hardcode it — grep -oE '[0-9]+$' /proc/self/uid_map | tail -1; no awk, the step images have none;
  2. make the runtime dir podman's pause process wants when a call runs outside the
    namespace (the pushes do);
  3. re-exec through unshare --map-users=0,0,$IDS --map-groups=0,0,$IDS --mount --propagation private;
  4. podman build --userns=host -v /proc:/proc --storage-driver=vfs …;
  5. push the tag with --digestfile, take the credential on stdin — never in
    argv (AGENTS.md) — and compare the registry's served manifest digest against
    the pushed one, the check §15d already runs over a derivation's push.

Behaviour worth fixing in the interface: a context, a tag, a registry
destination, a token on stdin; nothing else required.

What a consumer's step then looks like

      image: zot.faun-snapper.ts.net/milner/podman-builder:<pin>
      backend_options:
        kubernetes:
          hostUsers: false          # still named on that repo's allowlist
          securityContext:
            seccompProfile: { type: Unconfined }
      environment:
        ZOT_TOKEN: { from_secret: ZOT_TOKEN }
      commands: ["build-and-push . milner/annia:$TAG"]

The namespace, the namespace-and-proc recipe, the storage driver and the digest
check stop being a per-repository transcription, and the allowlist entry keeps
its meaning: one named step, in a namespace of its own.

How the builder itself is built: by this repo's workaround-free path —
nix build + skopeo, the same shape as oz/router — so there is no
bootstrapping paradox: the builder image needs none of the measures it provides.
That is worth stating in §9n, because it is the cleanest evidence that the
measures are podman's and not milner's.

2. docs/woodpecker-ci.md §9n made true again

  • Delete BUILDAH_ISOLATION: chroot (buildah#6947 at the pin) and keep
    STORAGE_DRIVER: vfs with its reason.
  • Carry the measured shape with a reason per line: hostUsers: false,
    seccompProfile: Unconfined, the runtime dir, the wide unshare, -v /proc:/proc, --userns=host, and the fetch of the map width from the pod.
  • Point §9n at the builder image first, and keep the raw recipe as what the image
    does (a repo that cannot use the image should still be able to read what
    happens).
  • The allowlist: the sentence and the snippet are "exactly one step per
    pipeline", and the real invariant is at most one namespaced step per run,
    which is now two entries on disjoint events — build-image on pull_request,
    publish on push (annia-infra's suite already asserts disjointness rather
    than a count; milner's own allowlist stays empty). Say it that way in §9n, and
    in .woodpecker/gates.yaml's "one named step per pipeline" prose.
  • The measurement paragraph must stop offering unshare -U true succeeding as
    evidence that a namespace is enough: a namespace that exists is not a namespace
    wide enough to chown in. That sentence is the one that misled this work.

3. Which path milner recommends, written down

docs/nix.md §15 and §9n should say plainly that a derivation
(dockerTools.buildLayeredImage + skopeo) needs none of these measures —
no user namespace, no procfs mount, no vfs, no pause process — and that a
Dockerfile-building step pays all of them. A repository that already has a flake
should prefer the derivation; a Dockerfile is for an app whose build genuinely is
one. annia-infra is the case in point: #310's own table removed
packages.annia-image in favour of the Dockerfile, and this issue is what makes
that removal cheap to revisit.

If story 20b needs rewording to stay true, reword it here — in the change that
contradicts it (AGENTS.md). Its "exactly one step per pipeline" half is still
true as written (the two namespaced steps never meet in one run); its "the same
command I run locally" half is the claim this work qualifies, and after item 1
it is true in the sense the story means: one command, with the pod-specific parts
in the tool.

4. Two experiments that could remove a measure each

Both are one lab pod in woodpecker-runtime, and the outcome — positive or
negative — goes into §9n as a measurement.

  1. A PVC for podman's storage, with overlay instead of vfs. Unprivileged
    overlayfs is supported on this kernel (6.x); what blocks it today is that the
    pod's root is itself an overlay. If it works, the re-pull cost of vfs goes
    away; this is the follow-up §9n already names, and it should be measured
    rather than assumed.
  2. unshare --pid --fork in the entrypoint. A pid namespace created inside
    the user namespace is owned by it, which is the precondition for mounting
    procfs — so this is the one candidate that could remove the measure that is a
    kernel rule, making -v /proc:/proc unnecessary. Worth one pod.
  3. (Micro, same pod) inside the wide unshare, does the build still need
    --userns=host, or does podman's own nested map now inherit the width?

5. What must not change

  • No privileged: true, in milner or in any consumer. The agent's
    capabilities field is Drop-only in v3.18.0, so a builder cannot even ask for
    more; #107's fallback stays unused.
  • No policy or PSA loosening: policy/k8s.rego unchanged, no exemption, and
    hostUsers: true is never the builder (that is node root).
  • No sysctl change and no node reboot: user.max_user_namespaces = 11255 is
    already raised and sufficient (measured).
  • No change to zot's credentials or its milner/** anonymous-push
    exception     — that is #110's and its own follow-up.
  • No stale story and no stale doc: where a story and this table disagree, the
    story is right and this table wins nothing.

The pause this issue puts on annia-infra

annia-infra #310/#318 is paused until this lands: #318 stays unmerged, and
that repo's .woodpecker/ci.yaml is not touched while this is in flight. When
this is done, annia-infra's step becomes the builder image in its own change, and
its merge runs publish — which is where the ZOT_TOKEN secret-extension proof
(#110) happens. Nothing about that proof is blocked by this issue; it is simply
not run twice.

Acceptance Criteria

  • A builder image exists as a flake output (dockerTools.buildLayeredImage),
    is built and pushed by this repo's pipeline with nix build + skopeo,
    and is pinned where a consumer step pins it
  • The image carries podman and nix at the dev shell's pins, the container
    configs, subuid/subgid, and an entrypoint whose required arguments are a
    context, a tag and a destination (token on stdin)
  • A Dockerfile whose RUN is nix build builds inside it, in a
    hostUsers: false pod, against the shared store, from a step that runs no
    unshare of its own — the recipe lives in the entrypoint
  • docs/woodpecker-ci.md §9n carries no BUILDAH_ISOLATION: chroot, carries
    the measured shape with a reason per line, and states the allowlist
    invariant as at most one namespaced step per run, two disjoint events
  • §9n's measurement paragraph no longer offers unshare -U true succeeding
    as evidence that a namespace is enough
  • docs/nix.md §15 and §9n say which path needs which measures, and story
    20b is either confirmed true as written or reworded in this change
  • .woodpecker/gates.yaml's "one named step per pipeline" prose matches the
    per-run invariant
  • Experiments 1–3 have recorded outcomes in §9n, and any measure they remove
    is removed from the doc and the entrypoint
  • python3 -m pytest tests/ green, NAMESPACED_ALLOWLIST still empty in this
    repo's suite, and conftest test --policy policy/ unchanged over the layers
  • annia-infra's step shrinks to the builder image in that repo's own change,
    after this lands

Testing Plan

  • python3 -m pytest tests/ -v — including test_static.py's allowlist
    invariant and test_docs.py's story guards
  • conftest test --policy policy/ over the rendered layers with the new image
    pinned: rule 3 reads only privileged, so nothing should change — that is the
    assertion
  • One lab pod in woodpecker-runtime with no resources (so the namespace
    LimitRange applies), running the entrypoint against a busybox context and
    against a nix build context, then deleted — the same harness this work used,
    and the reason the LimitRange matters: a step pod requests 5Gi
  • A real consumer pipeline (annia-infra #310) in its own repo, once the pause
    lifts

Related

  • #107 — the capability this corrects; its proof item 2 measured a distro podman
  • #110 — the zot push credential the publish half consumes
  • #109 — adjacent: the layered-store question, same store
  • annia-infra #310 / #318 — first consumer, paused until this lands
  • containers/buildah#6947 (error setting supplemental groups list, chroot
    isolation), #6910 (overlay over the build context)
  • docs/woodpecker-ci.md §9n, docs/nix.md §15, docs/storage.md §10g,
    .woodpecker/gates.yaml's builder comment block
## Overview #107 gave this cluster the ability to build a Dockerfile in a step: a `hostUsers: false` pod, `machine.sysctls.user.max_user_namespaces = 11255`, a `seccompProfile: Unconfined`, and a suite that names the one step allowed to ask. The capability is real. annia-infra #310's first pipeline built its image in that pod and is green (pipelines #10 and #11 in that repo), and no step anywhere is privileged — #107's stated fallback, one named `privileged: true` step, was never needed and stays unused. What is not real is the shape #107 published. Its proof item 2 measured `podman build --storage-driver=vfs` on `quay.io/podman/stable:v5.8.7-immutable` — a distro podman, on a distro filesystem, with setuid helpers, `/etc/subuid`, `/etc/containers`, fuse, and a login session, building a Dockerfile that never needed a foreign `chown`. The consumer's command is a different toolchain: nixpkgs' podman inside `nix develop`, over the shared `nix-store` PVC, building a Dockerfile whose `RUN` is `nix build`. The PoC passed for that toolchain and not for this one, which is why the shape in §9n — `BUILDAH_ISOLATION: chroot` and `STORAGE_DRIVER: vfs` and nothing else — does not work, and why the consumer's step has four measures around it instead. This issue does what #107's own Pre-Mortem 1 pointed at: move the measures into something milner owns and tests, make the documentation say what was measured, and write down which image path needs which measures so the next consumer is not re-deriving them from a doc that is wrong. ## What is measured today The consumer's step reaches green with four measures and one landmine. Each is a fact about the toolchain; only one is a kernel rule. | Measure | Why it is there (measured) | Fundamental? | |---|---|---| | `unshare --map-users=0,0,$IDS --map-groups=0,0,$IDS --mount --propagation private` + `podman build --userns=host` | podman in that pod runs rootless with a **single-id** nested map (`uid_map` reads `0 0 1` in every shape, plain and `--userns=host`). `chown` to any uid but 0 then fails `EINVAL`, which is what `nix profile install` and `nix build` do (`error: changing ownership of "/nix/store": Invalid argument`). A wider map through podman's own flags is refused (`write to \`gid_map\`: Operation not permitted`). With the wide map, `chown` succeeds for 0, 1, 1000, 30000 and 65534, and the whole Dockerfile builds | no — toolchain | | `-v /proc:/proc` | only the user namespace that owns the pid namespace may mount procfs. The build container reuses the pod's, so a fresh procfs mount is refused (`mount \`proc\` to \`proc\`: Operation not permitted`, in every shape tested). Binding the pod's own procfs works: the build then sees the same procfs superblock, and buildah leaves a caller-supplied mount alone | **yes** — kernel rule | | `mkdir -p /run/user/0/libpod/tmp` | rootless podman keeps a pause process and writes its pid under `$XDG_RUNTIME_DIR`; a pod has no login session to make `/run/user/0`. Without the directory every podman call *outside* the namespace dies (`unable to create a new pause process: … pause.pid: No such file or directory`). Inside the namespace it is not needed — podman there already holds the id range it would spawn a pause process to get | no | | `STORAGE_DRIVER: vfs` | the pod's own root is an overlay, so there is no overlay-on-overlay, and a pod has no `/dev/fuse` | no | | `BUILDAH_ISOLATION: chroot` — **in #107's YAML and in §9n** | must be *removed*, not documented: at the pinned buildah 1.43.2, chroot isolation dies at STEP 2 with `error setting supplemental groups list: operation not permitted` (containers/buildah#6947 — fixed on buildah main, in neither v1.43.2 nor v1.43.4) | a version bug | Two things that table is not. It is not a criticism of `hostUsers: false`, the sysctl, PSA or the allowlist — those are right, necessary, and unchanged here. And it is not "podman in a pod needs workarounds": a busybox-and-echo Dockerfile builds in that pod under the single-id map with no `unshare` at all. What pays these costs is the specific combination — `nix build` inside `podman build` inside a user-namespaced pod — because nix's build sandbox needs ids the single-id map cannot name. ## 1. A builder image, not a recipe in every consumer's YAML The measures are the same for every consumer, and they are pod facts rather than repository facts, yet today each consumer re-derives them from a doc. Move them into an image milner owns, publishes, and tests — in the shape §15a already gives images here: a `dockerTools.buildLayeredImage` derivation, no base image, and nothing mounted at runtime for the tool to work. **What it carries** - **podman** and its helpers (buildah, crun, conmon, netavark, util-linux, `mount`) at the dev shell's pins, so local and CI remain one tool. - **nix**, at the same pin as the store's seeded nix (docs/storage.md §10g), so a Dockerfile whose `RUN` is `nix profile install` / `nix build` works *inside* the image without fetching nix itself, and against the shared store when the consumer mounts it. This is what makes the image a *builder* rather than a podman wrapper, and it is why the entrypoint below is load-bearing for nix and not only for podman: nix's sandbox chowns to build users, which needs a map wider than one id (measured, table above). - `/etc/containers/policy.json` and `/etc/containers/registries.conf` — podman refuses to build without the former (`no policy.json file found at any of the above paths`). - `/etc/subuid` and `/etc/subgid`, and a writable runtime dir (the entrypoint may equally make both at start). - the entrypoint. **What its entrypoint does** (the measured sequence, in one command, so no consumer writes it again) 1. derive the map width from the pod rather than hardcode it — `grep -oE '[0-9]+$' /proc/self/uid_map | tail -1`; no `awk`, the step images have none; 2. make the runtime dir podman's pause process wants when a call runs outside the namespace (the pushes do); 3. re-exec through `unshare --map-users=0,0,$IDS --map-groups=0,0,$IDS --mount --propagation private`; 4. `podman build --userns=host -v /proc:/proc --storage-driver=vfs …`; 5. push the tag with `--digestfile`, take the credential on stdin — never in argv (AGENTS.md) — and compare the registry's served manifest digest against the pushed one, the check §15d already runs over a derivation's push. Behaviour worth fixing in the interface: a context, a tag, a registry destination, a token on stdin; nothing else required. **What a consumer's step then looks like** ```yaml image: zot.faun-snapper.ts.net/milner/podman-builder:<pin> backend_options: kubernetes: hostUsers: false # still named on that repo's allowlist securityContext: seccompProfile: { type: Unconfined } environment: ZOT_TOKEN: { from_secret: ZOT_TOKEN } commands: ["build-and-push . milner/annia:$TAG"] ``` The namespace, the namespace-and-proc recipe, the storage driver and the digest check stop being a per-repository transcription, and the allowlist entry keeps its meaning: one named step, in a namespace of its own. **How the builder itself is built**: by this repo's workaround-free path — `nix build` + `skopeo`, the same shape as `oz`/`router` — so there is no bootstrapping paradox: the builder image needs none of the measures it provides. That is worth stating in §9n, because it is the cleanest evidence that the measures are podman's and not milner's. ## 2. `docs/woodpecker-ci.md` §9n made true again - Delete `BUILDAH_ISOLATION: chroot` (buildah#6947 at the pin) and keep `STORAGE_DRIVER: vfs` with its reason. - Carry the measured shape with a reason per line: `hostUsers: false`, `seccompProfile: Unconfined`, the runtime dir, the wide `unshare`, `-v /proc:/proc`, `--userns=host`, and the fetch of the map width from the pod. - Point §9n at the builder image first, and keep the raw recipe as what the image does (a repo that cannot use the image should still be able to read what happens). - The allowlist: the sentence and the snippet are "exactly one step per pipeline", and the real invariant is **at most one namespaced step per run**, which is now two entries on disjoint events — `build-image` on `pull_request`, `publish` on `push` (annia-infra's suite already asserts disjointness rather than a count; milner's own allowlist stays empty). Say it that way in §9n, and in `.woodpecker/gates.yaml`'s "one named step per pipeline" prose. - The measurement paragraph must stop offering `unshare -U true` succeeding as evidence that a namespace is enough: a namespace that exists is not a namespace wide enough to `chown` in. That sentence is the one that misled this work. ## 3. Which path milner recommends, written down `docs/nix.md` §15 and §9n should say plainly that a derivation (`dockerTools.buildLayeredImage` + `skopeo`) needs **none** of these measures — no user namespace, no procfs mount, no vfs, no pause process — and that a Dockerfile-building step pays all of them. A repository that already has a flake should prefer the derivation; a Dockerfile is for an app whose build genuinely is one. annia-infra is the case in point: #310's own table *removed* `packages.annia-image` in favour of the Dockerfile, and this issue is what makes that removal cheap to revisit. If story 20b needs rewording to stay true, reword it here — in the change that contradicts it (AGENTS.md). Its "exactly one step per pipeline" half is still true as written (the two namespaced steps never meet in one run); its "the same command I run locally" half is the claim this work qualifies, and after item 1 it is true in the sense the story means: one command, with the pod-specific parts in the tool. ## 4. Two experiments that could remove a measure each Both are one lab pod in `woodpecker-runtime`, and the outcome — positive or negative — goes into §9n as a measurement. 1. **A PVC for podman's storage, with `overlay` instead of `vfs`.** Unprivileged overlayfs is supported on this kernel (6.x); what blocks it today is that the pod's root is itself an overlay. If it works, the re-pull cost of `vfs` goes away; this is the follow-up §9n already names, and it should be measured rather than assumed. 2. **`unshare --pid --fork` in the entrypoint.** A pid namespace created inside the user namespace is owned by it, which is the precondition for mounting procfs — so this is the one candidate that could remove the measure that is a kernel rule, making `-v /proc:/proc` unnecessary. Worth one pod. 3. (Micro, same pod) inside the wide `unshare`, does the build still need `--userns=host`, or does podman's own nested map now inherit the width? ## 5. What must not change - **No `privileged: true`**, in milner or in any consumer. The agent's `capabilities` field is Drop-only in v3.18.0, so a builder cannot even ask for more; #107's fallback stays unused. - **No policy or PSA loosening**: `policy/k8s.rego` unchanged, no exemption, and `hostUsers: true` is never the builder (that is node root). - **No sysctl change and no node reboot**: `user.max_user_namespaces = 11255` is already raised and sufficient (measured). - **No change to zot's credentials or its `milner/**` anonymous-push exception** — that is #110's and its own follow-up. - **No stale story and no stale doc**: where a story and this table disagree, the story is right and this table wins nothing. ## The pause this issue puts on annia-infra annia-infra #310/#318 is **paused until this lands**: #318 stays unmerged, and that repo's `.woodpecker/ci.yaml` is not touched while this is in flight. When this is done, annia-infra's step becomes the builder image in its own change, and its merge runs `publish` — which is where the `ZOT_TOKEN` secret-extension proof (#110) happens. Nothing about that proof is blocked by this issue; it is simply not run twice. ## Acceptance Criteria - [ ] A builder image exists as a flake output (`dockerTools.buildLayeredImage`), is built and pushed by this repo's pipeline with `nix build` + `skopeo`, and is pinned where a consumer step pins it - [ ] The image carries podman **and nix** at the dev shell's pins, the container configs, subuid/subgid, and an entrypoint whose required arguments are a context, a tag and a destination (token on stdin) - [ ] A Dockerfile whose `RUN` is `nix build` builds inside it, in a `hostUsers: false` pod, against the shared store, from a step that runs no `unshare` of its own — the recipe lives in the entrypoint - [ ] `docs/woodpecker-ci.md` §9n carries no `BUILDAH_ISOLATION: chroot`, carries the measured shape with a reason per line, and states the allowlist invariant as at most one namespaced step per run, two disjoint events - [ ] §9n's measurement paragraph no longer offers `unshare -U true` succeeding as evidence that a namespace is enough - [ ] `docs/nix.md` §15 and §9n say which path needs which measures, and story 20b is either confirmed true as written or reworded in this change - [ ] `.woodpecker/gates.yaml`'s "one named step per pipeline" prose matches the per-run invariant - [ ] Experiments 1–3 have recorded outcomes in §9n, and any measure they remove is removed from the doc *and* the entrypoint - [ ] `python3 -m pytest tests/` green, `NAMESPACED_ALLOWLIST` still empty in this repo's suite, and `conftest test --policy policy/` unchanged over the layers - [ ] annia-infra's step shrinks to the builder image in that repo's own change, after this lands ## Testing Plan - `python3 -m pytest tests/ -v` — including `test_static.py`'s allowlist invariant and `test_docs.py`'s story guards - `conftest test --policy policy/` over the rendered layers with the new image pinned: rule 3 reads only `privileged`, so nothing should change — that is the assertion - One lab pod in `woodpecker-runtime` with no `resources` (so the namespace `LimitRange` applies), running the entrypoint against a busybox context and against a `nix build` context, then deleted — the same harness this work used, and the reason the LimitRange matters: a step pod *requests* 5Gi - A real consumer pipeline (annia-infra #310) in its own repo, once the pause lifts ## Related - #107 — the capability this corrects; its proof item 2 measured a distro podman - #110 — the zot push credential the `publish` half consumes - #109 — adjacent: the layered-store question, same store - annia-infra #310 / #318 — first consumer, paused until this lands - containers/buildah#6947 (`error setting supplemental groups list`, chroot isolation), #6910 (overlay over the build context) - docs/woodpecker-ci.md §9n, docs/nix.md §15, docs/storage.md §10g, `.woodpecker/gates.yaml`'s builder comment block
tepichord added this to the MVP project 2026-10-07 06:56:21 -07:00
tepichord commented 2026-10-07 07:48:12 -07:00
Author
Owner
Copy link

Naina:

Overview

#107 gave the cluster a real unprivileged Dockerfile-building capability, but §9n publishes a shape measured against a distro podman on a distro filesystem, not the consumer's nixpkgs-podman-inside-nix develop toolchain. This plan moves the four working measures into a builder image this repo owns and builds as a dockerTools.buildLayeredImage flake output pushed with nix build + skopeo, then rewrites §9n, docs/nix.md §15 and .woodpecker/gates.yaml's builder block to say what was actually measured.

Technical Approach

  1. Put the whole recipe in one on-PATH script. The Woodpecker agent overrides a step image's entrypoint and runs commands as /bin/sh -c, so config.Cmd is never reached by a consumer step; the recipe must be an executable.1

    # flake.nix, in the same `let` as `skopeoPusher`
buildAndPush = pkgs.writeShellApplication {
  name = "build-and-push";
  runtimeInputs = with pkgs; [ podman util-linux curl coreutils gnugrep nixVersions.nix_2_35 ];
  text = ''
    ids=$(grep -oE '[0-9]+$' /proc/self/uid_map | tail -1)   # no `awk`: the step images have none
    mkdir -p /run/user/0/libpod/tmp                          # pause.pid's runtime dir
    if [ "''${1:-}" != --inside ]; then
      exec unshare --map-users=0,0:"$ids" --map-groups=0,0:"$ids" \
        --mount --propagation private "$0" --inside "$@"
    fi
    shift
    context=$1 tag=$2 registry=$3
    podman build --userns=host -v /proc:/proc --storage-driver=vfs --tag "$tag" "$context"
    printf '%s' "''${ZOT_TOKEN:?}" | podman login --password-stdin --username annia-ci "$registry"
    podman push --tls-verify=true --digestfile /tmp/push-digest "$tag"
    # served-digest compare, same shape as build-oz
  '';
};

  2. Add the image and its check to the derivation family.

    podmanBuilderTag = "nixpkgs-${builtins.substring 0 8 nixpkgs.rev}";  # moves when podman/nix move
podmanBuilderImage = pkgs.dockerTools.buildLayeredImage {
  name = "milner/podman-builder";
  tag = podmanBuilderTag;
  contents = [ buildAndPush pkgs.podman pkgs.nixVersions.nix_2_35 pkgs.pkgsStatic.busybox ];
  config = { Cmd = [ "/bin/build-and-push" ]; Env = [ "PATH=/bin" ]; };
  extraCommands = ''
    mkdir -p etc/containers
    printf '%s' '{ "default": [ { "type": "insecureAcceptAnything" } ] }' > etc/containers/policy.json
    printf 'unqualified-search-registries = ["docker.io"]\n' > etc/containers/registries.conf
    printf 'root:0:65536\n' > etc/subuid
    printf 'root:0:65536\n' > etc/subgid
  '';
};

    podmanBuilderCheck follows reaperCheck (docs/nix.md §15c)10: extract the image's own layers and assert /bin/build-and-push, /bin/podman, /bin/nix, /etc/containers/policy.json, /etc/subuid resolve inside, that the nix is nixVersions.nix_2_35 (not the pin's default nix 2.34.8)5 — nix eval on this pin gives nix = 2.34.8 and nixVersions.nix_2_35 = 2.35.2, and that config.Cmd reads /bin/build-and-push from the image's own config bytes.

  3. Expose the tag and image in packages so the step never retypes the pin.

    packages = { /* … */ podmanBuilderTag = pkgs.writeText "podman-builder-tag" podmanBuilderTag; podmanBuilderImage = …; podmanBuilderCheck = …; };

  4. Reword the builder comment block in .woodpecker/gates.yaml (~line 434): one named step per pipeline or none → at most one namespaced step per run.

  5. Add a build-podman-builder step that copies build-oz exactly (nixos/nix:2.35.2, volumes: [nix-store:/nix], then nix build → check → ./#skopeo copy --digestfile → curl+sha256sum served-digest compare).7 It never runs the builder image, so test_no_step_in_the_pipeline_is_privileged's image-name guard stays satisfied; the store mount is what realises the builder's closure into the store a consumer later mounts.2

    - name: build-podman-builder
  image: nixos/nix:2.35.2
  volumes: [nix-store:/nix]
  commands:
    - |
      $NIX build .#podmanBuilderCheck --out-link /tmp/podman-builder-check && cat /tmp/podman-builder-check
      $NIX build .#podmanBuilderImage --out-link /tmp/podman-builder
      TAG=$($NIX eval --raw .#podmanBuilderTag)
      /tmp/skopeo/bin/skopeo copy --dest-tls-verify=true --retry-times 3 --digestfile /tmp/pb-digest \
        docker-archive:/tmp/podman-builder "docker://zot.faun-snapper.ts.net/milner/podman-builder:$TAG"

  6. Rewrite docs/woodpecker-ci.md §9n: builder image first (consumer YAML, entrypoint arguments, token on stdin), raw recipe beneath as "what the image does"; delete only BUILDAH_ISOLATION: chroot and keep STORAGE_DRIVER: vfs with its reason (the allowlist test requires both hostUsers: false and STORAGE_DRIVER: vfs substrings3); restate the invariant as at most one namespaced step per run, with the two disjoint events (build-image on pull_request, publish on push); replace the unshare -U true sentence with the width argument.

  7. State which path needs which measures in docs/nix.md §15 and §9n: a derivation (buildLayeredImage + skopeo) needs none of the namespace/procfs/vfs/pause-process measures; a Dockerfile step pays all of them.

  8. Run experiments 1–3 in one lab pod in woodpecker-runtime (no resources, so the LimitRange applies; delete after) and record the outcome in §9n either way: overlay-on-a-PVC instead of vfs; unshare --pid --fork --mount-proc to drop the -v /proc:/proc bind; and whether --userns=host is still needed inside the wide map. Any measure removed comes out of the doc and the entrypoint in the same change.

Implementation Details

File Change
flake.nix buildAndPush, podmanBuilderTag, podmanBuilderImage, podmanBuilderCheck; packages entries. Use pkgs.nixVersions.nix_2_35, never pkgs.nix.
.woodpecker/gates.yaml comment-block reword; new build-podman-builder step. Watch ${...} substitution: only Woodpecker vars or :- defaults.9
docs/woodpecker-ci.md §9n rewrite (§9k/ZOT_TOKEN stays).
docs/nix.md §15 intro + §15d: builder image and which path needs which measures.
docs/storage.md §10g: the mounting-step count/list now includes build-podman-builder.
USER_STORIES.md 20b: confirm true as written; reword only if an experiment changes it (body ≤ 90 words, no code fences).
tests/test_static.py add build-podman-builder to the writers lists (lines ~1127, ~4712)8a 8b; extend test_a_user_namespaced_step_is_named_on_the_allowlist to assert §9n has no BUILDAH_ISOLATION, keeps hostUsers: false + STORAGE_DRIVER: vfs, and says per-run.

Constraints / out-of-scope notes

  • Do not touch policy/k8s.rego, zot credentials or the milner/** anonymous-push exception (#110), worker/main.tf's sysctl, or annia-infra's .woodpecker/ci.yaml (paused).
  • containers/storage#6968 in the context does not resolve on GitHub — dropped from the plan; Experiment 1 cites the kernel userxattr/NFS docs instead.
  • The consumer's pin lives in the consumer repo (annia-infra, its own paused change); milner has no in-tree manifest to guard, so no new ci/*-tag-guard.sh.
  • NAMESPACED_ALLOWLIST stays {} in this repo: milner builds the builder but never runs it.4

Acceptance Criteria

  • .#podmanBuilderImage is a dockerTools.buildLayeredImage output, built/checked/pushed by build-podman-builder with nix build + skopeo, at a tag derived from the flake pin.
  • The image carries podman, nix 2.35.2, policy.json/registries.conf, subuid/subgid, and build-and-push <context> <tag> <registry> with the token on stdin.
  • A Dockerfile whose RUN is nix build builds inside it, in a hostUsers: false pod, against the shared store, with no unshare in the step.
  • §9n has no BUILDAH_ISOLATION: chroot, keeps STORAGE_DRIVER: vfs with its reason, states per-run/at-most-one, and drops the unshare -U true evidence.
  • docs/nix.md §15 and §9n say which path needs which measures; story 20b is true as written or reworded here.
  • .woodpecker/gates.yaml's comment block matches the per-run invariant.
  • Experiments 1–3 have recorded §9n outcomes; any removed measure is gone from doc and entrypoint.
  • python3 -m pytest tests/ green, NAMESPACED_ALLOWLIST still empty, conftest test --policy policy/ unchanged.

Testing Plan

  • python3 -m pytest tests/ -v (incl. the allowlist and test_docs.py story guards); python3 -m pytest tests/ -k step_commands before editing .woodpecker/.
  • nix build .#podmanBuilderCheck && cat result — /bin/*, config files and Cmd resolve inside the image's own layers.
  • conftest test --policy policy/ over rendered layers: rule 3 reads only privileged, so no change is the assertion.
  • One lab pod in woodpecker-runtime (no resources) running build-and-push against a busybox context and a nix build context, plus Experiments 1–3, then deleted.
  • A real consumer pipeline (annia-infra #310) once the pause lifts.

Related Issues

  • #107 — the capability this corrects (its proof item 2 measured a distro podman).
  • #110 — the zot push credential the publish half consumes.
  • #109 — adjacent layered-store question, same store.
  • annia-infra #310 / #318 — first consumer, paused until this lands.
  • containers/buildah#6947 (chroot setgroups), #6910 (overlay over the build context).

Research Findings

  • buildah#6947 is fixed by v1.45.0, but the flake pin's podman 5.8.4 vendors buildah v1.43.2, so BUILDAH_ISOLATION: chroot still fails at the pin; pkgs.buildah being 1.45.0 does not help podman build.
  • Nix clears setuid bits on every store path, so the image can never ship newuidmap/newgidmap; the entrypoint's unshare --map-users is load-bearing, and the pin's util-linux 2.42.2 supports it (root writes maps directly since v2.40).
  • The Woodpecker agent overrides a step image's entrypoint and runs commands via /bin/sh -c, so build-and-push must be an executable on PATH, not config.Cmd.
  • dockerTools.buildLayeredImage merges contents bin/ into /bin and runs extraCommands unprivileged, which is the right place for /etc/containers/* and subuid/subgid.6
  • Unprivileged overlayfs needs kernel ≥ 5.11 plus the userxattr option and a non-NFS upper dir; milner's local-path PVCs meet the filesystem precondition (Experiment 1 is the measurement).
  • The pin's default pkgs.nix is 2.34.8 while the seeded store is nix 2.35.2, so the image must spell pkgs.nixVersions.nix_2_35 or it reads a newer store schema it cannot parse.
  • Zot's milner/** and preview/** are anonymous-push today, so the builder's own push needs no token, but the interface still takes one on stdin for consumer annia/** destinations.

Pre-Mortem

  1. The lab-pod experiments cannot be run in the implementation environment. — Acceptance requires Experiments 1–3 run in a live woodpecker-runtime pod and their outcomes recorded in §9n, but the coding agent has no cluster credentials or kubectl access, so the step either stalls or ships unverified prose. Because the suite only checks image contents and doc text, nothing catches the gap.
  2. The §9n rewrite trips a guard that encodes today's text. — test_a_user_namespaced_step_is_named_on_the_allowlist requires §9n to keep hostUsers: false and STORAGE_DRIVER: vfs, test_no_step_in_the_pipeline_is_privileged rejects any step image named podman, and test_docs.py caps story 20b at 90 words, so a rewrite that drops or rephrases one of these fails the build. The Woodpecker ${...} substitution rule for gates.yaml comments is a second way the same edit breaks a test.
  3. The image's closure and Nix version diverge from the shared store. — If pkgs.nix 2.34.8 slips in instead of pkgs.nixVersions.nix_2_35, or build-podman-builder fails to realise the full closure into the nix-store PVC, a consumer mounting the store over /nix either hits a schema-version error or loses the image's tools. This surfaces only in the paused downstream consumer (annia-infra #310), long after the change looks green here.
Naina: ## Overview #107 gave the cluster a real unprivileged Dockerfile-building capability, but §9n publishes a shape measured against a distro podman on a distro filesystem, not the consumer's nixpkgs-podman-inside-`nix develop` toolchain. This plan moves the four working measures into a builder image this repo owns and builds as a `dockerTools.buildLayeredImage` flake output pushed with `nix build` + `skopeo`, then rewrites §9n, `docs/nix.md` §15 and `.woodpecker/gates.yaml`'s builder block to say what was actually measured. ## Technical Approach 1. **Put the whole recipe in one on-PATH script.** The Woodpecker agent *overrides* a step image's entrypoint and runs `commands` as `/bin/sh -c`, so `config.Cmd` is never reached by a consumer step; the recipe must be an executable.<sup>[1](https://woodpecker-ci.org/docs/next/usage/workflow-syntax#entrypoint)</sup> ```nix # flake.nix, in the same `let` as `skopeoPusher` buildAndPush = pkgs.writeShellApplication { name = "build-and-push"; runtimeInputs = with pkgs; [ podman util-linux curl coreutils gnugrep nixVersions.nix_2_35 ]; text = '' ids=$(grep -oE '[0-9]+$' /proc/self/uid_map | tail -1) # no `awk`: the step images have none mkdir -p /run/user/0/libpod/tmp # pause.pid's runtime dir if [ "''${1:-}" != --inside ]; then exec unshare --map-users=0,0:"$ids" --map-groups=0,0:"$ids" \ --mount --propagation private "$0" --inside "$@" fi shift context=$1 tag=$2 registry=$3 podman build --userns=host -v /proc:/proc --storage-driver=vfs --tag "$tag" "$context" printf '%s' "''${ZOT_TOKEN:?}" | podman login --password-stdin --username annia-ci "$registry" podman push --tls-verify=true --digestfile /tmp/push-digest "$tag" # served-digest compare, same shape as build-oz ''; }; ``` 2. **Add the image and its check to the derivation family.** ```nix podmanBuilderTag = "nixpkgs-${builtins.substring 0 8 nixpkgs.rev}"; # moves when podman/nix move podmanBuilderImage = pkgs.dockerTools.buildLayeredImage { name = "milner/podman-builder"; tag = podmanBuilderTag; contents = [ buildAndPush pkgs.podman pkgs.nixVersions.nix_2_35 pkgs.pkgsStatic.busybox ]; config = { Cmd = [ "/bin/build-and-push" ]; Env = [ "PATH=/bin" ]; }; extraCommands = '' mkdir -p etc/containers printf '%s' '{ "default": [ { "type": "insecureAcceptAnything" } ] }' > etc/containers/policy.json printf 'unqualified-search-registries = ["docker.io"]\n' > etc/containers/registries.conf printf 'root:0:65536\n' > etc/subuid printf 'root:0:65536\n' > etc/subgid ''; }; ``` `podmanBuilderCheck` follows `reaperCheck` (docs/nix.md §15c)<sup>[10](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/docs/nix.md?display=source#L485)</sup>: extract the image's own layers and assert `/bin/build-and-push`, `/bin/podman`, `/bin/nix`, `/etc/containers/policy.json`, `/etc/subuid` resolve *inside*, that the nix is `nixVersions.nix_2_35` (not the pin's default `nix` 2.34.8)<sup>[5](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/flake.lock?display=source#L27) — `nix eval` on this pin gives `nix` = 2.34.8 and `nixVersions.nix_2_35` = 2.35.2</sup>, and that `config.Cmd` reads `/bin/build-and-push` from the image's own config bytes. 3. **Expose the tag and image in `packages`** so the step never retypes the pin. ```nix packages = { /* … */ podmanBuilderTag = pkgs.writeText "podman-builder-tag" podmanBuilderTag; podmanBuilderImage = …; podmanBuilderCheck = …; }; ``` 4. **Reword the builder comment block** in `.woodpecker/gates.yaml` (~line 434): `one named step per pipeline or none` → **at most one namespaced step per run**. 5. **Add a `build-podman-builder` step** that copies `build-oz` exactly (`nixos/nix:2.35.2`, `volumes: [nix-store:/nix]`, then `nix build` → check → `./#skopeo copy --digestfile` → curl+sha256sum served-digest compare).<sup>[7](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/.woodpecker/gates.yaml?display=source#L509)</sup> It never runs the builder image, so `test_no_step_in_the_pipeline_is_privileged`'s image-name guard stays satisfied; the store mount is what realises the builder's closure into the store a consumer later mounts.<sup>[2](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/tests/test_static.py?display=source#L2640)</sup> ```yaml - name: build-podman-builder image: nixos/nix:2.35.2 volumes: [nix-store:/nix] commands: - | $NIX build .#podmanBuilderCheck --out-link /tmp/podman-builder-check && cat /tmp/podman-builder-check $NIX build .#podmanBuilderImage --out-link /tmp/podman-builder TAG=$($NIX eval --raw .#podmanBuilderTag) /tmp/skopeo/bin/skopeo copy --dest-tls-verify=true --retry-times 3 --digestfile /tmp/pb-digest \ docker-archive:/tmp/podman-builder "docker://zot.faun-snapper.ts.net/milner/podman-builder:$TAG" ``` 6. **Rewrite `docs/woodpecker-ci.md` §9n**: builder image first (consumer YAML, entrypoint arguments, token on stdin), raw recipe beneath as "what the image does"; delete only `BUILDAH_ISOLATION: chroot` and keep `STORAGE_DRIVER: vfs` with its reason (the allowlist test requires both `hostUsers: false` and `STORAGE_DRIVER: vfs` substrings<sup>[3](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/tests/test_static.py?display=source#L2734)</sup>); restate the invariant as *at most one namespaced step per run*, with the two disjoint events (`build-image` on `pull_request`, `publish` on `push`); replace the `unshare -U true` sentence with the width argument. 7. **State which path needs which measures** in `docs/nix.md` §15 and §9n: a derivation (`buildLayeredImage` + `skopeo`) needs none of the namespace/procfs/vfs/pause-process measures; a Dockerfile step pays all of them. 8. **Run experiments 1–3 in one lab pod** in `woodpecker-runtime` (no `resources`, so the LimitRange applies; delete after) and record the outcome in §9n either way: overlay-on-a-PVC instead of `vfs`; `unshare --pid --fork --mount-proc` to drop the `-v /proc:/proc` bind; and whether `--userns=host` is still needed inside the wide map. Any measure removed comes out of the doc *and* the entrypoint in the same change. ## Implementation Details | File | Change | |---|---| | `flake.nix` | `buildAndPush`, `podmanBuilderTag`, `podmanBuilderImage`, `podmanBuilderCheck`; packages entries. Use `pkgs.nixVersions.nix_2_35`, never `pkgs.nix`. | | `.woodpecker/gates.yaml` | comment-block reword; new `build-podman-builder` step. Watch `${...}` substitution: only Woodpecker vars or `:-` defaults.<sup>[9](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/tests/test_static.py?display=source#L2741)</sup> | | `docs/woodpecker-ci.md` | §9n rewrite (§9k/ZOT_TOKEN stays). | | `docs/nix.md` | §15 intro + §15d: builder image and which path needs which measures. | | `docs/storage.md` | §10g: the mounting-step count/list now includes `build-podman-builder`. | | `USER_STORIES.md` | 20b: confirm true as written; reword only if an experiment changes it (body ≤ 90 words, no code fences). | | `tests/test_static.py` | add `build-podman-builder` to the `writers` lists (lines ~1127, ~4712)<sup>[8a](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/tests/test_static.py?display=source#L1156) [8b](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/tests/test_static.py?display=source#L4731)</sup>; extend `test_a_user_namespaced_step_is_named_on_the_allowlist` to assert §9n has no `BUILDAH_ISOLATION`, keeps `hostUsers: false` + `STORAGE_DRIVER: vfs`, and says per-run. | **Constraints / out-of-scope notes** - Do not touch `policy/k8s.rego`, zot credentials or the `milner/**` anonymous-push exception (#110), `worker/main.tf`'s sysctl, or annia-infra's `.woodpecker/ci.yaml` (paused). - `containers/storage#6968` in the context does not resolve on GitHub — dropped from the plan; Experiment 1 cites the kernel `userxattr`/NFS docs instead. - The consumer's pin lives in the consumer repo (`annia-infra`, its own paused change); milner has no in-tree manifest to guard, so no new `ci/*-tag-guard.sh`. - `NAMESPACED_ALLOWLIST` stays `{}` in this repo: milner builds the builder but never runs it.<sup>[4](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/tests/test_static.py?display=source#L2695)</sup> ## Acceptance Criteria - [ ] `.#podmanBuilderImage` is a `dockerTools.buildLayeredImage` output, built/checked/pushed by `build-podman-builder` with `nix build` + `skopeo`, at a tag derived from the flake pin. - [ ] The image carries podman, nix 2.35.2, `policy.json`/`registries.conf`, subuid/subgid, and `build-and-push <context> <tag> <registry>` with the token on stdin. - [ ] A Dockerfile whose `RUN` is `nix build` builds inside it, in a `hostUsers: false` pod, against the shared store, with no `unshare` in the step. - [ ] §9n has no `BUILDAH_ISOLATION: chroot`, keeps `STORAGE_DRIVER: vfs` with its reason, states per-run/at-most-one, and drops the `unshare -U true` evidence. - [ ] `docs/nix.md` §15 and §9n say which path needs which measures; story 20b is true as written or reworded here. - [ ] `.woodpecker/gates.yaml`'s comment block matches the per-run invariant. - [ ] Experiments 1–3 have recorded §9n outcomes; any removed measure is gone from doc and entrypoint. - [ ] `python3 -m pytest tests/` green, `NAMESPACED_ALLOWLIST` still empty, `conftest test --policy policy/` unchanged. ## Testing Plan - `python3 -m pytest tests/ -v` (incl. the allowlist and `test_docs.py` story guards); `python3 -m pytest tests/ -k step_commands` before editing `.woodpecker/`. - `nix build .#podmanBuilderCheck && cat result` — `/bin/*`, config files and `Cmd` resolve inside the image's own layers. - `conftest test --policy policy/` over rendered layers: rule 3 reads only `privileged`, so no change is the assertion. - One lab pod in `woodpecker-runtime` (no `resources`) running `build-and-push` against a busybox context and a `nix build` context, plus Experiments 1–3, then deleted. - A real consumer pipeline (annia-infra #310) once the pause lifts. ## Related Issues - #107 — the capability this corrects (its proof item 2 measured a distro podman). - #110 — the zot push credential the `publish` half consumes. - #109 — adjacent layered-store question, same store. - annia-infra #310 / #318 — first consumer, paused until this lands. - containers/buildah#6947 (chroot `setgroups`), #6910 (overlay over the build context). ## Research Findings - `buildah#6947` is fixed by v1.45.0, but the flake pin's `podman 5.8.4` vendors buildah v1.43.2, so `BUILDAH_ISOLATION: chroot` still fails at the pin; `pkgs.buildah` being 1.45.0 does not help `podman build`. - Nix clears setuid bits on every store path, so the image can never ship `newuidmap`/`newgidmap`; the entrypoint's `unshare --map-users` is load-bearing, and the pin's util-linux 2.42.2 supports it (root writes maps directly since v2.40). - The Woodpecker agent overrides a step image's entrypoint and runs `commands` via `/bin/sh -c`, so `build-and-push` must be an executable on `PATH`, not `config.Cmd`. - `dockerTools.buildLayeredImage` merges `contents` `bin/` into `/bin` and runs `extraCommands` unprivileged, which is the right place for `/etc/containers/*` and subuid/subgid.<sup>[6](https://nixos.org/manual/nixpkgs/stable/#ssec-pkgs-dockerTools-buildLayeredImage)</sup> - Unprivileged overlayfs needs kernel ≥ 5.11 plus the `userxattr` option and a non-NFS upper dir; milner's local-path PVCs meet the filesystem precondition (Experiment 1 is the measurement). - The pin's default `pkgs.nix` is 2.34.8 while the seeded store is nix 2.35.2, so the image must spell `pkgs.nixVersions.nix_2_35` or it reads a newer store schema it cannot parse. - Zot's `milner/**` and `preview/**` are anonymous-push today, so the builder's own push needs no token, but the interface still takes one on stdin for consumer `annia/**` destinations. ## Pre-Mortem 1. **The lab-pod experiments cannot be run in the implementation environment.** — Acceptance requires Experiments 1–3 run in a live `woodpecker-runtime` pod and their outcomes recorded in §9n, but the coding agent has no cluster credentials or kubectl access, so the step either stalls or ships unverified prose. Because the suite only checks image contents and doc text, nothing catches the gap. 2. **The §9n rewrite trips a guard that encodes today's text.** — `test_a_user_namespaced_step_is_named_on_the_allowlist` requires §9n to keep `hostUsers: false` and `STORAGE_DRIVER: vfs`, `test_no_step_in_the_pipeline_is_privileged` rejects any step image named `podman`, and `test_docs.py` caps story 20b at 90 words, so a rewrite that drops or rephrases one of these fails the build. The Woodpecker `${...}` substitution rule for `gates.yaml` comments is a second way the same edit breaks a test. 3. **The image's closure and Nix version diverge from the shared store.** — If `pkgs.nix` 2.34.8 slips in instead of `pkgs.nixVersions.nix_2_35`, or `build-podman-builder` fails to realise the full closure into the `nix-store` PVC, a consumer mounting the store over `/nix` either hits a schema-version error or loses the image's tools. This surfaces only in the paused downstream consumer (annia-infra #310), long after the change looks green here.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/109-nix-base
feat/109-layered-nix-store
feat/110-zot-push-credential
feat/90-annia-pause-lever-oz-console
feat/87-tailscale-egress-from-cluster
chore/ci-step-resources
docs/vault-secret-prompt
feat/42-support-backups-persistent-data
feat/41-integrate-restic-tfstate-backup
feat/3-setting-up-agent-development
kvm-kubernetes-setup
No results found.
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner#118
No description provided.