Make the podman builder run, and let a step build without pushing

tepichord commented

2026-10-07 22:17:58 -07:00

Owner

What happened

milner/podman-builder was merged in #119 without ever having been executed.
Built in a pod from the image the registry serves (nixpkgs-e5bdc4a4), the
entrypoint dies three times before a Dockerfile builds:

The re-exec's map is spelled with two separators. unshare --map-users=0,0:"$ids" is the hybrid 0,0:65536, and the util-linux the
image carries (2.42.2) accepts 0,0,65536 (and 0:0,65536) and refuses
the hybrid outright:
```
unshare: invalid mapping '0,0:65536'
```
The re-exec is the whole way into the namespace, so nothing after it ran —
build-and-push never reached podman.
/var/tmp does not exist in a pod, and buildah stats its temporary
directory rather than making it:
```
creating a temporary directory: stat /var/tmp: no such file or directory
```
Nothing trusted a registry. A pod carries no certificates and the image
is not a distribution, so every handshake died with x509: certificate signed by unknown authority — the base-image pull and the push alike.
/tmp does not exist either, and the push's --digestfile
/tmp/push-digest is written after the manifest is: the push looked like
it had worked and then died with open /tmp/push-digest: no such file or directory.

With those patched by hand the whole flow works end to end, and it was
re-measured against the fixed image afterwards, in a pod built from it: a
proxy Dockerfile carrying annia's own nix profile install --priority 4
builds, a call naming no registry stops after COMMIT, and a call naming the
registry pushes and reports the served manifest equal to the pushed one.

The fix

the map is all commas — --map-users=0,0,"$ids", --map-groups=0,0,"$ids";
mkdir -p /run/user/0/libpod/tmp /var/tmp, both for the measured reason;
pkgs.cacert rides contents and SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt
is in the image's Env, because Go — podman and buildah — does not read a
distribution's path by default;
the image carries gitMinimal: it is the step image, and naming a tag from
the checkout (git rev-list --count HEAD) is a step's own line.

The interface, and why it moved

A call now takes two required arguments and one optional registry, so a
step that must not push is one command with no registry and no secret:

build-and-push . annia:ci                     # built, nothing pushed
build-and-push . annia:$TAG zot.faun-snapper.ts.net   # built, pushed, verified

The login happens only when a token was handed over. milner/** and
preview/** are anonymous pushes (docs/woodpecker-ci.md §9n), so a caller
pushing to one of those is no longer stopped by a variable it has no reason to
set, and a destination that does need the annia-ci credential is refused by
the registry itself.

podmanBuilderCheck now holds each measure, the hybrid separator as a
negative, and names pkgs.pkgsStatic.busybox for the reason reaperCheck
documents — without it /bin/sh's chain leaves the extracted root for a store
the sandbox does not carry, and the check failed on a correct image under
nix build's default sandbox.

What merging this does

flake.nix is an input to two tag guards, so this change moves milner/reaper
to 0.2.4 and milner/annia-enqueuer to 0.1.3 (the same bump the previous
flake change made), with the docs and manifests that name each tag.

flake.lock does not move, so the builder's tag does not move. The
main-push build-podman-builder step re-pushes nixpkgs-e5bdc4a4 with the
fixed bytes, which means:

a consumer that cached the broken image keeps running it until it re-pulls,
so a step pinning a mutable tag sets pull: true (annia-infra#318 does);
the tag being a toolchain name rather than a content hash is the thing to
look at next — it is recorded in §9n, not fixed here.

Checks

nix build .#podmanBuilderCheck passes under the default sandbox (and the
message it prints is unchanged);
python -m pytest tests/ — 331 passed;
ci/{reaper,enqueuer,router,oz}-tag-guard.sh main all agree;
the flow itself, in a pod built from the pushed image, with the three fixes
applied by hand: DIGESTS-MATCH.

## What happened `milner/podman-builder` was merged in #119 without ever having been executed. Built in a pod from the image the registry serves (`nixpkgs-e5bdc4a4`), the entrypoint dies three times before a Dockerfile builds: 1. **The re-exec's map is spelled with two separators.** `unshare --map-users=0,0:"$ids"` is the hybrid `0,0:65536`, and the util-linux the image carries (2.42.2) accepts `0,0,65536` (and `0:0,65536`) and refuses the hybrid outright: ``` unshare: invalid mapping '0,0:65536' ``` The re-exec is the whole way into the namespace, so nothing after it ran — `build-and-push` never reached `podman`. 2. **`/var/tmp` does not exist in a pod**, and buildah *stats* its temporary directory rather than making it: ``` creating a temporary directory: stat /var/tmp: no such file or directory ``` 3. **Nothing trusted a registry.** A pod carries no certificates and the image is not a distribution, so every handshake died with `x509: certificate signed by unknown authority` — the base-image pull and the push alike. 4. **`/tmp` does not exist either**, and the push's `--digestfile` `/tmp/push-digest` is written *after* the manifest is: the push looked like it had worked and then died with `open /tmp/push-digest: no such file or directory`. With those patched by hand the whole flow works end to end, and it was re-measured against the fixed image afterwards, in a pod built from it: a proxy Dockerfile carrying annia's own `nix profile install --priority 4` builds, a call naming no registry stops after COMMIT, and a call naming the registry pushes and reports the served manifest equal to the pushed one. ## The fix * the map is all commas — `--map-users=0,0,"$ids"`, `--map-groups=0,0,"$ids"`; * `mkdir -p /run/user/0/libpod/tmp /var/tmp`, both for the measured reason; * `pkgs.cacert` rides `contents` and `SSL_CERT_FILE=/etc/ssl/certs/ca-bundle.crt` is in the image's `Env`, because Go — podman and buildah — does not read a distribution's path by default; * the image carries `gitMinimal`: it is the *step* image, and naming a tag from the checkout (`git rev-list --count HEAD`) is a step's own line. ## The interface, and why it moved A call now takes **two required arguments and one optional registry**, so a step that must not push is one command with no registry and no secret: ``` build-and-push . annia:ci # built, nothing pushed build-and-push . annia:$TAG zot.faun-snapper.ts.net # built, pushed, verified ``` The login happens only when a token was handed over. `milner/**` and `preview/**` are anonymous pushes (docs/woodpecker-ci.md §9n), so a caller pushing to one of those is no longer stopped by a variable it has no reason to set, and a destination that does need the `annia-ci` credential is refused by the registry itself. `podmanBuilderCheck` now holds each measure, the hybrid separator as a negative, and names `pkgs.pkgsStatic.busybox` for the reason `reaperCheck` documents — without it `/bin/sh`'s chain leaves the extracted root for a store the sandbox does not carry, and the check failed on a **correct** image under `nix build`'s default sandbox. ## What merging this does `flake.nix` is an input to two tag guards, so this change moves `milner/reaper` to `0.2.4` and `milner/annia-enqueuer` to `0.1.3` (the same bump the previous flake change made), with the docs and manifests that name each tag. `flake.lock` does not move, so the builder's tag does **not** move. The main-push `build-podman-builder` step re-pushes `nixpkgs-e5bdc4a4` with the fixed bytes, which means: * a consumer that cached the broken image keeps running it until it re-pulls, so a step pinning a mutable tag sets `pull: true` (annia-infra#318 does); * the tag being a toolchain name rather than a content hash is the thing to look at next — it is recorded in §9n, not fixed here. ## Checks * `nix build .#podmanBuilderCheck` passes under the default sandbox (and the message it prints is unchanged); * `python -m pytest tests/` — 331 passed; * `ci/{reaper,enqueuer,router,oz}-tag-guard.sh main` all agree; * the flow itself, in a pod built from the pushed image, with the three fixes applied by hand: `DIGESTS-MATCH`.

tepichord added 1 commit

2026-10-07 22:17:58 -07:00

fix(flake): make the podman builder run, and build or push on demand

ci/woodpecker/pr/gates Pipeline was canceled

Details

d6b75033a2

The builder image shipped without ever having been run: `/bin/build-and-push`
died on three things before a Dockerfile ever built, all measured in a pod
built from the pushed image.

  * `unshare --map-users=0,0:"$ids"` is a mixed separator, and the util-linux
    the image carries refuses the hybrid outright (`unshare: invalid mapping
    '0,0:65536'`) while taking `0,0,65536` (and `0:0,65536`). The re-exec is
    the whole way into the namespace, so nothing ran at all;
  * buildah *stats* its temporary directory rather than making it, and a pod
    carries no `/var/tmp` (`creating a temporary directory: stat /var/tmp: no
    such file or directory`), so the first thing a build does died;
  * nothing in the image trusted a registry: a pod carries no certificates and
    Go — podman and buildah — does not read a distribution's path by default,
    so every handshake failed with `x509: certificate signed by unknown
    authority`. `pkgs.cacert` rides `contents` now and `SSL_CERT_FILE` names
    the bundle in the image's `Env`.

The registry is optional as well: two required arguments and one, so a call
that names no registry builds and stops. That is the shape a pull-request
step runs — the Dockerfile is proved to build without writing a tag no
preview deploys, and without a secret it has no reason to hold. `milner/**`
and `preview/**` are anonymous pushes, so the login happens only when a token
was handed over; a destination that does need the `annia-ci` credential is
refused by the registry itself, which is the only party that can say so.

`podmanBuilderCheck` holds each of these, the hybrid separator as a negative,
and names `pkgs.pkgsStatic.busybox` for the reason reaperCheck documents:
without it `/bin/sh`'s chain leaves the extracted root for a store the
sandbox does not carry, and the check fails on a correct image.

`flake.nix` is an input to two tag guards, so `reaper` and `annia-enqueuer`
move to `0.2.4` and `0.1.3` — the same bump the previous flake change made,
and each with the docs that name the tag.

tepichord force-pushed fix/podman-builder-runs from d6b75033a2

ci/woodpecker/pr/gates Pipeline was canceled

Details

to 7590e10627

ci/woodpecker/pr/gates Pipeline was canceled

Details

2026-10-07 22:22:27 -07:00

Compare

tepichord force-pushed fix/podman-builder-runs from 7590e10627

ci/woodpecker/pr/gates Pipeline was canceled

Details

to 27cc28b345