tepichord/milner

Fork 0

ci: run every step in an image this repo builds (base image, no shared store) #123

Open

tepichord wants to merge 3 commits from feat/109-nix-base into main

tepichord commented

2026-10-22 20:32:09 -07:00

Owner

Closes #109.

Every step of both pipeline files now runs one image this repository builds,
zot.faun-snapper.ts.net/milner/nix-base:2.35.2, and nothing mounts a store.

What changed

flake.nix gains nixBaseImage: the pinned nix (2.35.2), ciEnv (a
buildEnv over ciTools), install-ci-env, /bin/sh (static busybox) and
/bin/bash (bashOnly), cacert, the flake's inputs transitively,
devShells.ci's build-time closure, and — via
buildLayeredImageWithNixDb — the store's database. extraCommands writes
/etc/nix/nix.conf (sandbox = false, no build users), /root with
HOME=/root, and /usr/bin/env. nixBaseCheck asserts all of it against the
image's own bytes and asks the image's own nix whether a path it carries is
valid.
devShells splits into default (the fat workstation shell, unchanged)
and ci (ciTools), so a step does not pay for qemu to run pytest.
helm and htpasswd join ciTools: the suite skips four tests without
them and still reports "passed".
.woodpecker/gates.yaml / cleanup.yaml: every step is the base image
with pull: true and no volumes; every nix develop names .#ci. A new
first step, build-nix-base, builds and checks the image on a pull request
and pushes it on main (digest-verified). A new nix-base-tag-guard holds
the tag and the flake's nix pin together in both directions.
docs: docs/nix.md §14 rewritten (what is in the image and why, who runs
it, the traps), §15d the hand-push; docs/storage.md §10a/§10g, OPERATIONS.md
7a, ARCHITECTURE.md, AGENTS.md, docs/woodpecker-ci.md §9m and the
nix-store manifests now say the shared store is retired rather than shared.
story 19 in USER_STORIES.md rewritten: a step starts from an image
carrying what earlier builds realised; nothing is mounted, seeded or topped up.

The bootstrap

The tag had to exist before any step could start, so it was published by hand
(the recipe is docs/nix.md §15d): skopeo copy to
milner/nix-base:2.35.2, digest sha256:8807059e…, served manifest digest
compared equal.

Measured, in the published image (`podman run`, arm64)

nix develop .#ci --command python3 -V — 2 s, --network=none.
nix build ./#gnused — 0 s, a store hit, --network=none.
the whole suite — 345 passed, 0 skipped (76 s), on a copy of this tree.

Three failures were found this way and are fixed here, each documented as a
trap in docs/nix.md §14d: a store without its database (every path invalid,
900 MB re-downloaded), a flake input's own inputs (nix-systems/default), and
nix develop needing the shell's build-time closure (stdenv).

Not in this PR

gitops/clusters/milner/infrastructure/nix-store/ (the PVC and the nix
StorageClass) is kept and documented as unmounted — deleting it is a separate
decision, since the durable disk behind it also holds what previous runs
realised.

Closes #109. Every step of both pipeline files now runs one image this repository builds, `zot.faun-snapper.ts.net/milner/nix-base:2.35.2`, and nothing mounts a store. ## What changed - **`flake.nix`** gains `nixBaseImage`: the pinned nix (2.35.2), `ciEnv` (a `buildEnv` over `ciTools`), `install-ci-env`, `/bin/sh` (static busybox) and `/bin/bash` (bashOnly), `cacert`, the flake's inputs *transitively*, `devShells.ci`'s build-time closure, and — via `buildLayeredImageWithNixDb` — the store's database. `extraCommands` writes `/etc/nix/nix.conf` (`sandbox = false`, no build users), `/root` with `HOME=/root`, and `/usr/bin/env`. `nixBaseCheck` asserts all of it against the image's own bytes and asks the image's own nix whether a path it carries is valid. - **`devShells`** splits into `default` (the fat workstation shell, unchanged) and `ci` (`ciTools`), so a step does not pay for qemu to run `pytest`. `helm` and `htpasswd` join `ciTools`: the suite *skips* four tests without them and still reports "passed". - **`.woodpecker/gates.yaml` / `cleanup.yaml`**: every step is the base image with `pull: true` and no `volumes`; every `nix develop` names `.#ci`. A new first step, `build-nix-base`, builds and checks the image on a pull request and pushes it on `main` (digest-verified). A new `nix-base-tag-guard` holds the tag and the flake's nix pin together in both directions. - **docs**: `docs/nix.md` §14 rewritten (what is in the image and why, who runs it, the traps), §15d the hand-push; `docs/storage.md` §10a/§10g, OPERATIONS.md 7a, ARCHITECTURE.md, AGENTS.md, `docs/woodpecker-ci.md` §9m and the `nix-store` manifests now say the shared store is retired rather than shared. - **story 19** in `USER_STORIES.md` rewritten: a step starts from an image carrying what earlier builds realised; nothing is mounted, seeded or topped up. ## The bootstrap The tag had to exist before any step could start, so it was published by hand (the recipe is `docs/nix.md` §15d): `skopeo copy` to `milner/nix-base:2.35.2`, digest `sha256:8807059e…`, served manifest digest compared equal. ## Measured, in the published image (`podman run`, arm64) - `nix develop .#ci --command python3 -V` — **2 s**, `--network=none`. - `nix build ./#gnused` — **0 s**, a store hit, `--network=none`. - the whole suite — **345 passed, 0 skipped** (76 s), on a copy of this tree. Three failures were found this way and are fixed here, each documented as a trap in `docs/nix.md` §14d: a store without its database (every path invalid, 900 MB re-downloaded), a flake input's own inputs (`nix-systems/default`), and `nix develop` needing the shell's build-time closure (`stdenv`). ## Not in this PR `gitops/clusters/milner/infrastructure/nix-store/` (the PVC and the `nix` StorageClass) is kept and documented as unmounted — deleting it is a separate decision, since the durable disk behind it also holds what previous runs realised.

tepichord added 2 commits

2026-10-22 20:32:09 -07:00

ci: run every step in an image this repo builds b1dec39571

Every step of both pipeline files ran `nixos/nix:2.35.2` with a shared store
mounted over `/nix`, seeded by hand and topped up by hand. The mount replaced
the image's own store, so a step could not start on an empty volume at all, the
seed and every mount had to name one image tag (the store database is versioned
by the nix that wrote it), population was a discipline rather than a lock, and
an image from another tag found its own `nix`, `bash` and `curl` hidden behind
it.

Replace it with one image built from this flake:
`zot.faun-snapper.ts.net/milner/nix-base:<nix version>` carries the pinned nix,
`ciEnv` over `ciTools` (`devShells.ci`, so a step and a developer run the same
programs from the same store paths), `/bin/sh` and `/bin/bash`, the trust
store, `/etc/nix/nix.conf`, and the flake inputs of the pin — so `nix develop`
and `nix build` in a step are store hits rather than fetches, and nothing
mounts anything.

- flake.nix: `ciTools`, `ciEnv`, `installCiEnvProgram`, the shell providers,
  `flakeInputs`, and `nixBaseImage`/`nixBaseTag`/`nixBaseCheck`, with
  `devShells` split into the fat `default` and the lean `ci`.
- `ci/nix-base-image.sh` + `ci/nix-base-tag-guard.sh`: the reader and the
  guard that hold the pipeline's tags to the flake's nix pin, in both
  directions, allowing the one case a tag cannot express (a nixpkgs bump that
  does not move the nix version — the push half re-pushes, `pull: true`
  carries it to the pods).
- `build-nix-base`: builds, checks and (on a push to `main`) publishes the
  image, running `nixos/nix` because that is the seed a pipeline needs before
  the image exists. The first publish is an operator action.
- tests: the base-image identity, the whole-file "one image, no mounts"
  guard, the installer run against fixture flakes (which caught
  `install-ci-env` installing an empty environment: at this pin `mkShell`
  keeps a consumer's packages in `buildInputs` and leaves
  `nativeBuildInputs` empty), and the tag guard's five cases.
- docs and story 19: the store was what a step mounted; it is now what its
  image carries. The shared directory stays on the durable disk and nothing
  mounts it (docs/storage.md §10g).

ci: make the base image a store steps can actually use 50e7059555

The base image built, checked and pushed, and every step still fetched its
toolchain: `buildLayeredImage` carries store *paths* and not the store's
database, so `nix` treated all 900 MB of them as invalid. Found in a pod,
against the published image, three fixes deep:

- `buildLayeredImageWithNixDb`: the database is what says a path is valid.
  Without it `nix develop` re-downloaded the tools the image carried and
  `nix build ./#gnused` began building readline from its tarball.
- `flakeInputs` walks the input tree transitively: `flake-utils` has an input
  of its own (`nix-systems/default`), resolved before its outputs run, so a
  step fetched a flake this repository never names.
- `ciShellInputs` carries `devShells.ci`'s build-time closure: `nix develop`
  enters a *derivation*, and `stdenv` is in no tool's runtime closure. Its
  `inputDerivation` names that closure but is a symlink `dockerTools` cannot
  mirror, so the paths come out of `closureInfo` into a directory.

Three more, each a pod away:

- `HOME=/root`, and the directory, because a container with no `/etc/passwd`
  leaves `$HOME` unset and nix then puts its fetcher locks in the checkout.
- `/usr/bin/env`, because a script here starts `#!/usr/bin/env bash` and the
  kernel runs the shebang, not the PATH — five tests failed on stubs that had
  become unrunnable files.
- every step names `.#ci`: bare `nix develop` enters the fat `default` shell,
  whose 4.6 GB closure the image does not carry (129 s and still going, against
  3 s). `helm` and `htpasswd` join `ciTools`, because the suite *skips* four
  tests without them and still says "passed".

Measured, in the image, `--network=none`: `nix develop .#ci` 3 s, and the whole
suite green.

tepichord added the

reviewing

label

2026-10-22 20:32:09 -07:00

tepichord added 1 commit

2026-10-22 20:37:57 -07:00

ci: re-trigger the gate after the base image was published 4bc9bcd535

Some required checks are missing.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/109-nix-base:feat/109-nix-base

git switch feat/109-nix-base

No reviewers

No milestone

No project

No assignees

1 participant

Notifications

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

tepichord/milner!123

No description provided.

Rows
Columns

ci: run every step in an image this repo builds (base image, no shared store) #123

What changed

The bootstrap

Measured, in the published image (podman run, arm64)

Not in this PR

Checkout

Measured, in the published image (`podman run`, arm64)