Layered Nix store: image as immutable lower, shared volume as writable upper

tepichord commented

2026-09-30 12:13:05 -07:00

Owner

Closes #109

Automated PR created by forgejo-done adapter.

Closes #109 Automated PR created by forgejo-done adapter.

tepichord added 6 commits

2026-09-30 12:13:05 -07:00

feat(ci): publish the store into a shared binary cache at /shared-nix 312086a513

The directory behind the `nix` class stops being a store mounted over
`/nix` and becomes a signed Nix binary cache at `/shared-nix`. Every
building step keeps the image's own `/nix`, substitutes from the cache
through `extra-substituters`, and publishes what it built with one
`ci/publish-nix-cache.sh` (`nix copy --to file:///shared-nix`, signed on
upload so `require-sigs = true` stays on).

The signing key is served by a third secret-extension rule on `push`
and `manual` only, so a pull request substitutes but never publishes.
Each building step therefore has a pull-request twin that carries the
commands and no key, and a push/manual twin that carries the key; the
commands are one YAML anchor so the twins cannot drift. No overlay, no
assembler and no privileged workload is involved.

Refs: #109

test(ci): hold the cache mount, the publish and the image's own /nix 0a6a53c377

Re-point the suite at the cache: a whole-tree sweep that no pod spec
under `gitops/` mounts anything at `/nix`, every mounting step carries
both `NIX_CONFIG` lines, every building step calls
`ci/publish-nix-cache.sh` behind the key guard, the publish has one
implementation, and no step sets `readOnlyRootFilesystem`. The §10g
seed-tag equality is deleted; what remains is the ordinary no-`:latest`
pin.

Refs: #109

docs(nix): the shared store becomes a cache beside the image 1037a508e4

`/var/mnt/data/nix/shared` is a signed binary cache at `/shared-nix`
now, not a store mounted over `/nix`. Rewrite the runbook (docs/nix.md
§14b/§14c/§14d, docs/storage.md §10a), drop §10g's seed Job and its awk
extraction for §10h's trim action (narinfo and nar together,
oldest-first), and cascade the wording through OPERATIONS.md, AGENTS.md,
ARCHITECTURE.md and storage/README.md. The on-node measurements the
implementer cannot take are marked `[measure]` rather than invented.

Stories 19 and 20 move with the architecture they describe, which the
Hard Rule requires; the §9k AppRole policy gains the cache path and the
secret extension gets its third rule.

Refs: #109

docs(ci): create the cache signing key, trim the publish comments dd552234e1

Fill the one gap in the cache runbook: §9k already names the
`secret/ci/nix-cache` rule and AppRole path but, unlike every other
pipeline secret there, never showed how the value is generated — add the
`nix key generate-secret` example beside the PAT's write, and drop the
"§9k's sed" pointer it never had.

Shorten the publish-guard comment in gates.yaml: seven lines repeated in
each building step restated what the twin steps' own comments already
say, so keep only why the guard exists. Fix step 7a's continuation
indent in OPERATIONS.md.

Refs: #109

docs(nix): fix the binary cache layout and the trim recipe 689251ab25

A local `file://` cache keeps its narinfos at the cache root, not in a
`narinfo/` subdirectory, and a narinfo's `URL` is the nar's path relative to
that root — `nar/<hash>.nar.xz` (xz is the default, so the file is not
`*.nar`). The trim runbook in docs/storage.md §10h used `narinfo/*.narinfo`
and `rm -f "$info" "nar/$nar"`, so it matched nothing and, had it matched,
would have removed `nar/nar/...`: a no-op runbook for the one action that keeps
the cache from filling the data disk. Verified against a real `nix copy --to
file:///…` cache with Nix 2.35.2.

Co-authored-by: milner-ci <ci@milner.local>

test(nix): hold every mounting step to one cache key

ci/woodpecker/pr/gates Pipeline was canceled

Details

b6ee3d1aa2

The public half of `milner-nix-cache-1` is copied into every mounting step's
`NIX_CONFIG`, and the guard only checked the `milner-nix-cache-1:` prefix — so
a step whose literal drifted passed while it refused every narinfo its
neighbours served. Assert the full value is one key across the mounts, and fix
the missing blank line before the next test.

Co-authored-by: milner-ci <ci@milner.local>

tepichord added the

pr

reviewing

label

2026-09-30 12:13:05 -07:00

tepichord referenced this pull request

2026-10-05 18:18:59 -07:00

Layered Nix store: image as immutable lower, shared volume as writable upper #109

tepichord added this to the MVP project

2026-10-20 12:25:38 -07:00

tepichord added 1 commit

2026-10-20 14:56:46 -07:00

fix(ci): pin the cache's public key to the key Vault holds

ci/woodpecker/pr/gates Pipeline was canceled

Details

e5600f81b0

The branch trusts `milner-nix-cache-1:wzztGob9…`, a key nothing can
produce: `vault kv metadata get secret/ci/nix-cache` answered "No value
found", so the first push run would have exited on the unset variable and
every mounting step would have refused every path the cache holds — a
warm-looking cache that substitutes nothing.

The pair now exists in Vault (field `NIX_CACHE_SIGNING_KEY`), and the
literal in `.woodpecker/gates.yaml` and `.woodpecker/cleanup.yaml` is
`nix key convert-secret-to-public` of that value read back from Vault —
computed, not retyped: 21 occurrences, one key.

Measured: publishing a single path into a throwaway `file://` cache with
the key read from Vault wrote `nix-cache-info`, `nar/`, `narinfo/` and one
`Sig: milner-nix-cache-1:` line, so the signing half works. Substitution
across a store boundary could not be driven from this CLI (a `file://`
store will not resolve a store-path argument), so the first live run is
what proves the verifying half — which is the measurement §14 still owes.

Held by `test_every_step_that_mounts_the_cache_reads_through_it`, which
compares the literals to each other, and by the Vault readback above; the
secret half is never in the tree.

tepichord added 1 commit

2026-10-20 15:05:54 -07:00

Merge branch 'main' into feat/109-layered-nix-store

ci/woodpecker/pr/gates Pipeline was canceled

Details

fcbc3ea7af

The branch forked before the builder image existed, so this is the
reconciliation: main's 40 commits of podman-builder, admission policy,
syncthing and the zot publish rule, against this branch's move from a
store mounted at /nix to a signed cache at /shared-nix.

Five files conflicted. How each was settled:

  .woodpecker/gates.yaml — main's `build-podman-builder` is a step of this
  shape now, so it becomes the twin every other build has:
  `build-podman-builder-pull-request` (cache read-only, no key) and
  `build-podman-builder` (push/manual, key, publish) over one commands
  anchor. Its `/nix` mount became `/shared-nix` + NIX_CONFIG, and its two
  "seeded store" sentences became the cache's (the store it realised into
  is gone with §10g).

  secret-extension — main moved the rules out of one YAML into
  `secret-extension/config.yml` and this branch added a rule to the old
  file, so the modify/delete conflict resolves by taking main's layout and
  re-homing the rule: `milner-nix-cache-publish` is the fourth rule, and
  the old `secret-extension.yaml` is deleted.

  docs/woodpecker-ci.md — both halves survive: the AppRole policy names
  four paths, `ci/zot` and `ci/nix-cache` side by side with their own
  comment blocks.

  docs/storage.md — §10g's seed job is the shape this branch replaced, so
  keeping it would document a mechanism nothing runs; the trim runbook
  (§10h) stays and its step list gains `build-podman-builder`.

  tests/test_static.py — main's `§10g seeds the store with one image`
  assertions fail by construction against a cache (there is no store
  database to match), so they go; main's builder steps join the branches'
  sets where they still hold.

Two things only a merge can break, both fixed here: `pkgs.nixVersions.
nix_2_35` was justified by "a nix younger than the store's schema", which
a binary cache does not have — the pin now earns its keep as "the version
every step image and the flake run"; and two `docs/storage.md §10g`
references pointed at a section this branch deleted.

Measured: 336 tests pass (328 on the branch alone — main's 8 arrived with
the merge), `terraform -chdir=storage test` passes, and every one of the
22 gates steps mounts `nix-store:/shared-nix`, with `NIX_CACHE_SIGNING_KEY`
on exactly the push/manual halves of the six building steps
(`test`, `build-oz`, `build-router`, `build-reaper`, `build-enqueuer`,
`build-podman-builder`).

tepichord added 1 commit

2026-10-22 14:03:03 -07:00

Merge remote-tracking branch 'origin/main' into feat/109-layered-nix-store

ci/woodpecker/pr/gates Pipeline was canceled

Details

d282564e67

ci/woodpecker/pr/gates Pipeline was canceled

Required

Details

Some required checks were not successful.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u origin feat/109-layered-nix-store:feat/109-layered-nix-store

git switch feat/109-layered-nix-store

Rows
Columns

Layered Nix store: image as immutable lower, shared volume as writable upper #114

Checkout