tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Layered Nix store: image as immutable lower, shared volume as writable upper #109

New issue
Open
opened 2026-09-29 16:27:35 -07:00 by tepichord · 4 comments
tepichord commented 2026-09-29 16:27:35 -07:00
Owner
Copy link

Why

Every step that wants a Nix store mounts the same claim (nix-store, class nix, /var/mnt/data/nix/shared) over /nix, which hides the image's own store. The volume therefore has to be filled by hand before anything can run: the one-shot nix-seed Job in docs/storage.md §10g, which cp -an /nix/. /mnt/ and then proves /bin/sh's target exists in the volume, because an unseeded volume does not fail as nix: not found — the pod cannot exec its entrypoint at all and Woodpecker reports nothing until the workflow deadline. Three consequences follow, and they are the whole of this issue:

  1. A fresh volume is a pod that cannot start, and the fix is an imperative step against state that Flux does not own.
  2. An image-tag bump needs a top-up, because the volume replaces the image's store. §10g states it plainly, and tests/test_static.py holds the seed tag and every consuming step's tag equal (the "one image, six consumers" assertions) — a coupling that is correct today and becomes unnecessary under a layered store.
  3. The annia pod (milner#25) wants the same store, in a namespace whose PSS is baseline, so it cannot take the escape hatch the CI steps could.

The shape

/nix/store becomes an overlay: the image's own /nix/store is the immutable lower, the durable directory is the writable upper. A fresh upper is then startable — the image is the base — and an image-tag bump needs no top-up at all.

store = local-overlay://?lower-store=/&upper-layer=<DIR>/upper&state=<DIR>/state

nix.conf needs extra-experimental-features = nix-command local-overlay-store and store = set before any nix use; the mount happens with mount -t overlay … (busybox/static mount is enough).

Recommended: the merge happens once, on the node, not in each pod. A privileged assembler (a DaemonSet, image nixos/nix:<pin>, hostPath on the durable disk) creates the merged store at a stable mount point; consumers bind-mount that mount point at /nix with mountPropagation: HostToContainer. Reasons:

  • annia is baseline — added capabilities and privileged are forbidden there, and hostPath is forbidden too, so an in-pod overlay cannot be the mechanism for the pod that most wants the store. One mechanism for both consumers beats two.
  • woodpecker-runtime is already privileged, so a step could do its own overlay — that is the tempting shortcut, and it is rejected precisely because it cannot be extended to annia.

The alternative — each pod overlays for itself with SYS_ADMIN — is worth stating and rejecting in the plan rather than leaving implicit.

What the plan must settle (not guess)

  • The propagation path. The consumer's volume has to receive the assembler's mount. Whether that works through the existing nix-class claim (a local-path PV is hostPath-shaped, so it may propagate) or requires an explicit hostPath volume to the assembler's mount point is the first thing to measure on the node, not assume. My recommendation: explicit hostPath + mountPropagation: HostToContainer, because the assembler owns the mount and the propagation mode should be visible in the manifest. The nix StorageClass then stays what it is for anything that genuinely wants a private durable directory, and stops being "the store".
  • Concurrent writers to one upper. Multiple mount namespaces (concurrent pipelines, and the annia pod) would share one upper layer. Nix's overlay store documents reduced guarantees compared with the local store, and the store DB plus links live in the shared state dir; whether its locking covers two pods building at once is the central risk here. Settle it from the manual and a two-writer smoke test. If it does not hold, the shape changes to one upper per writer-group over the shared immutable lower, and the plan must say what that costs (the per-group warm cache is no longer shared).
  • GC. The upper grows and nothing collects it today. The lower must never be collected (nix store gc from a consumer would try to delete into the overlay and, worse, a lower that changes under an existing upper corrupts it), so the owner of GC is the assembler, running nix-collect-garbage on its nix, and a consumer running GC has to be ruled out explicitly. docs/storage.md §10e already excludes /var/mnt/data/nix/ from backups as "reproducible from the package cache"; a layered store keeps that true — the upper is derived state — and the plan should say so rather than leave the exclusion looking like an accident.
  • The upgrade rule. Forward-only: a base-image bump moves the lower, and an older nix refuses a newer DB. So the assembler's tag, the consumers' effective nix (which comes from the lower) and the store's schema version are one thing — the same "one image" rule as today, restated as a pin rule. Bump the assembler first, then consumers, in one change.
  • The assembler's namespace and identity. A namespace that enforces privileged (the vault/woodpecker-runtime/local-path-storage precedent) or an explicit PSA label on a new one, automountServiceAccountToken: false, nodeSelector: {node-role: worker} (the durable disk is only on the worker).

What this deletes

  • The nix-seed Job block in docs/storage.md §10g, the extraction snippet under it, and §10g's explanation of why the mount has to be seeded.
  • tests/test_static.py's seed-tag assertions and its ### 10g. section regex (the "one image, six consumers" rule) — the rule survives, the seed does not, so the test has to be re-pointed at the assembler's pin.
  • milner#25's annia-nix claim, which is re-derived by this issue: the claim-at-/nix is the A-shape this replaces.

Verification

A consumer pod whose upper is empty starts and finds its own nix, bash and curl under the merged /nix with no seed step; nix develop --command cargo --version in a step succeeds; concurrent runs of two steps (a deliberate overlap) do not corrupt the store DB; a podman-less restart of the assembler re-establishes the mount without touching the upper; and a bump of the assembler's image tag, done as the pin rule requires, does not strand a single consumer.

Out of scope

The store's contents (which closures are warm) and the flake pins. This changes where the store's bytes come from and who owns them, not what gets built.

## Why Every step that wants a Nix store mounts the same claim (`nix-store`, class `nix`, `/var/mnt/data/nix/shared`) **over** `/nix`, which hides the image's own store. The volume therefore has to be filled by hand before anything can run: the one-shot `nix-seed` Job in `docs/storage.md` §10g, which `cp -an /nix/. /mnt/` and then proves `/bin/sh`'s target exists in the volume, because an unseeded volume does not fail as `nix: not found` — the pod cannot exec its entrypoint at all and Woodpecker reports nothing until the workflow deadline. Three consequences follow, and they are the whole of this issue: 1. **A fresh volume is a pod that cannot start**, and the fix is an imperative step against state that Flux does not own. 2. **An image-tag bump needs a top-up**, because the volume replaces the image's store. §10g states it plainly, and `tests/test_static.py` holds the seed tag and every consuming step's tag equal (the "one image, six consumers" assertions) — a coupling that is *correct today* and becomes unnecessary under a layered store. 3. **The annia pod (milner#25) wants the same store**, in a namespace whose PSS is `baseline`, so it cannot take the escape hatch the CI steps could. ## The shape `/nix/store` becomes an **overlay**: the image's own `/nix/store` is the **immutable lower**, the durable directory is the **writable upper**. A fresh upper is then startable — the image is the base — and an image-tag bump needs no top-up at all. ``` store = local-overlay://?lower-store=/&upper-layer=<DIR>/upper&state=<DIR>/state ``` `nix.conf` needs `extra-experimental-features = nix-command local-overlay-store` and `store =` set **before any nix use**; the `mount` happens with `mount -t overlay …` (busybox/static `mount` is enough). **Recommended: the merge happens once, on the node, not in each pod.** A privileged assembler (a DaemonSet, image `nixos/nix:<pin>`, hostPath on the durable disk) creates the merged store at a stable mount point; consumers bind-mount that mount point at `/nix` with `mountPropagation: HostToContainer`. Reasons: - **`annia` is `baseline`** — added capabilities and `privileged` are forbidden there, and `hostPath` is forbidden too, so an in-pod overlay cannot be the mechanism for the pod that most wants the store. One mechanism for both consumers beats two. - `woodpecker-runtime` is already `privileged`, so a step *could* do its own overlay — that is the tempting shortcut, and it is rejected precisely because it cannot be extended to `annia`. The alternative — each pod overlays for itself with `SYS_ADMIN` — is worth *stating and rejecting* in the plan rather than leaving implicit. ## What the plan must settle (not guess) - **The propagation path.** The consumer's volume has to receive the assembler's mount. Whether that works through the existing `nix`-class claim (a local-path PV is hostPath-shaped, so it may propagate) or requires an **explicit `hostPath` volume** to the assembler's mount point is the first thing to measure on the node, not assume. My recommendation: explicit `hostPath` + `mountPropagation: HostToContainer`, because the assembler owns the mount and the propagation mode should be visible in the manifest. The `nix` StorageClass then stays what it is for anything that genuinely wants a private durable directory, and stops being "the store". - **Concurrent writers to one upper.** Multiple mount namespaces (concurrent pipelines, and the annia pod) would share one upper layer. Nix's overlay store documents *reduced* guarantees compared with the local store, and the store DB plus `links` live in the shared `state` dir; whether its locking covers two pods building at once is the central risk here. Settle it from the manual **and** a two-writer smoke test. If it does not hold, the shape changes to **one upper per writer-group** over the shared immutable lower, and the plan must say what that costs (the per-group warm cache is no longer shared). - **GC.** The upper grows and nothing collects it today. The lower must never be collected (`nix store gc` from a consumer would try to delete into the overlay and, worse, a lower that changes under an existing upper corrupts it), so the owner of GC is the assembler, running `nix-collect-garbage` on **its** nix, and a consumer running GC has to be ruled out explicitly. `docs/storage.md` §10e already excludes `/var/mnt/data/nix/` from backups as "reproducible from the package cache"; a layered store keeps that true — the upper is derived state — and the plan should say so rather than leave the exclusion looking like an accident. - **The upgrade rule.** Forward-only: a base-image bump moves the lower, and an older `nix` refuses a newer DB. So the assembler's tag, the consumers' effective nix (which comes *from* the lower) and the store's schema version are one thing — the same "one image" rule as today, restated as a pin rule. Bump the assembler first, then consumers, in one change. - **The assembler's namespace and identity.** A namespace that enforces `privileged` (the `vault`/`woodpecker-runtime`/`local-path-storage` precedent) or an explicit PSA label on a new one, `automountServiceAccountToken: false`, `nodeSelector: {node-role: worker}` (the durable disk is only on the worker). ## What this deletes - The `nix-seed` Job block in `docs/storage.md` §10g, the extraction snippet under it, and §10g's explanation of *why* the mount has to be seeded. - `tests/test_static.py`'s seed-tag assertions and its `### 10g.` section regex (the "one image, six consumers" rule) — the rule survives, the seed does not, so the test has to be re-pointed at the assembler's pin. - **milner#25's `annia-nix` claim**, which is re-derived by this issue: the claim-at-`/nix` is the A-shape this replaces. ## Verification A consumer pod whose upper is **empty** starts and finds its own `nix`, `bash` and `curl` under the merged `/nix` with no seed step; `nix develop --command cargo --version` in a step succeeds; concurrent runs of two steps (a deliberate overlap) do not corrupt the store DB; a `podman`-less restart of the assembler re-establishes the mount without touching the upper; and a bump of the assembler's image tag, done as the pin rule requires, does not strand a single consumer. ## Out of scope The store's *contents* (which closures are warm) and the flake pins. This changes where the store's bytes come from and who owns them, not what gets built.
tepichord added this to the MVP project 2026-09-29 16:30:50 -07:00
tepichord commented 2026-09-29 16:32:40 -07:00
Author
Owner
Copy link

Waiting until annia-infra#310 is complete

Waiting until annia-infra#310 is complete
tepichord commented 2026-09-29 17:08:05 -07:00
Author
Owner
Copy link

Naina:
Updated on 2026-10-06 based on maintainer feedback.

Overview

Which shape stands (read this first). Issue #109's title and body specify the layered overlay (/nix = pinned image store as immutable lower + durable upper). Comments #10176/#10220 and the unmerged PR #114 specify a binary cache at /shared-nix and call it the operator's absolute decision; on 2026-10-05 the issue moved status/reviewing → status/planning while still titled "Layered Nix store" (/issues/109/timeline). This plan takes the title and body at their word and reinstates the layered shape; if the cache decision still stands, reject this and the withdrawn cache plan in #10176 holds.

Replace the hand-seeded store-at-/nix with an overlay assembled once per node: the pinned image's /nix is the immutable lower, the durable directory the writable upper, merged by a privileged DaemonSet and consumed through the existing nix-class claim11b. §10g's seed Job dies, a fresh upper starts, and an image bump moves the lower without a hand-run top-up.

Technical Approach

  1. Spike first — settle the four load-bearing facts on the worker (gating).

    # throwaway privileged pod; results go into docs/nix.md §14b before any manifest lands
mount -t overlay overlay -o lowerdir=/lower,upperdir=/upper,workdir=/work /var/mnt/data/nix/shared
# then, in a pod that mounts nix-store:/nix with NO mountPropagation:
grep ' /nix ' /proc/self/mounts            # (a) is the overlay visible at pod start?
nix --extra-experimental-features 'nix-command local-overlay-store' store ping   # (b) does it read?

    (c) two deliberately overlapping steps against one upper: does the SQLite DB stay consistent? (d) is check-mount off (the consumer sees the assembler's bind mount, not an overlay)? If (a) is false, Shape A cannot serve Woodpecker (its backend sets no mountPropagation — B6) — stop and report rather than guess.

  2. A privileged assembler namespace.

    # gitops/clusters/milner/infrastructure/namespaces/nix-assembler.yaml
apiVersion: v1
kind: Namespace
metadata: {name: nix-assembler, labels: {pod-security.kubernetes.io/enforce: privileged}}

  3. The assembler DaemonSet — materialise the lower, mount the overlay once. A host DaemonSet cannot see a consumer pod's mount namespace (A7)4, so the lower must be materialised on the durable disk; the assembler copies its own pinned image's /nix, and the pin rule makes it equal to the consumers'.

    # gitops/clusters/milner/infrastructure/nix-assembler/daemonset.yaml
spec:
  template:
    spec:
      nodeSelector: {node-role: worker}
      automountServiceAccountToken: false
      initContainers:
        - name: lower
          image: nixos/nix:2.35.2                 # THE pin: assembler == every consumer<sup>[3](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/.woodpecker/gates.yaml?display=source#L102)</sup>
          command: [sh, -c, "mkdir -p /d/lower && cp -an /nix/. /d/lower/"]
          volumeMounts: [{name: nix-data, mountPath: /d}]
      containers:
        - name: assemble
          image: busybox:1.36.1                   # ships `mount`; the nix image does not (A6)<sup>[7](https://github.com/NixOS/nix/blob/97990235454f0aa19b793c74c8f7fd8b7da3001b/docker.nix#L51)[7b](https://busybox.net/downloads/BusyBox.html)</sup>
          securityContext: {privileged: true}     # required for Bidirectional<sup>[8](https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation)</sup>
          command: [sh, -c, "mkdir -p /d/upper /d/work /d/shared; umount /d/shared 2>/dev/null || true; mount -t overlay overlay -o lowerdir=/d/lower,upperdir=/d/upper,workdir=/d/work /d/shared; exec sleep infinity"]
          volumeMounts:
            - {name: nix-data, mountPath: /d, mountPropagation: Bidirectional}
      volumes:
        - {name: nix-data, hostPath: {path: /var/mnt/data/nix, type: Directory}}

  4. Consumer contract — same mount path, overlay store URI.

    # every step that already has nix-store:/nix — the mount itself is unchanged<sup>[1](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/gitops/clusters/milner/infrastructure/local-path/values.yaml?display=source#L115)</sup>
volumes: [nix-store:/nix]
environment:
  NIX_CONFIG: |
    extra-experimental-features = nix-command flakes local-overlay-store
    store = local-overlay://?lower-store=/var/mnt/data/nix/lower&upper-layer=/var/mnt/data/nix/upper&state=/var/mnt/data/nix/state
    check-mount = false

    The state DB path a baseline pod can reach is spike (b); candidates are a path inside the merged mount (/nix/var/nix) or a second mount of the durable directory5.

  5. State and reject the per-pod SYS_ADMIN alternative. It is the only way to overlay a consumer's own image store directly, but annia is baseline (no SYS_ADMIN, no hostPath — A4)2 and Woodpecker's backend cannot add capabilities; one node-level mechanism for CI and #25 beats two.

  6. GC belongs to the assembler, never a consumer.

    # gitops/clusters/milner/infrastructure/nix-assembler/gc-cronjob.yaml — daily
command: [sh, -c, "nix --extra-experimental-features 'nix-command local-overlay-store' store gc"]
# same NIX_CONFIG `store=` URI as the consumers; image nixos/nix:2.35.2

    A lower that changes under a live upper corrupts it66b; a test asserts no step command contains store gc or collect-garbage.

  7. The pin rule survives, re-pointed at the assembler's manifest.

    # tests/test_static.py — read the tag, do not retype it
pin = re.search(r"image: (nixos/nix:[^\s]+)", assembler_yaml).group(1)
assert {step["image"] for step in mounting_steps} == {pin}

  8. Delete §10g and sweep every inbound reference.

    grep -rn 'nix-store:/nix\|§10g\|### 10g\|mounted at /nix\|one image, six consumers' \
  .woodpecker docs gitops storage tests AGENTS.md OPERATIONS.md USER_STORIES.md ARCHITECTURE.md data drills

    The Job, its awk extraction, the "seed before mounting" prose and the closing "one writer / avoid GC" note go; §10e's /var/mnt/data/nix/ backup exclusion is re-justified as "the upper is derived state"99b.

  9. Re-point the static tests.

    # tests/test_static.py
# - test_the_suite_runs_...: volumes == ["nix-store:/nix"] stays; §10g regex -> assembler-pin read
# - test_nothing_at_runtime_mounts_the_shared_store: whole-tree sweep for /nix mounts
# - kubernetes-client guard (~4718): drop the duplicate "one image" equality, keep pin vs assembler
# - new: every mounting step carries the local-overlay NIX_CONFIG; no step runs GC

  10. Amend stories 19/20 in this change (Hard Rule).

    19. A build starts from what earlier builds realised: I mount the shared Nix store at /nix; the image is its lower and the durable directory its upper, so paths earlier builds wrote are there without a seed.
    20. An app's image is the whole of what it runs, except its Nix store, which is an overlay over that image1010b10c.

  11. milner#25 — annia-nix is a class-nix claim mounted at /nix with the same NIX_CONFIG; a PVC is baseline-legal, so nothing about annia's PSA changes.

Implementation Details

Area File Change
Assembler gitops/…/infrastructure/nix-assembler/{daemonset,gc-cronjob,kustomization}.yaml new; lower copy + mount -t overlay + daily GC
Namespace gitops/…/infrastructure/namespaces/nix-assembler.yaml, …/infrastructure/kustomization.yaml new privileged namespace, listed
Steps .woodpecker/gates.yaml, .woodpecker/cleanup.yaml add the local-overlay NIX_CONFIG; mount path unchanged1
Policy policy/k8s.rego namespace-scoped carve-out for nix-assembler (rule 3)10
Docs docs/storage.md §10a/§10e/§10g, docs/nix.md §14b/§14c/§14d, docs/woodpecker-ci.md, storage/README.md, OPERATIONS.md, AGENTS.md, ARCHITECTURE.md, data/README.md, docs/durable-volumes.md, docs/logging.md store-at-/nix seeded -> overlay upper; bring-up loses the seed
Stories USER_STORIES.md 19 reworded, 20 amended
Tests tests/test_static.py, tests/test_docs.py §10g guards re-pointed; new overlay/GC/pin guards

Constraints:

  • The assembler is privileged, so policy/k8s.rego needs a carve-out and the "privileged tier is empty" claims10 in .woodpecker/gates.yaml/docs must read "empty except nix-assembler". This is Shape A's cost; it is stated, not hidden.
  • local-overlay-store is experimental and its RFC 0152 is still a draft (A1/A2)56 — name the stability risk in docs/nix.md §14d.

Out-of-scope notes:

  • The withdrawn cache plan (#10220, PR #114) is the fallback if the spike falsifies Shape A; it is not deleted here.
  • Store contents and flake pins are untouched; #25's claim is derived but added by #25.

Acceptance Criteria

  • The four spike measurements are recorded in docs/nix.md §14b; the propagation path (class-nix claim vs a new mechanism) is stated, not assumed — or the plan is stopped.
  • nix develop --command python3 -m pytest tests/ -v passes in the test step on a fresh upper — no seed job, no hand-run step.
  • A step pod on an empty upper execs its entrypoint and finds its own nix, bash and curl under /nix.
  • No step command runs GC; the assembler's CronJob is the only collector, and a test holds both.
  • Assembler tag == every mounting step's tag, read out of the manifest in one change; a bump strands no consumer.
  • §10g is gone; §10a/§10e read "the upper is derived state"; no document says the volume is seeded; stories 19/20 read as amended.
  • The policy/ carve-out is namespace-scoped to nix-assembler, and the docs say "the privileged tier is empty except the assembler".

Testing Plan

  • tests/test_static.py: assembler-pin equality; whole-tree "nothing mounts over /nix" sweep; every mounting step carries the overlay NIX_CONFIG; no step runs GC; the replaced §10g guards.
  • tests/test_docs.py: the story 19/20 guard and the §14b/§14d wording.
  • conftest test --policy policy/ stays clean with the namespace-scoped carve-out only.
  • On the node [measure]: spike (a)–(d); a cold upper's first step; two overlapping steps; assembler restart re-establishes the mount without touching the upper; a tag bump with a consumer already running.
  • One real main pipeline run after merge.

Related Issues/PRs

  • Supersedes the technical approach of comments #10176/#10220 (binary cache at /shared-nix); the unmerged PR #114 implemented that withdrawn shape.
  • milner#25 — its annia-nix claim is re-derived here at /nix under the overlay.
  • annia-infra#310 — holds acceptance runs (#10155), not planning or merge.
  • Precedent: #107/#110 — one secret-extension rule per consumer.

Research Findings

  • local-overlay is spelled store = local-overlay://?lower-store=…&upper-layer=…&state=… and needs extra-experimental-features = local-overlay-store; the upper DB is always ${state}/db (A1).
  • Its concurrency guarantee for one shared upper is not documented; RFC 0152 assumes a per-consumer upper, so the two-writer case must be measured (A2).
  • GC on a local-overlay store deletes only the upper copy, but an open bug (#16269) makes it free nothing under --max-freed; the schema is forward-only, so the writing and reading nix must be one version (A3).
  • Kubernetes Bidirectional propagation is privileged-only and recommended only with hostPath; Baseline PSS forbids both hostPath and SYS_ADMIN, so annia cannot use either directly (A4).
  • nixos/nix:2.35.2 ships no mount binary (A6), and a host-level assembler cannot use another pod's /nix as its lower (A7) — hence the materialised lower.

Pre-Mortem

  1. The gating propagation spike falsifies Shape A, and the change has no fallback — the overlay assembled on the worker never shows up at /nix inside a Woodpecker step pod, because the backend cannot set mountPropagation (B6) and propagation is not recommended on PVC-backed volumes (A4), which is the plan's own "(a) is false" stop condition. The whole change is then spent on an unreachable layered store.
  2. The consumer local-overlay:// contract is unworkable from a baseline pod — check-mount = false disables Nix's only check that the mount matches the configured lower/upper, the reachable state DB path is left as spike (b) with no default, and one shared upper has no documented concurrency guarantee (RFC 0152 assumes a per-consumer upper). Two overlapping steps corrupt the upper's SQLite DB or links, which is precisely the case the plan admits it must measure.
  3. The manifests land but the repository's own verifiers reject them — the suite runs inside the pipeline this change rewrites, so an altered step YAML or an unmoved §10g/pin assertion fails as a test fault with no functional cause, and the required policy/k8s.rego carve-out plus a privileged nix-assembler namespace break the "privileged tier is empty" invariant. A missed one of the ~28 inbound references, or a silently renumbered §10g, leaves the story 19/20 and tests/test_docs.py guards red in the same merge.
Naina: *Updated on 2026-10-06 based on maintainer feedback.* ## Overview > **Which shape stands (read this first).** Issue #109's title and body specify the **layered overlay** (`/nix` = pinned image store as immutable lower + durable upper). Comments #10176/#10220 and the unmerged PR #114 specify a **binary cache at `/shared-nix`** and call it the operator's absolute decision; on 2026-10-05 the issue moved `status/reviewing` → `status/planning` while still titled *"Layered Nix store"* (`/issues/109/timeline`). This plan takes the title and body at their word and **reinstates the layered shape**; if the cache decision still stands, reject this and the withdrawn cache plan in #10176 holds. Replace the hand-seeded store-at-`/nix` with an overlay assembled once per node: the pinned image's `/nix` is the immutable lower, the durable directory the writable upper, merged by a privileged DaemonSet and consumed through the existing `nix`-class claim<sup>[1](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/gitops/clusters/milner/infrastructure/local-path/values.yaml?display=source#L115)[1b](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/.woodpecker/gates.yaml?display=source#L104)</sup>. §10g's seed Job dies, a fresh upper starts, and an image bump moves the lower without a hand-run top-up. ## Technical Approach 1. **Spike first — settle the four load-bearing facts on the worker (gating).** ```sh # throwaway privileged pod; results go into docs/nix.md §14b before any manifest lands mount -t overlay overlay -o lowerdir=/lower,upperdir=/upper,workdir=/work /var/mnt/data/nix/shared # then, in a pod that mounts nix-store:/nix with NO mountPropagation: grep ' /nix ' /proc/self/mounts # (a) is the overlay visible at pod start? nix --extra-experimental-features 'nix-command local-overlay-store' store ping # (b) does it read? ``` (c) two deliberately overlapping steps against one upper: does the SQLite DB stay consistent? (d) is `check-mount` off (the consumer sees the assembler's bind mount, not an overlay)? If **(a) is false, Shape A cannot serve Woodpecker** (its backend sets no `mountPropagation` — B6) — stop and report rather than guess. 2. **A privileged assembler namespace.** ```yaml # gitops/clusters/milner/infrastructure/namespaces/nix-assembler.yaml apiVersion: v1 kind: Namespace metadata: {name: nix-assembler, labels: {pod-security.kubernetes.io/enforce: privileged}} ``` 3. **The assembler DaemonSet — materialise the lower, mount the overlay once.** A host DaemonSet cannot see a consumer pod's mount namespace (A7)<sup>[4](https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation)</sup>, so the lower must be materialised on the durable disk; the assembler copies **its own** pinned image's `/nix`, and the pin rule makes it equal to the consumers'. ```yaml # gitops/clusters/milner/infrastructure/nix-assembler/daemonset.yaml spec: template: spec: nodeSelector: {node-role: worker} automountServiceAccountToken: false initContainers: - name: lower image: nixos/nix:2.35.2 # THE pin: assembler == every consumer<sup>[3](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/.woodpecker/gates.yaml?display=source#L102)</sup> command: [sh, -c, "mkdir -p /d/lower && cp -an /nix/. /d/lower/"] volumeMounts: [{name: nix-data, mountPath: /d}] containers: - name: assemble image: busybox:1.36.1 # ships `mount`; the nix image does not (A6)<sup>[7](https://github.com/NixOS/nix/blob/97990235454f0aa19b793c74c8f7fd8b7da3001b/docker.nix#L51)[7b](https://busybox.net/downloads/BusyBox.html)</sup> securityContext: {privileged: true} # required for Bidirectional<sup>[8](https://kubernetes.io/docs/concepts/storage/volumes/#mount-propagation)</sup> command: [sh, -c, "mkdir -p /d/upper /d/work /d/shared; umount /d/shared 2>/dev/null || true; mount -t overlay overlay -o lowerdir=/d/lower,upperdir=/d/upper,workdir=/d/work /d/shared; exec sleep infinity"] volumeMounts: - {name: nix-data, mountPath: /d, mountPropagation: Bidirectional} volumes: - {name: nix-data, hostPath: {path: /var/mnt/data/nix, type: Directory}} ``` 4. **Consumer contract — same mount path, overlay store URI.** ```yaml # every step that already has nix-store:/nix — the mount itself is unchanged<sup>[1](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/gitops/clusters/milner/infrastructure/local-path/values.yaml?display=source#L115)</sup> volumes: [nix-store:/nix] environment: NIX_CONFIG: | extra-experimental-features = nix-command flakes local-overlay-store store = local-overlay://?lower-store=/var/mnt/data/nix/lower&upper-layer=/var/mnt/data/nix/upper&state=/var/mnt/data/nix/state check-mount = false ``` The `state` DB path a `baseline` pod can reach is spike **(b)**; candidates are a path inside the merged mount (`/nix/var/nix`) or a second mount of the durable directory<sup>[5](https://nix.dev/manual/nix/2.31/store/types/experimental-local-overlay-store.html)</sup>. 5. **State and reject the per-pod `SYS_ADMIN` alternative.** It is the only way to overlay a consumer's *own* image store directly, but `annia` is `baseline` (no `SYS_ADMIN`, no `hostPath` — A4)<sup>[2](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/gitops/clusters/milner/infrastructure/namespaces/annia.yaml?display=source#L6)</sup> and Woodpecker's backend cannot add capabilities; one node-level mechanism for CI and #25 beats two. 6. **GC belongs to the assembler, never a consumer.** ```yaml # gitops/clusters/milner/infrastructure/nix-assembler/gc-cronjob.yaml — daily command: [sh, -c, "nix --extra-experimental-features 'nix-command local-overlay-store' store gc"] # same NIX_CONFIG `store=` URI as the consumers; image nixos/nix:2.35.2 ``` A lower that changes under a live upper corrupts it<sup>[6](https://github.com/NixOS/rfcs/pull/152)[6b](https://github.com/NixOS/nix/issues/16269)</sup>; a test asserts no step command contains `store gc` or `collect-garbage`. 7. **The pin rule survives, re-pointed at the assembler's manifest.** ```python # tests/test_static.py — read the tag, do not retype it pin = re.search(r"image: (nixos/nix:[^\s]+)", assembler_yaml).group(1) assert {step["image"] for step in mounting_steps} == {pin} ``` 8. **Delete §10g and sweep every inbound reference.** ```sh grep -rn 'nix-store:/nix\|§10g\|### 10g\|mounted at /nix\|one image, six consumers' \ .woodpecker docs gitops storage tests AGENTS.md OPERATIONS.md USER_STORIES.md ARCHITECTURE.md data drills ``` The Job, its `awk` extraction, the "seed before mounting" prose and the closing "one writer / avoid GC" note go; §10e's `/var/mnt/data/nix/` backup exclusion is re-justified as "the upper is derived state"<sup>[9](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/tests/test_static.py?display=source#L1170)[9b](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/docs/storage.md?display=source#L126)</sup>. 9. **Re-point the static tests.** ```python # tests/test_static.py # - test_the_suite_runs_...: volumes == ["nix-store:/nix"] stays; §10g regex -> assembler-pin read # - test_nothing_at_runtime_mounts_the_shared_store: whole-tree sweep for /nix mounts # - kubernetes-client guard (~4718): drop the duplicate "one image" equality, keep pin vs assembler # - new: every mounting step carries the local-overlay NIX_CONFIG; no step runs GC ``` 10. **Amend stories 19/20 in this change (Hard Rule).** > **19.** A build starts from what earlier builds realised: I mount the shared Nix store at `/nix`; the image is its lower and the durable directory its upper, so paths earlier builds wrote are there without a seed. > **20.** An app's image is the whole of what it runs, **except its Nix store, which is an overlay over that image**<sup>[10](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/policy/k8s.rego?display=source#L97)[10b](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/.woodpecker/gates.yaml?display=source#L439)[10c](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/USER_STORIES.md?display=source#L96)</sup>. 11. **milner#25** — `annia-nix` is a class-`nix` claim mounted at `/nix` with the same `NIX_CONFIG`; a PVC is baseline-legal, so nothing about `annia`'s PSA changes. ## Implementation Details | Area | File | Change | |---|---|---| | Assembler | `gitops/…/infrastructure/nix-assembler/{daemonset,gc-cronjob,kustomization}.yaml` | new; lower copy + `mount -t overlay` + daily GC | | Namespace | `gitops/…/infrastructure/namespaces/nix-assembler.yaml`, `…/infrastructure/kustomization.yaml` | new privileged namespace, listed | | Steps | `.woodpecker/gates.yaml`, `.woodpecker/cleanup.yaml` | add the local-overlay `NIX_CONFIG`; mount path unchanged<sup>[1](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/gitops/clusters/milner/infrastructure/local-path/values.yaml?display=source#L115)</sup> | | Policy | `policy/k8s.rego` | namespace-scoped carve-out for `nix-assembler` (rule 3)<sup>[10](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/policy/k8s.rego?display=source#L97)</sup> | | Docs | `docs/storage.md` §10a/§10e/§10g, `docs/nix.md` §14b/§14c/§14d, `docs/woodpecker-ci.md`, `storage/README.md`, `OPERATIONS.md`, `AGENTS.md`, `ARCHITECTURE.md`, `data/README.md`, `docs/durable-volumes.md`, `docs/logging.md` | store-at-`/nix` seeded -> overlay upper; bring-up loses the seed | | Stories | `USER_STORIES.md` | 19 reworded, 20 amended | | Tests | `tests/test_static.py`, `tests/test_docs.py` | §10g guards re-pointed; new overlay/GC/pin guards | Constraints: - **The assembler is privileged**, so `policy/k8s.rego` needs a carve-out and the "privileged tier is empty" claims<sup>[10](https://git.gloom.garden/tepichord/milner/src/commit/626127114d15b39443a4aaac5534d53f96c5f06f/.woodpecker/gates.yaml?display=source#L439)</sup> in `.woodpecker/gates.yaml`/docs must read "empty except `nix-assembler`". This is Shape A's cost; it is stated, not hidden. - `local-overlay-store` is experimental and its RFC 0152 is still a draft (A1/A2)<sup>[5](https://nix.dev/manual/nix/2.31/store/types/experimental-local-overlay-store.html)[6](https://github.com/NixOS/rfcs/pull/152)</sup> — name the stability risk in `docs/nix.md §14d`. Out-of-scope notes: - The withdrawn cache plan (#10220, PR #114) is the fallback if the spike falsifies Shape A; it is not deleted here. - Store contents and flake pins are untouched; #25's claim is derived but added by #25. ## Acceptance Criteria - [ ] The four spike measurements are recorded in `docs/nix.md §14b`; the propagation path (class-`nix` claim vs a new mechanism) is **stated, not assumed** — or the plan is stopped. - [ ] `nix develop --command python3 -m pytest tests/ -v` passes in the `test` step on a **fresh upper** — no seed job, no hand-run step. - [ ] A step pod on an empty upper execs its entrypoint and finds its own `nix`, `bash` and `curl` under `/nix`. - [ ] No step command runs GC; the assembler's CronJob is the only collector, and a test holds both. - [ ] Assembler tag == every mounting step's tag, read out of the manifest in one change; a bump strands no consumer. - [ ] §10g is gone; §10a/§10e read "the upper is derived state"; no document says the volume is seeded; stories 19/20 read as amended. - [ ] The `policy/` carve-out is namespace-scoped to `nix-assembler`, and the docs say "the privileged tier is empty except the assembler". ## Testing Plan - `tests/test_static.py`: assembler-pin equality; whole-tree "nothing mounts over `/nix`" sweep; every mounting step carries the overlay `NIX_CONFIG`; no step runs GC; the replaced §10g guards. - `tests/test_docs.py`: the story 19/20 guard and the §14b/§14d wording. - `conftest test --policy policy/` stays clean with the namespace-scoped carve-out only. - **On the node [measure]**: spike (a)–(d); a cold upper's first step; two overlapping steps; assembler restart re-establishes the mount without touching the upper; a tag bump with a consumer already running. - One real `main` pipeline run after merge. ## Related Issues/PRs - Supersedes the technical approach of comments #10176/#10220 (binary cache at `/shared-nix`); the unmerged PR #114 implemented that withdrawn shape. - milner#25 — its `annia-nix` claim is re-derived here at `/nix` under the overlay. - annia-infra#310 — holds acceptance runs (#10155), not planning or merge. - Precedent: #107/#110 — one secret-extension rule per consumer. ## Research Findings - `local-overlay` is spelled `store = local-overlay://?lower-store=…&upper-layer=…&state=…` and needs `extra-experimental-features = local-overlay-store`; the upper DB is always `${state}/db` (A1). - Its concurrency guarantee for one shared upper is **not documented**; RFC 0152 assumes a per-consumer upper, so the two-writer case must be measured (A2). - GC on a `local-overlay` store deletes only the upper copy, but an open bug (#16269) makes it free nothing under `--max-freed`; the schema is forward-only, so the writing and reading `nix` must be one version (A3). - Kubernetes `Bidirectional` propagation is privileged-only and recommended only with `hostPath`; Baseline PSS forbids both `hostPath` and `SYS_ADMIN`, so `annia` cannot use either directly (A4). - `nixos/nix:2.35.2` ships **no `mount` binary** (A6), and a host-level assembler cannot use another pod's `/nix` as its lower (A7) — hence the materialised lower. ## Pre-Mortem 1. **The gating propagation spike falsifies Shape A, and the change has no fallback** — the overlay assembled on the worker never shows up at `/nix` inside a Woodpecker step pod, because the backend cannot set `mountPropagation` (B6) and propagation is not recommended on PVC-backed volumes (A4), which is the plan's own "(a) is false" stop condition. The whole change is then spent on an unreachable layered store. 2. **The consumer `local-overlay://` contract is unworkable from a `baseline` pod** — `check-mount = false` disables Nix's only check that the mount matches the configured lower/upper, the reachable `state` DB path is left as spike (b) with no default, and one shared upper has no documented concurrency guarantee (RFC 0152 assumes a per-consumer upper). Two overlapping steps corrupt the upper's SQLite DB or `links`, which is precisely the case the plan admits it must measure. 3. **The manifests land but the repository's own verifiers reject them** — the suite runs inside the pipeline this change rewrites, so an altered step YAML or an unmoved `§10g`/pin assertion fails as a test fault with no functional cause, and the required `policy/k8s.rego` carve-out plus a privileged `nix-assembler` namespace break the "privileged tier is empty" invariant. A missed one of the ~28 inbound references, or a silently renumbered §10g, leaves the story 19/20 and `tests/test_docs.py` guards red in the same merge.
tepichord commented 2026-09-30 07:52:57 -07:00
Author
Owner
Copy link

Naina:

Plan revision — the shared store becomes a cache at /shared-nix

This replaces the technical approach in comment #10176. The decision is the
operator's, and it is absolute: the base image's /nix is never mounted over.
Everything below restates the plan in that shape; anything in #10176 not named as
superseded still holds.

Superseded, explicitly

  • Technical approach 1–11 (assembler DaemonSet, overlay lower/upper/state,
    mountPropagation, the consumer local-overlay:// contract, the assembler pin
    rule, assembler-owned GC, the two-writer fallback, the Talos extraMounts
    patch) — withdrawn.
  • Implementation rows for nix-assembler/*, the policy/k8s.rego carve-out
    and worker/main.tf — withdrawn; nothing privileged is needed.
  • Acceptance criteria 1, 2, 4 and 7, and Pre-Mortem 1–3 — replaced
    below. All three old hazards dissolve with the overlay (no submount to
    propagate, no shared database, no store URI to spell wrong); two new ones take
    their place.
  • The Research Findings of #10176 stand as facts about
    local-overlay:// — they are simply no longer load-bearing.
  • One phrase is superseded everywhere: "the store". It is a cache, which is
    what docs/storage.md §10a has called the nix class all along.

Overview

/var/mnt/data/nix/shared stops being a store and becomes a Nix binary
cache. Every pod that wants it mounts the same claim at /shared-nix. A
pod's /nix is its image's /nix, untouched. Reads go through substituters;
writes go through one publish at the end of a step. No assembler, no overlay, no
privileged workload, no shared database, no mountPropagation, no experimental
store features.

Why the new root changes the mechanism (the one constraint)

Store paths are the absolute strings /nix/store/<hash>-<name>, and the store
directory is compiled into the nix binary. A path realised in a directory
mounted at /shared-nix is invisible to the processes that would run it, so a
Nix store can only be run from at /nix/store — mounting there was never a
preference. If /nix belongs to the image, the shared thing can only be read
from and written to: a cache.

Technical approach

  1. The cache. The host path /var/mnt/data/nix/shared, the nix class and
    the claim (woodpecker-runtime/nix-store) all keep their names; the contents
    change. It becomes a binary cache — nix-cache-info, narinfo/, nar/ —
    written by nix copy --to file:///shared-nix. The 6.4 GiB store sitting
    there now is deleted: it is a cache, it starts cold, and the first pipeline
    run warms it. (#10176's "migrating the seed is out of scope" becomes "the
    old directory is removed".)

  2. Reads. Consumers add one line to NIX_CONFIG:
    extra-substituters = file:///shared-nix. nix develop / nix build then
    take the paths the cache holds from local disk into the pod's own store
    instead of fetching them over the network or rebuilding them. Nothing else
    about a step changes: same image, same /etc/nix/nix.conf, same PATH.

  3. Writes. One script, ci/publish-nix-cache.sh, run at the end of every
    step that builds: nix copy --to "file:///shared-nix" --all. --all is the
    right extent on purpose: the pod's store at the end of a run is what the
    next pod wants — the dev shell and the tree's derivations (which the pod
    substituted, and which a post-build-hook would not see) as well as what it
    built. It is incremental — paths the cache already holds are skipped — so a
    warm run copies almost nothing.

  4. Signing, and who may publish. The cache is signed with a key generated
    once into Vault (secret/ci/nix-cache), so consumers keep
    require-sigs = true and the cache.nixos.org keys are untouched. That is the
    §9k trio: the value in Vault, the path in the secret extension's AppRole
    policy, and a third rule in
    apps/woodpecker/secret-extension.yaml's ConfigMap
    (milner-nix-cache-publish → ci/nix-cache,
    events: ["push", "manual"]). It also means a pull request substitutes
    but never publishes     — the boundary #110 draws for Zot, and what keeps
    arbitrary PR code out of the cache. Named fallback if the key plumbing
    proves fussy: require-sigs = false, recorded as an accepted widening
    rather than a silent one.

  5. Where the pod's writes land. Its own /nix/store is the image's store
    plus the container's writable layer, on the node's ephemeral filesystem,
    deleted with the pod. Measured on the worker today: 976 GB available of
    1.0 TB, 4% used     (df -h / in a pod: overlay … /), so per-step cost is
    affordable — and the plan measures one real step's peak before and after.

  6. GC and trimming. nix store gc does not exist for a binary cache —
    proved: nix store gc --store file:///tmp/cacheprobe → "Garbage collection
    not supported by store 'file:///…'"    . So the cache is trimmed by a runbook
    action: narinfo and its nar deleted together, oldest first, so no
    narinfo is ever left without its nar; a trim costs a re-fetch, never
    correctness. A pinned-root GC script is out of scope and named as the
    upgrade path. §10e's backup exclusion is re-justified in its own words:
    the class is a cache, and "reproducible from cache" is now literally what
    it is.

  7. §10g dies for a simpler reason. Nothing mounts a store at /nix, so
    there is nothing to seed. The seed Job, its awk extraction and the
    "one image in six places" invariant go with it, and so does the
    cross-consumer pin rule: a pod's image and its store are now the same
    thing. What remains is the ordinary "no :latest" policy.

  8. The experimental surface disappears. No local-overlay-store, no
    read-only-local-store, no store URI, no extra-experimental-features
    beyond the nix-command flakes the steps already ask for. That removes the
    largest early-adopter risk of #10176 (its Pre-Mortem 3) outright.

  9. Stories, in this change (Hard Rule: the story moves in the change that
    contradicts it). Story 19's promise stands, its mechanism changes; story 20
    is replaced by the rule this change implements.

    19. A build starts from what earlier builds realised. I mount the
    shared Nix cache at /shared-nix in whatever builds in this cluster — a
    pipeline step, any namespace — and the paths earlier builds produced are
    substituted from it, so repeated work is neither fetched nor built again.

    20. An image's own /nix is never mounted over. Whatever else a pod
    mounts, /nix stays the image's, so the tools that start its process come
    from the image and never from a directory another pod can write. The
    shared store lives at /shared-nix, where a build copies what it needs in
    and copies what it built out — a cache beside the pod, not the store it
    runs from.

    Cascade: docs/nix.md §14c (its table row, and the sentence
    "Nothing mounts it at runtime, and that is the design.", which now says
    the opposite rule and is re-marked rather than reintroduced), §14b/§14d,
    AGENTS.md's preview-console row, and
    tests/test_static.py's test_nothing_at_runtime_mounts_the_shared_store —
    which stops being a preview-namespace test and becomes a sweep of every
    pod spec under gitops/, plus the seven docstrings that cite story 20.

  10. milner#25. Its annia-nix claim stays a class-nix claim, mounted at
    /shared-nix, with the same NIX_CONFIG line; its plan's "mount at /nix"
    lines are re-derived against this shape (issue #25's revision §7 already
    calls itself provisional). A PVC is still baseline-legal and needs no
    propagation, so nothing about annia's PSA changes.

Implementation details

File Change
.woodpecker/gates.yaml, .woodpecker/cleanup.yaml mount nix-store at /shared-nix instead of /nix; add extra-substituters to NIX_CONFIG; every building step ends with ci/publish-nix-cache.sh
ci/publish-nix-cache.sh new — the one publish implementation (nix copy --to "file:///shared-nix" --all, key read from the environment, prints a one-line verdict)
gitops/…/apps/woodpecker/secret-extension.yaml third rule milner-nix-cache-publish → ci/nix-cache, events: ["push","manual"]
gitops/…/infrastructure/nix-store/persistentvolumeclaim.yaml unchanged except its comment: the claim is the cache handle
gitops/…/apps/annia/* drop what the stalled implementation added here: the nix-store-style claim, the /nix mount, NIX_CONFIG, HostToContainer. The enqueuer's deployment.yaml returns to untouched
gitops/…/infrastructure/nix-assembler/*, policy/k8s.rego, worker/main.tf withdrawn — delete what the branch added
USER_STORIES.md story 19 reworded, story 20 replaced (text above)
docs/storage.md §10a's nix row and §10e's exclusion reworded to "cache"; §10g deleted with its awk snippet; a trim action added to the runbook
docs/nix.md §14b/§14c/§14d, docs/woodpecker-ci.md §9m, OPERATIONS.md, AGENTS.md, storage/README.md the same move: the store mounted at /nix becomes the cache at /shared-nix; bring-up loses the seed step
tests/test_static.py, tests/test_docs.py §10g regex gone; "one image, six consumers" gone; new — no pod spec mounts anything at /nix; every building step calls ci/publish-nix-cache.sh; the step pods do not set readOnlyRootFilesystem (the design needs a writable store)

Acceptance criteria

  • No pod in gitops/ mounts anything over /nix; a static test holds it
    for the whole tree, not just preview namespaces (story 20).
  • nix develop --command python3 -m pytest tests/ -v passes in the test
    step with only the image's /nix and /shared-nix — no seed, no mount
    over /nix, no experimental store feature.
  • A cold pod substitutes the dev shell and the tree's derivations from
    /shared-nix, with no network fetch and no rebuild of them; the numbers
    (time, bytes) are recorded in the docs.
  • ci/publish-nix-cache.sh makes the next run warm — proved by re-running a
    step with the network substituter removed (--option substituters file:///shared-nix): it still succeeds.
  • A pull-request run cannot publish to the cache (the rule is push/
    manual only) and its steps still substitute from it.
  • Stories 19 and 20 read as amended; docs/nix.md §14c, AGENTS.md and the
    seven docstrings agree; no document still says a store is mounted at
    /nix or that the volume is seeded.
  • docs/storage.md §10g is gone, §10e is re-justified as "a cache", and the
    trim action is written down.
  • milner#25's annia-nix claim is a handle on the cache at /shared-nix,
    and nothing in #25's plan still mounts a store at /nix.
  • No privileged workload and no policy/ carve-out was needed; the
    pipeline's privileged tier is still empty.

Testing plan

  • tests/test_static.py: the new "nothing at /nix" sweep; "every building step
    publishes"; "steps keep a writable rootfs"; the replaced §10g guards; a
    ci/publish-nix-cache.sh presence + single-implementation assertion.
  • tests/test_docs.py: the story guard on the amended 19/20, and the §14c/§10e
    wording.
  • On the node [measure]: a cold pod's substitution from the cache (time,
    bytes, and that nothing is fetched from the network); a publish followed by a
    second pod that reads it back; a step's peak ephemeral-disk use; the trim
    action run once against a copy of the directory.
  • One real pipeline run on main after the merge — the first run is cold by
    definition, and its successor is the measurement that the cache warms.

Pre-mortem

  1. The cache never warms because the publish is forgotten or ineffective.
    The step succeeds, the story reads true, and every run quietly rebuilds from
    the internet. Guard: one ci/ implementation, a static test that every
    building step calls it, and the acceptance criterion that re-runs with the
    network substituter removed — a run that cannot substitute locally fails
    loudly.
  2. Substituting is slower than the mount it replaces. A cache hit is a local
    unpack into a fresh store per pod, not a zero-cost mount, and the pipeline
    could get slower while looking correct. Guard: measure one real step before
    and after and put both numbers in the docs; if substitution dominates, the
    answer is a Nix daemon on a node-level socket (named, not designed here) —
    not a quiet regression.
  3. The cache grows without bound. Every image version's store mirrors into
    it and nothing prunes unless the runbook is run. Guard: the trim action, a
    size check in the same runbook, and a du recorded after the first week.

Related

  • Supersedes the technical approach of comment #10176 (2026-09-29).
  • The #109 review's blocker — the enqueuer mounting the shared store — is
    resolved by construction: nothing mounts /nix, so
    apps/annia/deployment.yaml needs no mount, no NIX_CONFIG and no extra
    claim, and the deletion of annia-nix-pvc.yaml stands. That claim belongs to
    milner#25.
  • Reuses the pattern #107 and #110 established: a secret-extension rule per
    consumer, and publishing reachable only from push/manual.
  • milner#25 (annia-nix re-derived here), docs/storage.md §10a/§10e/§10g,
    docs/nix.md §14, docs/woodpecker-ci.md §9k/§9m, stories 19 and 20.

Research findings (verified today)

  • nix copy --to file:///tmp/cacheprobe <path> writes a real binary cache
    (.narinfo + nar/, 425 files for one derivation closure), and
    nix path-info --store file:///tmp/cacheprobe <path> reads it back.
  • nix store gc --store file:///… is refused for a binary cache — the trim
    action is a runbook, not a Nix command.
  • A pod's / is the node's containerd overlay with 976 GB of 1.0 TB free, so
    the per-pod copies land on space that exists.
  • docs/storage.md §10a already describes the class as a build cache, not
    backed up, "reproducible from cache" — the design is the repo's own wording
    applied literally.
  • The exact extra-substituters/trusted-public-keys spelling and the
    require-sigs interaction are standard Nix but not yet exercised by a
    pipeline step here unverified; the implementer confirms both in
    the first run, which is also acceptance criterion 2's run.
Naina: ## Plan revision — the shared store becomes a cache at `/shared-nix` **This replaces the technical approach in comment #10176.** The decision is the operator's, and it is absolute: *the base image's `/nix` is never mounted over*. Everything below restates the plan in that shape; anything in #10176 not named as superseded still holds. ### Superseded, explicitly - **Technical approach 1–11** (assembler DaemonSet, overlay lower/upper/state, `mountPropagation`, the consumer `local-overlay://` contract, the assembler pin rule, assembler-owned GC, the two-writer fallback, the Talos `extraMounts` patch) — **withdrawn**. - **Implementation rows** for `nix-assembler/*`, the `policy/k8s.rego` carve-out and `worker/main.tf` — **withdrawn**; nothing privileged is needed. - **Acceptance criteria 1, 2, 4 and 7**, and **Pre-Mortem 1–3** — replaced below. All three old hazards dissolve with the overlay (no submount to propagate, no shared database, no store URI to spell wrong); two new ones take their place. - The **Research Findings** of #10176 stand as facts about `local-overlay://` — they are simply no longer load-bearing. - One phrase is superseded everywhere: "the store". It is a **cache**, which is what `docs/storage.md` §10a has called the `nix` class all along. ### Overview `/var/mnt/data/nix/shared` stops being a store and becomes a **Nix binary cache**. Every pod that wants it mounts the same claim at **`/shared-nix`**. A pod's `/nix` is its image's `/nix`, untouched. Reads go through `substituters`; writes go through one publish at the end of a step. No assembler, no overlay, no privileged workload, no shared database, no `mountPropagation`, no experimental store features. ### Why the new root changes the mechanism (the one constraint) Store paths are the absolute strings `/nix/store/<hash>-<name>`, and the store directory is compiled into the nix binary. A path realised in a directory mounted at `/shared-nix` is invisible to the processes that would run it, so a Nix store can only be *run from* at `/nix/store` — mounting there was never a preference. If `/nix` belongs to the image, the shared thing can only be read from and written to: a cache. ### Technical approach 1. **The cache.** The host path `/var/mnt/data/nix/shared`, the `nix` class and the claim (`woodpecker-runtime/nix-store`) all keep their names; the contents change. It becomes a binary cache — `nix-cache-info`, `narinfo/`, `nar/` — written by `nix copy --to file:///shared-nix`. The 6.4 GiB store sitting there now is deleted: it is a cache, it starts cold, and the first pipeline run warms it. (#10176's "migrating the seed is out of scope" becomes "the old directory is removed".) 2. **Reads.** Consumers add one line to `NIX_CONFIG`: `extra-substituters = file:///shared-nix`. `nix develop` / `nix build` then take the paths the cache holds from local disk into the pod's own store instead of fetching them over the network or rebuilding them. Nothing else about a step changes: same image, same `/etc/nix/nix.conf`, same PATH. 3. **Writes.** One script, `ci/publish-nix-cache.sh`, run at the end of every step that builds: `nix copy --to "file:///shared-nix" --all`. `--all` is the right extent on purpose: the pod's store at the end of a run *is* what the next pod wants — the dev shell and the tree's derivations (which the pod substituted, and which a `post-build-hook` would not see) as well as what it built. It is incremental — paths the cache already holds are skipped — so a warm run copies almost nothing. 4. **Signing, and who may publish.** The cache is signed with a key generated once into Vault (`secret/ci/nix-cache`), so consumers keep `require-sigs = true` and the cache.nixos.org keys are untouched. That is the §9k trio: the value in Vault, the path in the secret extension's AppRole policy, and a third rule in `apps/woodpecker/secret-extension.yaml`'s ConfigMap (`milner-nix-cache-publish` → `ci/nix-cache`, `events: ["push", "manual"]`). It also means **a pull request substitutes but never publishes** — the boundary #110 draws for Zot, and what keeps arbitrary PR code out of the cache. Named fallback if the key plumbing proves fussy: `require-sigs = false`, recorded as an accepted widening rather than a silent one. 5. **Where the pod's writes land.** Its own `/nix/store` is the image's store plus the container's writable layer, on the node's ephemeral filesystem, deleted with the pod. Measured on the worker today: **976 GB available of 1.0 TB, 4% used** (`df -h /` in a pod: `overlay … /`), so per-step cost is affordable — and the plan measures one real step's peak before and after. 6. **GC and trimming.** `nix store gc` does not exist for a binary cache — proved: `nix store gc --store file:///tmp/cacheprobe` → *"Garbage collection not supported by store 'file:///…'"*. So the cache is trimmed by a runbook action: narinfo and its nar deleted **together**, oldest first, so no narinfo is ever left without its nar; a trim costs a re-fetch, never correctness. A pinned-root GC script is out of scope and named as the upgrade path. §10e's backup exclusion is re-justified in its own words: the class is a cache, and "reproducible from cache" is now literally what it is. 7. **§10g dies for a simpler reason.** Nothing mounts a store at `/nix`, so there is nothing to seed. The seed Job, its `awk` extraction and the "one image in six places" invariant go with it, and so does the cross-consumer pin rule: a pod's image and its store are now the same thing. What remains is the ordinary "no `:latest`" policy. 8. **The experimental surface disappears.** No `local-overlay-store`, no `read-only-local-store`, no store URI, no `extra-experimental-features` beyond the `nix-command flakes` the steps already ask for. That removes the largest early-adopter risk of #10176 (its Pre-Mortem 3) outright. 9. **Stories, in this change** (Hard Rule: the story moves in the change that contradicts it). Story 19's promise stands, its mechanism changes; story 20 is replaced by the rule this change implements. > **19. A build starts from what earlier builds realised.** I mount the > shared Nix cache at `/shared-nix` in whatever builds in this cluster — a > pipeline step, any namespace — and the paths earlier builds produced are > substituted from it, so repeated work is neither fetched nor built again. > **20. An image's own `/nix` is never mounted over.** Whatever else a pod > mounts, `/nix` stays the image's, so the tools that start its process come > from the image and never from a directory another pod can write. The > shared store lives at `/shared-nix`, where a build copies what it needs in > and copies what it built out — a cache beside the pod, not the store it > runs from. Cascade: `docs/nix.md` §14c (its table row, and the sentence "**Nothing mounts it at runtime, and that is the design.**", which now says the opposite rule and is re-marked rather than reintroduced), §14b/§14d, `AGENTS.md`'s preview-console row, and `tests/test_static.py`'s `test_nothing_at_runtime_mounts_the_shared_store` — which stops being a preview-namespace test and becomes a sweep of every pod spec under `gitops/`, plus the seven docstrings that cite story 20. 10. **milner#25.** Its `annia-nix` claim stays a class-`nix` claim, mounted at `/shared-nix`, with the same `NIX_CONFIG` line; its plan's "mount at `/nix`" lines are re-derived against this shape (issue #25's revision §7 already calls itself provisional). A PVC is still baseline-legal and needs no propagation, so nothing about annia's PSA changes. ### Implementation details | File | Change | |---|---| | `.woodpecker/gates.yaml`, `.woodpecker/cleanup.yaml` | mount `nix-store` at `/shared-nix` instead of `/nix`; add `extra-substituters` to `NIX_CONFIG`; every building step ends with `ci/publish-nix-cache.sh` | | `ci/publish-nix-cache.sh` | new — the one publish implementation (`nix copy --to "file:///shared-nix" --all`, key read from the environment, prints a one-line verdict) | | `gitops/…/apps/woodpecker/secret-extension.yaml` | third rule `milner-nix-cache-publish` → `ci/nix-cache`, `events: ["push","manual"]` | | `gitops/…/infrastructure/nix-store/persistentvolumeclaim.yaml` | unchanged except its comment: the claim is the cache handle | | `gitops/…/apps/annia/*` | drop what the stalled implementation added here: the `nix-store`-style claim, the `/nix` mount, `NIX_CONFIG`, `HostToContainer`. The enqueuer's `deployment.yaml` returns to untouched | | `gitops/…/infrastructure/nix-assembler/*`, `policy/k8s.rego`, `worker/main.tf` | withdrawn — delete what the branch added | | `USER_STORIES.md` | story 19 reworded, story 20 replaced (text above) | | `docs/storage.md` | §10a's `nix` row and §10e's exclusion reworded to "cache"; §10g deleted with its `awk` snippet; a trim action added to the runbook | | `docs/nix.md` §14b/§14c/§14d, `docs/woodpecker-ci.md` §9m, `OPERATIONS.md`, `AGENTS.md`, `storage/README.md` | the same move: the store mounted at `/nix` becomes the cache at `/shared-nix`; bring-up loses the seed step | | `tests/test_static.py`, `tests/test_docs.py` | §10g regex gone; "one image, six consumers" gone; new — no pod spec mounts anything at `/nix`; every building step calls `ci/publish-nix-cache.sh`; the step pods do not set `readOnlyRootFilesystem` (the design needs a writable store) | ### Acceptance criteria - [ ] No pod in `gitops/` mounts anything over `/nix`; a static test holds it for the whole tree, not just preview namespaces (story 20). - [ ] `nix develop --command python3 -m pytest tests/ -v` passes in the `test` step with only the image's `/nix` and `/shared-nix` — no seed, no mount over `/nix`, no experimental store feature. - [ ] A cold pod substitutes the dev shell and the tree's derivations from `/shared-nix`, with no network fetch and no rebuild of them; the numbers (time, bytes) are recorded in the docs. - [ ] `ci/publish-nix-cache.sh` makes the next run warm — proved by re-running a step with the network substituter removed (`--option substituters file:///shared-nix`): it still succeeds. - [ ] A pull-request run cannot publish to the cache (the rule is `push`/ `manual` only) and its steps still substitute from it. - [ ] Stories 19 and 20 read as amended; `docs/nix.md` §14c, `AGENTS.md` and the seven docstrings agree; no document still says a store is mounted at `/nix` or that the volume is seeded. - [ ] `docs/storage.md` §10g is gone, §10e is re-justified as "a cache", and the trim action is written down. - [ ] milner#25's `annia-nix` claim is a handle on the cache at `/shared-nix`, and nothing in #25's plan still mounts a store at `/nix`. - [ ] No privileged workload and no `policy/` carve-out was needed; the pipeline's privileged tier is still empty. ### Testing plan - `tests/test_static.py`: the new "nothing at `/nix`" sweep; "every building step publishes"; "steps keep a writable rootfs"; the replaced §10g guards; a `ci/publish-nix-cache.sh` presence + single-implementation assertion. - `tests/test_docs.py`: the story guard on the amended 19/20, and the §14c/§10e wording. - **On the node [measure]**: a cold pod's substitution from the cache (time, bytes, and that nothing is fetched from the network); a publish followed by a second pod that reads it back; a step's peak ephemeral-disk use; the trim action run once against a copy of the directory. - One real pipeline run on `main` after the merge — the first run is cold by definition, and its successor is the measurement that the cache warms. ### Pre-mortem 1. **The cache never warms because the publish is forgotten or ineffective.** The step succeeds, the story reads true, and every run quietly rebuilds from the internet. Guard: one `ci/` implementation, a static test that every building step calls it, and the acceptance criterion that re-runs with the network substituter removed — a run that cannot substitute locally fails loudly. 2. **Substituting is slower than the mount it replaces.** A cache hit is a local unpack into a fresh store per pod, not a zero-cost mount, and the pipeline could get slower while looking correct. Guard: measure one real step before and after and put both numbers in the docs; if substitution dominates, the answer is a Nix daemon on a node-level socket (named, not designed here) — not a quiet regression. 3. **The cache grows without bound.** Every image version's store mirrors into it and nothing prunes unless the runbook is run. Guard: the trim action, a size check in the same runbook, and a `du` recorded after the first week. ### Related - Supersedes the technical approach of comment #10176 (2026-09-29). - The #109 review's blocker — the enqueuer mounting the shared store — is **resolved by construction**: nothing mounts `/nix`, so `apps/annia/deployment.yaml` needs no mount, no `NIX_CONFIG` and no extra claim, and the deletion of `annia-nix-pvc.yaml` stands. That claim belongs to milner#25. - Reuses the pattern #107 and #110 established: a secret-extension rule per consumer, and publishing reachable only from `push`/`manual`. - milner#25 (`annia-nix` re-derived here), `docs/storage.md` §10a/§10e/§10g, `docs/nix.md` §14, `docs/woodpecker-ci.md` §9k/§9m, stories 19 and 20. ### Research findings (verified today) - `nix copy --to file:///tmp/cacheprobe <path>` writes a real binary cache (`.narinfo` + `nar/`, 425 files for one derivation closure), and `nix path-info --store file:///tmp/cacheprobe <path>` reads it back. - `nix store gc --store file:///…` is refused for a binary cache — the trim action is a runbook, not a Nix command. - A pod's `/` is the node's containerd overlay with 976 GB of 1.0 TB free, so the per-pod copies land on space that exists. - `docs/storage.md` §10a already describes the class as a build cache, not backed up, "reproducible from cache" — the design is the repo's own wording applied literally. - The exact `extra-substituters`/`trusted-public-keys` spelling and the `require-sigs` interaction are standard Nix but not yet exercised by a pipeline step here _<sup>unverified</sup>_; the implementer confirms both in the first run, which is also acceptance criterion 2's run.
tepichord commented 2026-10-22 17:13:11 -07:00
Author
Owner
Copy link

Superseded by milner/nix-base

This issue's goal — stop hand-filling the store volume, stop mounting over /nix,
and stop every step paying for the toolchain — is now met by an image rather than by
a shared volume, so the shape proposed here is not the one being built.

What the measurements settled (a full pull request on this branch, pipeline #186,
killed at the 60-minute cap):

  • A shared store mounted at /nix is the only zero-copy shape, and it costs
    verification (any PR pod can write a path every later step runs), cross-pod GC
    safety, and the failure this issue documents — an empty volume does not fail as
    nix: not found, the pod cannot exec its entrypoint at all.
  • A shared cache mounted at /shared-nix verified by a signature is safe, but a
    fresh pod copies its closure in: 573 paths, 4.6 GB, 435 s per step while the
    cache is empty — measured in l0-render's log, 573 copying path … from 'https://cache.nixos.org' lines. Not a number to build on.
  • The closure a step was substituting is the operator's shell: qemu 2.2 GB,
    SDL2 ~1.1 GB each, zenity ~1 GB, libvirt, minikube, tailscale. The tools the steps
    actually invoke are 147 paths / 902 MB.

So: milner/nix-base — one image this repository builds and publishes, carrying the
pinned nix, the shell a step's commands get, jq/curl/git/cacert, and
install-ci-env, a program that installs a repository's own dev shell into the
profile. A consumer's Dockerfile is FROM …:tag, COPY . /src/,
RUN install-ci-env /src; a consumer's CI step can skip the Dockerfile and run
install-ci-env .. /nix is never mounted over, no step names a substituter, and
there is no signing key.

Closing this in favour of that change once it lands; the reasons recorded here (why a
volume over /nix fails, why /nix belongs to the image) are being kept in
docs/nix.md.

## Superseded by `milner/nix-base` This issue's goal — stop hand-filling the store volume, stop mounting over `/nix`, and stop every step paying for the toolchain — is now met by an image rather than by a shared volume, so the shape proposed here is not the one being built. What the measurements settled (a full pull request on this branch, pipeline #186, killed at the 60-minute cap): - A shared *store* mounted at `/nix` is the only zero-copy shape, and it costs verification (any PR pod can write a path every later step runs), cross-pod GC safety, and the failure this issue documents — an empty volume does not fail as `nix: not found`, the pod cannot exec its entrypoint at all. - A shared *cache* mounted at `/shared-nix` verified by a signature is safe, but a fresh pod copies its closure in: **573 paths, 4.6 GB, 435 s** per step while the cache is empty — measured in `l0-render`'s log, 573 `copying path … from 'https://cache.nixos.org'` lines. Not a number to build on. - The closure a step was substituting is the *operator's* shell: qemu 2.2 GB, SDL2 ~1.1 GB each, zenity ~1 GB, libvirt, minikube, tailscale. The tools the steps actually invoke are **147 paths / 902 MB**. So: `milner/nix-base` — one image this repository builds and publishes, carrying the pinned nix, the shell a step's `commands` get, jq/curl/git/cacert, and `install-ci-env`, a program that installs a repository's own dev shell into the profile. A consumer's Dockerfile is `FROM …:tag`, `COPY . /src/`, `RUN install-ci-env /src`; a consumer's CI step can skip the Dockerfile and run `install-ci-env .`. `/nix` is never mounted over, no step names a substituter, and there is no signing key. Closing this in favour of that change once it lands; the reasons recorded here (why a volume over `/nix` fails, why `/nix` belongs to the image) are being kept in `docs/nix.md`.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/109-nix-base
feat/109-layered-nix-store
feat/110-zot-push-credential
feat/90-annia-pause-lever-oz-console
feat/87-tailscale-egress-from-cluster
chore/ci-step-resources
docs/vault-secret-prompt
feat/42-support-backups-persistent-data
feat/41-integrate-restic-tfstate-backup
feat/3-setting-up-agent-development
kvm-kubernetes-setup
No results found.
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner#109
No description provided.