Add a nix StorageClass whose claims all resolve to one directory on the durable data disk (/var/mnt/data/nix/shared), so any pod in any namespace can mount a pre-populated Nix store at /nix by declaring a PVC of class nix. local-path-provisioner normally gives every claim a private directory, so the class uses a constant pathPattern plus the allowUnsafePathPattern annotation to alias them together.¹² Retain is mandatory — Delete would rm -rf the shared store — and populating it is a documented one-off runbook, not Terraform state.³

Technical Approach

Add the nix class to storage/values.yaml — constant path, Retain, non-default:⁴⁵

  nix:
    storageClass:
      create: true
      defaultClass: false
      defaultVolumeType: hostPath
      reclaimPolicy: Retain
      volumeBindingMode: WaitForFirstConsumer
      pathPattern: "shared"            # constant -> every claim = /var/mnt/data/nix/shared
      annotations:
        allowUnsafePathPattern: "true" # chart exposes only params; provisioner reads this annotation
    nodePathMap:
      - node: DEFAULT_PATH_FOR_NON_LISTED_NODES
        paths:
          - /var/mnt/data/nix

Consume from any namespace — one PVC per namespace, same backing directory:⁶

apiVersion: v1
kind: PersistentVolumeClaim
metadata: {name: nix-store, namespace: woodpecker-runtime}
spec:
  storageClassName: nix
  accessModes: [ReadWriteOnce]
  resources: {requests: {storage: 30Gi}}
---
volumeMounts: [{name: nix, mountPath: /nix, readOnly: true}]
volumes:      [{name: nix, persistentVolumeClaim: {claimName: nix-store}}]

Populate it once (OPERATIONS.md §8g) — seed the Nix package into the volume, then install the tool closure; profile install leaves GC roots so the paths survive collection:⁷

initContainers:
  - name: seed
    image: nixos/nix:latest
    command: ["sh","-c","[ -e /mnt/store ] || cp -a /nix/. /mnt/"]
    volumeMounts: [{name: nix, mountPath: /mnt}]
containers:
  - name: populate
    image: nixos/nix:latest
    command: ["sh","-c","nix --extra-experimental-features 'nix-command flakes' profile install nixpkgs#terraform nixpkgs#kubectl nixpkgs#kubernetes-helm nixpkgs#talosctl nixpkgs#openssl nixpkgs#curl"]
    volumeMounts: [{name: nix, mountPath: /nix}]

Assert the contract in storage/tests/plan.tftest.hcl:⁸⁹

run "nix_class_is_one_shared_directory" {
  command = plan
  assert { condition = yamldecode(helm_release.local_path_provisioner.values[0]).storageClassConfigs.nix.storageClass.pathPattern == "shared" }
  assert { condition = yamldecode(helm_release.local_path_provisioner.values[0]).storageClassConfigs.nix.storageClass.annotations["allowUnsafePathPattern"] == "true" }
  assert { condition = yamldecode(helm_release.local_path_provisioner.values[0]).storageClassConfigs.nix.storageClass.reclaimPolicy == "Retain" }
  assert { condition = yamldecode(helm_release.local_path_provisioner.values[0]).storageClassConfigs.nix.nodePathMap[0].paths[0] == "/var/mnt/data/nix" }
}

Document — storage/README.md class table, ARCHITECTURE.md storage-tier note, OPERATIONS.md §8a table + §8e backup exclusion + new §8g runbook.¹⁰¹¹¹²

Implementation Details

Modify: storage/values.yaml, storage/tests/plan.tftest.hcl, storage/README.md, ARCHITECTURE.md, OPERATIONS.md.¹³¹⁴¹⁵
No new Terraform resources, no new qcow2, no worker//data/ changes — the existing 100 GiB data disk backs this.¹⁶¹⁷
Constraints: StorageClass parameters/reclaimPolicy are immutable → add a new class, never edit persistent/ephemeral; Retain is load-bearing (Delete calls rm -rf $VOL_DIR); capacity requests are ignored, so the store shares the data disk's capacity pool with persistent PVCs.¹⁸¹⁹²⁰
Out-of-scope notes: dedicated qcow2 (would need both UserVolumeConfig size bands reworked), nix-snapshotter/nix-csi/containix (runtime-layer, need patched containerd), a Terraform-managed population Job, and per-namespace store isolation.²¹²²

Acceptance Criteria

terraform -chdir=storage test passes with run "nix_class_is_one_shared_directory" asserting the constant pathPattern, the allowUnsafePathPattern annotation, Retain, and /var/mnt/data/nix.
python -m pytest tests/ -v passes (terraform fmt -check and the storage plan test).
storage/README.md, ARCHITECTURE.md, and OPERATIONS.md document the nix class, a cross-namespace consumption snippet, and the population runbook.
OPERATIONS backup section excludes /var/mnt/data/nix/ as reproducible from the cache, alongside /var/mnt/data/ephemeral/.

Testing Plan

terraform -chdir=storage test: new run "nix_class_is_one_shared_directory" block above.
python -m pytest tests/ -v: existing offline suite covering fmt + storage plan test.
Manual smoke (OPERATIONS §8g): create nix-store PVCs in two namespaces, write a marker file from a pod in one, read it from the other → same directory; ls /var/mnt/data/nix/shared/store.
Population smoke: run the §8g pod, then ls /nix/store | head from a consumer pod.

#26 — this issue
#42 — backups of persistent data (the Nix store must be excluded)
#46 — automate cluster deployments
#37 — introduced the storage root and the class-based tier model this extends

Research Findings

Per-claim dynamic provisioning never yields a shared directory: local-path-provisioner always joins the base path with a per-claim folderName, so sharing requires one directory aliased by every claim.²³
pathPattern normally must start with {{ .PVC.Namespace }}/{{ .PVC.Name }}/; setting allowUnsafePathPattern bypasses that check, and v0.0.37 reads it from a StorageClass annotation because the chart does not expose it as a parameter.²⁴²⁵
Nix store paths are content-addressed and immutable, so read-only sharing is safe, but the store's SQLite DB is not safe under concurrent writers — keep population to a single writer and avoid nix-collect-garbage on the shared store.²⁶
Talos volumeType: directory lives on the EPHEMERAL partition²⁷ and is wiped on re-image ^unverified, so the durable route is a directory on the data disk, which this plan uses.
A dedicated third qcow2 is not viable without reworking both UserVolumeConfig size bands (20 GiB and 100 GiB both match >= 40 GiB), so it is avoided.²⁸
nixos/nix publishes a linux/arm64 image, matching this aarch64-only cluster.²⁹

Pre-Mortem

The allowUnsafePathPattern annotation is never actually applied, so the constant pathPattern never aliases the claims together. Chart v0.0.37 exposes storageClassConfigs parameters but not arbitrary StorageClass annotations, so the class either loses the annotation (and provisioning rejects a pathPattern that does not start with {{ .PVC.Namespace }}/{{ .PVC.Name }}/) or silently provisions per-claim directories. Because plan.tftest.hcl asserts only the Helm values object and not live provisioning, this only surfaces on the cluster when two namespaces turn out to hold different data.
The store is shared and readable, but pods cannot actually run Nix against it. Mounting /nix readOnly: true leaves no writable /nix/var/nix/{profiles,gcroots,temproots} or unlocked store DB, so nix develop/nix build fail even though the ls /nix/store smoke test passes; the plan never assigns an owner for population versus normal use. The result is a mounted-but-unusable store, not the "any pod can utilize the Nix store" outcome the issue asks for.
The new plan assertion does not hold in the offline terraform test harness. yamldecode(helm_release.local_path_provisioner.values[0]) depends on the values argument being known at plan time and being a single-element list, which the mock-provider/offline suite used by pytest may not guarantee, so the acceptance criterion turns the suite red before the class can be merged. Any fallback of weakening the assertion would then remove the only automated guard on the Retain/constant-path contract.

Naina: ## Overview Add a `nix` StorageClass whose claims all resolve to one directory on the durable data disk (`/var/mnt/data/nix/shared`), so any pod in any namespace can mount a pre-populated Nix store at `/nix` by declaring a PVC of class `nix`. `local-path-provisioner` normally gives every claim a private directory, so the class uses a constant `pathPattern` plus the `allowUnsafePathPattern` annotation to alias them together.[1](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/README.md)[2](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/provisioner.go) `Retain` is mandatory — `Delete` would `rm -rf` the shared store — and populating it is a documented one-off runbook, not Terraform state.[3](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/values.yaml?display=source#L23) ## Technical Approach 1. **Add the `nix` class to `storage/values.yaml`** — constant path, `Retain`, non-default:[4](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/deploy/chart/local-path-provisioner/values.yaml)[5](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/deploy/chart/local-path-provisioner/templates/storageclass.yaml) ```yaml nix: storageClass: create: true defaultClass: false defaultVolumeType: hostPath reclaimPolicy: Retain volumeBindingMode: WaitForFirstConsumer pathPattern: "shared" # constant -> every claim = /var/mnt/data/nix/shared annotations: allowUnsafePathPattern: "true" # chart exposes only params; provisioner reads this annotation nodePathMap: - node: DEFAULT_PATH_FOR_NON_LISTED_NODES paths: - /var/mnt/data/nix ``` 2. **Consume from any namespace** — one PVC per namespace, same backing directory:[6](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/provisioner.go) ```yaml apiVersion: v1 kind: PersistentVolumeClaim metadata: {name: nix-store, namespace: woodpecker-runtime} spec: storageClassName: nix accessModes: [ReadWriteOnce] resources: {requests: {storage: 30Gi}} --- volumeMounts: [{name: nix, mountPath: /nix, readOnly: true}] volumes: [{name: nix, persistentVolumeClaim: {claimName: nix-store}}] ``` 3. **Populate it once (OPERATIONS.md §8g)** — seed the Nix package into the volume, then install the tool closure; `profile install` leaves GC roots so the paths survive collection:[7](https://releases.nixos.org/nix/nix-2.33.6/manual/command-ref/new-cli/nix3-profile.html) ```yaml initContainers: - name: seed image: nixos/nix:latest command: ["sh","-c","[ -e /mnt/store ] || cp -a /nix/. /mnt/"] volumeMounts: [{name: nix, mountPath: /mnt}] containers: - name: populate image: nixos/nix:latest command: ["sh","-c","nix --extra-experimental-features 'nix-command flakes' profile install nixpkgs#terraform nixpkgs#kubectl nixpkgs#kubernetes-helm nixpkgs#talosctl nixpkgs#openssl nixpkgs#curl"] volumeMounts: [{name: nix, mountPath: /nix}] ``` 4. **Assert the contract in `storage/tests/plan.tftest.hcl`**:[8](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/main.tf?display=source#L91)[9](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/tests/plan.tftest.hcl?display=source#L55) ```hcl run "nix_class_is_one_shared_directory" { command = plan assert { condition = yamldecode(helm_release.local_path_provisioner.values[0]).storageClassConfigs.nix.storageClass.pathPattern == "shared" } assert { condition = yamldecode(helm_release.local_path_provisioner.values[0]).storageClassConfigs.nix.storageClass.annotations["allowUnsafePathPattern"] == "true" } assert { condition = yamldecode(helm_release.local_path_provisioner.values[0]).storageClassConfigs.nix.storageClass.reclaimPolicy == "Retain" } assert { condition = yamldecode(helm_release.local_path_provisioner.values[0]).storageClassConfigs.nix.nodePathMap[0].paths[0] == "/var/mnt/data/nix" } } ``` 5. **Document** — `storage/README.md` class table, `ARCHITECTURE.md` storage-tier note, `OPERATIONS.md` §8a table + §8e backup exclusion + new §8g runbook.[10](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/OPERATIONS.md?display=source#L1158)[11](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/OPERATIONS.md?display=source#L1231)[12](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/ARCHITECTURE.md?display=source#L305) ## Implementation Details - Modify: `storage/values.yaml`, `storage/tests/plan.tftest.hcl`, `storage/README.md`, `ARCHITECTURE.md`, `OPERATIONS.md`.[13](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/values.yaml?display=source#L1)[14](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/tests/plan.tftest.hcl?display=source#L1)[15](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/README.md?display=source#L1) - No new Terraform resources, no new qcow2, no `worker/`/`data/` changes — the existing 100 GiB data disk backs this.[16](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/worker/main.tf?display=source#L273)[17](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/data/main.tf?display=source#L87) - Constraints: StorageClass `parameters`/`reclaimPolicy` are immutable → add a new class, never edit `persistent`/`ephemeral`; `Retain` is load-bearing (Delete calls `rm -rf $VOL_DIR`); capacity requests are ignored, so the store shares the data disk's capacity pool with `persistent` PVCs.[18](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/values.yaml?display=source#L40)[19](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/deploy/chart/local-path-provisioner/values.yaml)[20](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/README.md) - Out-of-scope notes: dedicated qcow2 (would need both `UserVolumeConfig` size bands reworked), `nix-snapshotter`/`nix-csi`/`containix` (runtime-layer, need patched containerd), a Terraform-managed population Job, and per-namespace store isolation.[21](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/worker/main.tf?display=source#L395)[22](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/worker/main.tf?display=source#L408) ## Acceptance Criteria - [ ] `terraform -chdir=storage test` passes with `run "nix_class_is_one_shared_directory"` asserting the constant `pathPattern`, the `allowUnsafePathPattern` annotation, `Retain`, and `/var/mnt/data/nix`. - [ ] `python -m pytest tests/ -v` passes (`terraform fmt -check` and the storage plan test). - [ ] `storage/README.md`, `ARCHITECTURE.md`, and `OPERATIONS.md` document the `nix` class, a cross-namespace consumption snippet, and the population runbook. - [ ] OPERATIONS backup section excludes `/var/mnt/data/nix/` as reproducible from the cache, alongside `/var/mnt/data/ephemeral/`. ## Testing Plan - `terraform -chdir=storage test`: new `run "nix_class_is_one_shared_directory"` block above. - `python -m pytest tests/ -v`: existing offline suite covering fmt + storage plan test. - Manual smoke (OPERATIONS §8g): create `nix-store` PVCs in two namespaces, write a marker file from a pod in one, read it from the other → same directory; `ls /var/mnt/data/nix/shared/store`. - Population smoke: run the §8g pod, then `ls /nix/store | head` from a consumer pod. ## Related Issues/PRs - #26 — this issue - #42 — backups of persistent data (the Nix store must be excluded) - #46 — automate cluster deployments - #37 — introduced the storage root and the class-based tier model this extends ## Research Findings - Per-claim dynamic provisioning never yields a shared directory: `local-path-provisioner` always joins the base path with a per-claim `folderName`, so sharing requires one directory aliased by every claim.[23](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/provisioner.go) - `pathPattern` normally must start with `{{ .PVC.Namespace }}/{{ .PVC.Name }}/`; setting `allowUnsafePathPattern` bypasses that check, and v0.0.37 reads it from a StorageClass annotation because the chart does not expose it as a parameter.[24](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/provisioner.go)[25](https://github.com/rancher/local-path-provisioner/blob/v0.0.37/deploy/chart/local-path-provisioner/values.yaml) - Nix store paths are content-addressed and immutable, so read-only sharing is safe, but the store's SQLite DB is not safe under concurrent writers — keep population to a single writer and avoid `nix-collect-garbage` on the shared store.[26](https://nix.dev/manual/nix/2.24/store/types/local-store) - Talos `volumeType: directory` lives on the EPHEMERAL partition[27](https://docs.siderolabs.com/talos/v1.13/configure-your-talos-cluster/storage-and-disk-management/disk-management/user) and is wiped on re-image _unverified_, so the durable route is a directory on the data disk, which this plan uses. - A dedicated third qcow2 is not viable without reworking both `UserVolumeConfig` size bands (20 GiB and 100 GiB both match `>= 40 GiB`), so it is avoided.[28](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/worker/main.tf?display=source#L408) - `nixos/nix` publishes a `linux/arm64` image, matching this aarch64-only cluster.[29](https://hub.docker.com/r/nixos/nix/tags) ## Pre-Mortem 1. **The `allowUnsafePathPattern` annotation is never actually applied, so the constant `pathPattern` never aliases the claims together.** Chart v0.0.37 exposes `storageClassConfigs` parameters but not arbitrary StorageClass annotations, so the class either loses the annotation (and provisioning rejects a `pathPattern` that does not start with `{{ .PVC.Namespace }}/{{ .PVC.Name }}/`) or silently provisions per-claim directories. Because `plan.tftest.hcl` asserts only the Helm values object and not live provisioning, this only surfaces on the cluster when two namespaces turn out to hold different data. 2. **The store is shared and readable, but pods cannot actually run Nix against it.** Mounting `/nix` `readOnly: true` leaves no writable `/nix/var/nix/{profiles,gcroots,temproots}` or unlocked store DB, so `nix develop`/`nix build` fail even though the `ls /nix/store` smoke test passes; the plan never assigns an owner for population versus normal use. The result is a mounted-but-unusable store, not the "any pod can utilize the Nix store" outcome the issue asks for. 3. **The new plan assertion does not hold in the offline `terraform test` harness.** `yamldecode(helm_release.local_path_provisioner.values[0])` depends on the `values` argument being known at plan time and being a single-element list, which the mock-provider/offline suite used by `pytest` may not guarantee, so the acceptance criterion turns the suite red before the class can be merged. Any fallback of weakening the assertion would then remove the only automated guard on the `Retain`/constant-path contract.

tepichord added

and removed

labels

2026-09-11 10:45:38 -07:00

tepichord referenced this issue from a commit

2026-09-11 12:30:00 -07:00

feat(storage): add shared nix StorageClass

tepichord referenced this issue from a commit