Utilize shared nix volume #51

tepichord · 2026-09-11T12:30:06-07:00

tepichord commented

2026-09-11 12:30:06 -07:00

Closes #26

Automated PR created by forgejo-done adapter.

Closes #26 Automated PR created by forgejo-done adapter.

tepichord added 9 commits

2026-09-11 12:30:06 -07:00

feat(storage): add shared nix StorageClass b2dfbf2f77

Add a `nix` StorageClass backed by /var/mnt/data/nix/shared via a
constant pathPattern and the allowUnsafePathPattern annotation.
local-path-provisioner v0.0.37 reads the annotation from the
StorageClass object (not a chart parameter) to bypass the
{{ .PVC.Namespace }}/{{ .PVC.Name }} path requirement.

Reclaim policy is Retain — Delete would rm -rf the shared store.
The backing directory is the existing 100 GiB data disk; no new qcow2
or Terraform resource is added.

Includes:
- storage/values.yaml: full chart values with persistent/ephemeral/nix
- storage/main.tf: Terraform root for test assertions
- storage/tests/plan.tftest.hcl: nix_class_is_one_shared_directory
- gitops values: nix class added to storageClassConfigs

Refs: #26

test: add storage/ to fmt check and terraform test suite a84b2bc651

- test_static.py: include storage/ in terraform fmt -check -recursive
  and storage/values.yaml in yamllint
- test_terraform.py: add test_terraform_test_storage following the
  existing pattern

Refs: #26

docs: document nix StorageClass and population runbook dcdb0b9d75

- ARCHITECTURE.md: update storage tiers section to list nix alongside
  persistent and ephemeral
- OPERATIONS.md: add nix to §8a table, §8e backup exclusion for
  /var/mnt/data/nix/, new §8g population runbook with initContainer
  seed pattern and tool closure install

Refs: #26

docs(storage): add README for storage Terraform root 9fa717d6ee

Documents the three StorageClasses (persistent, ephemeral, nix), the
cross-namespace PVC consumption snippet, and testing instructions.

Refs: #26

chore(storage): add terraform lock file bbb9cf76ce

Refs: #26

docs(repo): add storage/ to AGENTS.md repo map and testing examples 3355b173f3

Refs: #26

docs(operations): repair the §8g Nix population command 117cafa0cd

The §8g populate container's command carried literal \n escapes and ran
`profile install` as arguments, so the documented one-off job could never
have seeded a store. Spell the invocation on real lines, and pin
nixos/nix:latest to a concrete tag like every other image in the repo.

test(storage): assert the Flux-managed values, not a copy f45ea9d8bf

storage/values.yaml duplicated the Flux-owned
gitops/clusters/milner/infrastructure/local-path/values.yaml, and the
terraform test decoded that copy — so deleting the nix class from the
values that actually ship still passed. Point the stub HelmRelease at the
real file (the root is assertions-only; it configures no provider) and
drop the duplicate.

Also guard create/defaultClass for the nix class: create: false renders no
StorageClass at all, and a default nix class would send every unqualified
PVC into the shared store.

chore(storage): gitignore terraform state like every sibling root

ci/woodpecker/push/woodpecker Pipeline was successful

Details

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

e50bc3f1b1

tepichord added the

pr

feedback

label

2026-09-11 12:30:06 -07:00

tepichord added this to the MVP project

2026-09-11 19:43:33 -07:00

tepichord referenced this pull request from a commit

2026-09-11 22:01:53 -07:00

merge: resolve conflicts with main (#51)

tepichord referenced this pull request from a commit

2026-09-11 22:01:53 -07:00

docs(operations): remove duplicate §10 Bring-up order section

tepichord referenced this pull request from a commit

2026-09-11 22:01:53 -07:00

docs: renumber the shared Nix runbook under §10

tepichord referenced this pull request from a commit

2026-09-11 22:01:53 -07:00

docs(storage): correct stale class count and apply warning

tepichord added 4 commits

2026-09-11 22:01:53 -07:00

merge: resolve conflicts with main (#51 ) a15d4e8fec

docs(operations): remove duplicate §10 Bring-up order section e864fb2bb0

Merge conflict resolution left a stale copy of the Bring-up order
section (old §10 with §9 references) below §11. The correct version
already lives at §8 with the right section numbering. Remove the
duplicate.

Refs: #51

docs: renumber the shared Nix runbook under §10 95c68634eb

The merge of main renumbered Storage from §8 to §10, but the Nix
population subsection kept its old §8g label and the references to it
(§8a, §8g) still pointed at the retired number. Renumber the heading and
update the references in ARCHITECTURE.md, storage/README.md, and the
local-path values.yaml.

Refs: #51

docs(storage): correct stale class count and apply warning

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

c073096cf1

The local-path values header still said "both classes" after the nix
class made it three. The storage root also claimed an apply "cannot
reach a cluster"; the helm provider needs no explicit config and would
fall back to the ambient kubeconfig, so state the real hazard instead.

Refs: #51

tepichord added

and removed

labels

2026-09-11 22:01:55 -07:00

tepichord commented

2026-09-12 07:29:38 -07:00

Ensure that CI tests use a nix image and call nix develop and are able to utilize the shared volume for it's builds.

tepichord commented

2026-09-17 07:36:37 -07:00

Resolve merge conflicts with main.

tepichord added

and removed

labels

2026-09-17 07:36:42 -07:00

tepichord referenced this pull request from a commit

2026-09-17 10:12:35 -07:00

merge: resolve conflicts with main (#51)

tepichord referenced this pull request from a commit

2026-09-17 10:12:35 -07:00

feat(gitops): commit the CI agent's nix-store claim

tepichord referenced this pull request from a commit

2026-09-17 10:12:35 -07:00

feat(ci): run the suite from the nix dev shell on the shared store

tepichord referenced this pull request from a commit

2026-09-17 10:12:35 -07:00

docs(storage): the shared-store claim is Flux-owned and writable

tepichord referenced this pull request from a commit

2026-09-17 10:12:35 -07:00

docs(operations): fix stale §8c cross-reference to §10c

tepichord referenced this pull request from a commit

2026-09-17 10:12:35 -07:00

test(static): read the CI nix image out of the population runbook

tepichord added 6 commits

2026-09-17 10:12:35 -07:00

merge: resolve conflicts with main (#51 ) 06952619b9

feat(gitops): commit the CI agent's nix-store claim 1fe280e552

The `nix` StorageClass aliases every claim to one shared directory, so a
claim is how a pod mounts the store. Woodpecker mounts a claim by name and
never creates one, so the agent's (`nix-store`, in `woodpecker-runtime`) has
to be an applied manifest rather than a `kubectl create` an operator
remembers — the CI test step mounts it at `/nix`.

Refs #51.

feat(ci): run the suite from the nix dev shell on the shared store 00cdf57f59

The `test` step installed python, pytest and git onto alpine and ran
pytest there. That image carries nothing else the suite needs, so every
linter, render and terraform case skipped — "the suite passed" meant "the
part of it that needs no tools passed".

Run it the way a developer does instead: `nixos/nix` pinned to the tag the
population runbook pins, `nix develop --command`, and the `nix` class's
shared claim mounted at `/nix`. The devShell closure is then realised into
the store every claim of that class resolves to, so the first run pays for
it and later runs read it back — which is what makes the shared volume this
PR adds actually used.

Two consequences, both deliberate:

- `terraform` is on PATH inside the dev shell, so the terraform cases no
  longer skip on the binary alone. They now also require an initialized
  root: `terraform test` mocks its providers but still loads their schemas,
  so on a fresh clone it fails with "missing or corrupted provider plugins"
  rather than skipping. The init is manual and touches the network, and the
  suite is offline on purpose.
- `git` joins the flake's buildInputs. The suite shells out to it, and the
  shell the suite runs in has to carry it rather than inherit whatever the
  image happens to put on PATH.

Refs #51.

docs(storage): the shared-store claim is Flux-owned and writable a5d2f779a9

§10g told the operator to create the `nix-store` PVC by hand. That claim is
committed now (infrastructure/nix-store/), so the runbook would have had two
owners fighting over one object; the population job is the part that stays
a one-off, and it is the part the section still has to hand over.

The consumption snippet also said `readOnly: true`. Nix opens the store
database read-write even for a query — a read-only mount fails on the first
substitution — and the CI step builds into the store, so the mount is
writable on purpose.

Refs #51.

docs(operations): fix stale §8c cross-reference to §10c b2367525d2

Ref: #51

test(static): read the CI nix image out of the population runbook 0da2651085

The step guards in `test_the_suite_runs_in_the_nix_dev_shell_on_the_shared_store`
asserted the image twice: once that it starts with `nixos/nix:`, then that it
equals the literal `nixos/nix:2.35.2`. The first is subsumed by the second, and
the literal is a *copy* of the tag §10g seeds the store with — so bumping the
runbook without the step (or the reverse) left the suite green while the store's
DB schema and the `nix` reading it disagree, the one thing the second
assertion's message claims to prevent. That is the same failure `storage/`
avoids by reading the shipped `values.yaml` instead of copying it.

Pin the shape (a versioned `nixos/nix` tag, not a floating one) and compare the
step's image against the one §10g actually uses, so the two have to move
together. Verified by mutating the runbook tag: the test fails.

Refs #51.

tepichord reviewed

2026-09-17 13:59:27 -07:00

tepichord left a comment

Work needed

gitops/clusters/milner/infrastructure/nix-store/persistentvolumeclaim.yaml

					
				@ -0,0 +21,4 @@

				    # Ignored by local-path-provisioner: the store shares the data disk's

				    # capacity pool with the `persistent` PVCs (OPERATIONS.md §10e).

				    requests:

				      storage: 30Gi

Honestly the nix store can grow large, make this 200Gi

tepichord marked this conversation as resolved

storage/main.tf Outdated

					
				@ -0,0 +9,4 @@

				}

				# local-path-provisioner v0.0.37 serves three StorageClasses (persistent,

				# ephemeral, nix) backed by the worker's shared 100 GiB data disk. Flux owns

2 TiB now

tepichord marked this conversation as resolved

tepichord referenced this pull request from a commit

2026-09-17 15:39:29 -07:00

merge: resolve conflicts with main (#51)

tepichord referenced this pull request from a commit

2026-09-17 15:39:29 -07:00

fix(gitops): request 200Gi for the shared nix store claim

tepichord referenced this pull request from a commit

2026-09-17 15:39:29 -07:00

docs(storage): the provisioner's backing disk is 2 TiB, not 100 GiB

tepichord referenced this pull request from a commit

2026-09-17 15:39:29 -07:00

docs(storage): fix stale 100 GiB references to 2 TiB and update example PVC size

tepichord added 4 commits

2026-09-17 15:39:29 -07:00

merge: resolve conflicts with main (#51 ) 735804d973

fix(gitops): request 200Gi for the shared nix store claim 50ddaad33d

The store grows with every CI build and is never garbage-collected on its
own, so 30Gi undersold it. The request stays advisory: local-path-provisioner
ignores it and the store draws on the shared data disk's capacity pool.

Refs #51

docs(storage): the provisioner's backing disk is 2 TiB, not 100 GiB 3acaabf1ed

The 2026-09 grow took the shared data disk 100 GiB -> 2 TiB
(OPERATIONS.md §12d), which left this comment stale.

Refs #51

docs(storage): fix stale 100 GiB references to 2 TiB and update example PVC size 843929df99

The data disk was grown from 100 GiB to 2 TiB and the nix-store PVC
changed from 30Gi to 200Gi, but these documentation surfaces were not
updated consistently.

Refs: #51

tepichord added

and removed

labels

2026-09-17 15:39:33 -07:00

tepichord added 1 commit

2026-09-17 16:06:17 -07:00

Merge branch 'main' into feat/26-shared-nix-volume 9534bd1d04

tepichord commented

2026-09-17 16:12:44 -07:00

The woodpecker build failed:

  linter: Insufficient trust level to use `volumes`   (field: steps.test, file: gates)

The cause is in PR #51's own .woodpecker/gates.yaml: the test step gains image: nixos/nix:2.35.2 + a volumes: mount for the shared nix store. Woodpecker v3 splits the old
single "trusted" flag into three independent gates — network, security, volumes — and tepichord/milner is currently:

  trusted: {"network": false, "volumes": false, "security": true}

security: true is why the privileged: true buildah steps in my PR #70 ran fine; volumes: false is why this branch's config is rejected outright. So this isn't a webhook,
agent, or secret problem — the repo just isn't allowed to mount volumes.

The fix is a one-field repo setting: turn on Volumes trust (repo → Settings → Trusted → Volumes, or PATCH /api/repos/2 with "trusted": {"network": false, "volumes": true,
"security": true}).

The woodpecker build failed: ``` linter: Insufficient trust level to use `volumes` (field: steps.test, file: gates) ``` The cause is in PR #51's own .woodpecker/gates.yaml: the test step gains image: nixos/nix:2.35.2 + a volumes: mount for the shared nix store. Woodpecker v3 splits the old single "trusted" flag into three independent gates — network, security, volumes — and tepichord/milner is currently: ``` trusted: {"network": false, "volumes": false, "security": true} ``` security: true is why the privileged: true buildah steps in my PR #70 ran fine; volumes: false is why this branch's config is rejected outright. So this isn't a webhook, agent, or secret problem — the repo just isn't allowed to mount volumes. The fix is a one-field repo setting: turn on Volumes trust (repo → Settings → Trusted → Volumes, or PATCH /api/repos/2 with "trusted": {"network": false, "volumes": true, "security": true}).

tepichord added

and removed

labels

2026-09-17 16:12:52 -07:00

tepichord commented

2026-09-17 17:23:57 -07:00

Naina:
trusted.volumes is the one item in #9488 that cannot be fixed in this branch: it is a Woodpecker repo setting, not repo content. I re-checked GET /api/repos/2 and it still reports {"network": false, "volumes": false, "security": true}, and PATCH /api/repos/2 requires a Woodpecker admin token — this pipeline only carries FORGEJO_TOKEN, which Woodpecker rejects (401 from /api/user). So the switch has to be turned on from repo → Settings → Trusted → Volumes (or the PATCH quoted above) by someone with Woodpecker admin access; no step in this pipeline can do it.

Naina: `trusted.volumes` is the one item in #9488 that cannot be fixed in this branch: it is a Woodpecker *repo* setting, not repo content. I re-checked `GET /api/repos/2` and it still reports `{"network": false, "volumes": false, "security": true}`, and `PATCH /api/repos/2` requires a Woodpecker admin token — this pipeline only carries `FORGEJO_TOKEN`, which Woodpecker rejects (`401` from `/api/user`). So the switch has to be turned on from repo → Settings → Trusted → Volumes (or the PATCH quoted above) by someone with Woodpecker admin access; no step in this pipeline can do it.

tepichord added 1 commit

2026-09-17 18:13:41 -07:00

docs(operations): name the Terraform helm roots that actually remain c8beb970b7

The §2 note justified the imperative helm install by claiming `storage/` was
the only Terraform `helm` root left. It is not: `kubernetes-dashboard/`
still declares a `helm_release` (Headlamp, §7b) with a configured provider,
so the note named the wrong root as the exception and invited the reader to
believe Terraform does no helm work here. `storage/` is assertions-only and
configures no provider; say both.

tepichord added

and removed

labels

2026-09-17 18:14:04 -07:00

tepichord commented

2026-09-17 21:55:33 -07:00

Merge changes from main, the woodpecker repo setting has been changed.

tepichord added

and removed

labels

2026-09-17 21:55:40 -07:00

tepichord referenced this pull request from a commit

2026-09-17 23:53:21 -07:00

merge: resolve conflicts with main (#51)

tepichord referenced this pull request from a commit

2026-09-17 23:53:21 -07:00

docs(data): point the restore tail at the Flux-owned roots

tepichord added 2 commits

2026-09-17 23:53:22 -07:00

merge: resolve conflicts with main (#51 ) f351aa71a4

docs(data): point the restore tail at the Flux-owned roots

ci/woodpecker/pr/gates Pipeline was canceled

Details

ci/woodpecker/pull_request_closed/cleanup Pipeline was successful

Details

e9546e8e0e

`data/restore.sh` still printed the pre-Flux next steps. `woodpecker/` and
`vault/` are not Terraform roots any more (deleted with the Flux migration),
and this branch re-adds `storage/` as an assertions-only root whose own
README says an apply would resolve the ambient kubeconfig and fight Flux
over the provisioner release — so the block now instructs exactly the apply
that must not happen.

Replace it with the reinstall tail in docs/durable-volumes.md §11, which is
the authority for the order, and state that the StorageClasses, Woodpecker
and Vault are Flux-owned so no Terraform root applies them.

Refs #51.

tepichord added

and removed

labels

2026-09-17 23:53:44 -07:00

tepichord merged commit d461ede697 into main

2026-09-18 19:07:26 -07:00

tepichord deleted branch feat/26-shared-nix-volume

2026-09-18 19:07:27 -07:00