tepichord/milner

Fork 0

Support backups of persistent data #42

New issue

Open

opened 2026-09-10 10:24:27 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-10 10:24:27 -07:00

Owner

Sizing caveat: there's no churn data yet, so treat step 9c as a guess. Start from the current ~71 MiB of durable data and assume the cluster grows to 4× size and utilization;
even then snapshots stay under 10% of the drive after a year, which is why the policy below prunes on schedule rather than on capacity.

Stage the durable data somewhere that isn't the 4TB. Copy milner-worker-1-data.qcow2, milner-worker-1-vault.qcow2, and preserved/manifest.json to the host's internal disk
or a USB stick — it's only 71 MiB — and verify the SHA-256s against the manifest. This is your abort path, because step 5 destroys the drive.
Make the host capable of ZFS before anything depends on it. Install OpenZFS on Fedora Asahi and confirm modprobe zfs plus zpool status work against the Asahi kernel. Doing
this first means a kernel/module mismatch is discovered while there's nothing at stake.
Record the exact paths the new filesystem must reproduce. Read the pool path with virsh pool-dumpxml talos-images and the volume id/key host paths out of
data/terraform.tfstate. This matters because the libvirt provider resolves volumes by host path and silently forgets them if the path moves (§9e) — so these paths are a
hard requirement, not a preference.
Stop every writer — the only downtime in this entire exercise. Shut down both nodes, confirm virsh list --all shows them off, then virsh pool-destroy talos-images and
unmount /mnt/data. This is a one-time migration window; the recurring snapshot job in step 9 stays fully online.
Swap the filesystem while keeping the paths identical. Remove any old /etc/fstab entry for /mnt/data, then zpool create -o ashift=12 -O compression=lz4 -O atime=off -m
/mnt/data tank followed by zfs create -o recordsize=64K tank/milner (which auto-mounts at /mnt/data/milner). The dataset now occupies the same directory tree the
old filesystem did, so libvirt, Terraform, and the StorageClasses observe nothing at all.
Put the data back and make Terraform confirm it recognises it. Recreate the pool directory tree, copy the qcow2s to their recorded paths, verify digests, then run terraform
-chdir=data plan and expect No changes. A plan that wants to create the volumes means a path moved, and you stop rather than apply.
Bring the cluster back and prove it's healthy. virsh pool-start talos-images, start the domains (or terraform -chdir=control-plane apply), then check talosctl health,
kubectl get nodes, PVCs Bound, Vault unsealed, and a Woodpecker login. This reuses your §9d drill ladder as the acceptance evidence that the migration lost nothing.
Take the first snapshot and prove it's usable. Run zfs snapshot tank/milner@baseline, then zfs clone tank/milner@baseline tank/verify and confirm the qcow2s are present and
readable there. The offsite copy is deliberately deferred — see 9f.
Automate the snapshot policy

9a. Frequency is free; pick it from what you'll accept losing. Snapshots are instant and consume no space when created, so only retention costs anything. Recommendation:
15-minute snapshots to cover "I just broke Vault," hourly for the last day, daily for the last month, monthly for the last year — or drop the 15-minute tier entirely if a
1-hour RPO is fine for you.
9b. Retention is churn × days, and it's the only real cost. ZFS shares unchanged blocks, so a snapshot of a 40 GiB disk with 100 MB/day of churn costs ~100 MB/day, not 40
GiB. Sized for 4× growth at a generous ~1 GB/day, that's roughly 30 GB of dailies and a few hundred GB per year at worst — under 10% of 4 TB, so you'll never prune to
reclaim space.
9c. Let the tool enforce retention so it can't be forgotten. sanoid with autosnap = yes and autoprune = yes in the template gives you 70 retained snapshots on a fixed
schedule; zfs-auto-snapshot is the lighter alternative. The point is that pruning is config, not a chore you remember.
9d. Monitor so you find out before capacity bites. Check zpool list -H -o capacity tank weekly and alert at 80%, since ZFS performance degrades as the pool fills and writes
fail at 100%. USEDSNAP in zfs list -o space tank/milner is your snapshot bill — if it ever dominates, shorten retention rather than adding a disk.
9e. Know how a snapshot is actually consumed. Restore is path-preserving: stop the VM, zfs rollback tank/milner@autosnap_…, start the VM — the qcow2s revert in place, so the
paths libvirt and Terraform depend on never change. Use zfs clone for a what-if instead, since zfs rollback -r destroys newer snapshots.
9f. Defer the offsite leg, and write down that you deferred it. With no second machine, everything above is a restore-point layer, not a backup — it saves you from a bad
migration or a corrupted database, and dies with the drive. When a destination exists, add syncoid --recursive tank/milner root@dest:tank/milner, and zfs bookmark any
snapshot you intend to use as an incremental base, since a pruned base breaks the -i chain and forces a full resend.
9g. Replace the guess once real workloads are running. Sample zfs get -H -o value written tank/milner 24 hours apart to get true bytes/day, then re-check that 9b's retention
still fits. Until then, 9b is a placeholder sized to grow, not a measurement.

Done when: terraform -chdir=data plan is clean, talosctl health passes with both PVCs bound and Vault unsealed, zfs list -t snapshot tank/milner shows the scheduled tiers
filling in, one rollback drill has succeeded, and you've accepted in writing that until 9f lands, a lost 4 TB is a lost cluster.

Goal: replace the plain filesystem on the external 4TB with a ZFS dataset mounted at the identical path, so one zfs snapshot is an atomic point-in-time of every durable qcow2 — with no changes to Talos, Terraform, or the StorageClasses. Sizing caveat: there's no churn data yet, so treat step 9c as a guess. Start from the current ~71 MiB of durable data and assume the cluster grows to 4× size and utilization; even then snapshots stay under 10% of the drive after a year, which is why the policy below prunes on schedule rather than on capacity. 1. Stage the durable data somewhere that isn't the 4TB. Copy milner-worker-1-data.qcow2, milner-worker-1-vault.qcow2, and preserved/manifest.json to the host's internal disk or a USB stick — it's only 71 MiB — and verify the SHA-256s against the manifest. This is your abort path, because step 5 destroys the drive. 2. Make the host capable of ZFS before anything depends on it. Install OpenZFS on Fedora Asahi and confirm modprobe zfs plus zpool status work against the Asahi kernel. Doing this first means a kernel/module mismatch is discovered while there's nothing at stake. 3. Record the exact paths the new filesystem must reproduce. Read the pool path with virsh pool-dumpxml talos-images and the volume id/key host paths out of data/terraform.tfstate. This matters because the libvirt provider resolves volumes by host path and silently forgets them if the path moves (§9e) — so these paths are a hard requirement, not a preference. 4. Stop every writer — the only downtime in this entire exercise. Shut down both nodes, confirm virsh list --all shows them off, then virsh pool-destroy talos-images and unmount /mnt/data. This is a one-time migration window; the recurring snapshot job in step 9 stays fully online. 5. Swap the filesystem while keeping the paths identical. Remove any old /etc/fstab entry for /mnt/data, then zpool create -o ashift=12 -O compression=lz4 -O atime=off -m /mnt/data tank <device> followed by zfs create -o recordsize=64K tank/milner (which auto-mounts at /mnt/data/milner). The dataset now occupies the same directory tree the old filesystem did, so libvirt, Terraform, and the StorageClasses observe nothing at all. 6. Put the data back and make Terraform confirm it recognises it. Recreate the pool directory tree, copy the qcow2s to their recorded paths, verify digests, then run terraform -chdir=data plan and expect No changes. A plan that wants to create the volumes means a path moved, and you stop rather than apply. 7. Bring the cluster back and prove it's healthy. virsh pool-start talos-images, start the domains (or terraform -chdir=control-plane apply), then check talosctl health, kubectl get nodes, PVCs Bound, Vault unsealed, and a Woodpecker login. This reuses your §9d drill ladder as the acceptance evidence that the migration lost nothing. 8. Take the first snapshot and prove it's usable. Run zfs snapshot tank/milner@baseline, then zfs clone tank/milner@baseline tank/verify and confirm the qcow2s are present and readable there. The offsite copy is deliberately deferred — see 9f. 9. Automate the snapshot policy - 9a. Frequency is free; pick it from what you'll accept losing. Snapshots are instant and consume no space when created, so only retention costs anything. Recommendation: 15-minute snapshots to cover "I just broke Vault," hourly for the last day, daily for the last month, monthly for the last year — or drop the 15-minute tier entirely if a 1-hour RPO is fine for you. - 9b. Retention is churn × days, and it's the only real cost. ZFS shares unchanged blocks, so a snapshot of a 40 GiB disk with 100 MB/day of churn costs ~100 MB/day, not 40 GiB. Sized for 4× growth at a generous ~1 GB/day, that's roughly 30 GB of dailies and a few hundred GB per year at worst — under 10% of 4 TB, so you'll never prune to reclaim space. - 9c. Let the tool enforce retention so it can't be forgotten. sanoid with autosnap = yes and autoprune = yes in the template gives you 70 retained snapshots on a fixed schedule; zfs-auto-snapshot is the lighter alternative. The point is that pruning is config, not a chore you remember. - 9d. Monitor so you find out before capacity bites. Check zpool list -H -o capacity tank weekly and alert at 80%, since ZFS performance degrades as the pool fills and writes fail at 100%. USEDSNAP in zfs list -o space tank/milner is your snapshot bill — if it ever dominates, shorten retention rather than adding a disk. - 9e. Know how a snapshot is actually consumed. Restore is path-preserving: stop the VM, zfs rollback tank/milner@autosnap_…, start the VM — the qcow2s revert in place, so the paths libvirt and Terraform depend on never change. Use zfs clone for a what-if instead, since zfs rollback -r destroys newer snapshots. - 9f. Defer the offsite leg, and write down that you deferred it. With no second machine, everything above is a restore-point layer, not a backup — it saves you from a bad migration or a corrupted database, and dies with the drive. When a destination exists, add syncoid --recursive tank/milner root@dest:tank/milner, and zfs bookmark any snapshot you intend to use as an incremental base, since a pruned base breaks the -i chain and forces a full resend. - 9g. Replace the guess once real workloads are running. Sample zfs get -H -o value written tank/milner 24 hours apart to get true bytes/day, then re-check that 9b's retention still fits. Until then, 9b is a placeholder sized to grow, not a measurement. Done when: terraform -chdir=data plan is clean, talosctl health passes with both PVCs bound and Vault unsealed, zfs list -t snapshot tank/milner shows the scheduled tiers filling in, one rollback drill has succeeded, and you've accepted in writing that until 9f lands, a lost 4 TB is a lost cluster.

tepichord added the

labels

2026-09-10 10:24:27 -07:00

tepichord added this to the MVP project

2026-09-10 10:24:27 -07:00

tepichord commented

2026-09-10 10:53:06 -07:00

Author

Owner

Naina:

Overview

Migrate the external 4 TB drive from a plain filesystem to a ZFS dataset mounted at the identical pool path, so one zfs snapshot tank/milner@<name> is an atomic point-in-time of every durable qcow2. The work is a documented one-time runbook (README rejects per-change migration scripts) plus committed sanoid policy and systemd timers for scheduling, pruning, and capacity monitoring. Offsite replication via syncoid is deferred until a second machine exists and must be recorded as such.

Technical Approach

Freeze and stage the durable set off the drive (abort path).

virsh vol-list talos-images --details          # real names; there is no `-data.qcow2`
sudo mkdir -p ~/milner-preserved
sudo rsync -aHAX --sparse /mnt/data/milner/images/talos-kvm/ ~/milner-preserved/talos-kvm/
( cd ~/milner-preserved/talos-kvm && sha256sum ./* > ../manifest.sha256 )
sha256sum -c ~/milner-preserved/manifest.sha256

--sparse is mandatory: a plain cp materializes the 20 GiB vault qcow2 as 20 GiB of real bytes. ^unverified (rsync does need --sparse, but GNU cp --sparse=auto detects holes by default, so the cp half of this rationale is doubtful)

Build OpenZFS against the Asahi 16K kernel.

sudo dnf install -y kernel-16k-devel zlib-devel libuuid-devel libblkid-devel \
  libtirpc-devel libffi-devel openssl-devel elfutils-libelf-devel
git clone https://github.com/openzfs/zfs && cd zfs && git checkout zfs-2.3.9
./configure --with-config=kernel && make -j"$(nproc)" && sudo make install
sudo depmod -a && sudo modprobe zfs && zpool status

The Fedora zfs-dkms RPM is unusable here — its spec hard-requires kernel-devel, Asahi ships kernel-16k-devel (openzfs#16429, still open)¹. Do this before step 5 so a module failure costs nothing.

Record the path the new filesystem must reproduce.

virsh pool-dumpxml talos-images --xpath 'string(//target/path)'
# -> /mnt/data/milner/images/talos-kvm   (data_root_path=/mnt/data/milner + /images/talos-kvm)

The ${data_root_path}/images/talos-kvm composition is the pool's target.path².

Stop every writer (the only downtime).

talosctl shutdown --nodes 10.5.0.2,10.5.0.20
virsh list --all                    # both domains `shut off`
virsh pool-destroy talos-images
sudo umount /mnt/data

Swap the filesystem in place.

sudo sed -i '\|/mnt/data|d' /etc/fstab          # ZFS owns its mounts now
lsblk -o NAME,SIZE,TYPE,MOUNTPOINT              # identify the 4 TB device
sudo wipefs -a <device>                         # clear old ext4/xfs signatures
sudo zpool create -f -o ashift=12 -O compression=lz4 -O atime=off \
  -m /mnt/data tank <device>
sudo zfs create -o recordsize=64K tank/milner   # -> /mnt/data/milner

-f + wipefs -a are required; bare zpool create refuses a vdev holding a preexisting filesystem³.

Restore data, ownership, and the pool.

sudo mkdir -p /mnt/data/milner/images
sudo rsync -aHAX --sparse ~/milner-preserved/talos-kvm/ /mnt/data/milner/images/talos-kvm/
sha256sum -c ~/milner-preserved/manifest.sha256
sudo chown -R qemu:libvirt /mnt/data/milner/images/talos-kvm   # cleanup.sh's sudo-rm guard
virsh pool-start talos-images

Prove Terraform sees no drift (replaces the issue's -chdir=data).
```
terraform -chdir=control-plane plan   # No changes
terraform -chdir=worker plan          # No changes
```
No data/ root exists; the pool is declared in control-plane/main.tf⁴ and the durable volume in worker/main.tf⁵. Any plan that wants to create a volume means a path moved — stop, don't apply.

Bring the cluster back and run the §9d drill ladder. ^unverified (OPERATIONS.md at HEAD ends at §6, so no §9d exists to run yet)

virsh start milner-cp-1 && virsh start milner-worker-1
talosctl health && kubectl get nodes
kubectl get pvc -A
kubectl -n vault exec vault-0 -- vault status -address=http://127.0.0.1:8200  # sealed:false
curl -s https://vault.faun-snapper.ts.net/v1/sys/health

Baseline snapshot + proof it is usable.

sudo zfs snapshot tank/milner@baseline
sudo zfs clone tank/milner@baseline tank/verify
sudo qemu-img check /tank/verify/images/talos-kvm/milner-worker-1-vault.qcow2
sudo zfs destroy tank/verify

Commit and install the snapshot policy.

nix profile install nixpkgs#sanoid            # sanoid + syncoid, aarch64
sudo ln -sf "$(command -v sanoid)" /usr/local/bin/sanoid
sudo install -d /etc/sanoid
sudo ln -sf "$PWD/zfs/sanoid.conf" /etc/sanoid/sanoid.conf
sudo install -m0644 zfs/systemd/*.service zfs/systemd/*.timer /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now sanoid-prune.timer sanoid-monitor.timer
sudo sanoid --cron --verbose                  # first run creates every tier now

Rollback drill.
```
talosctl shutdown --nodes 10.5.0.20
zfs list -t snapshot tank/milner
sudo zfs rollback tank/milner@baseline
virsh start milner-worker-1 && talosctl health
```
zfs rollback is path-preserving, so the qcow2s revert in place; zfs clone is the non-destructive alternative (rollback -r destroys newer snapshots)⁶.
Write down the deferral. OPERATIONS.md §7 must state: until a second machine exists, snapshots are a restore-point layer, not a backup — a lost 4 TB is a lost cluster; the future command is syncoid --recursive tank/milner root@dest:tank/milner, with zfs bookmark on the incremental base so pruning cannot break the -i chain⁷.

Implementation Details

Path	Change
`OPERATIONS.md`	New §7 "ZFS storage, snapshots, restore": migration runbook, policy, monitoring, drill, deferral.
`zfs/sanoid.conf`	New — canonical policy, symlinked to `/etc/sanoid/sanoid.conf`.
`zfs/systemd/sanoid-prune.{service,timer}`	New — `sanoid --cron` every 15 min.
`zfs/systemd/sanoid-monitor.{service,timer}`	New — weekly `sanoid --monitor-health --monitor-capacity --monitor-snapshots`⁸.
`flake.nix`	Add `sanoid` to `buildInputs` (sanoid/syncoid/findoid on PATH in `nix develop`).
`tests/test_zfs.py`	New — parse and assert the committed policy.
`README.md`, `worker/README.md`, `vault/README.md`	One-line pointers: durable qcow2 now lives on dataset `tank/milner`; commands in OPERATIONS.md §7.

zfs/sanoid.conf:

# Canonical policy — symlink to /etc/sanoid/sanoid.conf (OPERATIONS.md §7).
# capacity_* must live in the root-pool section, not the dataset section.
[tank]
    capacity_warn = 80
    capacity_crit = 95

[tank/milner]
    frequently = 4    # frequent_period=15 -> last hour at 15-min RPO (set 0 if 1h RPO is fine)
    hourly = 24
    daily = 30
    monthly = 12
    yearly = 0
    autosnap = yes
    autoprune = yes

The capacity_* root-pool-section requirement and frequent_period = 15 default are documented in sanoid's own defaults file⁹.

zfs/systemd/sanoid-prune.timer → OnCalendar=*:0/15, Persistent=true, WantedBy=timers.target;
zfs/systemd/sanoid-prune.service → Type=oneshot, ExecStart=/usr/local/bin/sanoid --cron.

Dependencies: host-side OpenZFS built from source against kernel-16k-devel; wipefs (util-linux, already present); sanoid from nixpkgs¹⁰. No new Terraform providers.

Constraints:

virsh pool-dumpxml talos-images --xpath 'string(//target/path)' must stay /mnt/data/milner/images/talos-kvm — libvirt resolves volumes by host path.
No Terraform edits: data_root_path stays host-local (terraform.auto.tfvars)¹¹ and the pool stays type = "dir"¹².
No migration script committed (README: "Per-change migration scripts are rejected outright"¹³); docs + config only.
Preserve qemu:libvirt ownership, or libvirt volume operations fail¹⁴.
On every Asahi kernel update, rebuild the out-of-tree module and re-run modprobe zfs / zpool status.

Out-of-scope notes:

data/ Terraform root — does not exist and is not created; use the existing roots' plans.
Quiesced snapshots — scheduled snapshots are crash-consistent only (default Talos image has no QEMU guest agent); adding the Image Factory extension is a separate change¹⁵.
zpool scrub and SMART — recommended follow-up (a single-disk pool has no other integrity check) but not part of this issue.
Offsite syncoid — deferred per 9f; documented, not implemented. ^unverified (OPERATIONS.md at HEAD has no §9f)
ida / worker-sim — unaffected; the external drive is attached to naina only.

Acceptance Criteria

virsh pool-dumpxml talos-images --xpath 'string(//target/path)' returns /mnt/data/milner/images/talos-kvm; zfs list -H -o mountpoint tank/milner returns /mnt/data/milner; zfs get -H -o value recordsize tank/milner is 64K.
terraform -chdir=control-plane plan and terraform -chdir=worker plan both report "No changes".
Post-migration talosctl health passes, PVCs are Bound, and Vault reports "sealed":false.
zfs list -t snapshot tank/milner contains baseline plus the scheduled tiers, and a clone of @baseline yields qcow2s that pass qemu-img check.
One rollback drill succeeded: worker stopped, zfs rollback tank/milner@<snap>, worker boots, Vault unseals.
systemctl is-active sanoid-prune.timer sanoid-monitor.timer both active; sanoid --monitor-capacity uses the 80/95 thresholds.
python -m pytest tests/ -v passes (including tests/test_zfs.py).
OPERATIONS.md §7 contains the migration runbook and the explicit offsite-deferral statement.

Testing Plan

New tests/test_zfs.py (offline, no host dependencies):

import configparser, pathlib

def test_sanoid_policy_tiers():
    c = configparser.ConfigParser()
    c.read("zfs/sanoid.conf")
    d = c["tank/milner"]
    assert d["autosnap"] == "yes" and d["autoprune"] == "yes"
    assert d["hourly"] == "24" and d["daily"] == "30" and d["monthly"] == "12"
    assert c["tank"]["capacity_warn"] == "80"

def test_runbook_documents_migration():
    ops = pathlib.Path("OPERATIONS.md").read_text()
    assert "/mnt/data/milner/images/talos-kvm" in ops
    assert "syncoid" in ops

python -m pytest tests/ -v and terraform fmt -check -recursive control-plane/ worker/ must pass.
Manual (host, human-run): step 9 clone verification and step 11 rollback drill above.

#42 — this issue.
#22 — IaC control plane (libvirt_pool.talos-images, pool path).
#33 — Vault (owns the durable milner-worker-1-vault.qcow2).
OPERATIONS.md §4e, ARCHITECTURE.md §4/§8, worker/README.md "Vault data volume".

Research Findings

OpenZFS on Fedora Asahi: the module supports the 16K-page kernel, but the zfs-dkms RPM hard-requires kernel-devel while Asahi ships kernel-16k-devel (openzfs#16429, open since 2024) — build from source against kernel-16k-devel.
qcow2-on-ZFS under a libvirt dir pool is a documented configuration using recordsize=64K + compression=lz4, though OpenZFS prefers zvols/raw files; watch preallocation/detect-zeroes for txg_sync stalls.
zfs rollback restores a qcow2's data and metadata atomically and path-preservingly, so the VM boots immediately afterwards — the VM must be stopped first.
Snapshots of a running guest are crash-consistent only, since Talos ships no QEMU guest agent; the clean-shutdown @baseline is fully consistent.
sanoid is the actively maintained choice and nixpkgs ships sanoid/syncoid/findoid for aarch64; zfs-auto-snapshot's last upstream commit was 2019.
zpool create refuses a vdev holding an old ext4/xfs filesystem without -f; clear signatures with wipefs -a first and use ashift=12 (immutable per vdev).
syncoid plus zfs bookmark (or --create-bookmark) keeps the incremental chain alive after pruning, which matters for the deferred offsite leg.

Pre-Mortem

The source-built OpenZFS module never becomes reliable on the Asahi 16K kernel — the build against kernel-16k-devel fails on configure/make, or modprobe zfs stops working on the next Asahi kernel update, leaving the drive half-migrated with no supported fallback. Every step from 5 onward is blocked until the module loads.
The destructive swap irreversibly loses the durable qcow2s — wipefs -a plus zpool create -f on a single disk, with the only staging copy in ~/milner-preserved on a different but unverified-capacity filesystem, means a failed restore or a sparse-flag mistake is not recoverable. A wrong device selection in step 5 is equally fatal.
The plan passes review on documentation alone while the real migration is never proven — tests/test_zfs.py only asserts INI text and string presence, so the host-only acceptance criteria (rollback drill, clone qemu-img check, Vault unsealing, Terraform "No changes") can be skipped and the breakage surfaces later as libvirt path drift or a non-booting worker.

Naina: ## Overview Migrate the external 4 TB drive from a plain filesystem to a ZFS dataset mounted at the identical pool path, so one `zfs snapshot tank/milner@<name>` is an atomic point-in-time of every durable qcow2. The work is a documented one-time runbook (README rejects per-change migration scripts) plus committed sanoid policy and systemd timers for scheduling, pruning, and capacity monitoring. Offsite replication via `syncoid` is deferred until a second machine exists and must be recorded as such. ## Technical Approach 1. **Freeze and stage the durable set off the drive (abort path).** ```bash virsh vol-list talos-images --details # real names; there is no `-data.qcow2` sudo mkdir -p ~/milner-preserved sudo rsync -aHAX --sparse /mnt/data/milner/images/talos-kvm/ ~/milner-preserved/talos-kvm/ ( cd ~/milner-preserved/talos-kvm && sha256sum ./* > ../manifest.sha256 ) sha256sum -c ~/milner-preserved/manifest.sha256 ``` `--sparse` is mandatory: a plain `cp` materializes the 20 GiB vault qcow2 as 20 GiB of real bytes. _unverified_ (rsync does need `--sparse`, but GNU `cp --sparse=auto` detects holes by default, so the `cp` half of this rationale is doubtful) 2. **Build OpenZFS against the Asahi 16K kernel.** ```bash sudo dnf install -y kernel-16k-devel zlib-devel libuuid-devel libblkid-devel \ libtirpc-devel libffi-devel openssl-devel elfutils-libelf-devel git clone https://github.com/openzfs/zfs && cd zfs && git checkout zfs-2.3.9 ./configure --with-config=kernel && make -j"$(nproc)" && sudo make install sudo depmod -a && sudo modprobe zfs && zpool status ``` The Fedora `zfs-dkms` RPM is unusable here — its spec hard-requires `kernel-devel`, Asahi ships `kernel-16k-devel` (openzfs#16429, still open)[1](https://github.com/openzfs/zfs/issues/16429). Do this before step 5 so a module failure costs nothing. 3. **Record the path the new filesystem must reproduce.** ```bash virsh pool-dumpxml talos-images --xpath 'string(//target/path)' # -> /mnt/data/milner/images/talos-kvm (data_root_path=/mnt/data/milner + /images/talos-kvm) ``` The `${data_root_path}/images/talos-kvm` composition is the pool's `target.path`[2](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/control-plane/main.tf?display=source#L112). 4. **Stop every writer (the only downtime).** ```bash talosctl shutdown --nodes 10.5.0.2,10.5.0.20 virsh list --all # both domains `shut off` virsh pool-destroy talos-images sudo umount /mnt/data ``` 5. **Swap the filesystem in place.** ```bash sudo sed -i '\|/mnt/data|d' /etc/fstab # ZFS owns its mounts now lsblk -o NAME,SIZE,TYPE,MOUNTPOINT # identify the 4 TB device sudo wipefs -a <device> # clear old ext4/xfs signatures sudo zpool create -f -o ashift=12 -O compression=lz4 -O atime=off \ -m /mnt/data tank <device> sudo zfs create -o recordsize=64K tank/milner # -> /mnt/data/milner ``` `-f` + `wipefs -a` are required; bare `zpool create` refuses a vdev holding a preexisting filesystem[3](https://github.com/openzfs/zfs/issues/13171). 6. **Restore data, ownership, and the pool.** ```bash sudo mkdir -p /mnt/data/milner/images sudo rsync -aHAX --sparse ~/milner-preserved/talos-kvm/ /mnt/data/milner/images/talos-kvm/ sha256sum -c ~/milner-preserved/manifest.sha256 sudo chown -R qemu:libvirt /mnt/data/milner/images/talos-kvm # cleanup.sh's sudo-rm guard virsh pool-start talos-images ``` 7. **Prove Terraform sees no drift (replaces the issue's `-chdir=data`).** ```bash terraform -chdir=control-plane plan # No changes terraform -chdir=worker plan # No changes ``` No `data/` root exists; the pool is declared in `control-plane/main.tf`[4](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/control-plane/main.tf?display=source#L182) and the durable volume in `worker/main.tf`[5](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/worker/main.tf?display=source#L99). Any plan that wants to create a volume means a path moved — stop, don't apply. 8. **Bring the cluster back and run the §9d drill ladder.** _unverified_ (OPERATIONS.md at HEAD ends at §6, so no §9d exists to run yet) ```bash virsh start milner-cp-1 && virsh start milner-worker-1 talosctl health && kubectl get nodes kubectl get pvc -A kubectl -n vault exec vault-0 -- vault status -address=http://127.0.0.1:8200 # sealed:false curl -s https://vault.faun-snapper.ts.net/v1/sys/health ``` 9. **Baseline snapshot + proof it is usable.** ```bash sudo zfs snapshot tank/milner@baseline sudo zfs clone tank/milner@baseline tank/verify sudo qemu-img check /tank/verify/images/talos-kvm/milner-worker-1-vault.qcow2 sudo zfs destroy tank/verify ``` 10. **Commit and install the snapshot policy.** ```bash nix profile install nixpkgs#sanoid # sanoid + syncoid, aarch64 sudo ln -sf "$(command -v sanoid)" /usr/local/bin/sanoid sudo install -d /etc/sanoid sudo ln -sf "$PWD/zfs/sanoid.conf" /etc/sanoid/sanoid.conf sudo install -m0644 zfs/systemd/*.service zfs/systemd/*.timer /etc/systemd/system/ sudo systemctl daemon-reload sudo systemctl enable --now sanoid-prune.timer sanoid-monitor.timer sudo sanoid --cron --verbose # first run creates every tier now ``` 11. **Rollback drill.** ```bash talosctl shutdown --nodes 10.5.0.20 zfs list -t snapshot tank/milner sudo zfs rollback tank/milner@baseline virsh start milner-worker-1 && talosctl health ``` `zfs rollback` is path-preserving, so the qcow2s revert in place; `zfs clone` is the non-destructive alternative (`rollback -r` destroys newer snapshots)[6](https://openzfs.github.io/openzfs-docs/man/master/8/zfs-rollback.8.html). 12. **Write down the deferral.** OPERATIONS.md §7 must state: until a second machine exists, snapshots are a restore-point layer, not a backup — a lost 4 TB is a lost cluster; the future command is `syncoid --recursive tank/milner root@dest:tank/milner`, with `zfs bookmark` on the incremental base so pruning cannot break the `-i` chain[7](https://manpages.ubuntu.com/manpages/noble/man8/syncoid.8.html). ## Implementation Details | Path | Change | |---|---| | `OPERATIONS.md` | New §7 "ZFS storage, snapshots, restore": migration runbook, policy, monitoring, drill, deferral. | | `zfs/sanoid.conf` | New — canonical policy, symlinked to `/etc/sanoid/sanoid.conf`. | | `zfs/systemd/sanoid-prune.{service,timer}` | New — `sanoid --cron` every 15 min. | | `zfs/systemd/sanoid-monitor.{service,timer}` | New — weekly `sanoid --monitor-health --monitor-capacity --monitor-snapshots`[8](https://github.com/jimsalterjrs/sanoid/blob/master/README.md). | | `flake.nix` | Add `sanoid` to `buildInputs` (sanoid/syncoid/findoid on PATH in `nix develop`). | | `tests/test_zfs.py` | New — parse and assert the committed policy. | | `README.md`, `worker/README.md`, `vault/README.md` | One-line pointers: durable qcow2 now lives on dataset `tank/milner`; commands in OPERATIONS.md §7. | `zfs/sanoid.conf`: ```ini # Canonical policy — symlink to /etc/sanoid/sanoid.conf (OPERATIONS.md §7). # capacity_* must live in the root-pool section, not the dataset section. [tank] capacity_warn = 80 capacity_crit = 95 [tank/milner] frequently = 4 # frequent_period=15 -> last hour at 15-min RPO (set 0 if 1h RPO is fine) hourly = 24 daily = 30 monthly = 12 yearly = 0 autosnap = yes autoprune = yes ``` The `capacity_*` root-pool-section requirement and `frequent_period = 15` default are documented in sanoid's own defaults file[9](https://github.com/jimsalterjrs/sanoid/blob/master/sanoid.defaults.conf). `zfs/systemd/sanoid-prune.timer` → `OnCalendar=*:0/15`, `Persistent=true`, `WantedBy=timers.target`; `zfs/systemd/sanoid-prune.service` → `Type=oneshot`, `ExecStart=/usr/local/bin/sanoid --cron`. Dependencies: host-side OpenZFS built from source against `kernel-16k-devel`; `wipefs` (util-linux, already present); sanoid from nixpkgs[10](https://github.com/NixOS/nixpkgs/blob/nixpkgs-unstable/pkgs/by-name/sa/sanoid/package.nix). No new Terraform providers. Constraints: - `virsh pool-dumpxml talos-images --xpath 'string(//target/path)'` must stay `/mnt/data/milner/images/talos-kvm` — libvirt resolves volumes by host path. - No Terraform edits: `data_root_path` stays host-local (`terraform.auto.tfvars`)[11](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/control-plane/.gitignore?display=source#L6) and the pool stays `type = "dir"`[12](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/control-plane/main.tf?display=source#L184). - No migration script committed (README: "Per-change migration scripts are rejected outright"[13](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/README.md?display=source#L145)); docs + config only. - Preserve `qemu:libvirt` ownership, or libvirt volume operations fail[14](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/control-plane/cleanup.sh?display=source#L75). - On every Asahi kernel update, rebuild the out-of-tree module and re-run `modprobe zfs` / `zpool status`. Out-of-scope notes: - **`data/` Terraform root** — does not exist and is not created; use the existing roots' plans. - **Quiesced snapshots** — scheduled snapshots are crash-consistent only (default Talos image has no QEMU guest agent); adding the Image Factory extension is a separate change[15](https://docs.siderolabs.com/talos/v1.13/platform-specific-installations/virtualized-platforms/proxmox). - **`zpool scrub` and SMART** — recommended follow-up (a single-disk pool has no other integrity check) but not part of this issue. - **Offsite `syncoid`** — deferred per 9f; documented, not implemented. _unverified_ (OPERATIONS.md at HEAD has no §9f) - **ida / worker-sim** — unaffected; the external drive is attached to naina only. ## Acceptance Criteria - [ ] `virsh pool-dumpxml talos-images --xpath 'string(//target/path)'` returns `/mnt/data/milner/images/talos-kvm`; `zfs list -H -o mountpoint tank/milner` returns `/mnt/data/milner`; `zfs get -H -o value recordsize tank/milner` is `64K`. - [ ] `terraform -chdir=control-plane plan` and `terraform -chdir=worker plan` both report "No changes". - [ ] Post-migration `talosctl health` passes, PVCs are `Bound`, and Vault reports `"sealed":false`. - [ ] `zfs list -t snapshot tank/milner` contains `baseline` plus the scheduled tiers, and a clone of `@baseline` yields qcow2s that pass `qemu-img check`. - [ ] One rollback drill succeeded: worker stopped, `zfs rollback tank/milner@<snap>`, worker boots, Vault unseals. - [ ] `systemctl is-active sanoid-prune.timer sanoid-monitor.timer` both `active`; `sanoid --monitor-capacity` uses the 80/95 thresholds. - [ ] `python -m pytest tests/ -v` passes (including `tests/test_zfs.py`). - [ ] OPERATIONS.md §7 contains the migration runbook and the explicit offsite-deferral statement. ## Testing Plan New `tests/test_zfs.py` (offline, no host dependencies): ```python import configparser, pathlib def test_sanoid_policy_tiers(): c = configparser.ConfigParser() c.read("zfs/sanoid.conf") d = c["tank/milner"] assert d["autosnap"] == "yes" and d["autoprune"] == "yes" assert d["hourly"] == "24" and d["daily"] == "30" and d["monthly"] == "12" assert c["tank"]["capacity_warn"] == "80" def test_runbook_documents_migration(): ops = pathlib.Path("OPERATIONS.md").read_text() assert "/mnt/data/milner/images/talos-kvm" in ops assert "syncoid" in ops ``` - `python -m pytest tests/ -v` and `terraform fmt -check -recursive control-plane/ worker/` must pass. - Manual (host, human-run): step 9 clone verification and step 11 rollback drill above. ## Related Issues/PRs - #42 — this issue. - #22 — IaC control plane (`libvirt_pool.talos-images`, pool path). - #33 — Vault (owns the durable `milner-worker-1-vault.qcow2`). - `OPERATIONS.md` §4e, `ARCHITECTURE.md` §4/§8, `worker/README.md` "Vault data volume". ## Research Findings - OpenZFS on Fedora Asahi: the module supports the 16K-page kernel, but the `zfs-dkms` RPM hard-requires `kernel-devel` while Asahi ships `kernel-16k-devel` (openzfs#16429, open since 2024) — build from source against `kernel-16k-devel`. - qcow2-on-ZFS under a libvirt `dir` pool is a documented configuration using `recordsize=64K` + `compression=lz4`, though OpenZFS prefers zvols/raw files; watch preallocation/`detect-zeroes` for `txg_sync` stalls. - `zfs rollback` restores a qcow2's data and metadata atomically and path-preservingly, so the VM boots immediately afterwards — the VM must be stopped first. - Snapshots of a running guest are crash-consistent only, since Talos ships no QEMU guest agent; the clean-shutdown `@baseline` is fully consistent. - sanoid is the actively maintained choice and nixpkgs ships `sanoid`/`syncoid`/`findoid` for aarch64; zfs-auto-snapshot's last upstream commit was 2019. - `zpool create` refuses a vdev holding an old ext4/xfs filesystem without `-f`; clear signatures with `wipefs -a` first and use `ashift=12` (immutable per vdev). - syncoid plus `zfs bookmark` (or `--create-bookmark`) keeps the incremental chain alive after pruning, which matters for the deferred offsite leg. ## Pre-Mortem 1. **The source-built OpenZFS module never becomes reliable on the Asahi 16K kernel** — the build against `kernel-16k-devel` fails on configure/make, or `modprobe zfs` stops working on the next Asahi kernel update, leaving the drive half-migrated with no supported fallback. Every step from 5 onward is blocked until the module loads. 2. **The destructive swap irreversibly loses the durable qcow2s** — `wipefs -a` plus `zpool create -f` on a single disk, with the only staging copy in `~/milner-preserved` on a different but unverified-capacity filesystem, means a failed restore or a sparse-flag mistake is not recoverable. A wrong device selection in step 5 is equally fatal. 3. **The plan passes review on documentation alone while the real migration is never proven** — `tests/test_zfs.py` only asserts INI text and string presence, so the host-only acceptance criteria (rollback drill, clone `qemu-img check`, Vault unsealing, Terraform "No changes") can be skipped and the breakage surfaces later as libvirt path drift or a non-booting worker.

tepichord added

and removed

labels

2026-09-10 11:03:27 -07:00

tepichord referenced this issue from a commit

2026-09-10 16:22:41 -07:00

feat(zfs): add sanoid policy and systemd timers

tepichord referenced this issue from a commit