Milner

General-purpose Talos Linux cluster definition on libvirt/QEMU (aarch64). Hosts:

• HashiCorp Vault — secrets management (Flux-managed)
• CloudNativePG Postgres — cluster-wide PostgreSQL (Flux-managed)
• Pocket ID — OIDC identity provider (Flux-managed)
• Woodpecker CI — CI/CD (Flux-managed)
• IRC (Ergo) — IRC server at irc.faun-snapper.ts.net:6667, tailnet-only via Tailscale L3 (Flux-managed)
• Evolved-NPCs — architecture plan for simulation workloads

Getting Started

Install Nix, then enter the dev shell (terraform, talosctl, kubectl, vault, qemu, podman — everything the repo needs):

nix develop

On the libvirt host itself (Fedora Asahi, aarch64), install the system deps once. This is the repo's one dnf install: packages are nix first, always (host/README.md), and nix cannot serve this one — libvirtd has to be the host's own system service, with Fedora's udev, polkit, SELinux and kernel-module integration, and Terraform drives qemu:///system. Everything the repo uses (qemu-img, virsh, talosctl) comes from the dev shell.

# nix cannot provide libvirtd as the host's system service: udev, polkit,
# SELinux and kernel-module integration, plus the qemu:///system Terraform drives.
sudo dnf install -y edk2-aarch64 libvirt qemu-kvm
sudo usermod -a -G libvirt "$USER" && newgrp libvirt
sudo systemctl enable --now libvirtd

Terraform drives qemu:///system as your libvirt-group user — no sudo in the day-to-day workflow.

Host stability setup (once per Mac mini)

Every machine this repo runs on is an Apple Silicon Mac mini, and its only built-in network path is the Broadcom BCM4388 wifi driven by brcmfmac. That firmware can wedge mid-session (upstream AsahiLinux/linux#494 — still open): the interface keeps its IP but stops carrying traffic, so SSH, ping and the tailnet all go dark and the host looks frozen. It is not — the kernel is fine and the next reboot is clean. Confirm a wedge with journalctl -k | grep -c commonring, which climbs by ~20/min while it lasts. A second, unrelated bug makes the clock jump ~7 h forward at every boot.

Do all of this once per host, right after the libvirt deps above. In the commands, $WIFI is resolved from the active wifi connection so the same block works on any machine.

1. Prefer wired Ethernet. The mini's gigabit port (end0) is far more reliable than the wifi, and takes the radio out of the path for the host and the cluster VMs, whose NAT-bridge traffic rides the same chip. Plug it in — no configuration needed. Everything below keeps the wifi usable; this is what makes the host dependably reachable.

2. Disable wifi power save — power management is implicated in the firmware stalls:

WIFI=$(nmcli -t -f NAME,TYPE connection show --active \
        | awk -F: '$2=="802-11-wireless"{print $1; exit}')
sudo nmcli connection modify "$WIFI" 802-11-wireless.powersave 2   # 2 = off
sudo iw dev wlp1s0f0 set power_save off                            # immediate

3. Stop NetworkManager's MAC-randomisation churn. NetworkManager rewrites the MAC for every scan (the rewrite itself fails on this chip — set-hw-addr: failed …) and for every connect, hammering the firmware for nothing. Pin the profile to a fixed MAC and turn scan randomisation off:

sudo nmcli connection modify "$WIFI" 802-11-wireless.cloned-mac-address permanent
printf '[device]\nwifi.scan-rand-mac-address=no\n' \
  | sudo tee /etc/NetworkManager/conf.d/10-no-rand-mac.conf
sudo systemctl reload NetworkManager
sudo nmcli connection up "$WIFI"     # reactivate; wifi blips for a second

permanent uses the hardware MAC; use stable instead if you would rather not expose it (both stop the churn). Do not reach for the deprecated 802-11-wireless.mac-address-randomization: its enum is default(0) / never(1) / always(2), so the obvious-looking 2 sets always and drags cloned-mac-address back to random. Changing the MAC changes the DHCP lease — update any reservation that pins this host's address.

4. Install the wifi watchdog. There is one, and it lives in the brcmf repository now — this repo no longer carries its sources, because it belongs to the wifi driver rather than to the cluster. That README has the install steps (four files, sudo install, then systemctl enable --now wifi-watchdog.timer). It pings the default gateway every 30 s and escalates — reconnect, reload brcmfmac, reboot — so a wedge recovers itself instead of waiting for you.

5. Enable systemd's runtime watchdog so a genuine hard hang reboots the host instead of hanging forever (the shutdown watchdog is already armed; the runtime one is not):

sudo install -d /etc/systemd/system.conf.d          # the drop-in dir may not exist
printf '[Manager]\nRuntimeWatchdogSec=30s\n' \
  | sudo tee /etc/systemd/system.conf.d/10-watchdog.conf
sudo systemctl daemon-reexec

6. The 7-hour clock jump. If the hardware clock (macsmc-rtc) is kept in local time, the kernel still reads it as UTC at boot — so the clock starts one UTC-offset behind (25200 s here) and chronyd steps it forward a few seconds later. The clock is correct from then on; only pre-sync timestamps and last are affected. Check what the RTC actually holds:

cat /sys/class/rtc/rtc0/since_epoch; date +%s

Equal means the RTC is already UTC and there is nothing to do. If they differ by your UTC offset, the RTC holds local time; converting it is optional:

sudo timedatectl set-local-rtc 0          # declare the RTC is UTC
sudo hwclock --systohc --utc              # rewrite it in UTC
cat /sys/class/rtc/rtc0/since_epoch; date +%s   # re-check (see below)

timedatectl cannot confirm this on its own: it prints "RTC time" converted into local time, so it reads the same either way. Always compare since_epoch with date +%s.

If the write does not land, put the setting back with sudo timedatectl set-local-rtc 1 and accept the boot-time step — a RTC in local TZ: no next to a local-time RTC only makes /etc/adjtime lie about the hardware, which is what misleads hwclock/timedatectl later.

Measured on this host (M2 Mac mini, apple,j473) — the change does not stick. sudo hwclock --systohc --utc does reach the chip (--verbose ends ioctl(RTC_SET_TIME) was successful, and since_epoch momentarily equals date +%s), but the RTC is back in local time within ~1 minute, so the boot step returns. This is not chronyd: its RTC driver reports 513 RTC driver not running (despite rtcsync in /etc/chrony.conf), it logs nothing RTC-related, no systemd-hwclock.service exists, and nothing holds /dev/rtc0 open — the reversion looks like the SMC firmware reasserting its own value. The practical end state for this machine is therefore RTC in local TZ: yes (matching the hardware) plus the chronyd step at boot, which is cosmetic: the clock is exact once NTP lands, and only pre-sync timestamps and last are affected.

7. The external data disk is what the cluster runs from. Every VM disk lives on a USB drive (label Milner_Data) mounted at /mnt/data — the data_root_path pinned in control-plane/terraform.auto.tfvars points into it, and libvirt's talos-images pool is /mnt/data/milner/images/talos-kvm. The domains are autostart = true, but if the mount has not happened by the time virtstoraged starts, the pool cannot activate and both VMs stay shut off. Nothing retries, and nofail means the boot still looks clean. Diagnose it with:

findmnt /mnt/data                     # empty = not mounted
systemctl status mnt-data.mount
journalctl -b | grep -E 'mnt-data|talos-images|talos-kvm'
virsh -c qemu:///system list --all    # both domains `shut off`

On this host that was a race, not a fault: the drive enumerates at ~5.0 s and the fstab entry allowed exactly 5 s (x-systemd.device-timeout=5), so the device job timed out in the same second the disk appeared. Recover in order:

sudo systemctl start mnt-data.mount            # or: sudo mount /mnt/data
virsh -c qemu:///system pool-start talos-images
terraform -chdir=control-plane apply           # running = true starts the CP VM
terraform -chdir=worker apply                  # then the worker

Keep it from recurring by giving the USB drive time and making libvirt wait for the mount. Raise the timeout on the /mnt/data line in /etc/fstab to nofail,x-systemd.device-timeout=90, then:

for u in virtstoraged virtqemud; do
  sudo install -d /etc/systemd/system/$u.service.d
  printf '[Unit]\nRequiresMountsFor=/mnt/data\n' \
    | sudo tee /etc/systemd/system/$u.service.d/10-wait-for-data.conf
done
sudo systemctl daemon-reload

RequiresMountsFor makes the mount a hard dependency, so a missing data disk stops libvirt loudly instead of silently leaving the cluster down. The failure above is observed on this host; the ordering fix is not yet confirmed across a reboot.

Credentials

Copy the template and source it before working (never commit the result):

cp .env.example .env && chmod 600 .env
set -a && . ./.env && set +a

.env.example lists every credential used so far — Tailscale operator OAuth client, Vault runbook token (scoped, one year; root stays in your password manager — docs/vault.md §4h), Woodpecker/Forgejo OAuth app — with where each is created and used (OPERATIONS.md). .env is gitignored.

Control plane runbook

control-plane/ holds the entire Talos control plane as one Terraform graph: the libvirt NAT network, storage pool, installer-ISO and OS-disk volumes, the milner-cp-* VMs, and the Talos install/bootstrap + kubeconfig/talosconfig outputs.

control-plane/terraform.tfvars lists every tunable knob. Copy any value that must differ on a given host into the gitignored control-plane/terraform.auto.tfvars (on this host: an external data_root_path and installer_iso = false). Run everything from the repo root:

# --- 1. Fresh install (from a clean state; first apply boots the installer
#        ISO and bootstraps the cluster) ---
terraform -chdir=control-plane init
terraform -chdir=control-plane plan      # review every diff before applying
terraform -chdir=control-plane apply

# --- 2. Cluster is healthy: eject the installer ISO permanently ---
# Pin installer_iso = false in the gitignored host file (idempotent; keeps
# any other pins such as data_root_path):
touch control-plane/terraform.auto.tfvars
sed -i '/^installer_iso =/d' control-plane/terraform.auto.tfvars
echo 'installer_iso = false' >> control-plane/terraform.auto.tfvars
terraform -chdir=control-plane plan      # cdrom + ISO gone, hd boots first
terraform -chdir=control-plane apply     # node reboots once onto its disk

# --- 3. Daily use ---
./control-plane/use-talos.sh             # refresh the `milner` context in ~/.talos/config
talosctl health                          # other clusters' contexts untouched
talosctl version
./control-plane/use-kubectl.sh           # merge the current kubeconfig into ~/.kube/config
kubectl get nodes                         # (context admin@milner is made current)

# --- 4. Apply a change or patch (edit control-plane/main.tf or
#        terraform.tfvars / terraform.auto.tfvars first, then reconcile) ---
terraform -chdir=control-plane plan
terraform -chdir=control-plane apply

# --- 5. Replace a broken VM — the domain only, never its disk/network/pool ---
terraform -chdir=control-plane plan -replace='libvirt_domain.control_plane[0]'
terraform -chdir=control-plane apply -replace='libvirt_domain.control_plane[0]'

# --- 6. Full teardown (also the way back to step 1 for re-provisioning) ---
#     Order: worker -> preserve -> control-plane (docs/durable-volumes.md §11c).
./worker/cleanup.sh                      # type `destroy`: worker VM first (it
                                         # holds volumes in the pool the sweep
                                         # is about to delete)
./data/cleanup.sh                        # type `preserve`: moves the durable
                                         # qcow2s out of the pool (never deletes)
./control-plane/cleanup.sh               # type `destroy` when prompted

Re-provision from scratch (after step 6)

Reinstate the installer ISO, then fresh-install as in step 1; afterwards eject it again with step 2:

sed -i '/^installer_iso =/d' control-plane/terraform.auto.tfvars
echo 'installer_iso = true' >> control-plane/terraform.auto.tfvars
terraform -chdir=control-plane plan
terraform -chdir=control-plane apply

Worker node runbook

worker/ holds the Talos worker-large VM as one Terraform graph: the OS-disk volume, the milner-worker-1 domain attached to the control-plane NAT network, and the Talos machine config apply that joins it to the live cluster.

The libvirt network and storage pool are owned by the control-plane root — the worker root references them by name only and never re-declares them. It does declare its own installer-ISO volume (talos-metal-arm64-worker.iso, a distinct name from the CP's) so the first install and later eject stay independent of the CP root. The two durable data disks it attaches are owned by data/, which sits between the CP and worker roots.

# --- 1. Apply control-plane root first (must publish secrets + DHCP reservation) ---
terraform -chdir=control-plane apply

# --- 2. Apply the data root: the two durable disks the worker attaches ---
terraform -chdir=data init
terraform -chdir=data plan
terraform -chdir=data apply
#    (after a teardown with preserved data, `data/restore.sh` replaces this)

# --- 3. Apply worker root ---
terraform -chdir=worker init
terraform -chdir=worker plan
terraform -chdir=worker apply

# --- 4. Verify ---
./control-plane/use-kubectl.sh           # merge current kubeconfig into ~/.kube/config
kubectl get nodes -o wide                # (context admin@milner is made current)
kubectl get nodes --show-labels

# --- 5. Eject installer ISO after first install ---
touch worker/terraform.auto.tfvars
echo 'installer_iso = false' >> worker/terraform.auto.tfvars
terraform -chdir=worker plan
terraform -chdir=worker apply     # node reboots once onto its disk

# --- 6. Replace a broken worker VM ---
terraform -chdir=worker plan  -replace='libvirt_domain.worker'
terraform -chdir=worker apply -replace='libvirt_domain.worker'

# --- 7. Full teardown (worker -> preserve -> control-plane; docs/durable-volumes.md §11c) ---
./worker/cleanup.sh              # type `destroy` when prompted
./data/cleanup.sh                # type `preserve` (must precede the CP sweep)
./control-plane/cleanup.sh       # type `destroy` when prompted (sweeps the pool)

Postgres runbook

Postgres is a Flux-managed app: the CloudNativePG operator lives in gitops/clusters/milner/infrastructure/cloudnative-pg/ and the single-instance Cluster in gitops/clusters/milner/data/postgres/. Its 20 GiB PVC is served by the persistent StorageClass from the Flux-managed local-path provisioner (gitops/clusters/milner/infrastructure/local-path/), on the worker's shared data disk.

# --- 1. Worker root + storage tier (shared data disk + StorageClasses) ---
# The apply attaches the shared data disk; the VM reboots once and Talos
# mounts /var/mnt/data at boot (docs/storage.md §10b).
terraform -chdir=worker apply
# StorageClasses (persistent/ephemeral) are Flux-managed: ensure the
# infrastructure Kustomization is Ready (docs/storage.md §10c) before the
# data layer claims the PVC.

# --- 2. Postgres: applied by Flux with the data layer ---
# ./gitops/install.sh hands the tree to Flux; the operator and the Cluster are
# ordinary manifests, so there is no terraform -chdir=postgres step any more
# (docs/postgres.md §13a).

# --- 3. Verify ---
kubectl get pods,pvc -n postgres
kubectl -n postgres get secret postgres-app -o jsonpath='{.data.password}' | base64 -d

Out-of-band operations (not in Terraform)

OPERATIONS.md is the runbook and audit trail for every live change Terraform does not own: the Tailscale admin-console setup (ACL tags, OAuth client, service approval), the imperative tailscale-operator bootstrap (installed by hand because its credentials must never reach Terraform state), Vault init/unseal, and client daemon flags. Vault itself is exposed at https://vault.faun-snapper.ts.net by the operator — its namespace, Helm release, and Ingress are Terraform-owned, but the operator and the tailnet approvals that make the name routable are not.

How Changes Ship

In-cluster workloads (Vault, Pocket ID, Woodpecker, the Zot registry, storage) are managed by Flux v2 GitOps: edit a manifest under gitops/clusters/milner/, commit, and push. Flux pulls the change and reconciles within the sync interval. Rollback is git revert.

A change is finished when every story in USER_STORIES.md holds true — that file is the specification, and everything else in this repository is how it is kept.

Every pull request also gets a preview: L3 deploys it into its own preview-<n> namespace, preview.faun-snapper.ts.net serves whichever preview is selected at the root path, and the OIDC-gated console at oz.faun-snapper.ts.net selects, rebuilds and deletes them. All of it is Flux-owned under preview-system/; the runbook, the teardown paths and the preview drill are docs/preview-environment.md §11h.

Infrastructure that Flux cannot own (VMs, disks, Cloudflare) ships as reviewed Terraform deltas: one plan/apply reconciles exactly the declared change.

Layer	Managed by	In-place delta?	Recovery
VM, disk, libvirt network/pool	Terraform	No	-replace the domain only (runbook step 5)
Durable data volumes (data/)	Terraform	Never destroyed	data/cleanup.sh then data/restore.sh (runbook docs/durable-volumes.md §11c)
Talos machine configuration	Terraform	Yes	Reconcile (runbook step 4); stage config, never reset
Cloudflare Tunnel / WAF	Terraform	Yes	Reconcile the root
In-cluster workloads	Flux	Yes	git revert; Flux re-reconciles

Every disposable resource (VM, disk, libvirt network/pool, installer ISO) is ephemeral: the network/pool are definitions, and the volumes are a re-downloadable ISO and a disposable OS disk, so a plain terraform destroy or cleanup.sh removes them.

The two durable volumes are the exception, and they are handled by the mechanism this repo committed to before they existed: they live in their own data/ root (creation included, so a fresh install is still pure Terraform), worker/main.tf keeps removed { lifecycle { destroy = false } } blocks recording the handover, plus prevent_destroy guards that refuse a deliberate data/ destroy. Teardown never names that root; data/cleanup.sh additionally moves the qcow2s beside the pool before the pool sweep, and data/restore.sh puts them back and proves Terraform still recognises them. Runbook: docs/durable-volumes.md §11c.

control-plane/cleanup.sh has exactly one purpose — full cluster teardown — and is never used for partial recovery. Take the worker down first, then run data/cleanup.sh, before it: the CP sweep rm -rfs the pool directory, so a running worker would lose its OS-disk file and an unpreserved data volume would go with the pool. The script then runs the reviewed terraform destroy, removes leftover files under the pool directory (escalating to sudo rm -rf when the qemu:libvirt-owned directory blocks an unprivileged removal; the path comes from libvirt's live pool definition, not data_root_path) and undefines the pool.

Always:

• Review every plan before applying; never apply without user confirmation.
• Never commit Terraform state or secrets.
• Never hand-run virsh/qemu-img for infrastructure lifecycle.

Origin Repos

• evolved-npcs — the application this cluster supports