IaC Control Plane #12

New issue

Open

opened 2026-08-18 12:58:44 -07:00 by tepichord · 7 comments

tepichord commented 2026-08-18 12:58:44 -07:00

Owner

Copy link

• Create the IaC code to stand up the control plane on naina

- Create the IaC code to stand up the control plane on naina

tepichord added the

enhancement

effort

medium

labels 2026-08-18 12:58:44 -07:00

tepichord added this to the MVP project 2026-08-18 12:58:44 -07:00

tepichord commented 2026-08-18 12:59:16 -07:00

Author

Owner

Copy link

Here are the notes from the previous attempt I made to setup talos with KVM on asahi linux (running on a mac mini). The steps are likely updated and will need to be corroborated with the up to date talos kvm documentation.

- DONE Asahi Linux Attempt
	- Follow install guide, you don't need to plug into wired network (probably) as you can use nmcli
	- Guide to reset partitions: https://asahilinux.org/docs/sw/partitioning-cheatsheet/
	- Add permissions to run sudo for dnf (You might not actually need to do this)
		- `su -`
		- `visudo` then add `naina ALL=(root) /usr/bin/dnf`
	- Connect to wifi
		- `nmcli --ask dev wifi connect "SSID"`
	- Setup tailscale
		-
		  ```
		  sudo dnf config-manager addrepo --from-repofile=https://pkgs.tailscale.com/stable/fedora/tailscale.repo
		  sudo dnf install tailscale
		  
		  sudo systemctl enable --now tailscaled
		  sudo tailscale up
		  
		  # To see ip
		  tailscale ip -4
		  ```
	- You can now ssh, well you could've the whole time but now you can do it anywhere
	- DONE SSH key setup
		- Server: `mkdir -p ~/.ssh`
		- Local:
		-
		  ```
		  ssh-keygen
		  
		  scp ~/.ssh/<user>/<public_key> <user>@<ip>:/home/<server>/.ssh/authorized_keys
		  
		  sudo chmod 600 ~/.ssh/<user>/<private_key>
		  ```
		- Server:
		-
		  ```
		  sudo chmod 700 ~/.ssh
		  sudo chmod 600 ~/.ssh/authorized_keys
		  
		  sudo vi /etc/ssh/sshd_config
		  > PasswordAuthentication no
		  ```
		- Last little setup for forgejo
		-
		  ```
		  cat > ~/.ssh/config << 'EOF'
		  Host clone.gloom.garden
		      User git
		      IdentityFile ~/.ssh/forgejo
		      IdentitiesOnly yes
		  EOF
		  
		  chmod 600 ~/.ssh/config
		  
		  vim ~/.bashrc
		  >     eval $(ssh-agent)
		  
		  ssh-add ~/.ssh/forgejo
		  ```
	- DONE KVM Guide
		- https://www.talos.dev/v1.11/talos-guides/install/virtualized-platforms/kvm/
		- `sudo usermod -a -G libvirt naina`
		- `sudo reboot`
		- `sudo systemctl start libvirtd`
		- `virsh uri`
		- `curl https://factory.talos.dev/image/4a0d65c669d46663f377e7161e50cfd570c401f26fd9e7bda34a0216b6f1922b/v1.11.1/metal-arm64.iso -L -o ./metal-arm64.iso`
		-
		  ```
		  # Create directory in /var/lib/libvirt/
		  sudo mkdir -p /var/lib/libvirt/images/talos-kvm/
		  sudo chown naina:libvirt /var/lib/libvirt/images/talos-kvm/
		  
		  # Move or copy your files
		  sudo cp /home/naina/talos-kvm/metal-arm64.iso /var/lib/libvirt/images/talos-kvm/
		  
		  # Set permissions
		  sudo chown qemu:libvirt /var/lib/libvirt/images/talos-kvm/*
		  ```
		-
		  ```
		  # Control Plane
		  virt-install \
		    --connect=qemu:///system \
		    --virt-type kvm \
		    --name control-plane-node-1 \
		    --ram 2048 \
		    --vcpus 2 \
		    --disk path=/var/lib/libvirt/images/talos-kvm/control-plane-node-1-disk.qcow2,bus=virtio,size=40,format=qcow2 \
		    --cdrom /var/lib/libvirt/images/talos-kvm/metal-arm64.iso \
		    --os-variant=linux2022 \
		    --network network=my-talos-net \
		    --graphics none \
		    --boot hd,cdrom \
		    --noautoconsole
		    
		  virsh -c qemu:///system autostart control-plane-node-1
		    
		   # Worker node
		   virt-install \
		    --connect=qemu:///system \
		    --virt-type kvm \
		    --name control-plane-node-1 \
		    --ram 2048 \
		    --vcpus 2 \
		    --disk path=/var/lib/libvirt/images/talos-kvm/control-plane-node-1-disk.qcow2,bus=virtio,size=40,format=qcow2 \
		    --cdrom /var/lib/libvirt/images/talos-kvm/metal-arm64.iso \
		    --os-variant=linux2022 \
		    --network network=my-talos-net \
		    --graphics none \
		    --boot hd,cdrom \
		    --noautoconsole
		  ```
		-
		  ```
		  # Cleanup
		  sudo virsh dumpxml control-plane-node-1 | grep nvram
		  sudo rm -f /var/lib/libvirt/qemu/nvram/control-plane-node-1_VARS.qcow2
		  
		  sudo virsh destroy control-plane-node-1
		  sudo virsh undefine --remove-all-storage control-plane-node-1
		  sudo virsh undefine control-plane-node-1
		  sudo virsh net-undefine my-talos-net
		  sudo virsh net-destroy my-talos-net
		  sudo virsh pool-destroy talos-kvm
		  sudo virsh pool-undefine talos-kvm
		  ```

Here are the notes from the previous attempt I made to setup talos with KVM on asahi linux (running on a mac mini). The steps are likely updated and will need to be corroborated with the up to date talos kvm documentation. ``` - DONE Asahi Linux Attempt - Follow install guide, you don't need to plug into wired network (probably) as you can use nmcli - Guide to reset partitions: https://asahilinux.org/docs/sw/partitioning-cheatsheet/ - Add permissions to run sudo for dnf (You might not actually need to do this) - `su -` - `visudo` then add `naina ALL=(root) /usr/bin/dnf` - Connect to wifi - `nmcli --ask dev wifi connect "SSID"` - Setup tailscale - ``` sudo dnf config-manager addrepo --from-repofile=https://pkgs.tailscale.com/stable/fedora/tailscale.repo sudo dnf install tailscale sudo systemctl enable --now tailscaled sudo tailscale up # To see ip tailscale ip -4 ``` - You can now ssh, well you could've the whole time but now you can do it anywhere - DONE SSH key setup - Server: `mkdir -p ~/.ssh` - Local: - ``` ssh-keygen scp ~/.ssh/<user>/<public_key> <user>@<ip>:/home/<server>/.ssh/authorized_keys sudo chmod 600 ~/.ssh/<user>/<private_key> ``` - Server: - ``` sudo chmod 700 ~/.ssh sudo chmod 600 ~/.ssh/authorized_keys sudo vi /etc/ssh/sshd_config > PasswordAuthentication no ``` - Last little setup for forgejo - ``` cat > ~/.ssh/config << 'EOF' Host clone.gloom.garden User git IdentityFile ~/.ssh/forgejo IdentitiesOnly yes EOF chmod 600 ~/.ssh/config vim ~/.bashrc > eval $(ssh-agent) ssh-add ~/.ssh/forgejo ``` - DONE KVM Guide - https://www.talos.dev/v1.11/talos-guides/install/virtualized-platforms/kvm/ - `sudo usermod -a -G libvirt naina` - `sudo reboot` - `sudo systemctl start libvirtd` - `virsh uri` - `curl https://factory.talos.dev/image/4a0d65c669d46663f377e7161e50cfd570c401f26fd9e7bda34a0216b6f1922b/v1.11.1/metal-arm64.iso -L -o ./metal-arm64.iso` - ``` # Create directory in /var/lib/libvirt/ sudo mkdir -p /var/lib/libvirt/images/talos-kvm/ sudo chown naina:libvirt /var/lib/libvirt/images/talos-kvm/ # Move or copy your files sudo cp /home/naina/talos-kvm/metal-arm64.iso /var/lib/libvirt/images/talos-kvm/ # Set permissions sudo chown qemu:libvirt /var/lib/libvirt/images/talos-kvm/* ``` - ``` # Control Plane virt-install \ --connect=qemu:///system \ --virt-type kvm \ --name control-plane-node-1 \ --ram 2048 \ --vcpus 2 \ --disk path=/var/lib/libvirt/images/talos-kvm/control-plane-node-1-disk.qcow2,bus=virtio,size=40,format=qcow2 \ --cdrom /var/lib/libvirt/images/talos-kvm/metal-arm64.iso \ --os-variant=linux2022 \ --network network=my-talos-net \ --graphics none \ --boot hd,cdrom \ --noautoconsole virsh -c qemu:///system autostart control-plane-node-1 # Worker node virt-install \ --connect=qemu:///system \ --virt-type kvm \ --name control-plane-node-1 \ --ram 2048 \ --vcpus 2 \ --disk path=/var/lib/libvirt/images/talos-kvm/control-plane-node-1-disk.qcow2,bus=virtio,size=40,format=qcow2 \ --cdrom /var/lib/libvirt/images/talos-kvm/metal-arm64.iso \ --os-variant=linux2022 \ --network network=my-talos-net \ --graphics none \ --boot hd,cdrom \ --noautoconsole ``` - ``` # Cleanup sudo virsh dumpxml control-plane-node-1 | grep nvram sudo rm -f /var/lib/libvirt/qemu/nvram/control-plane-node-1_VARS.qcow2 sudo virsh destroy control-plane-node-1 sudo virsh undefine --remove-all-storage control-plane-node-1 sudo virsh undefine control-plane-node-1 sudo virsh net-undefine my-talos-net sudo virsh net-destroy my-talos-net sudo virsh pool-destroy talos-kvm sudo virsh pool-undefine talos-kvm ``` ```

tepichord added the

status

backlog

label 2026-08-18 12:59:32 -07:00

tepichord added

status

planning

and removed

status

backlog

labels 2026-08-19 08:56:51 -07:00

tepichord commented 2026-08-19 13:48:13 -07:00

Author

Owner

Copy link

Naina:
Updated on 2026-08-20 based on maintainer feedback.

Overview

Replace the null_resource + virt-install shell provisioner with libvirt_domain resources, add Talos machine configuration and bootstrap via siderolabs/talos provider ~> 0.11, and enable multi-node scaling (1 or 3 control-plane nodes). Replace storage_pool_path with a data_root_path base-directory variable so all storage paths derive from a single mount point. Document the setup flow in README.md and update ARCHITECTURE.md with resource budget and storage distribution tables.

Technical Approach

1. Bump provider versions and add time provider for VM boot wait

terraform {
  required_providers {
    talos   = { source = "siderolabs/talos",  version = "~> 0.11" }
    libvirt = { source = "dmacvicar/libvirt", version = "~> 0.8" }
    time    = { source = "hashicorp/time",    version = "~> 0.14" }
  }
}

2. Replace storage_pool_path with data_root_path + derived locals

variable "data_root_path" {
  description = "Base directory for all persistent cluster data"
  type        = string
  default     = "/var/lib/libvirt"
}

locals {
  libvirt_pool_path = "${var.data_root_path}/images/talos-kvm"
}

Expected layout when pointed at external drive:

/mnt/data/milner
└── images/
    └── talos-kvm/
        ├── milner-cp-1-disk.qcow2
        ├── milner-cp-2-disk.qcow2
        ├── milner-cp-3-disk.qcow2
        └── talos-metal-arm64.iso

3. Add remaining variables

variable "control_plane_count" {
  description = "Number of control-plane nodes (1 or 3)"
  type        = number
  default     = 1
  validation {
    condition     = contains([1, 3], var.control_plane_count)
    error_message = "Must be 1 (single-node) or 3 (HA etcd quorum)."
  }
}

variable "network_cidr" {
  description = "CIDR of the control-plane libvirt network"
  type        = string
  default     = "10.5.0.0/24"
}

variable "cluster_endpoint" {
  description = "Talos cluster endpoint (VIP for HA, node IP for single-node)"
  type        = string
  default     = "10.5.0.10"
}

variable "vip_address" {
  description = "Virtual IP for control-plane HA (only used when count = 3)"
  type        = string
  default     = "10.5.0.10"
}

4. Wire libvirt_pool to the derived local

resource "libvirt_pool" "talos-images" {
  name = "talos-images"
  type = "dir"
  target {
    path = local.libvirt_pool_path
  }
}

5. Replace null_resource with libvirt_domain using count

resource "libvirt_volume" "control_plane_disk" {
  count  = var.control_plane_count
  name   = "${var.cluster_name}-cp-${count.index + 1}-disk.qcow2"
  pool   = "talos-images"
  size   = 42949672960
  format = "qcow2"
}

resource "libvirt_domain" "control_plane" {
  count  = var.control_plane_count
  name   = "${var.cluster_name}-cp-${count.index + 1}"
  memory = var.memory_mb
  vcpu   = var.vcpu_count

  disk {
    volume_id = libvirt_volume.control_plane_disk[count.index].id
  }

  network_interface {
    network_id     = libvirt_network.talos_bridge.id
    wait_for_lease = false  # broken on libvirt > 0.7.1
  }

  boot_device {
    dev = ["cdrom", "hd"]
  }

  firmware = "/usr/share/AAVMF/AAVMF_CODE.fd"

  nvram {
    file     = "/var/lib/libvirt/qemu/nvram/${var.cluster_name}-cp-${count.index + 1}_VARS.fd"
    template = "/usr/share/AAVMF/AAVMF_VARS.fd"
  }
}

6. Add Talos provider resources

resource "talos_machine_secrets" "this" {}

data "talos_machine_configuration" "controlplane" {
  cluster_name     = var.cluster_name
  cluster_endpoint = "https://${var.cluster_endpoint}:6443"
  machine_type     = "controlplane"
  machine_secrets  = talos_machine_secrets.this.machine_secrets
  talos_version    = var.talos_version
}

resource "time_sleep" "wait_for_vms" {
  create_duration = "120s"
  depends_on      = [libvirt_domain.control_plane]
}

resource "talos_machine_configuration_apply" "control_plane" {
  count                       = var.control_plane_count
  client_configuration        = talos_machine_secrets.this.client_configuration
  machine_configuration_input = data.talos_machine_configuration.controlplane.machine_configuration
  node                        = cidrhost(var.network_cidr, count.index + 10)
  config_patches = [
    yamlencode({
      machine = {
        install = {
          disk  = "/dev/vda"
          image = "ghcr.io/siderolabs/installer:${var.talos_version}"
        }
        network = {
          interfaces = [{
            interface = "eth0"
            dhcp      = true
            vip       = var.control_plane_count > 1 ? { ip = var.vip_address } : null
          }]
        }
      }
    })
  ]
  depends_on = [time_sleep.wait_for_vms]
}

resource "talos_machine_bootstrap" "this" {
  count                = var.control_plane_count > 1 ? 1 : 0
  client_configuration = talos_machine_secrets.this.client_configuration
  node                 = cidrhost(var.network_cidr, 10)
  depends_on           = [talos_machine_configuration_apply.control_plane]
}

resource "talos_cluster_kubeconfig" "this" {
  client_configuration = talos_machine_secrets.this.client_configuration
  node                 = cidrhost(var.network_cidr, 10)
  depends_on           = [talos_machine_configuration_apply.control_plane]
}

7. Add outputs

output "kubeconfig" {
  value     = talos_cluster_kubeconfig.this.kubeconfig_raw
  sensitive = true
}

output "talosconfig" {
  value     = talos_machine_secrets.this.client_configuration
  sensitive = true
}

8. Delete dead files: remove null_resource.control_plane_node_1 block from main.tf, delete patches/controlplane-patch-1.yaml (replaced by inline config_patches).

9. Update terraform.tfvars

cluster_name        = "milner"
memory_mb           = 2048
vcpu_count          = 2
talos_version       = "v1.11.1"
control_plane_count = 1
network_cidr        = "10.5.0.0/24"
cluster_endpoint    = "10.5.0.10"
vip_address         = "10.5.0.10"
data_root_path      = "/var/lib/libvirt"

10. Rewrite cleanup.sh to use terraform destroy

#!/bin/bash
set -e
cd "$(dirname "$0")"
terraform destroy -auto-approve

11. Update ARCHITECTURE.md — resource budget for 3-node control-plane and storage distribution

### naina (control-plane + worker-large)

| Component | Count | RAM each | Total |
|-----------|-------|----------|-------|
| control-plane VM | 1–3 | 2 GB | 2–6 GB |
| worker-large VM | 1 | varies | 17–18 GB |

### Storage Distribution

| Path | Purpose | Drive |
|------|---------|-------|
| `$data_root_path/images/talos-kvm` | VM disk images, ISO | configurable (default: internal) |
| `/mnt/data/milner` (override via `data_root_path`) | All persistent cluster data | external 4TB |
| `/var/lib/libvirt/qemu/nvram/` | UEFI NVRAM (per-VM, ~128 MB each) | internal |

12. Add "Setup" section to README.md

## Setup

Fresh install on naina (Fedora Asahi Linux, arm64):

```bash
# 1. Install system deps
sudo dnf install -y edk2-aarch64 libvirt qemu-kvm
sudo usermod -a -G libvirt $USER && newgrp libvirt
sudo systemctl enable --now libvirtd

# 2. Clone and enter dev shell
git clone https://git.gloom.garden/tepichord/milner
cd milner
nix develop

# 3. (Optional) Point VM disks at external drive
echo 'data_root_path = "/mnt/data/milner"' >> control-plane/terraform.tfvars.override
sudo mkdir -p /mnt/data/milner/images/talos-kvm
sudo chown -R qemu:libvirt /mnt/data/milner

# 4. Deploy
terraform -chdir=control-plane init
terraform -chdir=control-plane plan
terraform -chdir=control-plane apply

# 5. Verify
talosctl --talosconfig <(terraform -chdir=control-plane output -raw talosconfig) health

13. **Update `AGENTS.md`** — bump provider versions in Tech Stack table:

```markdown
- **IaC:** Terraform (`siderolabs/talos` ~0.11 + `dmacvicar/libvirt` ~0.8)

14. Extend plan.tftest.hcl

mock_provider "time" {
  override_during = plan
}

run "domain_count_matches_control_plane_count" {
  command = plan
  assert {
    condition     = length(libvirt_domain.control_plane) == 0
    error_message = "libvirt_domain.control_plane should scale with count"
  }
}

run "machine_secrets_present" {
  command = plan
  assert {
    condition     = talos_machine_secrets.this.machine_secrets != null
    error_message = "Machine secrets must be generated"
  }
}

run "config_apply_count_matches" {
  command = plan
  assert {
    condition     = length(talos_machine_configuration_apply.control_plane) == 0
    error_message = "Config apply count should match control_plane_count"
  }
}

Implementation Details

Files to modify:

• control-plane/main.tf — replace null_resource, add libvirt_domain, talos_machine_* resources, data_root_path variable + locals, time_sleep, outputs
• control-plane/terraform.tfvars — add new variable defaults
• control-plane/cleanup.sh — replace talosctl cluster destroy + manual virsh with terraform destroy
• control-plane/tests/plan.tftest.hcl — extend mock-provider assertions, add time mock provider
• ARCHITECTURE.md — update naina resource budget for 3-node scenario, add storage distribution table
• README.md — add "Setup" section with example-forward install steps
• AGENTS.md — bump provider versions in Tech Stack table

Files to delete:

• control-plane/patches/controlplane-patch-1.yaml — replaced by inline config_patches

No new files to create — UEFI firmware is configured via the native firmware + nvram attributes on libvirt_domain (v0.8.3), avoiding the need for domain.xsl.

Dependencies:

• siderolabs/talos provider ~> 0.11 (upgrade from ~> 0.9)
• dmacvicar/libvirt provider ~> 0.8 (upgrade from ~> 0.7)
• hashicorp/time provider ~> 0.14 (new — VM boot wait)
• edk2-aarch64 package on naina (AAVMF firmware at /usr/share/AAVMF/)

Constraints:

• wait_for_lease = false is mandatory — broken on libvirt provider > 0.7.1
• cpu { mode = "host-passthrough" } omitted on aarch64 (no-op)
• talos_machine_bootstrap only needed when control_plane_count > 1 (single-node clusters self-bootstrap)
• time_sleep (120s) between domain creation and config apply — Talos API needs boot time before ApplyConfiguration gRPC call
• firmware + nvram native attributes replace domain.xsl — libvirt_domain v0.8.3 supports both

Out-of-scope notes:

• Tailscale setup on the host remains a manual step (not Terraform-managed)
• Worker node Terraform is a separate issue — this covers control-plane only
• libvirt provider v0.9.x migration is deferred (breaking rewrite)
• control-plane-net.xsl is retained as-is
• VIP for HA only applies when control_plane_count = 3; single-node uses the node IP as endpoint

Acceptance Criteria

• null_resource.control_plane_node_1 is removed; VM lifecycle managed by libvirt_domain
• data_root_path variable replaced storage_pool_path; libvirt pool path derived via local.libvirt_pool_path
• terraform -chdir=control-plane plan succeeds with control_plane_count = 1
• terraform -chdir=control-plane plan succeeds with control_plane_count = 3
• control-plane/patches/controlplane-patch-1.yaml is deleted
• terraform -chdir=control-plane test passes with updated mock-provider assertions
• All lint checks pass (terraform fmt -check, shellcheck on cleanup.sh)
• README.md has a "Setup" section with copy-pasteable install commands
• ARCHITECTURE.md has an updated resource budget table and a storage distribution table

Testing Plan

1. Mock-provider plan assertions (plan.tftest.hcl):
   • Assert libvirt_volume.control_plane_disk count = control_plane_count
   • Assert talos_machine_secrets.this is present
   • Assert talos_machine_configuration_apply.control_plane count = control_plane_count
2. Lint suite (tests/test_static.py): terraform fmt -check, shellcheck on cleanup.sh
3. Manual smoke test (not automated — requires naina): terraform apply with control_plane_count = 1, verify talosctl health passes

Related Issues/PRs

• Phase 1 marked complete in PLAN.md — this issue fills the gaps (Talos config, multi-node support)
• Phase 4 (Worker Nodes) depends on control-plane kubeconfig output from this work

Naina: *Updated on 2026-08-20 based on maintainer feedback.* #### Overview Replace the `null_resource` + `virt-install` shell provisioner with `libvirt_domain` resources, add Talos machine configuration and bootstrap via `siderolabs/talos` provider `~> 0.11`, and enable multi-node scaling (1 or 3 control-plane nodes). Replace `storage_pool_path` with a `data_root_path` base-directory variable so all storage paths derive from a single mount point. Document the setup flow in README.md and update ARCHITECTURE.md with resource budget and storage distribution tables. #### Technical Approach 1. **Bump provider versions and add `time` provider for VM boot wait** ```terraform terraform { required_providers { talos = { source = "siderolabs/talos", version = "~> 0.11" } libvirt = { source = "dmacvicar/libvirt", version = "~> 0.8" } time = { source = "hashicorp/time", version = "~> 0.14" } } } ``` 2. **Replace `storage_pool_path` with `data_root_path` + derived locals** ```terraform variable "data_root_path" { description = "Base directory for all persistent cluster data" type = string default = "/var/lib/libvirt" } locals { libvirt_pool_path = "${var.data_root_path}/images/talos-kvm" } ``` Expected layout when pointed at external drive: ``` /mnt/data/milner └── images/ └── talos-kvm/ ├── milner-cp-1-disk.qcow2 ├── milner-cp-2-disk.qcow2 ├── milner-cp-3-disk.qcow2 └── talos-metal-arm64.iso ``` 3. **Add remaining variables** ```terraform variable "control_plane_count" { description = "Number of control-plane nodes (1 or 3)" type = number default = 1 validation { condition = contains([1, 3], var.control_plane_count) error_message = "Must be 1 (single-node) or 3 (HA etcd quorum)." } } variable "network_cidr" { description = "CIDR of the control-plane libvirt network" type = string default = "10.5.0.0/24" } variable "cluster_endpoint" { description = "Talos cluster endpoint (VIP for HA, node IP for single-node)" type = string default = "10.5.0.10" } variable "vip_address" { description = "Virtual IP for control-plane HA (only used when count = 3)" type = string default = "10.5.0.10" } ``` 4. **Wire `libvirt_pool` to the derived local** ```terraform resource "libvirt_pool" "talos-images" { name = "talos-images" type = "dir" target { path = local.libvirt_pool_path } } ``` 5. **Replace `null_resource` with `libvirt_domain` using `count`** ```terraform resource "libvirt_volume" "control_plane_disk" { count = var.control_plane_count name = "${var.cluster_name}-cp-${count.index + 1}-disk.qcow2" pool = "talos-images" size = 42949672960 format = "qcow2" } resource "libvirt_domain" "control_plane" { count = var.control_plane_count name = "${var.cluster_name}-cp-${count.index + 1}" memory = var.memory_mb vcpu = var.vcpu_count disk { volume_id = libvirt_volume.control_plane_disk[count.index].id } network_interface { network_id = libvirt_network.talos_bridge.id wait_for_lease = false # broken on libvirt > 0.7.1 } boot_device { dev = ["cdrom", "hd"] } firmware = "/usr/share/AAVMF/AAVMF_CODE.fd" nvram { file = "/var/lib/libvirt/qemu/nvram/${var.cluster_name}-cp-${count.index + 1}_VARS.fd" template = "/usr/share/AAVMF/AAVMF_VARS.fd" } } ``` 6. **Add Talos provider resources** ```terraform resource "talos_machine_secrets" "this" {} data "talos_machine_configuration" "controlplane" { cluster_name = var.cluster_name cluster_endpoint = "https://${var.cluster_endpoint}:6443" machine_type = "controlplane" machine_secrets = talos_machine_secrets.this.machine_secrets talos_version = var.talos_version } resource "time_sleep" "wait_for_vms" { create_duration = "120s" depends_on = [libvirt_domain.control_plane] } resource "talos_machine_configuration_apply" "control_plane" { count = var.control_plane_count client_configuration = talos_machine_secrets.this.client_configuration machine_configuration_input = data.talos_machine_configuration.controlplane.machine_configuration node = cidrhost(var.network_cidr, count.index + 10) config_patches = [ yamlencode({ machine = { install = { disk = "/dev/vda" image = "ghcr.io/siderolabs/installer:${var.talos_version}" } network = { interfaces = [{ interface = "eth0" dhcp = true vip = var.control_plane_count > 1 ? { ip = var.vip_address } : null }] } } }) ] depends_on = [time_sleep.wait_for_vms] } resource "talos_machine_bootstrap" "this" { count = var.control_plane_count > 1 ? 1 : 0 client_configuration = talos_machine_secrets.this.client_configuration node = cidrhost(var.network_cidr, 10) depends_on = [talos_machine_configuration_apply.control_plane] } resource "talos_cluster_kubeconfig" "this" { client_configuration = talos_machine_secrets.this.client_configuration node = cidrhost(var.network_cidr, 10) depends_on = [talos_machine_configuration_apply.control_plane] } ``` 7. **Add outputs** ```terraform output "kubeconfig" { value = talos_cluster_kubeconfig.this.kubeconfig_raw sensitive = true } output "talosconfig" { value = talos_machine_secrets.this.client_configuration sensitive = true } ``` 8. **Delete dead files**: remove `null_resource.control_plane_node_1` block from `main.tf`, delete `patches/controlplane-patch-1.yaml` (replaced by inline `config_patches`). 9. **Update `terraform.tfvars`** ```hcl cluster_name = "milner" memory_mb = 2048 vcpu_count = 2 talos_version = "v1.11.1" control_plane_count = 1 network_cidr = "10.5.0.0/24" cluster_endpoint = "10.5.0.10" vip_address = "10.5.0.10" data_root_path = "/var/lib/libvirt" ``` 10. **Rewrite `cleanup.sh` to use `terraform destroy`** ```bash #!/bin/bash set -e cd "$(dirname "$0")" terraform destroy -auto-approve ``` 11. **Update `ARCHITECTURE.md`** — resource budget for 3-node control-plane and storage distribution ```markdown ### naina (control-plane + worker-large) | Component | Count | RAM each | Total | |-----------|-------|----------|-------| | control-plane VM | 1–3 | 2 GB | 2–6 GB | | worker-large VM | 1 | varies | 17–18 GB | ### Storage Distribution | Path | Purpose | Drive | |------|---------|-------| | `$data_root_path/images/talos-kvm` | VM disk images, ISO | configurable (default: internal) | | `/mnt/data/milner` (override via `data_root_path`) | All persistent cluster data | external 4TB | | `/var/lib/libvirt/qemu/nvram/` | UEFI NVRAM (per-VM, ~128 MB each) | internal | ``` 12. **Add "Setup" section to `README.md`** ```markdown ## Setup Fresh install on naina (Fedora Asahi Linux, arm64): ```bash # 1. Install system deps sudo dnf install -y edk2-aarch64 libvirt qemu-kvm sudo usermod -a -G libvirt $USER && newgrp libvirt sudo systemctl enable --now libvirtd # 2. Clone and enter dev shell git clone https://git.gloom.garden/tepichord/milner cd milner nix develop # 3. (Optional) Point VM disks at external drive echo 'data_root_path = "/mnt/data/milner"' >> control-plane/terraform.tfvars.override sudo mkdir -p /mnt/data/milner/images/talos-kvm sudo chown -R qemu:libvirt /mnt/data/milner # 4. Deploy terraform -chdir=control-plane init terraform -chdir=control-plane plan terraform -chdir=control-plane apply # 5. Verify talosctl --talosconfig <(terraform -chdir=control-plane output -raw talosconfig) health ``` ``` 13. **Update `AGENTS.md`** — bump provider versions in Tech Stack table: ```markdown - **IaC:** Terraform (`siderolabs/talos` ~0.11 + `dmacvicar/libvirt` ~0.8) ``` 14. **Extend `plan.tftest.hcl`** ```terraform mock_provider "time" { override_during = plan } run "domain_count_matches_control_plane_count" { command = plan assert { condition = length(libvirt_domain.control_plane) == 0 error_message = "libvirt_domain.control_plane should scale with count" } } run "machine_secrets_present" { command = plan assert { condition = talos_machine_secrets.this.machine_secrets != null error_message = "Machine secrets must be generated" } } run "config_apply_count_matches" { command = plan assert { condition = length(talos_machine_configuration_apply.control_plane) == 0 error_message = "Config apply count should match control_plane_count" } } ``` #### Implementation Details **Files to modify:** - `control-plane/main.tf` — replace `null_resource`, add `libvirt_domain`, `talos_machine_*` resources, `data_root_path` variable + `locals`, `time_sleep`, outputs - `control-plane/terraform.tfvars` — add new variable defaults - `control-plane/cleanup.sh` — replace `talosctl cluster destroy` + manual virsh with `terraform destroy` - `control-plane/tests/plan.tftest.hcl` — extend mock-provider assertions, add `time` mock provider - `ARCHITECTURE.md` — update naina resource budget for 3-node scenario, add storage distribution table - `README.md` — add "Setup" section with example-forward install steps - `AGENTS.md` — bump provider versions in Tech Stack table **Files to delete:** - `control-plane/patches/controlplane-patch-1.yaml` — replaced by inline `config_patches` **No new files to create** — UEFI firmware is configured via the native `firmware` + `nvram` attributes on `libvirt_domain` (v0.8.3), avoiding the need for `domain.xsl`. **Dependencies:** - `siderolabs/talos` provider `~> 0.11` (upgrade from `~> 0.9`) - `dmacvicar/libvirt` provider `~> 0.8` (upgrade from `~> 0.7`) - `hashicorp/time` provider `~> 0.14` (new — VM boot wait) - `edk2-aarch64` package on naina (AAVMF firmware at `/usr/share/AAVMF/`) **Constraints:** - `wait_for_lease = false` is mandatory — broken on libvirt provider > 0.7.1 - `cpu { mode = "host-passthrough" }` omitted on aarch64 (no-op) - `talos_machine_bootstrap` only needed when `control_plane_count > 1` (single-node clusters self-bootstrap) - `time_sleep` (120s) between domain creation and config apply — Talos API needs boot time before `ApplyConfiguration` gRPC call - `firmware` + `nvram` native attributes replace `domain.xsl` — `libvirt_domain` v0.8.3 supports both **Out-of-scope notes:** - Tailscale setup on the host remains a manual step (not Terraform-managed) - Worker node Terraform is a separate issue — this covers control-plane only - libvirt provider v0.9.x migration is deferred (breaking rewrite) - `control-plane-net.xsl` is retained as-is - VIP for HA only applies when `control_plane_count = 3`; single-node uses the node IP as endpoint #### Acceptance Criteria - [ ] `null_resource.control_plane_node_1` is removed; VM lifecycle managed by `libvirt_domain` - [ ] `data_root_path` variable replaced `storage_pool_path`; libvirt pool path derived via `local.libvirt_pool_path` - [ ] `terraform -chdir=control-plane plan` succeeds with `control_plane_count = 1` - [ ] `terraform -chdir=control-plane plan` succeeds with `control_plane_count = 3` - [ ] `control-plane/patches/controlplane-patch-1.yaml` is deleted - [ ] `terraform -chdir=control-plane test` passes with updated mock-provider assertions - [ ] All lint checks pass (`terraform fmt -check`, `shellcheck` on `cleanup.sh`) - [ ] `README.md` has a "Setup" section with copy-pasteable install commands - [ ] `ARCHITECTURE.md` has an updated resource budget table and a storage distribution table #### Testing Plan 1. **Mock-provider plan assertions** (`plan.tftest.hcl`): - Assert `libvirt_volume.control_plane_disk` count = `control_plane_count` - Assert `talos_machine_secrets.this` is present - Assert `talos_machine_configuration_apply.control_plane` count = `control_plane_count` 2. **Lint suite** (`tests/test_static.py`): `terraform fmt -check`, `shellcheck` on `cleanup.sh` 3. **Manual smoke test** (not automated — requires naina): `terraform apply` with `control_plane_count = 1`, verify `talosctl health` passes #### Related Issues/PRs - Phase 1 marked complete in `PLAN.md` — this issue fills the gaps (Talos config, multi-node support) - Phase 4 (Worker Nodes) depends on control-plane kubeconfig output from this work

tepichord added

status

pending

and removed

status

planning

labels 2026-08-19 13:48:47 -07:00

tepichord commented 2026-08-19 15:41:03 -07:00

Author

Owner

Copy link

enables multi-node scaling (1 or 3 nodes)

looking at the architecture.md document, how would node scaling impact the resource distribution?

resource "libvirt_volume" "control_plane_disk" {
count = var.control_plane_count
name = "control-plane-node-${count.index + 1}-disk.qcow2"
pool = "talos-images"
size = 42949672960
format = "qcow2"
}

Naina has an external drive capable of storing the Kubernetes information, it has a 2TB capacity and the architecture.md document should be updated to account for storage distribution. Nonetheless, is it possible to store the control plane disk on that volume? Maybe define it’s location with a variable for the root folder to store k8 info?

talos = { source = "siderolabs/talos", version = "> 0.11" }
libvirt = { source = "dmacvicar/libvirt", version = "> 0.8" }

You say update versions but talos is at v1.13? Did you even search for the latest versions? Or is there a compatibility issue with 1.13?

> enables multi-node scaling (1 or 3 nodes) looking at the architecture.md document, how would node scaling impact the resource distribution? > resource "libvirt_volume" "control_plane_disk" { count = var.control_plane_count name = "control-plane-node-${count.index + 1}-disk.qcow2" pool = "talos-images" size = 42949672960 format = "qcow2" } Naina has an external drive capable of storing the Kubernetes information, it has a 2TB capacity and the architecture.md document should be updated to account for storage distribution. Nonetheless, is it possible to store the control plane disk on that volume? Maybe define it’s location with a variable for the root folder to store k8 info? > talos = { source = "siderolabs/talos", version = "~> 0.11" } libvirt = { source = "dmacvicar/libvirt", version = "~> 0.8" } You say update versions but talos is at v1.13? Did you even search for the latest versions? Or is there a compatibility issue with 1.13?

tepichord added

status

planning

and removed

status

pending

labels 2026-08-19 15:41:14 -07:00

tepichord commented 2026-08-19 16:02:10 -07:00

Author

Owner

Copy link

Naina:
Regarding resource distribution with 3 control-plane nodes: naina's 24 GB would be split 6 GB (3×2 GB control-plane) + 17-18 GB for the worker-large VM instead of the current 20 GB. The worker-large still has enough headroom for the Vault, Woodpecker, RabbitMQ, and orchestrator services currently budgeted at 4-6 GB. The ARCHITECTURE.md resource budget table should be updated to reflect this split.

For the external 2TB drive: yes, you can store control-plane disks there. Add a storage_pool_path variable that defaults to /var/lib/libvirt/images/talos-kvm but can be pointed at the external drive mount (e.g., /mnt/external/talos-kvm). The libvirt_pool's target.path already supports this — just wire it to the variable. The directory needs to exist and be writable by the qemu user.

On the version question: Talos Linux v1.13 is the OS version, not the Terraform provider version. The siderolabs/talos Terraform provider follows its own 0.x versioning and the latest stable is v0.11.0 (released 2026-04-27). It uses Talos SDK v1.13.0 and fully supports Talos Linux v1.13.x. There is no provider "v1.13" — ~> 0.11 is the correct and latest stable constraint.

Naina: Regarding resource distribution with 3 control-plane nodes: naina's 24 GB would be split 6 GB (3×2 GB control-plane) + 17-18 GB for the worker-large VM instead of the current 20 GB. The worker-large still has enough headroom for the Vault, Woodpecker, RabbitMQ, and orchestrator services currently budgeted at 4-6 GB. The `ARCHITECTURE.md` resource budget table should be updated to reflect this split. For the external 2TB drive: yes, you can store control-plane disks there. Add a `storage_pool_path` variable that defaults to `/var/lib/libvirt/images/talos-kvm` but can be pointed at the external drive mount (e.g., `/mnt/external/talos-kvm`). The `libvirt_pool`'s `target.path` already supports this — just wire it to the variable. The directory needs to exist and be writable by the `qemu` user. On the version question: Talos Linux v1.13 is the OS version, not the Terraform provider version. The `siderolabs/talos` Terraform provider follows its own 0.x versioning and the latest stable is v0.11.0 (released 2026-04-27). It uses Talos SDK v1.13.0 and fully supports Talos Linux v1.13.x. There is no provider "v1.13" — `~> 0.11` is the correct and latest stable constraint.

tepichord added

status

pending

and removed

status

planning

labels 2026-08-19 16:03:22 -07:00

tepichord commented 2026-08-19 16:19:03 -07:00

Author

Owner

Copy link

For the external 2TB drive: yes, you can store control-plane disks there. Add a storage_pool_path variable that defaults to /var/lib/libvirt/images/talos-kvm but can be pointed at the external drive mount (e.g., /mnt/external/talos-kvm). The libvirt_pool's target.path already supports this — just wire it to the variable. The directory needs to exist and be writable by the qemu user.

add a section to the README.md called setup detailing not only this but how to standup the control plane with terraform on naina from a fresh Linux install. It should be example forward with limited prose. Also the drive storage (and persistent kubernetes volumes should be added to ARCHITECTURE.MD as well as a table or diagram.

> For the external 2TB drive: yes, you can store control-plane disks there. Add a storage_pool_path variable that defaults to /var/lib/libvirt/images/talos-kvm but can be pointed at the external drive mount (e.g., /mnt/external/talos-kvm). The libvirt_pool's target.path already supports this — just wire it to the variable. The directory needs to exist and be writable by the qemu user. add a section to the README.md called setup detailing not only this but how to standup the control plane with terraform on naina from a fresh Linux install. It should be example forward with limited prose. Also the drive storage (and persistent kubernetes volumes should be added to ARCHITECTURE.MD as well as a table or diagram.

tepichord added

status

planning

and removed

status

pending

labels 2026-08-19 16:19:13 -07:00

tepichord added

status

pending

and removed

status

planning

labels 2026-08-19 18:58:16 -07:00

tepichord commented 2026-08-19 22:22:10 -07:00

Author

Owner

Copy link

variable "storage_pool_path" {
description = "Root path for libvirt storage pool (e.g. external drive mount)"
type = string
default = "/var/lib/libvirt/images/talos-kvm"
}

I want to know, this is one variable as it is that I have to define as a path, what happens when the next storage volume needs to be created? Will there be a second variable that I also have to point to this drive that I have? I would rather point to a base folder on the mounted drive and have an expected file structure exist from there (e.g. base is /mnt/t9_drive/milner_data, images get stored at /mnt/t9_drive/milner_data/libvirt/images/talos-kvm)

> variable "storage_pool_path" { description = "Root path for libvirt storage pool (e.g. external drive mount)" type = string default = "/var/lib/libvirt/images/talos-kvm" } I want to know, this is one variable as it is that I have to define as a path, what happens when the next storage volume needs to be created? Will there be a second variable that I also have to point to this drive that I have? I would rather point to a base folder on the mounted drive and have an expected file structure exist from there (e.g. base is `/mnt/t9_drive/milner_data`, images get stored at `/mnt/t9_drive/milner_data/libvirt/images/talos-kvm`)

tepichord added

status

planning

and removed

status

pending

labels 2026-08-19 22:22:16 -07:00

tepichord commented 2026-08-19 22:45:03 -07:00

Author

Owner

Copy link

Naina:
Good call — a single data_root_path variable that defines a base directory and derives all sub-paths from it is the right design. Here's how it works:

variable "data_root_path" {
  description = "Base directory for all persistent cluster data (VM disks, ISOs, etc.)"
  type        = string
  default     = "/var/lib/libvirt"
}

locals {
  libvirt_pool_path = "${var.data_root_path}/images/talos-kvm"
}

The libvirt_pool resource uses the local: target { path = local.libvirt_pool_path }. When you point it at your external drive (/mnt/t9_drive/milner_data), the expected layout becomes:

/mnt/t9_drive/milner_data/
└── images/
    └── talos-kvm/
        ├── milner-cp-1-disk.qcow2
        ├── milner-cp-2-disk.qcow2
        ├── milner-cp-3-disk.qcow2
        └── talos-metal-arm64.iso

Future storage needs (worker node disks, persistent volumes, etc.) can derive from the same data_root_path — add a new local like worker_pool_path = "${var.data_root_path}/images/worker-disks" without introducing a second variable. The updated plan will use this design.

Naina: Good call — a single `data_root_path` variable that defines a base directory and derives all sub-paths from it is the right design. Here's how it works: ```terraform variable "data_root_path" { description = "Base directory for all persistent cluster data (VM disks, ISOs, etc.)" type = string default = "/var/lib/libvirt" } locals { libvirt_pool_path = "${var.data_root_path}/images/talos-kvm" } ``` The `libvirt_pool` resource uses the local: `target { path = local.libvirt_pool_path }`. When you point it at your external drive (`/mnt/t9_drive/milner_data`), the expected layout becomes: ``` /mnt/t9_drive/milner_data/ └── images/ └── talos-kvm/ ├── milner-cp-1-disk.qcow2 ├── milner-cp-2-disk.qcow2 ├── milner-cp-3-disk.qcow2 └── talos-metal-arm64.iso ``` Future storage needs (worker node disks, persistent volumes, etc.) can derive from the same `data_root_path` — add a new local like `worker_pool_path = "${var.data_root_path}/images/worker-disks"` without introducing a second variable. The updated plan will use this design.

tepichord added

status

implement

and removed

status

planning

labels 2026-08-19 22:46:32 -07:00

tepichord referenced this issue 2026-08-19 23:20:48 -07:00

IaC Worker Node - Naina #17

tepichord referenced this issue from a commit 2026-08-20 00:06:58 -07:00

feat(control-plane): replace null_resource with libvirt_domain and Talos provider

tepichord referenced this issue from a commit 2026-08-20 00:06:58 -07:00

chore(cleanup): rewrite cleanup.sh to use terraform destroy

tepichord referenced this issue from a commit 2026-08-20 00:06:58 -07:00

test(control-plane): extend plan tests with new mock-provider assertions

tepichord referenced this issue from a commit 2026-08-20 00:06:58 -07:00

docs(architecture): add resource budget table and storage distribution table

tepichord referenced this issue from a commit 2026-08-20 00:06:58 -07:00

docs(readme): add Setup section with install commands

tepichord referenced this issue from a commit 2026-08-20 00:06:58 -07:00

docs(agents): bump provider versions in tech stack table

tepichord referenced this issue from a commit 2026-08-20 00:06:58 -07:00

docs(control-plane): add output descriptions

tepichord referenced this issue from a pull request that will close it, 2026-08-20 00:07:07 -07:00

IaC Control Plane #22

tepichord added

status

reviewing

and removed

status

implement

labels 2026-08-20 00:07:07 -07:00

tepichord referenced this issue 2026-08-21 18:41:30 -07:00

Determine incremental deployment methodology #27

tepichord referenced this issue 2026-08-21 18:57:21 -07:00

Determine incremental deployment methodology #27

tepichord referenced this issue from a commit 2026-08-23 09:54:18 -07:00

docs(control-plane): shorten cleanup.sh header comment

tepichord added

effort

high

and removed

effort

medium

labels 2026-09-05 21:26:29 -07:00

tepichord referenced this issue 2026-09-06 10:40:04 -07:00

IaC Control Plane #22

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

feat/12-iac-control-plane

feat/3-setting-up-agent-development

kvm-kubernetes-setup

No results found.

Labels

Clear labels   

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

effort

high

  

effort

low

  

effort

medium

  

effort

minimal

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

pr

feedback

Used for PRs, indicates that feedback has been given and agent needs to make changes

  

pr

reviewing

Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  

question

More information is needed

  

status

backlog

Ignore, no work needed yet, human will assign this work by changing to status/planning

  

status

document

Record findings, create flashcards, and document knowledge in knowledgebase for the future

  

status

done

Completed when PR is merged, no more work needed.

  

status

implement

Make the code changes. Should be done by a faster and cheaper model

  

status

pending

Give human a chance to review planning details

  

status

planning

Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  

status

reviewing

PR is created and feedback loop begins between human and agent

  

wontfix

This won't be fixed

No labels

bug

duplicate

effort

high

effort

low

effort

medium

effort

minimal

enhancement

help wanted

invalid

pr

feedback

pr

reviewing

question

status

backlog

status

document

status

done

status

implement

status

pending

status

planning

status

reviewing

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

MVP

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

tepichord/milner#12

No description provided.