Add a worker/ Terraform root that provisions the naina worker-large VM (20 GB RAM / 6 vCPU¹ / 40 GiB disk, per ARCHITECTURE.md §1 — §1 sizes RAM/vCPU only and specifies no disk; the 40 GiB is control-plane parity²) attached to the existing cp-bridge NAT network³, and joins it to the live cluster as a Talos worker using the control-plane root's machine secrets. The shell-based provisioning (worker/vm-setup/) is superseded and removed; the control-plane graph gains two sensitive outputs plus a DHCP reservation for the worker.

Technical Approach

Extend control-plane/main.tf — expose cluster identity to the worker root⁴⁵

output "machine_secrets" {
  value     = talos_machine_secrets.this.machine_secrets
  sensitive = true
}

output "client_configuration" {
  value     = talos_machine_secrets.this.client_configuration
  sensitive = true
}

output "cluster_name"       { value = var.cluster_name }
output "cluster_endpoint"   { value = var.cluster_endpoint }
output "talos_version"      { value = var.talos_version }
output "kubernetes_version" { value = var.kubernetes_version }

variable "worker_dhcp_hosts" {
  description = "Static DHCP reservations for worker nodes on control-plane-net"
  type        = list(object({ mac = string, ip = string }))
  default     = []
}

# ...in libvirt_network.talos_bridge.ips[0].dhcp — append after the count-driven CP hosts
hosts = concat(
  [for i in range(var.control_plane_count) : {
    mac = local.control_plane_macs[i]
    ip  = cidrhost(var.network_cidr, i + 2)
  }],
  var.worker_dhcp_hosts
)

# control-plane/terraform.tfvars
worker_dhcp_hosts = [{ mac = "52:54:00:10:00:04", ip = "10.5.0.20" }] # milner-worker-1 (worker-large, naina)

Scaffold worker/main.tf — provider block + secrets read from the CP state

terraform {
  required_version = ">= 1.7"
  required_providers {
    talos   = { source = "siderolabs/talos",   version = "~> 0.11" }
    libvirt = { source = "dmacvicar/libvirt", version = "~> 0.9" }
    time    = { source = "hashicorp/time",    version = "~> 0.14" }
  }
}

provider "libvirt" { uri = "qemu:///system" }

variable "control_plane_state_path" {
  type    = string
  default = "../control-plane/terraform.tfstate"
}

data "terraform_remote_state" "cp" {
  backend = "local"
  config  = { path = var.control_plane_state_path }
}

locals {
  cp = data.terraform_remote_state.cp.outputs
}

Create volumes + libvirt_domain.worker — join the CP-owned network/pool by name⁶⁷⁸

variable "memory_mb"    { type = number, default = 20480 }  # ARCHITECTURE.md: 20 GB
variable "vcpu_count"   { type = number, default = 6 }      # ARCHITECTURE.md: 6 vCPU
variable "worker_mac"   { type = string, default = "52:54:00:10:00:04" } # matches control-plane tfvars reservation
variable "worker_ip"    { type = string, default = "10.5.0.20" }         # matches control-plane tfvars reservation
variable "network_name" { type = string, default = "control-plane-net" } # owned by control-plane root
variable "pool_name"    { type = string, default = "talos-images" }      # owned by control-plane root

resource "libvirt_volume" "worker_disk" {
  name     = "${var.cluster_name}-worker-1-disk.qcow2"
  pool     = var.pool_name
  capacity = 42949672960 # 40 GiB, control-plane parity
  target   = { format = { type = "qcow2" } }
}

resource "libvirt_volume" "worker_iso" {
  count = var.installer_iso ? 1 : 0
  name  = "talos-metal-arm64-worker.iso" # distinct name — never re-declare the CP-owned ISO volume
  pool  = var.pool_name
  create = { content = { url = "https://factory.talos.dev/image/4a0d65c669d46663f377e7161e50cfd570c401f26fd9e7bda34a0216b6f1922b/${local.cp.talos_version}/metal-arm64.iso" } }
  target = { format = { type = "iso" } }
}

resource "libvirt_domain" "worker" {
  name  = "${var.cluster_name}-worker-1"
  memory      = var.memory_mb
  memory_unit = "MiB"
  vcpu        = var.vcpu_count
  # cpu host-passthrough / features acpi + gic v3 / AAVMF os block / scsi+qcow2 disks /
  # serial console: mirror the control-plane domain block (installer_iso toggle identical)
  devices = {
    interfaces = [{
      type = "network"
      mac  = { address = var.worker_mac } # statically reserved on control-plane-net
      source = { network = { network = var.network_name } } # attach by libvirt network NAME, not bridge
      wait_for_ip = { source = "lease", timeout = 300 }
    }]
  }
}

Generate + apply the worker machine config — same secrets, no bootstrap⁹

resource "time_sleep" "wait_for_worker" {
  create_duration = "120s"
  depends_on      = [libvirt_domain.worker]
}

data "talos_machine_configuration" "worker" {
  cluster_name       = local.cp.cluster_name
  cluster_endpoint   = "https://${local.cp.cluster_endpoint}:6443"
  machine_type       = "worker"
  machine_secrets    = local.cp.machine_secrets   # MUST be the CP's secrets; a fresh resource would mint a CA that cannot join
  talos_version      = local.cp.talos_version
  kubernetes_version = local.cp.kubernetes_version
}

resource "talos_machine_configuration_apply" "worker" {
  client_configuration        = local.cp.client_configuration
  machine_configuration_input = data.talos_machine_configuration.worker.machine_configuration
  node                        = var.worker_ip
  config_patches = [yamlencode({
    machine = {
      install = { disk = "/dev/vda", image = "ghcr.io/siderolabs/installer:${local.cp.talos_version}" }
      network = { interfaces = [{ interface = "eth0", dhcp = true }] }
      nodeLabels = { "node-role" = "worker" } # ARCHITECTURE.md affinity key
    }
  })]
  depends_on = [time_sleep.wait_for_worker]
}
# No talos_machine_bootstrap and no talos_cluster_kubeconfig — a worker joins the
# existing cluster in its role as soon as its config is applied.

Hermetic tests — fixture CP state + mock providers¹⁰

# worker/tests/fixtures/cp.tfstate — minimal committed state (fake values) so tests run offline:
# { "version": 4, "outputs": { "machine_secrets": { "value": "fake", "type": "string" }, ... } }

# worker/tests/plan.tftest.hcl
mock_provider "libvirt" { override_during = plan }
mock_provider "talos"   { override_during = plan }
mock_provider "time"    { override_during = plan }

run "reads_control_plane_outputs" {
  command = plan
  variables {
    control_plane_state_path = "tests/fixtures/cp.tfstate"
  }
  assert {
    condition     = data.terraform_remote_state.cp.outputs.cluster_name == "milner"
    error_message = "worker root must read cluster identity from the control-plane state"
  }
}

run "worker_is_a_worker" {
  command = plan
  assert {
    condition     = data.talos_machine_configuration.worker.machine_type == "worker"
    error_message = "machine_type must be worker (not controlplane)"
  }
}

Docs + harness updates (root tests/test_terraform.py runs terraform test in both roots¹¹; tests/test_static.py fmt/shellcheck lists extended to worker/¹²; README.md + worker/README.md runbook mirroring control-plane/; AGENTS.md repo-map and "How Changes Ship" updated; worker/cleanup.sh = confirm-gated terraform destroy of worker resources only — no pool sweep, the pool is CP-owned).
Delete superseded files: worker/vm-setup/create-worker-vm.sh, worker/tests/test_create_worker_vm.bats, worker/patches/worker-patch-1.yaml (placeholders are now inline config_patches).¹³¹⁴¹⁵

Implementation Details

Files to create:

worker/main.tf — remote-state read, worker volumes/domain, worker talos config + apply (conventions mirror control-plane/main.tf)
worker/terraform.tfvars — repo-default knobs (mirrors CP file style)
worker/.gitignore — .terraform/, *.tfstate*, *.auto.tfvars
worker/tests/plan.tftest.hcl + worker/tests/fixtures/cp.tfstate
worker/cleanup.sh, worker/README.md

Files to modify:

control-plane/main.tf — append 6 outputs + worker_dhcp_hosts variable + hosts = concat(...) in libvirt_network¹⁶
control-plane/terraform.tfvars — add worker_dhcp_hosts entry¹⁷¹⁸
control-plane/tests/plan.tftest.hcl — add run asserting the worker reservation and the new outputs
tests/test_terraform.py, tests/test_static.py, README.md, AGENTS.md

Files to delete: worker/vm-setup/create-worker-vm.sh, worker/tests/test_create_worker_vm.bats, worker/patches/worker-patch-1.yaml, plus the now-empty worker/vm-setup/ dir. The bats pytest in tests/test_shell.py is removed with the bats dir.¹⁹

Dependencies: none new — provider pins (talos ~> 0.11, libvirt ~> 0.9, time ~> 0.14) identical to the control-plane root²⁰; worker/.terraform.lock.hcl generated by terraform init.

Constraints:

Worker graph must not re-declare the network, pool, or CP's ISO volume (one real object under two states); it references control-plane-net/talos-images by name string only²¹.
talos_machine_configuration_apply has no machine_type and no join-token input — role + join material come from the machine_type = "worker" data source and the shared machine secrets.²²²³
Apply order: control-plane root first (state must exist), then worker root.
worker_ip/worker_mac in worker/ must match the CP worker_dhcp_hosts reservation; comments cross-reference both sides and tests lock the values.
Re-applying the CP root to add the reservation is an in-place network update (dnsmasq restart) ^unverified; run talosctl health afterwards.
node-role: worker via machine.nodeLabels is outside NodeRestriction's restricted prefixes (NodeRestriction forbids only node-role.kubernetes.io/* and other kubernetes.io/k8s.io-domain labels), but is empirically unverified on this cluster — confirm with kubectl get node --show-labels after join.²⁴²⁵

Out-of-scope notes:

ida's worker-sim node and any further workers (Phase 4 stays 🚧 until 3+ nodes; per-node DHCP reservations would need the same pattern).
The single-graph consolidation framed in ARCHITECTURE.md §8 (clean-install) is not done here; worker/ is built standalone and can be folded in later.
No provider upgrades (talos stays ~> 0.11; post-0.11 ephemeral-secrets features and the v1.14 KubeNodeConfig nodeLabels refactor are future work).
Deploying Vault/Woodpecker/RabbitMQ/orchestrator workloads onto the node is out of scope — this issue provisions the node only.

Acceptance Criteria

terraform -chdir=control-plane plan (with worker_dhcp_hosts set) shows only the additive DHCP host + outputs; existing control-plane terraform test suite still passes with the new reservation/output runs
terraform -chdir=worker plan against the real CP state creates exactly the worker disk/ISO volumes, milner-worker-1 domain, and talos_machine_configuration_apply.worker — no network, pool, bootstrap, or kubeconfig resources
Worker domain: 20480 MiB (memory_unit = "MiB"), 6 vCPU, 40 GiB qcow2 disk, MAC 52:54:00:10:00:04 on network control-plane-net, installer_iso toggle with hd/cdrom boot + SCSI cdrom, serial console, no prevent_destroy
Worker joins with the CP's machine secrets: kubectl get nodes shows milner-worker-1 Ready with node-role=worker label; a second terraform -chdir=worker apply is a no-op
Superseded shell artifacts deleted and repo test harness green offline: pytest tests/, terraform test in both roots, fmt/shellcheck over worker/
Docs updated: worker/README.md runbook, root README.md/AGENTS.md, PLAN.md Phase 4 notes the worker-large node is now Terraform-managed

Testing Plan

CP suite (control-plane/tests/plan.tftest.hcl) — mock-provider plan assertions that dhcp.hosts length = control_plane_count + 1 with hosts[1] == { mac = "52:54:00:10:00:04", ip = "10.5.0.20" }, and that the machine_secrets/client_configuration outputs are non-null and sensitive.
Worker suite (worker/tests/plan.tftest.hcl) — mock-provider plan assertions (fixture CP state): domain shape (memory/unit/vcpu, boot order hd→cdrom, qcow2 driver, virtio-scsi controller, AAVMF os block, serial console, autostart/running), interface MAC/network-name/wait_for_ip, disk 40 GiB qcow2, ISO volume present only when installer_iso = true (and ejected-run variant with installer_iso = false), machine_type == "worker" + nodeLabels patch, apply count 1, time_sleep dependency chain.
Root pytest suite — terraform test both roots, terraform fmt -check -recursive, shellcheck on both cleanup.sh scripts, all offline.
Manual smoke on naina (not automated — needs the live cluster): apply CP root (adds reservation/outputs), then terraform -chdir=worker apply; verify talosctl health and kubectl get nodes -o wide + --show-labels. Recovery check: terraform -chdir=worker apply -replace='libvirt_domain.worker' re-joins the node while the CP stays up.

#12 — IaC Control Plane (closed): the reference graph; extended here with secrets outputs + worker DHCP reservation
#16 — Update architecture document for network topology (closed): cp-bridge topology this worker attaches to
#14 — Determine testing methodology (closed): mock-provider/offline test conventions reused
PR #22 — IaC Control Plane: merged implementation of #12

Research Findings

Worker role is set by machine_type = "worker" on the talos_machine_configuration data source; the apply resource has no machine_type or join-token input — the generated config carries the join material, so every node must derive from the same talos_machine_secrets or the new node cannot join. (talos provider docs, Talos scaling-up guide)
The libvirt provider v0.9.8 has no network/pool data source, so the worker attaches to the existing network by its libvirt name string (control-plane-net, not bridge cp-bridge); network/pool stay owned by the control-plane root to avoid one real object under two states. (provider data-source tree at v0.9.8)
Cross-root secrets sharing is supported via talos_machine_secrets import from a secrets file or terraform_remote_state (local backend); the plan adopts remote state, noting it exposes the whole CP state snapshot to the worker root. (machine_secrets import, remote-state data)
Talos 1.11 exposes node labels as machine.nodeLabels; NodeRestriction forbids kubelet self-assignment of node-role.kubernetes.io/*, but the plain node-role: worker key used by ARCHITECTURE.md affinity is outside restricted prefixes (still worth empirical confirmation). (node labels guide)
ARCHITECTURE.md sizes the worker-large as 20 GB RAM / 6 vCPU but specifies no disk; Talos requires ≥10 GiB and recommends 100 GiB, so the plan keeps 40 GiB control-plane parity behind a variable. (Talos system requirements)

Pre-Mortem

Apply-order and remote-state coupling between the two roots — The worker graph reads the control-plane state via terraform_remote_state and requires the CP root to have been re-applied first to publish the secrets outputs and the DHCP reservation; if the roots are applied out of order or the CP tfstate path/lock is stale, the worker plan/apply errors out or the VM never receives its reserved lease and wait_for_ip times out. This dependency is only documented, never enforced, so it is the easiest step to get wrong on the live host.
The worker boots but never joins the cluster as Ready — talos_machine_configuration_apply fails or the node stays NotReady when the secrets pulled from remote state do not match the live cluster's CA (sensitive-output handling quirks) or when the installer-ISO talos_version drifts from the running cluster version. The failure surfaces only in the manual smoke step, so hermetic tests stay green while the node never appears in kubectl get nodes.
In-place network changes disrupt or desync the live control plane — Adding the reservation rewrites libvirt_network DHCP (dnsmasq restart), and if the MAC/IP pair drifts between the two tfvars files, the network update can disturb the running CP node or leave the worker IP-less, while the "second apply is a no-op" and node-role: worker label assumptions remain empirically unverified until the live-cluster smoke test.

Naina: ## Overview Add a `worker/` Terraform root that provisions the naina worker-large VM (20 GB RAM / 6 vCPU[1](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/ARCHITECTURE.md?display=source#L22) / 40 GiB disk, per ARCHITECTURE.md §1 — §1 sizes RAM/vCPU only and specifies no disk; the 40 GiB is control-plane parity[2](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L204)) attached to the existing `cp-bridge` NAT network[3](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/ARCHITECTURE.md?display=source#L229), and joins it to the live cluster as a Talos worker using the control-plane root's machine secrets. The shell-based provisioning (`worker/vm-setup/`) is superseded and removed; the control-plane graph gains two sensitive outputs plus a DHCP reservation for the worker. ## Technical Approach 1. **Extend `control-plane/main.tf` — expose cluster identity to the worker root**[4](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L339)[5](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L31) ```hcl output "machine_secrets" { value = talos_machine_secrets.this.machine_secrets sensitive = true } output "client_configuration" { value = talos_machine_secrets.this.client_configuration sensitive = true } output "cluster_name" { value = var.cluster_name } output "cluster_endpoint" { value = var.cluster_endpoint } output "talos_version" { value = var.talos_version } output "kubernetes_version" { value = var.kubernetes_version } variable "worker_dhcp_hosts" { description = "Static DHCP reservations for worker nodes on control-plane-net" type = list(object({ mac = string, ip = string })) default = [] } ``` ```hcl # ...in libvirt_network.talos_bridge.ips[0].dhcp — append after the count-driven CP hosts hosts = concat( [for i in range(var.control_plane_count) : { mac = local.control_plane_macs[i] ip = cidrhost(var.network_cidr, i + 2) }], var.worker_dhcp_hosts ) ``` ```hcl # control-plane/terraform.tfvars worker_dhcp_hosts = [{ mac = "52:54:00:10:00:04", ip = "10.5.0.20" }] # milner-worker-1 (worker-large, naina) ``` 2. **Scaffold `worker/main.tf` — provider block + secrets read from the CP state** ```hcl terraform { required_version = ">= 1.7" required_providers { talos = { source = "siderolabs/talos", version = "~> 0.11" } libvirt = { source = "dmacvicar/libvirt", version = "~> 0.9" } time = { source = "hashicorp/time", version = "~> 0.14" } } } provider "libvirt" { uri = "qemu:///system" } variable "control_plane_state_path" { type = string default = "../control-plane/terraform.tfstate" } data "terraform_remote_state" "cp" { backend = "local" config = { path = var.control_plane_state_path } } locals { cp = data.terraform_remote_state.cp.outputs } ``` 3. **Create volumes + `libvirt_domain.worker` — join the CP-owned network/pool by name**[6](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L117)[7](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L175)[8](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L185) ```hcl variable "memory_mb" { type = number, default = 20480 } # ARCHITECTURE.md: 20 GB variable "vcpu_count" { type = number, default = 6 } # ARCHITECTURE.md: 6 vCPU variable "worker_mac" { type = string, default = "52:54:00:10:00:04" } # matches control-plane tfvars reservation variable "worker_ip" { type = string, default = "10.5.0.20" } # matches control-plane tfvars reservation variable "network_name" { type = string, default = "control-plane-net" } # owned by control-plane root variable "pool_name" { type = string, default = "talos-images" } # owned by control-plane root resource "libvirt_volume" "worker_disk" { name = "${var.cluster_name}-worker-1-disk.qcow2" pool = var.pool_name capacity = 42949672960 # 40 GiB, control-plane parity target = { format = { type = "qcow2" } } } resource "libvirt_volume" "worker_iso" { count = var.installer_iso ? 1 : 0 name = "talos-metal-arm64-worker.iso" # distinct name — never re-declare the CP-owned ISO volume pool = var.pool_name create = { content = { url = "https://factory.talos.dev/image/4a0d65c669d46663f377e7161e50cfd570c401f26fd9e7bda34a0216b6f1922b/${local.cp.talos_version}/metal-arm64.iso" } } target = { format = { type = "iso" } } } resource "libvirt_domain" "worker" { name = "${var.cluster_name}-worker-1" memory = var.memory_mb memory_unit = "MiB" vcpu = var.vcpu_count # cpu host-passthrough / features acpi + gic v3 / AAVMF os block / scsi+qcow2 disks / # serial console: mirror the control-plane domain block (installer_iso toggle identical) devices = { interfaces = [{ type = "network" mac = { address = var.worker_mac } # statically reserved on control-plane-net source = { network = { network = var.network_name } } # attach by libvirt network NAME, not bridge wait_for_ip = { source = "lease", timeout = 300 } }] } } ``` 4. **Generate + apply the worker machine config — same secrets, no bootstrap**[9](https://docs.siderolabs.com/talos/v1.11/deploy-and-manage-workloads/scaling-up) ```hcl resource "time_sleep" "wait_for_worker" { create_duration = "120s" depends_on = [libvirt_domain.worker] } data "talos_machine_configuration" "worker" { cluster_name = local.cp.cluster_name cluster_endpoint = "https://${local.cp.cluster_endpoint}:6443" machine_type = "worker" machine_secrets = local.cp.machine_secrets # MUST be the CP's secrets; a fresh resource would mint a CA that cannot join talos_version = local.cp.talos_version kubernetes_version = local.cp.kubernetes_version } resource "talos_machine_configuration_apply" "worker" { client_configuration = local.cp.client_configuration machine_configuration_input = data.talos_machine_configuration.worker.machine_configuration node = var.worker_ip config_patches = [yamlencode({ machine = { install = { disk = "/dev/vda", image = "ghcr.io/siderolabs/installer:${local.cp.talos_version}" } network = { interfaces = [{ interface = "eth0", dhcp = true }] } nodeLabels = { "node-role" = "worker" } # ARCHITECTURE.md affinity key } })] depends_on = [time_sleep.wait_for_worker] } # No talos_machine_bootstrap and no talos_cluster_kubeconfig — a worker joins the # existing cluster in its role as soon as its config is applied. ``` 5. **Hermetic tests — fixture CP state + mock providers**[10](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/tests/plan.tftest.hcl?display=source#L1) ```hcl # worker/tests/fixtures/cp.tfstate — minimal committed state (fake values) so tests run offline: # { "version": 4, "outputs": { "machine_secrets": { "value": "fake", "type": "string" }, ... } } ``` ```hcl # worker/tests/plan.tftest.hcl mock_provider "libvirt" { override_during = plan } mock_provider "talos" { override_during = plan } mock_provider "time" { override_during = plan } run "reads_control_plane_outputs" { command = plan variables { control_plane_state_path = "tests/fixtures/cp.tfstate" } assert { condition = data.terraform_remote_state.cp.outputs.cluster_name == "milner" error_message = "worker root must read cluster identity from the control-plane state" } } run "worker_is_a_worker" { command = plan assert { condition = data.talos_machine_configuration.worker.machine_type == "worker" error_message = "machine_type must be worker (not controlplane)" } } ``` 6. **Docs + harness updates** (root `tests/test_terraform.py` runs `terraform test` in both roots[11](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/tests/test_terraform.py?display=source#L12); `tests/test_static.py` fmt/`shellcheck` lists extended to `worker/`[12](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/tests/test_static.py?display=source#L14); `README.md` + `worker/README.md` runbook mirroring `control-plane/`; `AGENTS.md` repo-map and "How Changes Ship" updated; `worker/cleanup.sh` = confirm-gated `terraform destroy` of worker resources only — no pool sweep, the pool is CP-owned). 7. **Delete superseded files**: `worker/vm-setup/create-worker-vm.sh`, `worker/tests/test_create_worker_vm.bats`, `worker/patches/worker-patch-1.yaml` (placeholders are now inline `config_patches`).[13](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/worker/vm-setup/create-worker-vm.sh?display=source#L1)[14](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/worker/tests/test_create_worker_vm.bats?display=source#L1)[15](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/worker/patches/worker-patch-1.yaml?display=source#L1) ## Implementation Details **Files to create:** - `worker/main.tf` — remote-state read, worker volumes/domain, worker talos config + apply (conventions mirror `control-plane/main.tf`) - `worker/terraform.tfvars` — repo-default knobs (mirrors CP file style) - `worker/.gitignore` — `.terraform/`, `*.tfstate*`, `*.auto.tfvars` - `worker/tests/plan.tftest.hcl` + `worker/tests/fixtures/cp.tfstate` - `worker/cleanup.sh`, `worker/README.md` **Files to modify:** - `control-plane/main.tf` — append 6 outputs + `worker_dhcp_hosts` variable + `hosts = concat(...)` in `libvirt_network`[16](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L147) - `control-plane/terraform.tfvars` — add `worker_dhcp_hosts` entry[17](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L142)[18](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L108) - `control-plane/tests/plan.tftest.hcl` — add run asserting the worker reservation and the new outputs - `tests/test_terraform.py`, `tests/test_static.py`, `README.md`, `AGENTS.md` **Files to delete:** `worker/vm-setup/create-worker-vm.sh`, `worker/tests/test_create_worker_vm.bats`, `worker/patches/worker-patch-1.yaml`, plus the now-empty `worker/vm-setup/` dir. The bats pytest in `tests/test_shell.py` is removed with the bats dir.[19](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/tests/test_shell.py?display=source#L13) **Dependencies:** none new — provider pins (`talos ~> 0.11`, `libvirt ~> 0.9`, `time ~> 0.14`) identical to the control-plane root[20](https://git.gloom.garden/tepichord/milner/src/commit/aad15b83856a2be1019d942ce4a98eb3ef6307a5/control-plane/main.tf?display=source#L4); `worker/.terraform.lock.hcl` generated by `terraform init`. **Constraints:** - Worker graph must not re-declare the network, pool, or CP's ISO volume (one real object under two states); it references `control-plane-net`/`talos-images` by name string only[21](https://github.com/dmacvicar/terraform-provider-libvirt/tree/v0.9.8/docs/data-sources). - `talos_machine_configuration_apply` has no `machine_type` and no join-token input — role + join material come from the `machine_type = "worker"` data source and the shared machine secrets.[22](https://registry.terraform.io/providers/siderolabs/talos/latest/docs/resources/machine_configuration_apply)[23](https://registry.terraform.io/providers/siderolabs/talos/latest/docs/data-sources/machine_configuration) - Apply order: control-plane root first (state must exist), then worker root. - `worker_ip`/`worker_mac` in `worker/` must match the CP `worker_dhcp_hosts` reservation; comments cross-reference both sides and tests lock the values. - Re-applying the CP root to add the reservation is an in-place network update (dnsmasq restart) _unverified_; run `talosctl health` afterwards. - `node-role: worker` via `machine.nodeLabels` is outside NodeRestriction's restricted prefixes (NodeRestriction forbids only `node-role.kubernetes.io/*` and other `kubernetes.io`/`k8s.io`-domain labels), but is empirically unverified on this cluster — confirm with `kubectl get node --show-labels` after join.[24](https://docs.siderolabs.com/kubernetes-guides/advanced-guides/node-labels)[25](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#noderestriction) **Out-of-scope notes:** - ida's `worker-sim` node and any further workers (Phase 4 stays 🚧 until 3+ nodes; per-node DHCP reservations would need the same pattern). - The single-graph consolidation framed in ARCHITECTURE.md §8 (clean-install) is not done here; `worker/` is built standalone and can be folded in later. - No provider upgrades (talos stays `~> 0.11`; post-0.11 ephemeral-secrets features and the v1.14 `KubeNodeConfig` nodeLabels refactor are future work). - Deploying Vault/Woodpecker/RabbitMQ/orchestrator workloads onto the node is out of scope — this issue provisions the node only. ## Acceptance Criteria - [ ] `terraform -chdir=control-plane plan` (with `worker_dhcp_hosts` set) shows only the additive DHCP host + outputs; existing control-plane `terraform test` suite still passes with the new reservation/output runs - [ ] `terraform -chdir=worker plan` against the real CP state creates exactly the worker disk/ISO volumes, `milner-worker-1` domain, and `talos_machine_configuration_apply.worker` — no network, pool, bootstrap, or kubeconfig resources - [ ] Worker domain: 20480 MiB (`memory_unit = "MiB"`), 6 vCPU, 40 GiB qcow2 disk, MAC `52:54:00:10:00:04` on network `control-plane-net`, `installer_iso` toggle with hd/cdrom boot + SCSI cdrom, serial console, no `prevent_destroy` - [ ] Worker joins with the CP's machine secrets: `kubectl get nodes` shows `milner-worker-1` `Ready` with `node-role=worker` label; a second `terraform -chdir=worker apply` is a no-op - [ ] Superseded shell artifacts deleted and repo test harness green offline: `pytest tests/`, `terraform test` in both roots, fmt/`shellcheck` over `worker/` - [ ] Docs updated: `worker/README.md` runbook, root `README.md`/`AGENTS.md`, `PLAN.md` Phase 4 notes the worker-large node is now Terraform-managed ## Testing Plan 1. **CP suite (`control-plane/tests/plan.tftest.hcl`)** — mock-provider plan assertions that `dhcp.hosts` length = `control_plane_count + 1` with `hosts[1] == { mac = "52:54:00:10:00:04", ip = "10.5.0.20" }`, and that the `machine_secrets`/`client_configuration` outputs are non-null and sensitive. 2. **Worker suite (`worker/tests/plan.tftest.hcl`)** — mock-provider plan assertions (fixture CP state): domain shape (memory/unit/vcpu, boot order hd→cdrom, qcow2 driver, virtio-scsi controller, AAVMF os block, serial console, autostart/running), interface MAC/network-name/`wait_for_ip`, disk 40 GiB qcow2, ISO volume present only when `installer_iso = true` (and ejected-run variant with `installer_iso = false`), `machine_type == "worker"` + `nodeLabels` patch, apply count 1, `time_sleep` dependency chain. 3. **Root pytest suite** — `terraform test` both roots, `terraform fmt -check -recursive`, `shellcheck` on both `cleanup.sh` scripts, all offline. 4. **Manual smoke on naina** (not automated — needs the live cluster): apply CP root (adds reservation/outputs), then `terraform -chdir=worker apply`; verify `talosctl health` and `kubectl get nodes -o wide` + `--show-labels`. Recovery check: `terraform -chdir=worker apply -replace='libvirt_domain.worker'` re-joins the node while the CP stays up. ## Related Issues/PRs - #12 — IaC Control Plane (closed): the reference graph; extended here with secrets outputs + worker DHCP reservation - #16 — Update architecture document for network topology (closed): cp-bridge topology this worker attaches to - #14 — Determine testing methodology (closed): mock-provider/offline test conventions reused - PR #22 — IaC Control Plane: merged implementation of #12 ## Research Findings - Worker role is set by `machine_type = "worker"` on the `talos_machine_configuration` data source; the apply resource has no `machine_type` or join-token input — the generated config carries the join material, so every node must derive from the same `talos_machine_secrets` or the new node cannot join. ([talos provider docs](https://registry.terraform.io/providers/siderolabs/talos/latest/docs/data-sources/machine_configuration), [Talos scaling-up guide](https://docs.siderolabs.com/talos/v1.11/deploy-and-manage-workloads/scaling-up)) - The libvirt provider v0.9.8 has no network/pool data source, so the worker attaches to the existing network by its libvirt **name** string (`control-plane-net`, not bridge `cp-bridge`); network/pool stay owned by the control-plane root to avoid one real object under two states. ([provider data-source tree at v0.9.8](https://github.com/dmacvicar/terraform-provider-libvirt/tree/v0.9.8/docs/data-sources)) - Cross-root secrets sharing is supported via `talos_machine_secrets` import from a secrets file or `terraform_remote_state` (local backend); the plan adopts remote state, noting it exposes the whole CP state snapshot to the worker root. ([machine_secrets import](https://registry.terraform.io/providers/siderolabs/talos/latest/docs/resources/machine_secrets), [remote-state data](https://developer.hashicorp.com/terraform/language/state/remote-state-data)) - Talos 1.11 exposes node labels as `machine.nodeLabels`; NodeRestriction forbids kubelet self-assignment of `node-role.kubernetes.io/*`, but the plain `node-role: worker` key used by ARCHITECTURE.md affinity is outside restricted prefixes (still worth empirical confirmation). ([node labels guide](https://docs.siderolabs.com/kubernetes-guides/advanced-guides/node-labels)) - ARCHITECTURE.md sizes the worker-large as 20 GB RAM / 6 vCPU but specifies no disk; Talos requires ≥10 GiB and recommends 100 GiB, so the plan keeps 40 GiB control-plane parity behind a variable. ([Talos system requirements](https://docs.siderolabs.com/talos/v1.11/getting-started/system-requirements)) ## Pre-Mortem 1. **Apply-order and remote-state coupling between the two roots** — The worker graph reads the control-plane state via `terraform_remote_state` and requires the CP root to have been re-applied first to publish the secrets outputs and the DHCP reservation; if the roots are applied out of order or the CP tfstate path/lock is stale, the worker `plan`/`apply` errors out or the VM never receives its reserved lease and `wait_for_ip` times out. This dependency is only documented, never enforced, so it is the easiest step to get wrong on the live host. 2. **The worker boots but never joins the cluster as Ready** — `talos_machine_configuration_apply` fails or the node stays NotReady when the secrets pulled from remote state do not match the live cluster's CA (sensitive-output handling quirks) or when the installer-ISO `talos_version` drifts from the running cluster version. The failure surfaces only in the manual smoke step, so hermetic tests stay green while the node never appears in `kubectl get nodes`. 3. **In-place network changes disrupt or desync the live control plane** — Adding the reservation rewrites `libvirt_network` DHCP (dnsmasq restart), and if the MAC/IP pair drifts between the two tfvars files, the network update can disturb the running CP node or leave the worker IP-less, while the "second apply is a no-op" and `node-role: worker` label assumptions remain empirically unverified until the live-cluster smoke test.

tepichord added

and removed

labels

2026-09-08 12:41:03 -07:00

tepichord referenced this issue from a commit

2026-09-08 13:32:19 -07:00

feat(worker): add cleanup script and runbook

tepichord referenced this issue from a commit