Pod - Hashicorp Vault #18

New issue

Closed

opened 2026-08-19 23:30:07 -07:00 by tepichord · 1 comment

tepichord commented 2026-08-19 23:30:07 -07:00

Owner

Copy link

• Define IaC code to create vault pod on worker-large worker node
• For any manual work needed to stand up the vault pod, detail the setup steps taken in vault/README.md under a # Setup section
• Vault data should be stored under the data_root_path

- Define IaC code to create vault pod on `worker-large` worker node - For any manual work needed to stand up the vault pod, detail the setup steps taken in `vault/README.md` under a `# Setup` section - Vault data should be stored under the data_root_path

tepichord added the

effort

medium

enhancement

status

backlog

labels 2026-08-19 23:30:07 -07:00

tepichord added this to the MVP project 2026-08-19 23:30:07 -07:00

tepichord added

status

planning

and removed

status

backlog

labels 2026-09-08 15:43:06 -07:00

tepichord commented 2026-09-08 16:03:17 -07:00

Author

Owner

Copy link

Naina:

Overview

Migrate Vault out of podman-compose into the official hashicorp/vault Helm chart as a single-replica StatefulSet in a new vault namespace on worker-large. The nginx Tailscale-TLS proxy survives as a pod sidecar, and Vault's data moves to a dedicated 20 GiB qcow2 volume that is physically stored under data_root_path and reaches the pod via a Talos user volume mounted at /var/mnt/vault. Three Terraform roots change (control-plane NodePort range, worker data disk, new vault/ root driving the Helm release); every manual step is documented under vault/README.md → # Setup.

Technical Approach

1. Widen the NodePort range on the control plane so ARCHITECTURE.md's 8200/8443/8080 ports stay valid (K8s default range is 30000–32767)¹².

# control-plane/main.tf — append to talos_machine_configuration_apply.control_plane config_patches
yamlencode({
  cluster = {
    apiServer = {
      extraArgs = {
        "service-node-port-range" = "8080-32767"
      }
    }
  }
})

2. Add a dedicated Vault data disk to worker-large (volume in the CP-owned talos-images pool, so its backing file lands at ${data_root_path}/images/talos-kvm/milner-worker-1-vault.qcow2)³⁴.

# worker/main.tf
resource "libvirt_volume" "vault_data" {
  name     = "${var.cluster_name}-worker-1-vault.qcow2"
  pool     = var.pool_name
  capacity = 21474836480 # 20 GiB
  target   = { format = { type = "qcow2" } }
}
# ...and append a second disk entry inside libvirt_domain.worker devices.disks (before the ISO concat):
{ driver = { type = "qcow2" }
  source = { volume = { pool = libvirt_volume.vault_data.pool, volume = libvirt_volume.vault_data.name } }
  target = { dev = "vdb", bus = "virtio" } }

3. Surface the disk to the kubelet as a Talos user volume (machine-config document; /var/mnt/vault is auto-propagated into the kubelet so pods can hostPath it)⁵⁶.

# worker/main.tf — extra config_patches entry on talos_machine_configuration_apply.worker
apiVersion: v1alpha1
kind: UserVolumeConfig
name: vault
provisioning:
  diskSelector:
    match: disk.system_disk == false  # verify CEL against Talos v1.11 docs at implement time
  minSize: 10GB

⚠️ check-assumptions: the diskSelector CEL expression disk.system_disk == false is not shown verbatim in the Talos v1.11 docs — the documented CEL evaluation context exposes a system_disk (bool) variable plus a disk resource (official examples use !system_disk)⁷; the plan itself flags re-verification of the exact selector at implement time ^unverified.

talosctl get volumestatus u-vault   # expect PHASE=ready, mounted at /var/mnt/vault

4. Create the vault/ Terraform root, driving the cluster with credentials parsed from the CP kubeconfig output (same remote-state pattern as worker/)⁸⁹.

# vault/main.tf
locals { kc = yamldecode(data.terraform_remote_state.cp.outputs.kubeconfig) }

provider "kubernetes" {
  host                   = local.kc.clusters[0].cluster.server
  cluster_ca_certificate = base64decode(local.kc.clusters[0].cluster["certificate-authority-data"])
  client_certificate     = base64decode(local.kc.users[0].user["client-certificate-data"])
  client_key             = base64decode(local.kc.users[0].user["client-key-data"])
}
provider "helm" {
  kubernetes {
    host = local.kc.clusters[0].cluster.server
    # same base64-decoded cert/key fields as the kubernetes provider above
  }
}

resource "kubernetes_namespace_v1" "vault" {
  metadata {
    name = "vault"
    labels = { "pod-security.kubernetes.io/enforce" = "privileged" } # hostPath data (baseline forbids hostPath)
  }
}

Also here: two kubernetes_secret_v1 (certs read from gitignored paths in terraform.auto.tfvars), a ConfigMap for nginx, helm_release.vault, and the nginx Service.

5. Install the chart with the current config ported into server.standalone.config (chart PVC disabled — data on the hostPath volume; TLS and mlock settings carried over)¹⁰¹¹.

# vault/values.yaml (excerpt — block style for yamllint)
global:
  tlsDisable: false
server:
  nodeSelector:
    node-role: worker
  image:
    repository: hashicorp/vault
    tag: "2.0.4"                 # repo pin 1.15.4 is EOL; re-verify current line at implement time
  dataStorage:
    enabled: false
  volumes:
    - name: vault-data
      hostPath:
        path: /var/mnt/vault
    - name: vault-server-tls
      secret:
        secretName: vault-tls
  volumeMounts:
    - name: vault-data
      mountPath: /vault/data
    - name: vault-server-tls
      mountPath: /vault/userconfig/vault-server-tls
      readOnly: true
  standalone:
    enabled: true
    config: |-
      ui = true
      disable_mlock = true
      default_lease_ttl = "2304h"
      max_lease_ttl = "2304h"
      api_addr = "https://vault.vault.svc.cluster.local:8200"
      storage "file" { path = "/vault/data" }
      listener "tcp" {
        address = "[::]:8200"
        tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt"
        tls_key_file = "/vault/userconfig/vault-server-tls/vault.key"
        tls_client_ca_file = "/vault/userconfig/vault-server-tls/rootCA.pem"
      }
      user_lockout "all" {
        lockout_threshold = 25
        lockout_duration = "60m"
        lockout_counter_reset = "60m"
      }
  service:
    type: NodePort
    nodePort: 8200

6. Carry nginx over as server.extraContainers (same-pod proxy to 127.0.0.1:8200, Tailscale cert secret) and expose it with its own NodePort Service¹²¹³.

# vault/values.yaml (excerpt)
server:
  volumes:
    - name: tailscale-tls
      secret:
        secretName: tailscale-tls
    - name: nginx-conf
      configMap:
        name: vault-nginx-conf
  extraContainers:
    - name: nginx
      image: nginx:alpine
      ports:
        - containerPort: 443
        - containerPort: 80
      volumeMounts:
        - name: tailscale-tls
          mountPath: /etc/ssl
          readOnly: true
        - name: nginx-conf
          mountPath: /etc/nginx/conf.d/default.conf  # replaces stock default.conf (its :80 server would clash)
          subPath: default.conf
          readOnly: true

# vault/main.tf → kubernetes_config_map_v1 "vault-nginx-conf" (key: default.conf)
server {
    listen 443 ssl;
    server_name <tailnet-name>.faun-snapper.ts.net;
    ssl_certificate     /etc/ssl/tailscale-cert.crt;
    ssl_certificate_key /etc/ssl/tailscale-key.key;
    location / {
        proxy_pass https://127.0.0.1:8200;
        proxy_ssl_verify off;                  # mkcert upstream, as today
        proxy_set_header Host $host;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Upgrade $http_upgrade;   # Vault UI websockets
        proxy_set_header Connection "upgrade";
    }
}
server { listen 80; return 301 https://$host$request_uri; }

# vault/main.tf
resource "kubernetes_service_v1" "nginx" {
  metadata { name = "vault-nginx"; namespace = kubernetes_namespace_v1.vault.metadata[0].name }
  spec {
    type = "NodePort"
    selector = { "app.kubernetes.io/name" = "vault", component = "server" } # chart pod labels
    port { port = 443; node_port = 8443; target_port = 443 }
    port { port = 80;  node_port = 8080; target_port = 80 }
  }
}

7. Document the manual steps under vault/README.md → # Setup (issue criterion 2) and decommission the podman stack once the in-cluster Vault is verified:

1. Certs: mkcert rootCA + vault.crt/key → secret vault-tls; `tailscale cert` → tailscale-tls.
   Paths live in gitignored vault/terraform.auto.tfvars — certs are never committed.
2. Apply order: control-plane → worker → vault.
3. vault operator init + unseal x3; store unseal keys/root token off-cluster.
4. Host reachability: retarget naina's existing tailnet→podman port forwarding to
   worker NodePorts 10.5.0.20:8443/8200 (the podman DNAT rules die with the old stack).
5. Existing data: re-seed via API or start fresh (file backend is not copyable between instances).
6. podman-compose down; delete hc-vault/ after cutover.

Implementation Details

Root / file	Change
control-plane/main.tf, control-plane/tests/	apiserver extraArg patch + plan-test assert
worker/main.tf	libvirt_volume.vault_data (vdb) + UserVolumeConfig patch in config_patches
worker/tests/plan.tftest.hcl	asserts for volume name/capacity/qcow2, vdb disk, user-volume patch
worker/cleanup.sh	banner says "no persistent data" — must now warn the vault volume is destroyed on destroy14
vault/ (new root)	main.tf, values.yaml, terraform.tfvars, .gitignore, tests/plan.tftest.hcl + fixture
vault/README.md	runbook with # Setup (criterion 2)
tests/test_static.py, tests/test_terraform.py	fmt/lint lists add vault/ and drop hc-vault entries; add vault terraform test run1516
hc-vault/	deleted at cutover (config.hcl/nginx.conf content folded into values.yaml + nginx ConfigMap)
README.md, AGENTS.md, PLAN.md	repoint Vault references from hc-vault/ → vault/

Dependencies and constraints:

• New providers hashicorp/kubernetes + hashicorp/helm (repo first use); no flake change — helm_release removes the helm CLI dependency¹⁷¹⁸.
• Chart tag/version drift is real (research-time pin: chart 0.34.x, image 2.0.4; repo pin 1.15.4 is EOL) — re-verify at implement time; keep disable_mlock = true and the arm64 multi-arch official image^1920212223.
• Only the vault namespace gets the PSA privileged label; the chart's own securityContext (runAsNonRoot, drop ALL) stays baseline-compliant²⁴²⁵.
• Worker VM replacement (-replace='libvirt_domain.worker') keeps the vault volume; only a full terraform destroy removes it²⁶²⁷.
• Vault test fixture needs the CP state to publish a fake kubeconfig output; secret file() sources resolve to fixture files during plan tests (never real certs)²⁸²⁹.

Out-of-scope notes (discovered, not requested):

• Vault Secrets Operator (ARCHITECTURE.md §6) is not installed here — namespace name and service address keep it forward-compatible.
• No local-path-provisioner: woodpecker/values.yaml already assumes a local-path StorageClass, so that lands with issue #19, not this one³⁰.
• cert-manager rotation and HA/raft are future work — a single worker cannot host a raft quorum today.

Acceptance Criteria

• vault/ root applies cleanly: StatefulSet vault-0 runs on worker-large in the vault namespace with the official hashicorp/vault image + nginx sidecar, and reaches Ready after unseal.
• Vault data is written to /var/mnt/vault, whose backing qcow2 lives under data_root_path, and survives both a pod delete and terraform apply -replace='libvirt_domain.worker'.
• https://<tailnet-name>:8443 serves the Vault UI through the nginx sidecar and direct :8200 responds, via NodePorts 8443/8200 (range widened to 8080–32767).
• vault/README.md contains a # Setup section covering cert prep/import, apply order, init/unseal, host reachability, and legacy-stack decommission.
• Offline tests green: updated CP/worker plan tests, new vault/ plan tests, and updated pytest lint/fmt lists (terraform fmt, yamllint on values.yaml).
• hc-vault/ podman-compose stack removed from the repo and all doc references repointed after in-cluster cutover is verified.

Testing Plan

• terraform -chdir=control-plane test — assert the apiserver patch sets service-node-port-range = "8080-32767".
• terraform -chdir=worker test — vault volume is 20 GiB qcow2 named milner-worker-1-vault.qcow2, domain has a vdb virtio disk, and a config_patches entry yaml-decodes to UserVolumeConfig name=vault.
• terraform -chdir=vault test (mocked helm/kubernetes + CP fixture) — namespace carries the PSA privileged label; helm_release.vault has chart vault, namespace vault, and values markers (nodeSelector, tlsDisable=false, dataStorage.enabled=false); nginx Service NodePorts are 8443/8080.
• python -m pytest tests/ -v — fmt covers vault/, yamllint passes vault/values.yaml, no stale hc-vault lints.
• Manual (documented in # Setup, not automated): init/unseal; UI login over tailnet :8443; sealed→unsealed survives a worker reboot; cert renewal path unchanged from today.

Related Issues/PRs

• #18 — this issue
• #19 — Pod - Woodpecker CI (sibling migration; local-path StorageClass assumed there)
• #17 — IaC Worker Node (hosts the new data disk / user volume)
• #12 — IaC Control Plane (pool under data_root_path, kubeconfig output)
• #27 — Incremental deployment methodology (layer/removed doctrine applied here)
• #30 — Add kubectl bootstrap script (ergonomics for the # Setup manual steps)

Research Findings

• The official hashicorp/vault Helm chart is the recommended install path; it always renders a StatefulSet (standalone = 1 replica, file storage, OnDelete upgrades), configured via server.standalone.config.
• Repo pin hashicorp/vault:1.15.4 is EOL (1.15 line ended 2024-10); official images are multi-arch (arm64 OK), and disable_mlock = true remains the right setting in containers since image 2.0.2 removed IPC_LOCK.
• Talos enforces Pod Security baseline on every namespace except kube-system, and baseline forbids hostPath volumes — the vault namespace must be labeled enforce=privileged; a PVC + local-path provisioner would avoid it but adds a cluster component.
• Talos has no default StorageClass; user volumes (UserVolumeConfig, Talos ≥ 1.10) partition a dedicated disk and auto-propagate /var/mnt/<name> into the kubelet for hostPath/subPath use (schema verified against the Talos v1.11 docs).
• Kubernetes' default NodePort range (30000–32767) conflicts with ARCHITECTURE.md's 8200/8443/8080 — widening via the apiserver --service-node-port-range extraArg is a reconcile-only control-plane change.
• The vault-helm standalone-TLS pattern (generic Secret mounted under /vault/userconfig/…, referenced by listener tcp tls_*_file) matches the mkcert material the repo already owns, so certs are imported, not regenerated.

Pre-Mortem

1. The new Talos user volume never surfaces /var/mnt/vault to the kubelet, stranding the Vault pod — the plan itself flags the diskSelector CEL for verification "against Talos v1.11 docs at implement time", and a subtle mismatch means Talos rejects the worker machine config at apply or the volume isn't auto-propagated, leaving vault-0 Pending/CrashLooping behind a hostPath mount that never appears. Because the offline plan tests only mock the machine-config patch, this only shows up as a live-cluster failure where recovering risks reformatting or replacing the wrong disk.

2. Chart/image drift or a config regression breaks the real helm_release apply that mocked tests can't see — the plan pins chart 0.34.x / image 2.0.4 at research time with an explicit "re-verify at implement time" caveat, and if the live chart schema differs (dataStorage.enabled=false, extraContainers shape) or the tag is gone, the apply fails only against the cluster. A ported standalone.config HCL error (wrong storage/listener stanza for the 2.x line) then crash-loops the pod, and each retry round-trips a full apply cycle on the shared worker.

3. Cutover mishap loses existing Vault data or the tailnet endpoint — the file backend is not copyable between instances, so re-seeding via API is the only data path; if unseal keys/root token are mishandled or the old podman DNAT rules die before the NodePort retarget is verified, the live UI becomes unreachable and prior secrets are unrecoverable.

Naina: ## Overview Migrate Vault out of podman-compose into the official `hashicorp/vault` Helm chart as a single-replica StatefulSet in a new `vault` namespace on worker-large. The nginx Tailscale-TLS proxy survives as a pod sidecar, and Vault's data moves to a dedicated 20 GiB qcow2 volume that is physically stored under `data_root_path` and reaches the pod via a Talos user volume mounted at `/var/mnt/vault`. Three Terraform roots change (control-plane NodePort range, worker data disk, new `vault/` root driving the Helm release); every manual step is documented under `vault/README.md` → `# Setup`. ## Technical Approach 1. **Widen the NodePort range on the control plane** so ARCHITECTURE.md's 8200/8443/8080 ports stay valid (K8s default range is 30000–32767)^{[1](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/ARCHITECTURE.md?display=source#L271)}^{[2](https://kubernetes.io/docs/reference/command-line-tools-reference/kube-apiserver/)}. ```hcl # control-plane/main.tf — append to talos_machine_configuration_apply.control_plane config_patches yamlencode({ cluster = { apiServer = { extraArgs = { "service-node-port-range" = "8080-32767" } } } }) ``` 2. **Add a dedicated Vault data disk to worker-large** (volume in the CP-owned `talos-images` pool, so its backing file lands at `${data_root_path}/images/talos-kvm/milner-worker-1-vault.qcow2`)^{[3](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/control-plane/main.tf?display=source#L112)}^{[4](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/worker/main.tf?display=source#L185)}. ```hcl # worker/main.tf resource "libvirt_volume" "vault_data" { name = "${var.cluster_name}-worker-1-vault.qcow2" pool = var.pool_name capacity = 21474836480 # 20 GiB target = { format = { type = "qcow2" } } } # ...and append a second disk entry inside libvirt_domain.worker devices.disks (before the ISO concat): { driver = { type = "qcow2" } source = { volume = { pool = libvirt_volume.vault_data.pool, volume = libvirt_volume.vault_data.name } } target = { dev = "vdb", bus = "virtio" } } ``` 3. **Surface the disk to the kubelet as a Talos user volume** (machine-config document; `/var/mnt/vault` is auto-propagated into the kubelet so pods can hostPath it)^{[5](https://docs.siderolabs.com/talos/v1.11/configure-your-talos-cluster/storage-and-disk-management/disk-management/user)}^{[6](https://docs.siderolabs.com/talos/v1.11/reference/configuration/block/uservolumeconfig)}. ```yaml # worker/main.tf — extra config_patches entry on talos_machine_configuration_apply.worker apiVersion: v1alpha1 kind: UserVolumeConfig name: vault provisioning: diskSelector: match: disk.system_disk == false # verify CEL against Talos v1.11 docs at implement time minSize: 10GB ``` > ⚠️ check-assumptions: the `diskSelector` CEL expression `disk.system_disk == false` is not shown verbatim in the Talos v1.11 docs — the documented CEL evaluation context exposes a `system_disk` (`bool`) variable plus a `disk` resource (official examples use `!system_disk`)^{[7](https://docs.siderolabs.com/talos/v1.11/configure-your-talos-cluster/storage-and-disk-management/disk-management/common)}; the plan itself flags re-verification of the exact selector at implement time _^unverified_. ```bash talosctl get volumestatus u-vault # expect PHASE=ready, mounted at /var/mnt/vault ``` 4. **Create the `vault/` Terraform root**, driving the cluster with credentials parsed from the CP kubeconfig output (same remote-state pattern as `worker/`)^{[8](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/control-plane/main.tf?display=source#L416)}^{[9](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/worker/main.tf?display=source#L89)}. ```hcl # vault/main.tf locals { kc = yamldecode(data.terraform_remote_state.cp.outputs.kubeconfig) } provider "kubernetes" { host = local.kc.clusters[0].cluster.server cluster_ca_certificate = base64decode(local.kc.clusters[0].cluster["certificate-authority-data"]) client_certificate = base64decode(local.kc.users[0].user["client-certificate-data"]) client_key = base64decode(local.kc.users[0].user["client-key-data"]) } provider "helm" { kubernetes { host = local.kc.clusters[0].cluster.server # same base64-decoded cert/key fields as the kubernetes provider above } } resource "kubernetes_namespace_v1" "vault" { metadata { name = "vault" labels = { "pod-security.kubernetes.io/enforce" = "privileged" } # hostPath data (baseline forbids hostPath) } } ``` Also here: two `kubernetes_secret_v1` (certs read from gitignored paths in `terraform.auto.tfvars`), a ConfigMap for nginx, `helm_release.vault`, and the nginx Service. 5. **Install the chart with the current config ported into `server.standalone.config`** (chart PVC disabled — data on the hostPath volume; TLS and mlock settings carried over)^{[10](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/hc-vault/hc_vault/config/config.hcl?display=source#L1)}^{[11](https://developer.hashicorp.com/vault/docs/deploy/kubernetes/helm/configuration)}. ```yaml # vault/values.yaml (excerpt — block style for yamllint) global: tlsDisable: false server: nodeSelector: node-role: worker image: repository: hashicorp/vault tag: "2.0.4" # repo pin 1.15.4 is EOL; re-verify current line at implement time dataStorage: enabled: false volumes: - name: vault-data hostPath: path: /var/mnt/vault - name: vault-server-tls secret: secretName: vault-tls volumeMounts: - name: vault-data mountPath: /vault/data - name: vault-server-tls mountPath: /vault/userconfig/vault-server-tls readOnly: true standalone: enabled: true config: |- ui = true disable_mlock = true default_lease_ttl = "2304h" max_lease_ttl = "2304h" api_addr = "https://vault.vault.svc.cluster.local:8200" storage "file" { path = "/vault/data" } listener "tcp" { address = "[::]:8200" tls_cert_file = "/vault/userconfig/vault-server-tls/vault.crt" tls_key_file = "/vault/userconfig/vault-server-tls/vault.key" tls_client_ca_file = "/vault/userconfig/vault-server-tls/rootCA.pem" } user_lockout "all" { lockout_threshold = 25 lockout_duration = "60m" lockout_counter_reset = "60m" } service: type: NodePort nodePort: 8200 ``` 6. **Carry nginx over as `server.extraContainers`** (same-pod proxy to `127.0.0.1:8200`, Tailscale cert secret) and expose it with its own NodePort Service^{[12](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/hc-vault/nginx/conf/nginx.conf?display=source#L11)}^{[13](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/hc-vault/docker-compose.yml?display=source#L34)}. ```yaml # vault/values.yaml (excerpt) server: volumes: - name: tailscale-tls secret: secretName: tailscale-tls - name: nginx-conf configMap: name: vault-nginx-conf extraContainers: - name: nginx image: nginx:alpine ports: - containerPort: 443 - containerPort: 80 volumeMounts: - name: tailscale-tls mountPath: /etc/ssl readOnly: true - name: nginx-conf mountPath: /etc/nginx/conf.d/default.conf # replaces stock default.conf (its :80 server would clash) subPath: default.conf readOnly: true ``` ```nginx # vault/main.tf → kubernetes_config_map_v1 "vault-nginx-conf" (key: default.conf) server { listen 443 ssl; server_name <tailnet-name>.faun-snapper.ts.net; ssl_certificate /etc/ssl/tailscale-cert.crt; ssl_certificate_key /etc/ssl/tailscale-key.key; location / { proxy_pass https://127.0.0.1:8200; proxy_ssl_verify off; # mkcert upstream, as today proxy_set_header Host $host; proxy_set_header X-Forwarded-Proto $scheme; proxy_set_header Upgrade $http_upgrade; # Vault UI websockets proxy_set_header Connection "upgrade"; } } server { listen 80; return 301 https://$host$request_uri; } ``` ```hcl # vault/main.tf resource "kubernetes_service_v1" "nginx" { metadata { name = "vault-nginx"; namespace = kubernetes_namespace_v1.vault.metadata[0].name } spec { type = "NodePort" selector = { "app.kubernetes.io/name" = "vault", component = "server" } # chart pod labels port { port = 443; node_port = 8443; target_port = 443 } port { port = 80; node_port = 8080; target_port = 80 } } } ``` 7. **Document the manual steps under `vault/README.md` → `# Setup`** (issue criterion 2) and decommission the podman stack once the in-cluster Vault is verified: ```text 1. Certs: mkcert rootCA + vault.crt/key → secret vault-tls; `tailscale cert` → tailscale-tls. Paths live in gitignored vault/terraform.auto.tfvars — certs are never committed. 2. Apply order: control-plane → worker → vault. 3. vault operator init + unseal x3; store unseal keys/root token off-cluster. 4. Host reachability: retarget naina's existing tailnet→podman port forwarding to worker NodePorts 10.5.0.20:8443/8200 (the podman DNAT rules die with the old stack). 5. Existing data: re-seed via API or start fresh (file backend is not copyable between instances). 6. podman-compose down; delete hc-vault/ after cutover. ``` ## Implementation Details | Root / file | Change | |---|---| | `control-plane/main.tf`, `control-plane/tests/` | apiserver extraArg patch + plan-test assert | | `worker/main.tf` | `libvirt_volume.vault_data` (vdb) + `UserVolumeConfig` patch in `config_patches` | | `worker/tests/plan.tftest.hcl` | asserts for volume name/capacity/qcow2, vdb disk, user-volume patch | | `worker/cleanup.sh` | banner says "no persistent data" — must now warn the vault volume is destroyed on `destroy`^{[14](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/worker/cleanup.sh?display=source#L12)} | | `vault/` (new root) | `main.tf`, `values.yaml`, `terraform.tfvars`, `.gitignore`, `tests/plan.tftest.hcl` + fixture | | `vault/README.md` | runbook with `# Setup` (criterion 2) | | `tests/test_static.py`, `tests/test_terraform.py` | fmt/lint lists add `vault/` and drop `hc-vault` entries; add vault `terraform test` run^{[15](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/tests/test_static.py?display=source#L14)}^{[16](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/tests/test_terraform.py?display=source#L12)} | | `hc-vault/` | deleted at cutover (config.hcl/nginx.conf content folded into `values.yaml` + nginx ConfigMap) | | `README.md`, `AGENTS.md`, `PLAN.md` | repoint Vault references from `hc-vault/` → `vault/` | Dependencies and constraints: - New providers `hashicorp/kubernetes` + `hashicorp/helm` (repo first use); no flake change — `helm_release` removes the `helm` CLI dependency^{[17](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/control-plane/main.tf?display=source#L5)}^{[18](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/flake.nix?display=source#L21)}. - Chart tag/version drift is real (research-time pin: chart 0.34.x, image 2.0.4; repo pin 1.15.4 is EOL) — re-verify at implement time; keep `disable_mlock = true` and the arm64 multi-arch official image^{[19](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/hc-vault/hc_vault/Dockerfile?display=source#L1)}^{[20](https://endoflife.ai/hashicorp-vault/1.15)}^{[21](https://github.com/hashicorp/vault/releases/tag/v2.0.2)}^{[22](https://hub.docker.com/r/hashicorp/vault/tags)}^{[23](https://developer.hashicorp.com/vault/docs/deploy/kubernetes/helm)}. - Only the `vault` namespace gets the PSA `privileged` label; the chart's own securityContext (runAsNonRoot, drop ALL) stays baseline-compliant^{[24](https://docs.siderolabs.com/kubernetes-guides/security/pod-security)}^{[25](https://kubernetes.io/docs/concepts/security/pod-security-standards/)}. - Worker VM replacement (`-replace='libvirt_domain.worker'`) keeps the vault volume; only a full `terraform destroy` removes it^{[26](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/worker/main.tf?display=source#L105)}^{[27](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/worker/cleanup.sh?display=source#L22)}. - Vault test fixture needs the CP state to publish a fake `kubeconfig` output; secret `file()` sources resolve to fixture files during plan tests (never real certs)^{[28](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/worker/tests/plan.tftest.hcl?display=source#L20)}^{[29](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/worker/tests/fixtures/cp.tfstate?display=source#L1)}. Out-of-scope notes (discovered, not requested): - Vault Secrets Operator (ARCHITECTURE.md §6) is not installed here — namespace name and service address keep it forward-compatible. - No local-path-provisioner: `woodpecker/values.yaml` already assumes a `local-path` StorageClass, so that lands with issue #19, not this one^{[30](https://git.gloom.garden/tepichord/milner/src/commit/e7521360b7f61983968249854415e1e938e6f5cf/woodpecker/values.yaml?display=source#L4)}. - cert-manager rotation and HA/raft are future work — a single worker cannot host a raft quorum today. ## Acceptance Criteria - [ ] `vault/` root applies cleanly: StatefulSet `vault-0` runs on worker-large in the `vault` namespace with the official `hashicorp/vault` image + nginx sidecar, and reaches `Ready` after unseal. - [ ] Vault data is written to `/var/mnt/vault`, whose backing qcow2 lives under `data_root_path`, and survives both a pod delete and `terraform apply -replace='libvirt_domain.worker'`. - [ ] `https://<tailnet-name>:8443` serves the Vault UI through the nginx sidecar and direct `:8200` responds, via NodePorts 8443/8200 (range widened to 8080–32767). - [ ] `vault/README.md` contains a `# Setup` section covering cert prep/import, apply order, init/unseal, host reachability, and legacy-stack decommission. - [ ] Offline tests green: updated CP/worker plan tests, new `vault/` plan tests, and updated pytest lint/fmt lists (terraform fmt, yamllint on `values.yaml`). - [ ] `hc-vault/` podman-compose stack removed from the repo and all doc references repointed after in-cluster cutover is verified. ## Testing Plan - `terraform -chdir=control-plane test` — assert the apiserver patch sets `service-node-port-range = "8080-32767"`. - `terraform -chdir=worker test` — vault volume is 20 GiB qcow2 named `milner-worker-1-vault.qcow2`, domain has a vdb virtio disk, and a `config_patches` entry yaml-decodes to `UserVolumeConfig name=vault`. - `terraform -chdir=vault test` (mocked helm/kubernetes + CP fixture) — namespace carries the PSA `privileged` label; `helm_release.vault` has chart `vault`, namespace `vault`, and values markers (nodeSelector, `tlsDisable=false`, `dataStorage.enabled=false`); nginx Service NodePorts are 8443/8080. - `python -m pytest tests/ -v` — fmt covers `vault/`, yamllint passes `vault/values.yaml`, no stale `hc-vault` lints. - Manual (documented in `# Setup`, not automated): init/unseal; UI login over tailnet :8443; sealed→unsealed survives a worker reboot; cert renewal path unchanged from today. ## Related Issues/PRs - https://git.gloom.garden/tepichord/milner/issues/18 — this issue - https://git.gloom.garden/tepichord/milner/issues/19 — Pod - Woodpecker CI (sibling migration; local-path StorageClass assumed there) - https://git.gloom.garden/tepichord/milner/issues/17 — IaC Worker Node (hosts the new data disk / user volume) - https://git.gloom.garden/tepichord/milner/issues/12 — IaC Control Plane (pool under `data_root_path`, kubeconfig output) - https://git.gloom.garden/tepichord/milner/issues/27 — Incremental deployment methodology (layer/`removed` doctrine applied here) - https://git.gloom.garden/tepichord/milner/issues/30 — Add kubectl bootstrap script (ergonomics for the `# Setup` manual steps) ## Research Findings - The official `hashicorp/vault` Helm chart is the recommended install path; it always renders a StatefulSet (standalone = 1 replica, `file` storage, `OnDelete` upgrades), configured via `server.standalone.config`. - Repo pin `hashicorp/vault:1.15.4` is EOL (1.15 line ended 2024-10); official images are multi-arch (arm64 OK), and `disable_mlock = true` remains the right setting in containers since image 2.0.2 removed `IPC_LOCK`. - Talos enforces Pod Security `baseline` on every namespace except `kube-system`, and `baseline` forbids hostPath volumes — the `vault` namespace must be labeled `enforce=privileged`; a PVC + local-path provisioner would avoid it but adds a cluster component. - Talos has no default StorageClass; user volumes (`UserVolumeConfig`, Talos ≥ 1.10) partition a dedicated disk and auto-propagate `/var/mnt/<name>` into the kubelet for hostPath/subPath use (schema verified against the Talos v1.11 docs). - Kubernetes' default NodePort range (30000–32767) conflicts with ARCHITECTURE.md's 8200/8443/8080 — widening via the apiserver `--service-node-port-range` extraArg is a reconcile-only control-plane change. - The vault-helm standalone-TLS pattern (generic Secret mounted under `/vault/userconfig/…`, referenced by `listener tcp` `tls_*_file`) matches the mkcert material the repo already owns, so certs are imported, not regenerated. ## Pre-Mortem 1. **The new Talos user volume never surfaces `/var/mnt/vault` to the kubelet, stranding the Vault pod** — the plan itself flags the `diskSelector` CEL for verification "against Talos v1.11 docs at implement time", and a subtle mismatch means Talos rejects the worker machine config at apply or the volume isn't auto-propagated, leaving `vault-0` Pending/CrashLooping behind a hostPath mount that never appears. Because the offline plan tests only mock the machine-config patch, this only shows up as a live-cluster failure where recovering risks reformatting or replacing the wrong disk. 2. **Chart/image drift or a config regression breaks the real `helm_release` apply that mocked tests can't see** — the plan pins chart 0.34.x / image 2.0.4 at research time with an explicit "re-verify at implement time" caveat, and if the live chart schema differs (`dataStorage.enabled=false`, `extraContainers` shape) or the tag is gone, the apply fails only against the cluster. A ported `standalone.config` HCL error (wrong `storage`/`listener` stanza for the 2.x line) then crash-loops the pod, and each retry round-trips a full apply cycle on the shared worker. 3. **Cutover mishap loses existing Vault data or the tailnet endpoint** — the file backend is not copyable between instances, so re-seeding via API is the only data path; if unseal keys/root token are mishandled or the old podman DNAT rules die before the NodePort retarget is verified, the live UI becomes unreachable and prior secrets are unrecoverable.

tepichord added

status

implement

and removed

status

planning

labels 2026-09-08 16:12:03 -07:00

tepichord referenced this issue from a commit 2026-09-08 16:50:56 -07:00

feat(control-plane): widen NodePort range for Vault services

tepichord referenced this issue from a commit 2026-09-08 16:50:56 -07:00

feat(worker): add Vault data volume and Talos UserVolumeConfig

tepichord referenced this issue from a commit 2026-09-08 16:50:56 -07:00

feat(vault): add Terraform root for in-cluster Vault via Helm

tepichord referenced this issue from a commit 2026-09-08 16:50:56 -07:00

test: update pytest lint/fmt lists for vault migration

tepichord referenced this issue from a commit 2026-09-08 16:50:56 -07:00

docs: repoint Vault references from hc-vault/ to vault/

tepichord referenced this issue from a commit 2026-09-08 16:50:56 -07:00

docs(infra): shorten verbose comments, add missing section context

tepichord referenced this issue from a commit 2026-09-08 16:50:56 -07:00

fix(vault): ignore vault/certs holding real TLS private keys

tepichord referenced this issue from a commit 2026-09-08 16:50:56 -07:00

refactor(vault): drop meaningless nginx server_name from apiserver URL

tepichord referenced this issue from a pull request that will close it, 2026-09-08 16:51:00 -07:00

Pod - Hashicorp Vault #33

tepichord 2026-09-08 16:51:01 -07:00

• closed this issue
• added
  status
  reviewing
  and removed
  status
  implement
  labels

tepichord referenced this issue from a commit 2026-09-09 12:31:07 -07:00

Pod - Hashicorp Vault (#33)

tepichord added

status

document

and removed

status

reviewing

labels 2026-09-09 12:31:15 -07:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

feat/109-nix-base

feat/109-layered-nix-store

feat/110-zot-push-credential

feat/90-annia-pause-lever-oz-console

feat/87-tailscale-egress-from-cluster

chore/ci-step-resources

docs/vault-secret-prompt

feat/42-support-backups-persistent-data

feat/41-integrate-restic-tfstate-backup

feat/3-setting-up-agent-development

kvm-kubernetes-setup

No results found.

Labels

Clear labels   

blocked

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

effort

high

  

effort

low

  

effort

medium

  

effort

minimal

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

pr

feedback

Used for PRs, indicates that feedback has been given and agent needs to make changes

  

pr

reviewing

Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  

question

More information is needed

  

status

backlog

Ignore, no work needed yet, human will assign this work by changing to status/planning

  

status

document

Record findings, create flashcards, and document knowledge in knowledgebase for the future

  

status

done

Completed when PR is merged, no more work needed.

  

status

implement

Make the code changes. Should be done by a faster and cheaper model

  

status

pending

Give human a chance to review planning details

  

status

planning

Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  

status

reviewing

PR is created and feedback loop begins between human and agent

  

waiting

  

wontfix

This won't be fixed

No labels

blocked

bug

duplicate

effort

high

effort

low

effort

medium

effort

minimal

enhancement

help wanted

invalid

pr

feedback

pr

reviewing

question

status

backlog

status

document

status

done

status

implement

status

pending

status

planning

status

reviewing

waiting

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

MVP

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

tepichord/milner#18

No description provided.