tepichord/milner

Fork 0

Automate Cluster Deployments with Flux #46

New issue

Closed

opened 2026-09-10 22:32:00 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-10 22:32:00 -07:00

Owner

What GitOps actually is

Your desired state lives in Git, and a controller running inside the cluster continuously reads Git and makes the cluster match it. The canonical four properties (CNCF
OpenGitOps): declarative, versioned and immutable, pulled automatically, continuously reconciled.

The load-bearing word is pulled. CI running kubectl apply is not GitOps — that's still push. When the cluster pulls:

No cluster credentials ever leave the cluster. CI holds none.
Deploys still happen when CI is down or your laptop is closed.
git log is the audit trail and git revert is the rollback button.

And the reason it works for Kubernetes specifically: Kubernetes already has a declarative API and a reconciliation loop. GitOps doesn't add a second desired-state system — it
only supplies the desired state. That's precisely the redundancy Terraform introduces, which is why GitOps beats Terraform here and not in general.

It is not a wholesale Terraform replacement

This is the part to be clear about: GitOps only applies where an in-cluster controller exists. Nothing inside the cluster can create a libvirt domain or format a host disk. So
it's not a swap — it's a split by layer:

┌───────────────────────────────────────────┬─────────────────────────────────────────────┐
│ Root │ Owner after │
├───────────────────────────────────────────┼─────────────────────────────────────────────┤
│ control-plane/, worker/, data/ │ Terraform (or a script) — unchanged │
├───────────────────────────────────────────┼─────────────────────────────────────────────┤
│ cloudflare/ │ Terraform — unchanged; its genuine best fit │
├───────────────────────────────────────────┼─────────────────────────────────────────────┤
│ storage/, vault/, woodpecker/, pocket-id/ │ Flux │
├───────────────────────────────────────────┼─────────────────────────────────────────────┤
│ evolved-npcs/ + future apps │ Flux │
└───────────────────────────────────────────┴─────────────────────────────────────────────┘

The tree you'd end up with

  milner/
  ├── control-plane/  worker/  data/  cloudflare/    # Terraform, untouched
  └── gitops/
      └── clusters/milner/
          ├── flux-system/          # GitRepository + Kustomization — Flux writes these for itself
          ├── infrastructure/
          │   └── storage/          # local-path HelmRelease + your two StorageClasses
          └── apps/
              ├── vault/            # HelmRelease + values.yaml + Tailscale ingress
              ├── woodpecker/
              ├── pocket-id/
              └── evolved-npcs/     # plain Deployment/Service/ScaledObject + KEDA

Each app directory is a kustomization.yaml plus manifests. For the charts you already deploy, the four Terraform roots collapse into HelmRelease objects — same chart, same
values.yaml, roughly 40 lines of YAML instead of 200 lines of HCL. Only flux-system/ is special: Flux bootstraps it, then manages its own config from Git like everything else.

One deploy, before and after

Today, bumping Vault's chart: edit vault/main.tf → terraform -chdir=vault plan → apply, from a laptop on the tailnet holding a kubeconfig.

GitOps: edit the HelmRelease → commit → push. Flux notices within the sync interval and applies. Rollback is git revert. And drift is actively corrected — if you kubectl -n
vault edit deploy/vault, Flux reverts it. Under Terraform that edit is invisible until the next plan, and then it shows up as a change you didn't make.

What you actually give up

plan as a mandatory gate. A merge is the apply. Flux's nearest equivalent is flux diff kustomization , a dry-run against the live cluster — but you have to make it a
gate yourself in the pipeline. This is the real loss.
Terraform's dependency graph. Flux has dependsOn and healthChecks; they're coarser. Adequate at your scale, weaker in general.
Your tests/plan.tftest.hcl suites. Replaced by kustomize build plus schema/policy validation.

Two gotchas

Secrets. Everything is in Git now, so nothing sensitive can be. Your current flow hand-creates Secrets from Vault (OPERATIONS §7c). Use Vault Secrets Operator — already in
PLAN.md — so VSO pulls from Vault and creates the Secret at runtime while Git holds only a reference. Flux's alternative is SOPS+age, but you already run Vault, so VSO is
the natural fit.
Bootstrap. Something must install Flux and point it at the repo; that's the one unavoidable push step. Either flux bootstrap git from the laptop (Flux commits its own
config and self-manages after that), or a small gitops/ Terraform root that installs the controllers and then stops.

The one-line summary: Terraform keeps the machine; Flux keeps the cluster.

(A footnote worth knowing: external-dns with the Cloudflare provider can generate DNS records from your Ingresses, which would move part of cloudflare/ into GitOps too. Not
necessary now — the Tunnel and account-level objects stay in Terraform regardless.)

What GitOps actually is Your desired state lives in Git, and a controller running inside the cluster continuously reads Git and makes the cluster match it. The canonical four properties (CNCF OpenGitOps): declarative, versioned and immutable, pulled automatically, continuously reconciled. The load-bearing word is pulled. CI running kubectl apply is not GitOps — that's still push. When the cluster pulls: - No cluster credentials ever leave the cluster. CI holds none. - Deploys still happen when CI is down or your laptop is closed. - git log is the audit trail and git revert is the rollback button. And the reason it works for Kubernetes specifically: Kubernetes already has a declarative API and a reconciliation loop. GitOps doesn't add a second desired-state system — it only supplies the desired state. That's precisely the redundancy Terraform introduces, which is why GitOps beats Terraform here and not in general. It is not a wholesale Terraform replacement This is the part to be clear about: GitOps only applies where an in-cluster controller exists. Nothing inside the cluster can create a libvirt domain or format a host disk. So it's not a swap — it's a split by layer: ┌───────────────────────────────────────────┬─────────────────────────────────────────────┐ │ Root │ Owner after │ ├───────────────────────────────────────────┼─────────────────────────────────────────────┤ │ control-plane/, worker/, data/ │ Terraform (or a script) — unchanged │ ├───────────────────────────────────────────┼─────────────────────────────────────────────┤ │ cloudflare/ │ Terraform — unchanged; its genuine best fit │ ├───────────────────────────────────────────┼─────────────────────────────────────────────┤ │ storage/, vault/, woodpecker/, pocket-id/ │ Flux │ ├───────────────────────────────────────────┼─────────────────────────────────────────────┤ │ evolved-npcs/ + future apps │ Flux │ └───────────────────────────────────────────┴─────────────────────────────────────────────┘ The tree you'd end up with ``` milner/ ├── control-plane/ worker/ data/ cloudflare/ # Terraform, untouched └── gitops/ └── clusters/milner/ ├── flux-system/ # GitRepository + Kustomization — Flux writes these for itself ├── infrastructure/ │ └── storage/ # local-path HelmRelease + your two StorageClasses └── apps/ ├── vault/ # HelmRelease + values.yaml + Tailscale ingress ├── woodpecker/ ├── pocket-id/ └── evolved-npcs/ # plain Deployment/Service/ScaledObject + KEDA ``` Each app directory is a kustomization.yaml plus manifests. For the charts you already deploy, the four Terraform roots collapse into HelmRelease objects — same chart, same values.yaml, roughly 40 lines of YAML instead of 200 lines of HCL. Only flux-system/ is special: Flux bootstraps it, then manages its own config from Git like everything else. One deploy, before and after Today, bumping Vault's chart: edit vault/main.tf → terraform -chdir=vault plan → apply, from a laptop on the tailnet holding a kubeconfig. GitOps: edit the HelmRelease → commit → push. Flux notices within the sync interval and applies. Rollback is git revert. And drift is actively corrected — if you kubectl -n vault edit deploy/vault, Flux reverts it. Under Terraform that edit is invisible until the next plan, and then it shows up as a change you didn't make. What you actually give up - plan as a mandatory gate. A merge is the apply. Flux's nearest equivalent is flux diff kustomization <name>, a dry-run against the live cluster — but you have to make it a gate yourself in the pipeline. This is the real loss. - Terraform's dependency graph. Flux has dependsOn and healthChecks; they're coarser. Adequate at your scale, weaker in general. - Your tests/plan.tftest.hcl suites. Replaced by kustomize build plus schema/policy validation. Two gotchas 1. Secrets. Everything is in Git now, so nothing sensitive can be. Your current flow hand-creates Secrets from Vault (OPERATIONS §7c). Use Vault Secrets Operator — already in PLAN.md — so VSO pulls from Vault and creates the Secret at runtime while Git holds only a reference. Flux's alternative is SOPS+age, but you already run Vault, so VSO is the natural fit. 2. Bootstrap. Something must install Flux and point it at the repo; that's the one unavoidable push step. Either flux bootstrap git from the laptop (Flux commits its own config and self-manages after that), or a small gitops/ Terraform root that installs the controllers and then stops. The one-line summary: Terraform keeps the machine; Flux keeps the cluster. (A footnote worth knowing: external-dns with the Cloudflare provider can generate DNS records from your Ingresses, which would move part of cloudflare/ into GitOps too. Not necessary now — the Tunnel and account-level objects stay in Terraform regardless.)

tepichord added the

labels

2026-09-10 22:32:00 -07:00

tepichord added this to the MVP project

2026-09-10 22:32:00 -07:00

tepichord commented

2026-09-10 23:09:16 -07:00

Author

Owner

Naina:

Overview

Move ownership of the four in-cluster Terraform roots (storage/, vault/, woodpecker/, pocket-id/) to Flux v2 GitOps under gitops/clusters/milner/, while control-plane/, worker/, data/ and cloudflare/ stay in Terraform.^{Verified (in-repo): all four roots exist and are listed in the AGENTS repo map — AGENTS.md#L22} flux bootstrap git is the single one-time push; after that every cluster change is a commit and the old roots are removed from Terraform state only once Flux reports Healthy. Secrets move from hand-run kubectl create secret (OPERATIONS §7c) to Vault Secrets Operator, so git holds only Vault references.^{Verified (in-repo): §7c runs kubectl -n woodpecker create secret generic — OPERATIONS.md#L771}

Technical Approach

Add the Flux CLI and validators to the dev shell^{Verified (in-repo): the dev shell already declares buildInputs — flake.nix#L21}

# flake.nix — buildInputs
fluxcd
kubeconform

Create four Flux Kustomizations, one per dependency layer^{Verified (external): kustomize.toolkit.fluxcd.io/v1 and helm.toolkit.fluxcd.io/v2 are the GA APIs from Flux v2.3 — Flux 2.3 GA}

Flux Kustomization	`spec.path`	`dependsOn`	Owns
`infrastructure`	`./gitops/clusters/milner/infrastructure`	—	namespaces, local-path, `ingress-proxies`
`vault`	`./gitops/clusters/milner/vault`	`infrastructure`	Vault HelmRelease + Ingress
`secrets`	`./gitops/clusters/milner/secrets`	`vault`	VSO, VaultAuth, VaultStaticSecrets
`apps`	`./gitops/clusters/milner/apps`	`secrets`	Woodpecker, Pocket ID, evolved-npcs stub

# gitops/clusters/milner/vault.yaml
apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata: {name: vault, namespace: flux-system}
spec:
  interval: 10m
  path: ./gitops/clusters/milner/vault
  prune: true
  wait: true
  dependsOn: [{name: infrastructure}]
  sourceRef: {kind: GitRepository, name: flux-system}

Declare all namespaces (with PSA labels) in the infrastructure layer^{Verified (in-repo): the labels match the roots that already declare them, e.g. vault privileged — vault/main.tf#L75, woodpecker baseline — woodpecker/main.tf#L79, pocket-id baseline — pocket-id/main.tf#L81}

Namespace	`pod-security.kubernetes.io/enforce`
`local-path-storage`	`privileged`
`vault`	`privileged`
`woodpecker`	`baseline`
`woodpecker-runtime`	`privileged`
`pocket-id`	`baseline`
`vault-secrets-operator`	`baseline`

Collapse storage/ into an OCIRepository + HelmRelease (chart version pinned exactly)^{Verified (in-repo): storage/ already pulls oci://ghcr.io/rancher/local-path-provisioner/charts at 0.0.37 — storage/main.tf#L88; the OCI layerSelector needs the Helm chart media type — Flux HelmRelease guide}

# gitops/clusters/milner/infrastructure/local-path/helmrelease.yaml
apiVersion: source.toolkit.fluxcd.io/v1
kind: OCIRepository
metadata: {name: local-path-provisioner, namespace: local-path-storage}
spec:
  interval: 10m
  url: oci://ghcr.io/rancher/local-path-provisioner/charts
  layerSelector: {mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"}
  ref: {tag: "0.0.37"}
---
apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: {name: local-path-provisioner, namespace: local-path-storage}
spec:
  interval: 10m
  chartRef: {kind: OCIRepository, name: local-path-provisioner}
  valuesFrom: [{kind: ConfigMap, name: local-path-provisioner-values}]

Keep values.yaml files as-is and expose each as a ConfigMap

# gitops/clusters/milner/apps/pocket-id/kustomization.yaml
generatorOptions: {disableNameSuffixHash: true}
configMapGenerator:
  - {name: pocket-id-values, files: [values.yaml]}
resources: [persistentvolumeclaim.yaml, helmrelease.yaml, ingress.yaml]

Move the shared ingress-proxies ProxyGroup out of vault/ into infrastructure

# gitops/clusters/milner/infrastructure/tailscale.yaml
apiVersion: tailscale.com/v1alpha1
kind: ProxyGroup
metadata: {name: ingress-proxies}
spec: {type: ingress, replicas: 1}

Vault uses a HelmRepository + spec.chart.spec.sourceRef (HTTP repo, not OCI)^{Verified (in-repo): Vault is installed from the HTTP repo https://helm.releases.hashicorp.com, chart vault 0.34.1 — vault/main.tf#L89}

apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: {name: vault, namespace: vault}
spec:
  interval: 10m
  chart:
    spec:
      chart: vault
      version: "0.34.1"
      sourceRef: {kind: HelmRepository, name: hashicorp}
  valuesFrom: [{kind: ConfigMap, name: vault-values}]

Install VSO, then reference Vault — no secret material in git^{Verified (external): the hashicorp/vault-secrets-operator chart is at 1.5.0 — VSO installation; the repo already uses the secrets.hashicorp.com/v1beta1 CRD — ARCHITECTURE.md#L200}

# gitops/clusters/milner/secrets/vaultauth.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultAuth
metadata: {name: default, namespace: woodpecker}
spec:
  method: kubernetes
  mount: kubernetes
  kubernetes: {role: woodpecker, serviceAccount: default}
---
# gitops/clusters/milner/secrets/woodpecker.yaml
apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: {name: woodpecker-forgejo-oauth, namespace: woodpecker}
spec:
  type: kv-v2
  mount: secret
  path: woodpecker/forgejo-oauth
  refreshAfter: 1h
  destination: {name: woodpecker-forgejo-oauth}
  vaultAuthRef: default
  rolloutRestartTargets: [{kind: Deployment, name: woodpecker-server}]

Chart hashicorp/vault-secrets-operator pinned to 1.5.0; VaultConnection points at http://vault.vault.svc.cluster.local:8200.
One VaultAuth/VaultStaticSecret per Secret: woodpecker/{forgejo-oauth,grpc} and pocket-id/{encryption-key}.
Replaces OPERATIONS §7c's kubectl create secret; Vault paths already exist from §7b/§5a.^{Verified (in-repo): secret/woodpecker/forgejo-oauth — OPERATIONS.md#L754, secret/pocket-id/encryption-key — OPERATIONS.md#L400}

Woodpecker and Pocket ID HelmReleases reuse today's values unchanged^{Verified (in-repo): oci://ghcr.io/woodpecker-ci/helm 3.7.3 — woodpecker/main.tf#L101; oci://ghcr.io/hobbit44/pocket-id-helm 2.4.0 — pocket-id/main.tf#L135}

oci://ghcr.io/woodpecker-ci/helm woodpecker 3.7.3, namespace woodpecker, Ingress unchanged.
oci://ghcr.io/hobbit44/pocket-id-helm pocket-id 2.4.0, namespace pocket-id.
Pocket ID's PVC becomes a Flux-managed manifest copied verbatim from pocket-id/main.tf (persistent, 5Gi, no bind wait) — no chart change needed.^{Verified (in-repo): the claim is on persistent with 5Gi and wait_until_bound = false — pocket-id/main.tf#L101, asserted by tests/test_static.py#L246}

Bootstrap script (the one unavoidable push), credentials from env only^{Verified (external): flux bootstrap git accepts --url, --branch, --path and --private-key-file — flux bootstrap git}

# gitops/bootstrap.sh <deploy-key-path>
flux bootstrap git \
  --url=ssh://git@git.gloom.garden/tepichord/milner.git \
  --branch=main \
  --path=gitops/clusters/milner \
  --private-key-file="$1"

flux bootstrap writes gitops/clusters/milner/flux-system/ (committed, reviewed); patch controller node selectors there via gitops/clusters/milner/flux-system/kustomization.yaml:^{Verified (external): bootstrap generates gotk-components.yaml, gotk-sync.yaml and kustomization.yaml under the target path — Bootstrap customization}

patches:
  - target: {kind: Deployment}
    patch: |-
      - op: add
        path: /spec/template/spec/nodeSelector
        value: {node-role: worker}

Hand off ownership from Terraform after Flux is Healthy^{Verified (in-repo): the listed Vault addresses exist — kubernetes_namespace_v1.vault vault/main.tf#L71, helm_release.vault #L87, kubernetes_manifest.vault_proxy_group #L107, kubernetes_ingress_v1.vault #L121}

flux get kustomizations -A          # all Ready, all HelmReleases Healthy
terraform -chdir=vault state rm \
  kubernetes_namespace_v1.vault helm_release.vault \
  kubernetes_manifest.vault_proxy_group kubernetes_ingress_v1.vault
# ...same for storage/ woodpecker/ pocket-id/...
git rm -r storage/ vault/ woodpecker/ pocket-id/

Release name + namespace must match exactly so Flux upgrades the existing Helm releases instead of erroring; chart versions are pinned to the deployed versions to avoid an unintended bump on first reconcile. ^{unverified: helm-controller's adoption/upgrade-in-place semantics for a pre-existing release could not be confirmed from docs; needs a live cluster.}
Order matters: state rm before deleting the root, or a later terraform destroy would delete Flux-owned objects.

Port the existing regression guards instead of deleting them^{Verified (in-repo): all five named guards exist — tests/test_static.py#L208, #L246, #L320, #L353, #L376}

Guard in `tests/test_static.py`	New target
`test_vault_values_keep_data_mount_under_server`	`gitops/clusters/milner/vault/values.yaml`
`test_pocket_id_persists_to_the_durable_storage_class`	`.../apps/pocket-id/values.yaml` + `persistentvolumeclaim.yaml`
`test_pocket_id_external_url_matches_ingress_magicdns_name`	`.../apps/pocket-id/ingress.yaml`
`test_funnel_is_retired_from_the_woodpecker_root`	`.../apps/woodpecker/ingress.yaml`
`test_woodpecker_webhook_host_matches_cloudflare_route`	split: woodpecker half reads the new values path

CI gate: keep it credential-free^{Verified (external): ghcr.io/fluxcd/flux-cli:v2.4.0 publishes a linux/arm64 manifest — flux-cli package; the woodpecker-runtime namespace already exists — woodpecker/main.tf#L84}

# .woodpecker.yaml — new; agent runs in woodpecker-runtime
steps:
  - name: build
    image: ghcr.io/fluxcd/flux-cli:v2.4.0   # arm64
    commands:
      - flux build kustomization infrastructure --path=gitops/clusters/milner/infrastructure --kustomization-file=gitops/clusters/milner/infrastructure.yaml --dry-run > /tmp/rendered.yaml
      - kubeconform -strict -ignore-missing-schemas /tmp/rendered.yaml
  - name: test
    commands: [python -m pytest tests/ -v]

flux diff kustomization <name> needs a kubeconfig, so it stays a manual laptop-side pre-merge check documented in OPERATIONS (exit 0 = no drift, 1 = diff, >1 = error) — CI keeps holding no cluster credentials, per the issue.^{Verified (external): documented exit status 0 no differences, 1 differences, >1 error — flux diff kustomization}

Update docs: README, AGENTS repo map / "How Changes Ship", OPERATIONS bring-up order (§7), and a new Flux section (bootstrap, unseal-blocks-reconcile note, manual flux diff, VaultAuth k8s-auth enablement).

Implementation Details

Created

Path	Purpose
`gitops/bootstrap.sh`	one-time `flux bootstrap git` wrapper
`gitops/clusters/milner/{infrastructure,vault,secrets,apps}.yaml`	the four Flux Kustomizations
`gitops/clusters/milner/infrastructure/`	namespaces, local-path HelmRelease, `ingress-proxies`
`gitops/clusters/milner/vault/`	Vault HelmRelease + Ingress + `values.yaml`
`gitops/clusters/milner/secrets/`	VSO HelmRelease, VaultConnection, VaultAuth, VaultStaticSecrets
`gitops/clusters/milner/apps/{woodpecker,pocket-id,evolved-npcs}/`	app HelmReleases, Ingresses, PVC, stub
`gitops/clusters/milner/flux-system/`	written by bootstrap, then committed
`tests/test_gitops.py`	build + schema + ported guards
`.woodpecker.yaml`	offline validation pipeline

Modified: flake.nix (fluxcd, kubeconform), tests/test_static.py (repoint guards, drop the four roots from test_terraform_fmt/test_yamllint), tests/test_terraform.py (drop the four roots), README.md, AGENTS.md, OPERATIONS.md, .env.example (deploy-key path only if needed).^{Verified (in-repo): test_terraform_fmt/test_yamllint currently reference the roots — tests/test_static.py#L22; tests/test_terraform.py has per-root jobs — #L46}

Deleted after handoff: storage/, vault/, woodpecker/, pocket-id/ (Terraform roots incl. tests/fixtures).

Dependencies / constraints

Pinned chart versions: local-path-provisioner 0.0.37, vault 0.34.1, woodpecker 3.7.3, pocket-id 2.4.0, VSO 1.5.0.^{Verified (in-repo + external): storage/main.tf#L88, vault/main.tf#L91, woodpecker/main.tf#L101, pocket-id/main.tf#L135; VSO 1.5.0 per Hashicorp docs}
All images must be arm64; flux controllers and CI images included.
All pods keep nodeSelector: {node-role: worker}, including Flux controllers.^{Verified (in-repo): present in the existing values — vault/values.yaml#L89, woodpecker/values.yaml#L24, pocket-id/values.yaml#L68}
No secret material in git — VSO references only; kubectl create secret steps retired.
vault Kustomization uses wait: true, so a sealed Vault blocks secrets/apps until §4c unseal runs — intentional, documented. ^{unverified: depends on live reconcile behavior of the sealed StatefulSet; not confirmable offline.}
VaultAuth needs Vault's auth/kubernetes enabled once (OPERATIONS); static-token VaultAuth is the fallback.^{Verified (external): the Kubernetes auth method must be enabled and configured in Vault — Vault Kubernetes auth}

Out-of-scope notes

The Tailscale operator stays imperative (OPERATIONS §3): its OAuth secret has no safe GitOps home yet. Only its ingress-proxies ProxyGroup moves to Flux.
apps/evolved-npcs/ ships as a README + empty kustomization, not wired into apps/kustomization.yaml; no KEDA/RabbitMQ workloads here.
Cloudflare and external-dns are not touched.
HA control plane, VSO for evolved-npcs, and Flux image automation are not part of this change.

Acceptance Criteria

gitops/bootstrap.sh runs flux bootstrap git successfully; flux get kustomizations -A shows flux-system, infrastructure, vault, secrets, apps all Ready.
All four workloads reconcile from Git: kubectl -n pocket-id delete svc pocket-id is recreated by Flux within one interval.
terraform state rm has been run for every migrated resource and storage/, vault/, woodpecker/, pocket-id/ are deleted; terraform -chdir=control-plane plan, worker, data, cloudflare still plan clean.
No secret material is committed (git grep finds no Vault values / tokens); Woodpecker and Pocket ID start without the OPERATIONS §7c kubectl create secret steps.
Every ported regression guard in tests/test_gitops.py passes, and python -m pytest tests/ -v is green.
The offline CI gate (.woodpecker.yaml) fails on a deliberately broken manifest and passes on the committed tree.
README/AGENTS/OPERATIONS describe Flux as the cluster owner; no stale terraform -chdir=vault apply instructions remain.

Testing Plan

tests/test_gitops.py: kustomize build every Flux path; assert every namespace carries an explicit PSA enforce label; assert every HelmRelease is helm.toolkit.fluxcd.io/v2 with a pinned version; assert no Secret has literal data/stringData.
Ported guards: Vault data mount under server:, Pocket ID PVC on persistent with /app/data mount, Pocket ID appUrl vs Ingress tls.hosts, Woodpecker webhook host == published Cloudflare route, no tailscale.com/funnel.
flux build kustomization <name> --dry-run for all four layers + kubeconform -strict -ignore-missing-schemas.
Manual pre-merge: flux diff kustomization apps --path=./gitops/clusters/milner/apps returns 0 on a clean tree, non-zero-but-not-error when a manifest changes.
Live drift check: hand-edit deploy/vault and confirm Flux reverts it.
Fresh-cluster path: operator bootstrap → flux bootstrap → unseal Vault → flux get helmreleases -A all Healthy.

Closes #46
#38 (Pocket ID), #37 (Woodpecker), #33 (Vault) — the Terraform roots being migrated
#32 / #31 (kubectl bootstrap script) — the imperative-bootstrap pattern this replaces
PLAN.md target architecture (VSO, Woodpecker CI/CD), ARCHITECTURE.md §6 (VSO for evolved-npcs)

Research Findings

flux bootstrap git is Flux's recommended install path and is idempotent; Forgejo is Gitea-compatible, so flux bootstrap gitea --token-auth also works but needs a PAT (installation, gitea).
Helm APIs reached GA in Flux v2.3: use helm.toolkit.fluxcd.io/v2, source.toolkit.fluxcd.io/v1, kustomize.toolkit.fluxcd.io/v1; v1beta2 examples are stale (v2.3 GA).
.spec.chartRef references OCIRepository/HelmChart; HTTP repos keep spec.chart.spec.sourceRef with a HelmRepository (helmreleases guide).
Ordering is dependsOn (ready-gated, not apply-gated) plus healthChecks/wait: true; Flux already applies Namespaces before namespaced resources, so PSA labels land before pods (kustomizations).
VSO is the idiomatic fit while Vault is in-cluster: chart 1.5.0, CRD secrets.hashicorp.com/v1beta1, and VaultStaticSecret syncs Vault paths into native Secrets (VSO install).
flux diff kustomization exits 0/1/>1 (no diff / diff / error) and needs cluster access; flux build kustomization --dry-run renders offline (flux diff).
flux bootstrap generates and commits flux-system/ (gotk-components.yaml, gotk-sync.yaml, kustomization.yaml), which is the patch point for controller node selectors (bootstrap customization).

Pre-Mortem

The Terraform→Flux handoff runs out of order and destroys live objects instead of transferring them. Four roots must each have terraform state rm run before git rm -r deletes the directory, and any root still owning a namespace in state will delete that namespace — taking the Pocket ID PVC and the local-path StorageClasses with it — on the next terraform destroy. The plan documents this ordering in prose but adds no automated guard (e.g. a pre-destroy check or a test asserting the roots are gone from state), so a forgotten state rm is only discovered when data is already deleted.
Flux fails to adopt the existing Helm releases, so the first reconcile errors or replaces workloads. Flux derives the release name and namespace from the HelmRelease object and diffs its rendered values against the live release, so any divergence from what Terraform installed — re-expressing values through valuesFrom ConfigMaps instead of file("values.yaml"), a renamed Pocket ID PVC (pocket_id_data), or a namespace/PSA difference — surfaces only during the live cutover. Because the acceptance criterion requires only that Flux reports Ready, an unintended upgrade or reinstall can pass while Woodpecker and Vault are briefly unavailable.
The dependsOn/wait: true chain stalls behind manual, out-of-band steps. infrastructure still depends on the imperatively-installed Tailscale operator for the ProxyGroup CRD, and vault is wait: true behind a Vault that must be unsealed by hand, so secrets and apps never become Ready if either prerequisite is skipped. This reproduces on every cluster rebuild and on any Vault pod restart, leaving the cluster serving stale manifests with no automatic recovery.

Naina: ## Overview Move ownership of the four in-cluster Terraform roots (`storage/`, `vault/`, `woodpecker/`, `pocket-id/`) to Flux v2 GitOps under `gitops/clusters/milner/`, while `control-plane/`, `worker/`, `data/` and `cloudflare/` stay in Terraform.Verified (in-repo): all four roots exist and are listed in the AGENTS repo map — [AGENTS.md#L22](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/AGENTS.md?display=source#L22) `flux bootstrap git` is the single one-time push; after that every cluster change is a commit and the old roots are removed from Terraform state only once Flux reports Healthy. Secrets move from hand-run `kubectl create secret` (OPERATIONS §7c) to Vault Secrets Operator, so git holds only Vault references.Verified (in-repo): §7c runs `kubectl -n woodpecker create secret generic` — [OPERATIONS.md#L771](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/OPERATIONS.md?display=source#L771) ## Technical Approach 1. **Add the Flux CLI and validators to the dev shell**Verified (in-repo): the dev shell already declares `buildInputs` — [flake.nix#L21](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/flake.nix?display=source#L21) ```nix # flake.nix — buildInputs fluxcd kubeconform ``` 2. **Create four Flux Kustomizations, one per dependency layer**Verified (external): `kustomize.toolkit.fluxcd.io/v1` and `helm.toolkit.fluxcd.io/v2` are the GA APIs from Flux v2.3 — [Flux 2.3 GA](https://fluxcd.io/blog/2024/05/flux-v2.3.0/) | Flux Kustomization | `spec.path` | `dependsOn` | Owns | |---|---|---|---| | `infrastructure` | `./gitops/clusters/milner/infrastructure` | — | namespaces, local-path, `ingress-proxies` | | `vault` | `./gitops/clusters/milner/vault` | `infrastructure` | Vault HelmRelease + Ingress | | `secrets` | `./gitops/clusters/milner/secrets` | `vault` | VSO, VaultAuth, VaultStaticSecrets | | `apps` | `./gitops/clusters/milner/apps` | `secrets` | Woodpecker, Pocket ID, evolved-npcs stub | ```yaml # gitops/clusters/milner/vault.yaml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: {name: vault, namespace: flux-system} spec: interval: 10m path: ./gitops/clusters/milner/vault prune: true wait: true dependsOn: [{name: infrastructure}] sourceRef: {kind: GitRepository, name: flux-system} ``` 3. **Declare all namespaces (with PSA labels) in the infrastructure layer**Verified (in-repo): the labels match the roots that already declare them, e.g. vault `privileged` — [vault/main.tf#L75](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/vault/main.tf?display=source#L75), woodpecker `baseline` — [woodpecker/main.tf#L79](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/main.tf?display=source#L79), pocket-id `baseline` — [pocket-id/main.tf#L81](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/pocket-id/main.tf?display=source#L81) | Namespace | `pod-security.kubernetes.io/enforce` | |---|---| | `local-path-storage` | `privileged` | | `vault` | `privileged` | | `woodpecker` | `baseline` | | `woodpecker-runtime` | `privileged` | | `pocket-id` | `baseline` | | `vault-secrets-operator` | `baseline` | 4. **Collapse `storage/` into an OCIRepository + HelmRelease (chart version pinned exactly)**Verified (in-repo): `storage/` already pulls `oci://ghcr.io/rancher/local-path-provisioner/charts` at `0.0.37` — [storage/main.tf#L88](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/main.tf?display=source#L88); the OCI `layerSelector` needs the Helm chart media type — [Flux HelmRelease guide](https://fluxcd.io/flux/guides/helmreleases/) ```yaml # gitops/clusters/milner/infrastructure/local-path/helmrelease.yaml apiVersion: source.toolkit.fluxcd.io/v1 kind: OCIRepository metadata: {name: local-path-provisioner, namespace: local-path-storage} spec: interval: 10m url: oci://ghcr.io/rancher/local-path-provisioner/charts layerSelector: {mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"} ref: {tag: "0.0.37"} --- apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: {name: local-path-provisioner, namespace: local-path-storage} spec: interval: 10m chartRef: {kind: OCIRepository, name: local-path-provisioner} valuesFrom: [{kind: ConfigMap, name: local-path-provisioner-values}] ``` 5. **Keep `values.yaml` files as-is and expose each as a ConfigMap** ```yaml # gitops/clusters/milner/apps/pocket-id/kustomization.yaml generatorOptions: {disableNameSuffixHash: true} configMapGenerator: - {name: pocket-id-values, files: [values.yaml]} resources: [persistentvolumeclaim.yaml, helmrelease.yaml, ingress.yaml] ``` 6. **Move the shared `ingress-proxies` ProxyGroup out of `vault/` into infrastructure** ```yaml # gitops/clusters/milner/infrastructure/tailscale.yaml apiVersion: tailscale.com/v1alpha1 kind: ProxyGroup metadata: {name: ingress-proxies} spec: {type: ingress, replicas: 1} ``` 7. **Vault uses a HelmRepository + `spec.chart.spec.sourceRef` (HTTP repo, not OCI)**Verified (in-repo): Vault is installed from the HTTP repo `https://helm.releases.hashicorp.com`, chart `vault` `0.34.1` — [vault/main.tf#L89](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/vault/main.tf?display=source#L89) ```yaml apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: {name: vault, namespace: vault} spec: interval: 10m chart: spec: chart: vault version: "0.34.1" sourceRef: {kind: HelmRepository, name: hashicorp} valuesFrom: [{kind: ConfigMap, name: vault-values}] ``` 8. **Install VSO, then reference Vault — no secret material in git**Verified (external): the `hashicorp/vault-secrets-operator` chart is at `1.5.0` — [VSO installation](https://developer.hashicorp.com/vault/docs/deploy/kubernetes/vso/installation); the repo already uses the `secrets.hashicorp.com/v1beta1` CRD — [ARCHITECTURE.md#L200](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/ARCHITECTURE.md?display=source#L200) ```yaml # gitops/clusters/milner/secrets/vaultauth.yaml apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultAuth metadata: {name: default, namespace: woodpecker} spec: method: kubernetes mount: kubernetes kubernetes: {role: woodpecker, serviceAccount: default} --- # gitops/clusters/milner/secrets/woodpecker.yaml apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: {name: woodpecker-forgejo-oauth, namespace: woodpecker} spec: type: kv-v2 mount: secret path: woodpecker/forgejo-oauth refreshAfter: 1h destination: {name: woodpecker-forgejo-oauth} vaultAuthRef: default rolloutRestartTargets: [{kind: Deployment, name: woodpecker-server}] ``` - Chart `hashicorp/vault-secrets-operator` pinned to `1.5.0`; `VaultConnection` points at `http://vault.vault.svc.cluster.local:8200`. - One `VaultAuth`/`VaultStaticSecret` per Secret: `woodpecker/{forgejo-oauth,grpc}` and `pocket-id/{encryption-key}`. - Replaces OPERATIONS §7c's `kubectl create secret`; Vault paths already exist from §7b/§5a.Verified (in-repo): `secret/woodpecker/forgejo-oauth` — [OPERATIONS.md#L754](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/OPERATIONS.md?display=source#L754), `secret/pocket-id/encryption-key` — [OPERATIONS.md#L400](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/OPERATIONS.md?display=source#L400) 9. **Woodpecker and Pocket ID HelmReleases reuse today's values unchanged**Verified (in-repo): `oci://ghcr.io/woodpecker-ci/helm` `3.7.3` — [woodpecker/main.tf#L101](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/main.tf?display=source#L101); `oci://ghcr.io/hobbit44/pocket-id-helm` `2.4.0` — [pocket-id/main.tf#L135](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/pocket-id/main.tf?display=source#L135) - `oci://ghcr.io/woodpecker-ci/helm` `woodpecker` `3.7.3`, namespace `woodpecker`, Ingress unchanged. - `oci://ghcr.io/hobbit44/pocket-id-helm` `pocket-id` `2.4.0`, namespace `pocket-id`. - Pocket ID's PVC becomes a Flux-managed manifest copied verbatim from `pocket-id/main.tf` (`persistent`, 5Gi, no bind wait) — no chart change needed.Verified (in-repo): the claim is on `persistent` with `5Gi` and `wait_until_bound = false` — [pocket-id/main.tf#L101](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/pocket-id/main.tf?display=source#L101), asserted by [tests/test_static.py#L246](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_static.py?display=source#L246) 10. **Bootstrap script (the one unavoidable push), credentials from env only**Verified (external): `flux bootstrap git` accepts `--url`, `--branch`, `--path` and `--private-key-file` — [flux bootstrap git](https://fluxcd.io/flux/cmd/flux_bootstrap_git/) ```bash # gitops/bootstrap.sh <deploy-key-path> flux bootstrap git \ --url=ssh://git@git.gloom.garden/tepichord/milner.git \ --branch=main \ --path=gitops/clusters/milner \ --private-key-file="$1" ``` - `flux bootstrap` writes `gitops/clusters/milner/flux-system/` (committed, reviewed); patch controller node selectors there via `gitops/clusters/milner/flux-system/kustomization.yaml`:Verified (external): bootstrap generates `gotk-components.yaml`, `gotk-sync.yaml` and `kustomization.yaml` under the target path — [Bootstrap customization](https://fluxcd.io/flux/installation/configuration/bootstrap-customization/) ```yaml patches: - target: {kind: Deployment} patch: |- - op: add path: /spec/template/spec/nodeSelector value: {node-role: worker} ``` 11. **Hand off ownership from Terraform after Flux is Healthy**Verified (in-repo): the listed Vault addresses exist — `kubernetes_namespace_v1.vault` [vault/main.tf#L71](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/vault/main.tf?display=source#L71), `helm_release.vault` [#L87](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/vault/main.tf?display=source#L87), `kubernetes_manifest.vault_proxy_group` [#L107](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/vault/main.tf?display=source#L107), `kubernetes_ingress_v1.vault` [#L121](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/vault/main.tf?display=source#L121) ```bash flux get kustomizations -A # all Ready, all HelmReleases Healthy terraform -chdir=vault state rm \ kubernetes_namespace_v1.vault helm_release.vault \ kubernetes_manifest.vault_proxy_group kubernetes_ingress_v1.vault # ...same for storage/ woodpecker/ pocket-id/... git rm -r storage/ vault/ woodpecker/ pocket-id/ ``` - Release name + namespace must match exactly so Flux upgrades the existing Helm releases instead of erroring; chart versions are pinned to the deployed versions to avoid an unintended bump on first reconcile. _unverified: helm-controller's adoption/upgrade-in-place semantics for a pre-existing release could not be confirmed from docs; needs a live cluster._ - Order matters: `state rm` **before** deleting the root, or a later `terraform destroy` would delete Flux-owned objects. 12. **Port the existing regression guards instead of deleting them**Verified (in-repo): all five named guards exist — [tests/test_static.py#L208](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_static.py?display=source#L208), [#L246](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_static.py?display=source#L246), [#L320](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_static.py?display=source#L320), [#L353](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_static.py?display=source#L353), [#L376](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_static.py?display=source#L376) | Guard in `tests/test_static.py` | New target | |---|---| | `test_vault_values_keep_data_mount_under_server` | `gitops/clusters/milner/vault/values.yaml` | | `test_pocket_id_persists_to_the_durable_storage_class` | `.../apps/pocket-id/values.yaml` + `persistentvolumeclaim.yaml` | | `test_pocket_id_external_url_matches_ingress_magicdns_name` | `.../apps/pocket-id/ingress.yaml` | | `test_funnel_is_retired_from_the_woodpecker_root` | `.../apps/woodpecker/ingress.yaml` | | `test_woodpecker_webhook_host_matches_cloudflare_route` | split: woodpecker half reads the new values path | 13. **CI gate: keep it credential-free**Verified (external): `ghcr.io/fluxcd/flux-cli:v2.4.0` publishes a `linux/arm64` manifest — [flux-cli package](https://github.com/fluxcd/flux2/pkgs/container/flux-cli); the `woodpecker-runtime` namespace already exists — [woodpecker/main.tf#L84](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/main.tf?display=source#L84) ```yaml # .woodpecker.yaml — new; agent runs in woodpecker-runtime steps: - name: build image: ghcr.io/fluxcd/flux-cli:v2.4.0 # arm64 commands: - flux build kustomization infrastructure --path=gitops/clusters/milner/infrastructure --kustomization-file=gitops/clusters/milner/infrastructure.yaml --dry-run > /tmp/rendered.yaml - kubeconform -strict -ignore-missing-schemas /tmp/rendered.yaml - name: test commands: [python -m pytest tests/ -v] ``` - `flux diff kustomization <name>` needs a kubeconfig, so it stays a **manual laptop-side pre-merge check** documented in OPERATIONS (exit `0` = no drift, `1` = diff, `>1` = error) — CI keeps holding no cluster credentials, per the issue.Verified (external): documented exit status `0` no differences, `1` differences, `>1` error — [flux diff kustomization](https://fluxcd.io/flux/cmd/flux_diff_kustomization/) 14. **Update docs**: README, AGENTS repo map / "How Changes Ship", OPERATIONS bring-up order (§7), and a new Flux section (bootstrap, unseal-blocks-reconcile note, manual `flux diff`, VaultAuth k8s-auth enablement). ## Implementation Details **Created** | Path | Purpose | |---|---| | `gitops/bootstrap.sh` | one-time `flux bootstrap git` wrapper | | `gitops/clusters/milner/{infrastructure,vault,secrets,apps}.yaml` | the four Flux Kustomizations | | `gitops/clusters/milner/infrastructure/` | namespaces, local-path HelmRelease, `ingress-proxies` | | `gitops/clusters/milner/vault/` | Vault HelmRelease + Ingress + `values.yaml` | | `gitops/clusters/milner/secrets/` | VSO HelmRelease, VaultConnection, VaultAuth, VaultStaticSecrets | | `gitops/clusters/milner/apps/{woodpecker,pocket-id,evolved-npcs}/` | app HelmReleases, Ingresses, PVC, stub | | `gitops/clusters/milner/flux-system/` | written by bootstrap, then committed | | `tests/test_gitops.py` | build + schema + ported guards | | `.woodpecker.yaml` | offline validation pipeline | **Modified**: `flake.nix` (`fluxcd`, `kubeconform`), `tests/test_static.py` (repoint guards, drop the four roots from `test_terraform_fmt`/`test_yamllint`), `tests/test_terraform.py` (drop the four roots), `README.md`, `AGENTS.md`, `OPERATIONS.md`, `.env.example` (deploy-key path only if needed).Verified (in-repo): `test_terraform_fmt`/`test_yamllint` currently reference the roots — [tests/test_static.py#L22](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_static.py?display=source#L22); `tests/test_terraform.py` has per-root jobs — [#L46](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_terraform.py?display=source#L46) **Deleted after handoff**: `storage/`, `vault/`, `woodpecker/`, `pocket-id/` (Terraform roots incl. tests/fixtures). **Dependencies / constraints** - Pinned chart versions: local-path-provisioner `0.0.37`, vault `0.34.1`, woodpecker `3.7.3`, pocket-id `2.4.0`, VSO `1.5.0`.Verified (in-repo + external): [storage/main.tf#L88](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/main.tf?display=source#L88), [vault/main.tf#L91](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/vault/main.tf?display=source#L91), [woodpecker/main.tf#L101](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/main.tf?display=source#L101), [pocket-id/main.tf#L135](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/pocket-id/main.tf?display=source#L135); VSO `1.5.0` per [Hashicorp docs](https://developer.hashicorp.com/vault/docs/deploy/kubernetes/vso/installation) - All images must be arm64; `flux` controllers and CI images included. - All pods keep `nodeSelector: {node-role: worker}`, including Flux controllers.Verified (in-repo): present in the existing values — [vault/values.yaml#L89](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/vault/values.yaml?display=source#L89), [woodpecker/values.yaml#L24](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/values.yaml?display=source#L24), [pocket-id/values.yaml#L68](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/pocket-id/values.yaml?display=source#L68) - No secret material in git — VSO references only; `kubectl create secret` steps retired. - `vault` Kustomization uses `wait: true`, so a sealed Vault blocks `secrets`/`apps` until §4c unseal runs — intentional, documented. _unverified: depends on live reconcile behavior of the sealed StatefulSet; not confirmable offline._ - VaultAuth needs Vault's `auth/kubernetes` enabled once (OPERATIONS); static-token `VaultAuth` is the fallback.Verified (external): the Kubernetes auth method must be enabled and configured in Vault — [Vault Kubernetes auth](https://developer.hashicorp.com/vault/docs/auth/kubernetes) **Out-of-scope notes** - The Tailscale operator stays imperative (OPERATIONS §3): its OAuth secret has no safe GitOps home yet. Only its `ingress-proxies` ProxyGroup moves to Flux. - `apps/evolved-npcs/` ships as a README + empty kustomization, not wired into `apps/kustomization.yaml`; no KEDA/RabbitMQ workloads here. - Cloudflare and `external-dns` are not touched. - HA control plane, VSO for evolved-npcs, and Flux image automation are not part of this change. ## Acceptance Criteria - [ ] `gitops/bootstrap.sh` runs `flux bootstrap git` successfully; `flux get kustomizations -A` shows `flux-system`, `infrastructure`, `vault`, `secrets`, `apps` all `Ready`. - [ ] All four workloads reconcile from Git: `kubectl -n pocket-id delete svc pocket-id` is recreated by Flux within one interval. - [ ] `terraform state rm` has been run for every migrated resource and `storage/`, `vault/`, `woodpecker/`, `pocket-id/` are deleted; `terraform -chdir=control-plane plan`, `worker`, `data`, `cloudflare` still plan clean. - [ ] No secret material is committed (`git grep` finds no Vault values / tokens); Woodpecker and Pocket ID start without the OPERATIONS §7c `kubectl create secret` steps. - [ ] Every ported regression guard in `tests/test_gitops.py` passes, and `python -m pytest tests/ -v` is green. - [ ] The offline CI gate (`.woodpecker.yaml`) fails on a deliberately broken manifest and passes on the committed tree. - [ ] README/AGENTS/OPERATIONS describe Flux as the cluster owner; no stale `terraform -chdir=vault apply` instructions remain. ## Testing Plan - `tests/test_gitops.py`: `kustomize build` every Flux path; assert every namespace carries an explicit PSA `enforce` label; assert every `HelmRelease` is `helm.toolkit.fluxcd.io/v2` with a pinned version; assert no `Secret` has literal `data`/`stringData`. - Ported guards: Vault data mount under `server:`, Pocket ID PVC on `persistent` with `/app/data` mount, Pocket ID `appUrl` vs Ingress `tls.hosts`, Woodpecker webhook host == published Cloudflare route, no `tailscale.com/funnel`. - `flux build kustomization <name> --dry-run` for all four layers + `kubeconform -strict -ignore-missing-schemas`. - Manual pre-merge: `flux diff kustomization apps --path=./gitops/clusters/milner/apps` returns `0` on a clean tree, non-zero-but-not-error when a manifest changes. - Live drift check: hand-edit `deploy/vault` and confirm Flux reverts it. - Fresh-cluster path: operator bootstrap → `flux bootstrap` → unseal Vault → `flux get helmreleases -A` all Healthy. ## Related Issues/PRs - Closes #46 - #38 (Pocket ID), #37 (Woodpecker), #33 (Vault) — the Terraform roots being migrated - #32 / #31 (kubectl bootstrap script) — the imperative-bootstrap pattern this replaces - `PLAN.md` target architecture (VSO, Woodpecker CI/CD), `ARCHITECTURE.md` §6 (VSO for evolved-npcs) ## Research Findings - `flux bootstrap git` is Flux's recommended install path and is idempotent; Forgejo is Gitea-compatible, so `flux bootstrap gitea --token-auth` also works but needs a PAT ([installation](https://fluxcd.io/flux/installation/), [gitea](https://fluxcd.io/flux/installation/bootstrap/gitea/)). - Helm APIs reached GA in Flux v2.3: use `helm.toolkit.fluxcd.io/v2`, `source.toolkit.fluxcd.io/v1`, `kustomize.toolkit.fluxcd.io/v1`; `v1beta2` examples are stale ([v2.3 GA](https://fluxcd.io/blog/2024/05/flux-v2.3.0/)). - `.spec.chartRef` references `OCIRepository`/`HelmChart`; HTTP repos keep `spec.chart.spec.sourceRef` with a `HelmRepository` ([helmreleases guide](https://fluxcd.io/flux/guides/helmreleases/)). - Ordering is `dependsOn` (ready-gated, not apply-gated) plus `healthChecks`/`wait: true`; Flux already applies Namespaces before namespaced resources, so PSA labels land before pods ([kustomizations](https://fluxcd.io/flux/components/kustomize/kustomizations/)). - VSO is the idiomatic fit while Vault is in-cluster: chart `1.5.0`, CRD `secrets.hashicorp.com/v1beta1`, and `VaultStaticSecret` syncs Vault paths into native Secrets ([VSO install](https://developer.hashicorp.com/vault/docs/deploy/kubernetes/vso/installation)). - `flux diff kustomization` exits `0`/`1`/`>1` (no diff / diff / error) and needs cluster access; `flux build kustomization --dry-run` renders offline ([flux diff](https://fluxcd.io/flux/cmd/flux_diff_kustomization/)). - `flux bootstrap` generates and commits `flux-system/` (`gotk-components.yaml`, `gotk-sync.yaml`, `kustomization.yaml`), which is the patch point for controller node selectors ([bootstrap customization](https://fluxcd.io/flux/installation/configuration/bootstrap-customization/)). ## Pre-Mortem 1. **The Terraform→Flux handoff runs out of order and destroys live objects instead of transferring them.** Four roots must each have `terraform state rm` run before `git rm -r` deletes the directory, and any root still owning a namespace in state will delete that namespace — taking the Pocket ID PVC and the local-path StorageClasses with it — on the next `terraform destroy`. The plan documents this ordering in prose but adds no automated guard (e.g. a pre-destroy check or a test asserting the roots are gone from state), so a forgotten `state rm` is only discovered when data is already deleted. 2. **Flux fails to adopt the existing Helm releases, so the first reconcile errors or replaces workloads.** Flux derives the release name and namespace from the `HelmRelease` object and diffs its rendered values against the live release, so any divergence from what Terraform installed — re-expressing values through `valuesFrom` ConfigMaps instead of `file("values.yaml")`, a renamed Pocket ID PVC (`pocket_id_data`), or a namespace/PSA difference — surfaces only during the live cutover. Because the acceptance criterion requires only that Flux reports `Ready`, an unintended upgrade or reinstall can pass while Woodpecker and Vault are briefly unavailable. 3. **The `dependsOn`/`wait: true` chain stalls behind manual, out-of-band steps.** `infrastructure` still depends on the imperatively-installed Tailscale operator for the `ProxyGroup` CRD, and `vault` is `wait: true` behind a Vault that must be unsealed by hand, so `secrets` and `apps` never become `Ready` if either prerequisite is skipped. This reproduces on every cluster rebuild and on any Vault pod restart, leaving the cluster serving stale manifests with no automatic recovery.

tepichord added

and removed

labels

2026-09-10 23:19:11 -07:00

tepichord referenced this issue from a commit

2026-09-11 07:46:33 -07:00

test: repoint regression guards to gitops paths, drop migrated roots

tepichord referenced this issue from a commit