Automate Cluster Deployments with Flux

tepichord commented

2026-09-11 07:46:53 -07:00

Owner

Closes #46

Automated PR created by forgejo-done adapter.

Closes #46 Automated PR created by forgejo-done adapter.

tepichord added 11 commits

2026-09-11 07:46:53 -07:00

feat(gitops): add Flux v2 GitOps manifests and CI gate a1c1ec928c

Create the complete Flux GitOps tree under gitops/clusters/milner/ with
four Kustomization layers (infrastructure, vault, secrets, apps), a
bootstrap script, and all workload manifests ported from the four
Terraform roots.

- infrastructure: namespaces with PSA labels, local-path-provisioner
  HelmRelease + OCIRepository, Tailscale ProxyGroup
- vault: HelmRelease + HelmRepository (HTTP) + Ingress + values
- secrets: VSO HelmRelease, VaultConnection, VaultAuth, VaultStaticSecrets
- apps: woodpecker, pocket-id (HelmRelease + Ingress + PVC), evolved-npcs stub

Also adds:
- gitops/bootstrap.sh: one-time flux bootstrap git wrapper
- .woodpecker.yaml: offline CI gate (flux build + kubeconform + pytest)
- tests/test_gitops.py: kustomize build + kubeconform schema validation +
  ported regression guards
- flake.nix: add fluxcd and kubeconform to dev shell

Refs: #46

test: repoint regression guards to gitops paths, drop migrated roots 0f9bab19b2

Update test_static.py:
- Drop storage/, vault/, woodpecker/, pocket-id/ from test_terraform_fmt
- Repoint test_yamllint to gitops values files
- Port all five regression guards to read from gitops/clusters/milner/

Update test_terraform.py:
- Drop test_terraform_test_vault, test_terraform_test_pocket_id,
  test_terraform_test_woodpecker, test_terraform_test_storage

Refs: #46

docs: describe Flux as cluster owner, update how changes ship eb5b7ccb87

Update README.md:
- Link to gitops/ paths for Vault, Pocket ID, Woodpecker
- Add Flux to How Changes Ship table

Update AGENTS.md:
- Replace Terraform roots with gitops/ in Repo Map
- Update How Changes Ship and Testing sections

Update OPERATIONS.md:
- Update What lives where table for Flux-managed resources
- Rewrite bring-up order (§7): Flux bootstrap replaces per-root applies
- Replace §4a/§5b/§7c/§7d/§8c with Flux-managed equivalents
- Retire manual kubectl create secret steps (VSO handles syncing)
- Note sealed Vault blocks Flux reconciliation chain

Refs: #46

chore: delete migrated Terraform roots (storage, vault, woodpecker, pocket-id) 250e0d7719

Remove the four in-cluster Terraform roots that are now fully managed by
Flux under gitops/clusters/milner/. Operator must run terraform state rm
for every migrated resource BEFORE this commit reaches the live cluster.

Migration notes:
- storage/ → gitops/clusters/milner/infrastructure/local-path/
- vault/ → gitops/clusters/milner/vault/
- woodpecker/ → gitops/clusters/milner/apps/woodpecker/
- pocket-id/ → gitops/clusters/milner/apps/pocket-id/

Helm chart versions pinned identically: local-path-provisioner 0.0.37,
vault 0.34.1, woodpecker 3.7.3, pocket-id 2.4.0, VSO 1.5.0.

Refs: #46

style(test): apply ruff formatting to touched test files b7b95066f0

test: dedupe ported regression guards and lint the bootstrap script 9f257c8867

The five gitops regression guards were live in both tests/test_static.py and
tests/test_gitops.py, so pytest ran every assertion twice. tests/test_gitops.py
is their declared home (its docstring documents the port), so drop the
test_static.py copies.

Also add gitops/bootstrap.sh to the shellcheck list: every other script in the
repo is covered, and the new one was not.

test(gitops): use importorskip for yaml and drop the dead helper ea18eb34d5

The three rendered-manifest checks used a bare 'import yaml', which turns an
absent PyYAML into a collection error instead of a skip; the ported guards
already use pytest.importorskip. _block_at had no callers left, and the inner
'import pathlib' shadowed the module-level import.

build(gitops): add kustomize to the dev shell d3ecb28bda

AGENTS.md documents kustomize build as the manifest validation command and the
test suite shells out to it, but the dev shell only provided fluxcd and
kubeconform, so every build/schema test silently skipped.

ci: render all layers and install kubeconform in the offline gate a4038ad8f7

The build step ran kubeconform inside ghcr.io/fluxcd/flux-cli, which ships only
flux and kubectl, so the gate would fail on the committed tree. Split rendering
(flux-cli) from validation (alpine + pinned kubeconform, whose upstream image is
scratch and has no shell), render all four layers instead of infrastructure
only, and give the pytest step a real image. Verified locally: render +
kubeconform pass on the tree and fail on an injected schema error.

docs: describe Flux as the cluster owner and document Vault k8s auth d4e99af4b5

OPERATIONS §7 gained duplicate section numbers (two each of 7g/7h/7i) because
the replaced 7c/7d were appended as 7e/7g; renumber the new sections back into
the replaced slots so the existing §7e-§7i cross-references stay valid.

Also: drop the leftover hand-run 'kubectl create secret' blocks that the VSO
flow supersedes (their new section says no such steps are needed), add the
missing §4c-ter (Vault Kubernetes auth: enable, config, policies, roles - VSO
cannot authenticate without it), restore §4c-bis/§5a to the bring-up order, and
repoint the remaining references to the deleted Terraform roots and README.

fix(gitops): drop TLS-only fields from the plaintext VaultConnection cd9e91462c

Vault listens plaintext in-cluster (tlsDisable), so caCertBundle/skipTLSVerify
never apply to this address and only suggest TLS is configured.

tepichord added the

pr

reviewing

label

2026-09-11 07:46:53 -07:00

tepichord added this to the MVP project

2026-09-11 09:28:15 -07:00

tepichord added 4 commits

2026-09-11 11:34:39 -07:00

fix(gitops): rotate Woodpecker StatefulSets, not Deployments cdce5097d7

The chart renders both the server and the agent as StatefulSets, so the
Deployment rolloutRestartTargets never matched anything and a Vault secret
rotation was synced without ever rolling the pods that read it.

Point the Forgejo OAuth target at the server StatefulSet, and the gRPC
secret at both pods (the agent reads that Secret too).

feat(gitops): install Flux from committed manifests, not flux bootstrap d5fb22ab34

`flux bootstrap git` commits flux-system/ and pushes it, so using it would
put a repo-write credential in the loop while the manifests themselves are
already reviewable repo content. Split the halves: the manifests are
ordinary commits pushed like any other change, and gitops/install.sh only
applies them.

- flux-system/gotk-components.yaml: `flux install --export --namespace
  flux-system` (v2.9.4 — the same embedded base and options bootstrap
  would have used; verified against pkg/bootstrap/bootstrap_plain_git.go).
- flux-system/gotk-sync.yaml: mirrors pkg/manifestgen/sync.Generate, minus
  `spec.secretRef`. The repository is anonymously readable over HTTPS, so
  the clone needs no credential and no key material lands in the cluster.
- clusters/milner/kustomization.yaml: the root Kustomization's spec.path is
  this directory, so this is the list of what Flux owns. Bootstrapping left
  it to kustomize-controller's synthesise-from-scanning fallback, which made
  the four layer Kustomizations reconciled-by-accident and left it unstated
  whether flux-system/ itself was included. It is now explicit, which also
  makes a Flux version bump a commit rather than a CLI re-run.
- install.sh replaces bootstrap.sh: applies the components, waits for the
  controllers, then applies the root kustomization. Idempotent, and only
  needed once per cluster (or after a reinstall wipes etcd).

docs(operations): record the cluster-side Flux install a7f9388bef

§1 gains the flux-system row (self-managed from git, applied once by
install.sh), §4a/§8c/§7-step-7 and the §9 recovery flow stop pointing at
`flux bootstrap`, and §7 step 7 spells out why the install is a plain
`kubectl apply` with no deploy key.

test(gitops): guard the cluster root, lint install.sh, fix import order

ci/woodpecker/push/woodpecker Pipeline failed

Details

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

8ff591a9ef

The root Kustomization's spec.path is gitops/clusters/milner, so what Flux
owns is whatever that directory builds to. Assert it renders, and assert the
kustomization.yaml there names all four layers plus flux-system/ — the list
that used to be implicit in kustomize-controller's directory scan. That second
check is deliberately not gated on kustomize: it only reads YAML, so it runs
in the CI gate too, where no kustomize binary exists and every render test
skips.

Also swap gitops/bootstrap.sh for gitops/install.sh in the shellcheck list,
and drop the blank line that made ruff's isort check fail (the offline gate
ran ruff over tests/, so it was red).

tepichord added 2 commits

2026-09-11 11:46:31 -07:00

ci: retry the kubeconform download instead of trusting one DNS lookup bc14b9389a

The gate went red on its first real run, in the validate step:

    curl: (6) Could not resolve host: release-assets.githubusercontent.com
    tar: invalid magic / short read / kubeconform: not found in archive

github.com redirects release downloads to that host, and the lookup failed
once. The pipeline on the same commit passed a minute later, so the check
itself is sound: the fetch was simply a single point of failure, and the pipe
into tar reported it as an archive error rather than a network one.

- --retry-all-errors, because a DNS failure is not a "transient error" to
  plain --retry
- download to a file, then extract, so a failed fetch fails as a failed fetch

Verified in-cluster in alpine:3.21/arm64 — the step's own image and arch:
fetch, extract, and `kubeconform -strict -ignore-missing-schemas` over all
four rendered layers exit 0.

ci: scope the gate to push, pull_request and manual

ci/woodpecker/push/woodpecker Pipeline was successful

Details

ci/woodpecker/pr/woodpecker Pipeline was successful

Details

ed50082734

Every step without a `when` block draws a bad-habit warning, which the UI
surfaces next to the errors — it reads like a parse failure but is
IsWarning: true, and the same three warnings were present on the pipeline that
passed. The rule clears only when the *root* has a `when` and every item in it
carries an `event` filter (pipeline/frontend/yaml/linter/linter.go,
lintBadHabits); a step-level filter would just move the warning around.

The list is the gate's actual intent: run where the tree can change. Tag,
release, deployment, cron, pull_request_closed and pull_request_metadata are
now filtered instead of ignored-by-accident, so a delivery that no longer
starts a pipeline is a deliberate outcome rather than the mysterious
"ignoring hook: 'when' filters filtered out all steps" seen once in the server
log.

Checked with the real linter, woodpeckerci/woodpecker-cli:v3.18.0 — the
server's own version — in-cluster, over a ConfigMap of the file:

  before: ⚠️ 3 warnings ... exit 0   (so the warnings never failed the build)
  after:  ✅ Config is valid ... exit 0

tepichord merged commit 5fe8513224 into main

2026-09-11 11:59:42 -07:00

tepichord referenced this pull request from a commit

2026-09-11 11:59:42 -07:00

Automate Cluster Deployments with Flux (#47)

tepichord removed the

pr

reviewing

label

2026-09-11 11:59:48 -07:00

tepichord referenced this pull request from a commit

2026-09-11 16:47:32 -07:00