tepichord/milner

Fork 0

Create staging builds to validate Flux deploys #49

New issue

Closed

opened 2026-09-11 10:24:32 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-11 10:24:32 -07:00

Owner

One-time prerequisites

Put the CI identity where the pipeline pods actually run. Woodpecker's agent sets WOODPECKER_BACKEND_K8S_NAMESPACE: woodpecker-runtime, so pipeline pods land there, and a
pod's serviceAccountName must resolve in its own namespace. So create preview-deployer in woodpecker-runtime, and grant it RoleBindings in the target namespaces — a
RoleBinding may reference an SA from another namespace, so cross-namespace binding works even though the SA can't move.
Scope that identity tightly. ClusterRole granting only namespaces: [get,list,create,delete] (namespace names can't be wildcarded in RBAC, so keep the cluster-scoped grant
to that one resource), plus a Role in preview-system covering gitrepositories, kustomizations, and configmaps. No write verbs anywhere else. The design point: CI creates
intent, Flux applies workloads — so CI never needs write access to apps or the platform namespaces.
Attach the identity to the pipeline. Preferred: backend_options.kubernetes.serviceAccountName: preview-deployer in .woodpecker.yaml (confirm the field name for your
Woodpecker version). Fallback: mint a token Secret for the SA, build a kubeconfig from it, store that as a Woodpecker repo secret, and use from_secret. The SA route avoids
exporting a cluster credential entirely, which is why it's preferred.
Add a reaper before you add previews. A CronJob in preview-system that deletes any preview Kustomization/GitRepository/Namespace whose milner.dev/preview-expires annotation
is in the past. Teardown must not depend on a PR-close webhook arriving — Forgejo being off-tailnet makes that a real failure mode.

Repo layout

Make apps/base environment-agnostic. No hardcoded namespace, and placeholders for everything that differs: ${registry}, ${image_tag}, ${domain}, ${replicas},
${storage_class}. This is what lets one manifest tree serve prod and previews without forking.
Add apps/prod as a Kustomization with postBuild.substituteFrom pointing at a cluster-config-prod ConfigMap. That Kustomization is now the unit flux build renders and flux
diff diffs — so keeping all env variance in substitution, not in patches, is what keeps the pipeline simple.
Add apps/preview as an overlay that de-fangs side effects: PVC storageClassName: ephemeral (your Delete tier, so teardown leaves no retained PV on the data disk), the
Tailscale Ingress removed (a preview should not mint tailnet nodes), and replicas: 1.

L0 — offline render

Render every Kustomization the PR touches. flux build kustomization --path=./apps/... --kustomization-file= for each, plus a plain kustomize build apps/prod.
This catches bad patches, malformed Kustomizations, and undefined substitution variables with no cluster involved.
Note the gap, because it will bite you. flux build renders Kustomizations; it does not render HelmRelease chart output. If the app layer uses HelmReleases, add flux-local
build (third-party) for offline chart rendering — otherwise L0 silently validates nothing about the chart.

L1 — schema and policy

Run kubeconform with a CRD schema location. The default schema set has no Flux, KEDA, or VSO types, so without pointing at the CRDs catalog every custom resource is
skipped rather than checked — a passing L1 that means nothing. Use -strict so unknown fields fail instead of being ignored.
Keep policy to two or three rules, not a framework. conftest with checks for no :latest tag, resource limits present, and no privileged containers. Anything more elaborate
than that will be bypassed within a month at this scale.

L2 — server-side dry run (the highest-value gate)

Diff each touched Kustomization against the live API server. flux diff kustomization apps-prod --path=./apps/prod, or the explicit fallback kustomize build apps/prod |
kubectl diff -f -. This runs through real CRDs, real admission, real PSA labels, and Kubernetes' immutable-field rules, and persists nothing — it's the direct answer to
"will this work before we deploy it."
Accept the RBAC consequence deliberately. Server-side dry-run authorizes as an apply, so the CI identity needs namespaced write Roles in the namespaces it diffs. That's
namespaced-only and cannot escalate to cluster scope, so it's acceptable — but if you'd rather not, run L2 for prod from a maintainer's dev shell and keep CI at L0+L1 for
prod while L2 runs freely against preview-system.

L3 — real Flux preview

Build and push the PR's image first, tagged by commit SHA. Otherwise you preview the previous image and learn nothing. With the registry still deferred, L3 currently
validates manifest wiring and Flux reconciliation only.

Have CI create four objects, then let Flux do the work. Namespace preview-, ConfigMap preview--values, GitRepository milner-pr- (branch = PR head, reusing the
main repo's secretRef), and the Kustomization:

  apiVersion: kustomize.toolkit.fluxcd.io/v1
  kind: Kustomization
  metadata:
    name: milner-pr-123
    namespace: preview-system
  spec:
    interval: 1m
    path: ./apps/preview
    prune: true          # deleting this object garbage-collects the preview
    wait: true
    timeout: 10m
    targetNamespace: preview-123
    sourceRef: { kind: GitRepository, name: milner-pr-123 }
    postBuild:
      substituteFrom: [{ kind: ConfigMap, name: preview-123-values }]
    healthChecks:
      - { apiVersion: apps/v1, kind: Deployment, name: orchestrator, namespace: preview-123 }

Copy prod's PSA labels onto the preview namespace. An unlabeled namespace defaults to privileged, so it would happily admit exactly what prod rejects — the preview would
pass while the deploy fails. pod-security.kubernetes.io/enforce: baseline must be set at namespace creation.
Wait on healthChecks, not on pods existing. kubectl -n preview-system wait --for=condition=Ready kustomization/milner-pr-123 --timeout=10m, and on failure dump kubectl
describe kustomization, the namespace events, and container logs. The health check is the actual assertion that Flux deployed it; a Running pod is not.
Give previews their own Vault path, or none at all. If the app syncs secrets through VSO, the preview namespace needs its own Vault role bound to its ServiceAccount, or
pods hang in ContainerCreating and you'll blame Flux. Cheaper alternative: have the preview overlay reference a pre-created Secret so previews never talk to Vault.
Cap the blast radius, because you have none structurally. Previews share the single 20 GB worker with production, and Flannel doesn't enforce NetworkPolicy, so namespace
boundaries are bookkeeping rather than isolation. Set a ResourceQuota and LimitRange in every preview namespace, and give previews a lower PriorityClass so they're evicted
before prod under memory pressure.
Teardown in the right order. Delete the Kustomization (prune collects the workloads), then the namespace, then the GitRepository. Set milner.dev/preview-expires on all
three so the reaper from step 4 catches anything a closed PR misses.

Where this stops working

L3 belongs to the apps/ layer only. For vault/, woodpecker/, pocket-id/, CRDs, and operators, L0–L2 is the ceiling — you can't meaningfully run a second Vault in a preview
namespace, and targetNamespace cannot apply cluster-scoped resources at all. And since a preview starts from nothing, it never exercises the Helm upgrade path from the
currently-running release, which is where most real breakage lives; wait: true on the prod Kustomization plus a fast rollback is your only cover for that.

Done when: a PR touching apps/ produces a rendered diff, a kubeconform pass, a server-side dry-run against Milner, and a preview- namespace that reaches Kustomization
Ready=True with health checks passing — and closing the PR (or letting the reaper fire) leaves no orphaned namespace, no GitRepository, and no retained PV behind.

One-time prerequisites 1. Put the CI identity where the pipeline pods actually run. Woodpecker's agent sets WOODPECKER_BACKEND_K8S_NAMESPACE: woodpecker-runtime, so pipeline pods land there, and a pod's serviceAccountName must resolve in its own namespace. So create preview-deployer in woodpecker-runtime, and grant it RoleBindings in the target namespaces — a RoleBinding may reference an SA from another namespace, so cross-namespace binding works even though the SA can't move. 2. Scope that identity tightly. ClusterRole granting only namespaces: [get,list,create,delete] (namespace names can't be wildcarded in RBAC, so keep the cluster-scoped grant to that one resource), plus a Role in preview-system covering gitrepositories, kustomizations, and configmaps. No write verbs anywhere else. The design point: CI creates intent, Flux applies workloads — so CI never needs write access to apps or the platform namespaces. 3. Attach the identity to the pipeline. Preferred: backend_options.kubernetes.serviceAccountName: preview-deployer in .woodpecker.yaml (confirm the field name for your Woodpecker version). Fallback: mint a token Secret for the SA, build a kubeconfig from it, store that as a Woodpecker repo secret, and use from_secret. The SA route avoids exporting a cluster credential entirely, which is why it's preferred. 4. Add a reaper before you add previews. A CronJob in preview-system that deletes any preview Kustomization/GitRepository/Namespace whose milner.dev/preview-expires annotation is in the past. Teardown must not depend on a PR-close webhook arriving — Forgejo being off-tailnet makes that a real failure mode. Repo layout 5. Make apps/base environment-agnostic. No hardcoded namespace, and placeholders for everything that differs: ${registry}, ${image_tag}, ${domain}, ${replicas}, ${storage_class}. This is what lets one manifest tree serve prod and previews without forking. 6. Add apps/prod as a Kustomization with postBuild.substituteFrom pointing at a cluster-config-prod ConfigMap. That Kustomization is now the unit flux build renders and flux diff diffs — so keeping all env variance in substitution, not in patches, is what keeps the pipeline simple. 7. Add apps/preview as an overlay that de-fangs side effects: PVC storageClassName: ephemeral (your Delete tier, so teardown leaves no retained PV on the data disk), the Tailscale Ingress removed (a preview should not mint tailnet nodes), and replicas: 1. L0 — offline render 8. Render every Kustomization the PR touches. flux build kustomization <name> --path=./apps/... --kustomization-file=<file> for each, plus a plain kustomize build apps/prod. This catches bad patches, malformed Kustomizations, and undefined substitution variables with no cluster involved. 9. Note the gap, because it will bite you. flux build renders Kustomizations; it does not render HelmRelease chart output. If the app layer uses HelmReleases, add flux-local build (third-party) for offline chart rendering — otherwise L0 silently validates nothing about the chart. L1 — schema and policy 10. Run kubeconform with a CRD schema location. The default schema set has no Flux, KEDA, or VSO types, so without pointing at the CRDs catalog every custom resource is skipped rather than checked — a passing L1 that means nothing. Use -strict so unknown fields fail instead of being ignored. 11. Keep policy to two or three rules, not a framework. conftest with checks for no :latest tag, resource limits present, and no privileged containers. Anything more elaborate than that will be bypassed within a month at this scale. L2 — server-side dry run (the highest-value gate) 12. Diff each touched Kustomization against the live API server. flux diff kustomization apps-prod --path=./apps/prod, or the explicit fallback kustomize build apps/prod | kubectl diff -f -. This runs through real CRDs, real admission, real PSA labels, and Kubernetes' immutable-field rules, and persists nothing — it's the direct answer to "will this work before we deploy it." 13. Accept the RBAC consequence deliberately. Server-side dry-run authorizes as an apply, so the CI identity needs namespaced write Roles in the namespaces it diffs. That's namespaced-only and cannot escalate to cluster scope, so it's acceptable — but if you'd rather not, run L2 for prod from a maintainer's dev shell and keep CI at L0+L1 for prod while L2 runs freely against preview-system. L3 — real Flux preview 14. Build and push the PR's image first, tagged by commit SHA. Otherwise you preview the previous image and learn nothing. With the registry still deferred, L3 currently validates manifest wiring and Flux reconciliation only. 15. Have CI create four objects, then let Flux do the work. Namespace preview-<n>, ConfigMap preview-<n>-values, GitRepository milner-pr-<n> (branch = PR head, reusing the main repo's secretRef), and the Kustomization: ```yaml apiVersion: kustomize.toolkit.fluxcd.io/v1 kind: Kustomization metadata: name: milner-pr-123 namespace: preview-system spec: interval: 1m path: ./apps/preview prune: true # deleting this object garbage-collects the preview wait: true timeout: 10m targetNamespace: preview-123 sourceRef: { kind: GitRepository, name: milner-pr-123 } postBuild: substituteFrom: [{ kind: ConfigMap, name: preview-123-values }] healthChecks: - { apiVersion: apps/v1, kind: Deployment, name: orchestrator, namespace: preview-123 } ``` 16. Copy prod's PSA labels onto the preview namespace. An unlabeled namespace defaults to privileged, so it would happily admit exactly what prod rejects — the preview would pass while the deploy fails. pod-security.kubernetes.io/enforce: baseline must be set at namespace creation. 17. Wait on healthChecks, not on pods existing. kubectl -n preview-system wait --for=condition=Ready kustomization/milner-pr-123 --timeout=10m, and on failure dump kubectl describe kustomization, the namespace events, and container logs. The health check is the actual assertion that Flux deployed it; a Running pod is not. 18. Give previews their own Vault path, or none at all. If the app syncs secrets through VSO, the preview namespace needs its own Vault role bound to its ServiceAccount, or pods hang in ContainerCreating and you'll blame Flux. Cheaper alternative: have the preview overlay reference a pre-created Secret so previews never talk to Vault. 19. Cap the blast radius, because you have none structurally. Previews share the single 20 GB worker with production, and Flannel doesn't enforce NetworkPolicy, so namespace boundaries are bookkeeping rather than isolation. Set a ResourceQuota and LimitRange in every preview namespace, and give previews a lower PriorityClass so they're evicted before prod under memory pressure. 20. Teardown in the right order. Delete the Kustomization (prune collects the workloads), then the namespace, then the GitRepository. Set milner.dev/preview-expires on all three so the reaper from step 4 catches anything a closed PR misses. Where this stops working L3 belongs to the apps/ layer only. For vault/, woodpecker/, pocket-id/, CRDs, and operators, L0–L2 is the ceiling — you can't meaningfully run a second Vault in a preview namespace, and targetNamespace cannot apply cluster-scoped resources at all. And since a preview starts from nothing, it never exercises the Helm upgrade path from the currently-running release, which is where most real breakage lives; wait: true on the prod Kustomization plus a fast rollback is your only cover for that. Done when: a PR touching apps/ produces a rendered diff, a kubeconform pass, a server-side dry-run against Milner, and a preview-<n> namespace that reaches Kustomization Ready=True with health checks passing — and closing the PR (or letting the reaper fire) leaves no orphaned namespace, no GitRepository, and no retained PV behind.

tepichord added the

labels

2026-09-11 10:24:32 -07:00

tepichord added this to the MVP project

2026-09-11 10:24:32 -07:00

tepichord commented

2026-09-11 11:42:08 -07:00

Author

Owner

Naina:

Overview

Milner requires a multi-layer CI/CD validation pipeline and GitOps preview workflow for Kubernetes manifests. This implementation introduces a 4-tier Woodpecker validation pipeline (L0 offline render, L1 schema/policy, L2 server-side dry-run, L3 live preview) alongside automated preview lifecycle management and resource isolation on the shared worker.

Technical Approach

Add validation CLI tooling to flake.nix¹

buildInputs = with pkgs; [
  # ... existing tools ...
  fluxcd
  kubeconform
  conftest
];

Enable step ServiceAccount assignment in woodpecker/values.yaml²

agent:
  env:
    WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true"

(Permits Woodpecker pipeline steps to bind step-level ServiceAccounts³)

Provision preview management namespace, RBAC, and PriorityClass in preview-system/

# preview-system/rbac.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: preview-deployer
  namespace: woodpecker-runtime
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: preview-namespace-manager
rules:
  - apiGroups: [""]
    resources: ["namespaces"]
    verbs: ["get", "list", "create", "delete"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: preview-manifest-manager
  namespace: preview-system
rules:
  - apiGroups: ["kustomize.toolkit.fluxcd.io", "source.toolkit.fluxcd.io", ""]
    resources: ["kustomizations", "gitrepositories", "configmaps"]
    verbs: ["get", "list", "create", "update", "patch", "delete"]

(Pipeline pods execute in woodpecker-runtime namespace⁴)

Deploy TTL Reaper CronJob in preview-system/

# preview-system/reaper.yaml
apiVersion: batch/v1
kind: CronJob
metadata:
  name: preview-reaper
  namespace: preview-system
spec:
  schedule: "*/15 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          serviceAccountName: preview-reaper
          containers:
            - name: reaper
              image: bitnami/kubectl:latest
              command:
                - /bin/sh
                - -c
                - |
                  NOW=$(date +%s)
                  for ns in $(kubectl get ns -l milner.dev/preview-expires -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.annotations.milner\.dev/preview-expires}{"\n"}{end}'); do
                    NAME=$(echo "$ns" | awk '{print $1}')
                    EXP=$(echo "$ns" | awk '{print $2}')
                    if [ -n "$EXP" ] && [ "$NOW" -gt "$EXP" ]; then
                      kubectl delete kustomization -n preview-system "$NAME" || true
                      kubectl delete ns "$NAME" || true
                      kubectl delete gitrepository -n preview-system "$NAME" || true
                    fi
                  done

Establish environment-agnostic apps/base, apps/prod, and apps/preview overlays

# apps/preview/kustomization.yaml
apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - ../base
patches:
  - target:
      kind: PersistentVolumeClaim
    patch: |-
      - op: replace
        path: /spec/storageClassName
        value: ephemeral
  - target:
      kind: Deployment
    patch: |-
      - op: replace
        path: /spec/replicas
        value: 1

(Ephemeral StorageClass uses reclaimPolicy: Delete on /var/mnt/data/ephemeral⁵)

Add baseline Conftest OPA policies in policy/k8s.rego⁶

package main

# Reject privileged containers
deny[msg] {
  input.kind == "Deployment"
  c := input.spec.template.spec.containers[_]
  c.securityContext.privileged == true
  msg := sprintf("Container '%v' in %v/%v must not run as privileged", [c.name, input.kind, input.metadata.name])
}

# Reject :latest or untagged container images
deny[msg] {
  input.kind == "Deployment"
  c := input.spec.template.spec.containers[_]
  endswith(c.image, ":latest")
  msg := sprintf("Container '%v' in %v/%v uses forbidden :latest tag", [c.name, input.kind, input.metadata.name])
}

# Require CPU and memory limits
deny[msg] {
  input.kind == "Deployment"
  c := input.spec.template.spec.containers[_]
  not c.resources.limits.memory
  msg := sprintf("Container '%v' in %v/%v missing resources.limits.memory", [c.name, input.kind, input.metadata.name])
}

Define Woodpecker CI pipeline gates in .woodpecker.yaml

pipeline:
  # L0: Offline Render
  l0-render:
    image: nixpkgs/nix-flakes
    commands:
      - kustomize build apps/prod > /tmp/prod.yaml
      - kustomize build apps/preview > /tmp/preview.yaml

  # L1: Schema & Policy
  l1-validate:
    image: nixpkgs/nix-flakes
    commands:
      - kubeconform -strict -ignore-missing-schemas -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' /tmp/*.yaml
      - conftest test /tmp/*.yaml --policy policy/

  # L2: Server-Side Dry-Run
  l2-dry-run:
    image: bitnami/kubectl:latest
    backend_options:
      kubernetes:
        serviceAccountName: preview-deployer
    commands:
      - kubectl diff -f /tmp/prod.yaml || [ $? -eq 1 ]

  # L3: Live Preview Lifecycle (on PRs)
  l3-preview:
    image: bitnami/kubectl:latest
    backend_options:
      kubernetes:
        serviceAccountName: preview-deployer
    when:
      event: [pull_request]
    commands:
      - EXPIRY=$(date -d "+4 hours" +%s)
      - kubectl create ns preview-${CI_COMMIT_PULL_REQUEST} --dry-run=client -o yaml | kubectl annotate --local -f - milner.dev/preview-expires=$EXPIRY -o yaml | kubectl label --local -f - pod-security.kubernetes.io/enforce=baseline -o yaml | kubectl apply -f -
      - kubectl apply -f preview-manifests/
      - kubectl -n preview-system wait --for=condition=Ready kustomization/milner-pr-${CI_COMMIT_PULL_REQUEST} --timeout=10m

(CRD schemas fetched from Datree catalog⁷; step ServiceAccount specified in backend_options³)

Implementation Details

Files to Modify / Create:
- flake.nix: Add fluxcd, kubeconform, and conftest to devShells.default.buildInputs.¹
- woodpecker/values.yaml: Add WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true".²
- preview-system/: Add RBAC (rbac.yaml), reaper CronJob (reaper.yaml), and preview ResourceQuota/LimitRange templates.
- apps/base/, apps/prod/, apps/preview/: Environment-agnostic base app and overlay manifests.
- policy/k8s.rego: Conftest Rego checks (no :latest, memory/CPU limits required, non-privileged).⁶
- .woodpecker.yaml: 4-gate CI validation workflow (l0-render, l1-validate, l2-dry-run, l3-preview).
- tests/test_static.py: Guard tests validating PSA baseline labels, ephemeral storage class on previews, and reaper annotations.⁸
Dependencies & Tools:
- fluxcd (v2.8.x)⁹, kubeconform (v0.7.x)¹⁰, conftest (v0.63.x)¹¹, Datree CRD catalog schema registry.⁷
Constraints & Edge Cases:
- CI ServiceAccount preview-deployer is confined to woodpecker-runtime with strict namespace-scoped Roles and cluster-scoped namespace CRUD only.⁴
- Previews must strictly use storageClassName: ephemeral to prevent orphaned local-path volumes on /var/mnt/data/persistent.¹²
- Previews must strip Tailscale Ingress annotations to avoid generating stray tailnet nodes.
Out-of-scope notes:
- In-cluster container registry provisioning is deferred; image builds in L3 use mock/pre-built images or validate manifest reconciliation only.
- Multi-tenant NetworkPolicy enforcement is out of scope as Flannel CNI does not enforce NetworkPolicies.¹³

Acceptance Criteria

flake.nix provides fluxcd, kubeconform, and conftest in the dev shell.
woodpecker/values.yaml configures the agent with WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true".
RBAC manifests define preview-deployer in woodpecker-runtime scoped to preview-system and namespace CRUD.
preview-system manifests include a TTL reaper CronJob deleting expired preview resources by milner.dev/preview-expires.
apps/base, apps/prod, and apps/preview overlays isolate env variance and enforce storageClassName: ephemeral and no Tailscale Ingress on previews.
Conftest policy policy/k8s.rego enforces no :latest tags, defined resource limits, and non-privileged containers.
.woodpecker.yaml implements L0 offline render, L1 kubeconform/conftest validation, L2 dry-run diffing, and L3 live preview wait.
Static guard tests in tests/test_static.py validate preview storage tier and PSA labels.

Testing Plan

pytest tests/test_static.py: Verify new yamllint, PSA baseline labeling, and preview storage class guard tests.
conftest test /tmp/*.yaml --policy policy/: Verify pass on valid overlays and failure when inserting a :latest tag or privileged container.
kubeconform -strict -ignore-missing-schemas -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json': Verify schema validation against Flux and Kubernetes CRDs.
Local dry-run verification: Validate kustomize build apps/prod and kustomize build apps/preview render valid manifests without undefined substitution variables.

Closes #49

Research Findings

pkgs.fluxcd, pkgs.kubeconform, and pkgs.conftest are available in nixpkgs for aarch64-linux and installable via flake.nix.¹
Offline validation of Flux custom resources requires pointing kubeconform to the Datree CRDs catalog schema location.⁷
Woodpecker's Kubernetes backend requires WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP=true on agents to permit pipeline steps to bind preview-deployer.³
Flux v2 postBuild.substituteFrom requires substitution ConfigMap resources to reside in the same namespace as the Kustomization object (preview-system).¹⁴
Previews require lower PriorityClass and ResourceQuota limits to avoid resource starvation on the single 20GB shared worker node.¹⁵

Pre-Mortem

Resource exhaustion on the shared worker node — Concurrent live preview environments could exceed the single 20GB worker's memory and ephemeral disk capacity if ResourceQuotas or PriorityClasses are misconfigured. This would trigger pod evictions or OOM kills that disrupt co-located production workloads.
RBAC permission or step ServiceAccount assignment failures — The preview-deployer ServiceAccount may lack cluster-level namespace permissions, or the Woodpecker agent may reject step-level ServiceAccount overrides. This would cause L2 server-side dry-run and L3 preview lifecycle stages to fail immediately in CI.
Reaper CronJob failures causing preview resource leaks — If the TTL reaper fails to parse expiration annotations or lacks permissions across preview namespaces, abandoned preview environments will persist indefinitely. This would gradually leak ephemeral storage volumes and consume cluster resources when PR close webhooks fail.

Naina: ## Overview Milner requires a multi-layer CI/CD validation pipeline and GitOps preview workflow for Kubernetes manifests. This implementation introduces a 4-tier Woodpecker validation pipeline (L0 offline render, L1 schema/policy, L2 server-side dry-run, L3 live preview) alongside automated preview lifecycle management and resource isolation on the shared worker. ## Technical Approach 1. **Add validation CLI tooling to `flake.nix`**[1](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/flake.nix?display=source#L20) ```nix buildInputs = with pkgs; [ # ... existing tools ... fluxcd kubeconform conftest ]; ``` 2. **Enable step ServiceAccount assignment in `woodpecker/values.yaml`**[2](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/values.yaml?display=source#L40) ```yaml agent: env: WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true" ``` *(Permits Woodpecker pipeline steps to bind step-level ServiceAccounts[3](https://woodpecker-ci.org/docs/administration/configuration/backends/kubernetes))* 3. **Provision preview management namespace, RBAC, and PriorityClass in `preview-system/`** ```yaml # preview-system/rbac.yaml apiVersion: v1 kind: ServiceAccount metadata: name: preview-deployer namespace: woodpecker-runtime --- apiVersion: rbac.authorization.k8s.io/v1 kind: ClusterRole metadata: name: preview-namespace-manager rules: - apiGroups: [""] resources: ["namespaces"] verbs: ["get", "list", "create", "delete"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: preview-manifest-manager namespace: preview-system rules: - apiGroups: ["kustomize.toolkit.fluxcd.io", "source.toolkit.fluxcd.io", ""] resources: ["kustomizations", "gitrepositories", "configmaps"] verbs: ["get", "list", "create", "update", "patch", "delete"] ``` *(Pipeline pods execute in `woodpecker-runtime` namespace[4](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/values.yaml?display=source#L41))* 4. **Deploy TTL Reaper CronJob in `preview-system/`** ```yaml # preview-system/reaper.yaml apiVersion: batch/v1 kind: CronJob metadata: name: preview-reaper namespace: preview-system spec: schedule: "*/15 * * * *" jobTemplate: spec: template: spec: serviceAccountName: preview-reaper containers: - name: reaper image: bitnami/kubectl:latest command: - /bin/sh - -c - | NOW=$(date +%s) for ns in $(kubectl get ns -l milner.dev/preview-expires -o jsonpath='{range .items[*]}{.metadata.name}{" "}{.metadata.annotations.milner\.dev/preview-expires}{"\n"}{end}'); do NAME=$(echo "$ns" | awk '{print $1}') EXP=$(echo "$ns" | awk '{print $2}') if [ -n "$EXP" ] && [ "$NOW" -gt "$EXP" ]; then kubectl delete kustomization -n preview-system "$NAME" || true kubectl delete ns "$NAME" || true kubectl delete gitrepository -n preview-system "$NAME" || true fi done ``` 5. **Establish environment-agnostic `apps/base`, `apps/prod`, and `apps/preview` overlays** ```yaml # apps/preview/kustomization.yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization resources: - ../base patches: - target: kind: PersistentVolumeClaim patch: |- - op: replace path: /spec/storageClassName value: ephemeral - target: kind: Deployment patch: |- - op: replace path: /spec/replicas value: 1 ``` *(Ephemeral StorageClass uses `reclaimPolicy: Delete` on `/var/mnt/data/ephemeral`[5](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/values.yaml?display=source#L76))* 6. **Add baseline Conftest OPA policies in `policy/k8s.rego`**[6](https://www.conftest.dev) ```rego package main # Reject privileged containers deny[msg] { input.kind == "Deployment" c := input.spec.template.spec.containers[_] c.securityContext.privileged == true msg := sprintf("Container '%v' in %v/%v must not run as privileged", [c.name, input.kind, input.metadata.name]) } # Reject :latest or untagged container images deny[msg] { input.kind == "Deployment" c := input.spec.template.spec.containers[_] endswith(c.image, ":latest") msg := sprintf("Container '%v' in %v/%v uses forbidden :latest tag", [c.name, input.kind, input.metadata.name]) } # Require CPU and memory limits deny[msg] { input.kind == "Deployment" c := input.spec.template.spec.containers[_] not c.resources.limits.memory msg := sprintf("Container '%v' in %v/%v missing resources.limits.memory", [c.name, input.kind, input.metadata.name]) } ``` 7. **Define Woodpecker CI pipeline gates in `.woodpecker.yaml`** ```yaml pipeline: # L0: Offline Render l0-render: image: nixpkgs/nix-flakes commands: - kustomize build apps/prod > /tmp/prod.yaml - kustomize build apps/preview > /tmp/preview.yaml # L1: Schema & Policy l1-validate: image: nixpkgs/nix-flakes commands: - kubeconform -strict -ignore-missing-schemas -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' /tmp/*.yaml - conftest test /tmp/*.yaml --policy policy/ # L2: Server-Side Dry-Run l2-dry-run: image: bitnami/kubectl:latest backend_options: kubernetes: serviceAccountName: preview-deployer commands: - kubectl diff -f /tmp/prod.yaml || [ $? -eq 1 ] # L3: Live Preview Lifecycle (on PRs) l3-preview: image: bitnami/kubectl:latest backend_options: kubernetes: serviceAccountName: preview-deployer when: event: [pull_request] commands: - EXPIRY=$(date -d "+4 hours" +%s) - kubectl create ns preview-${CI_COMMIT_PULL_REQUEST} --dry-run=client -o yaml | kubectl annotate --local -f - milner.dev/preview-expires=$EXPIRY -o yaml | kubectl label --local -f - pod-security.kubernetes.io/enforce=baseline -o yaml | kubectl apply -f - - kubectl apply -f preview-manifests/ - kubectl -n preview-system wait --for=condition=Ready kustomization/milner-pr-${CI_COMMIT_PULL_REQUEST} --timeout=10m ``` *(CRD schemas fetched from Datree catalog[7](https://github.com/datreeio/CRDs-catalog); step ServiceAccount specified in `backend_options`[3](https://woodpecker-ci.org/docs/administration/configuration/backends/kubernetes))* ## Implementation Details - **Files to Modify / Create:** - `flake.nix`: Add `fluxcd`, `kubeconform`, and `conftest` to `devShells.default.buildInputs`.[1](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/flake.nix?display=source#L20) - `woodpecker/values.yaml`: Add `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true"`.[2](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/values.yaml?display=source#L40) - `preview-system/`: Add RBAC (`rbac.yaml`), reaper CronJob (`reaper.yaml`), and preview `ResourceQuota`/`LimitRange` templates. - `apps/base/`, `apps/prod/`, `apps/preview/`: Environment-agnostic base app and overlay manifests. - `policy/k8s.rego`: Conftest Rego checks (no `:latest`, memory/CPU limits required, non-privileged).[6](https://www.conftest.dev) - `.woodpecker.yaml`: 4-gate CI validation workflow (`l0-render`, `l1-validate`, `l2-dry-run`, `l3-preview`). - `tests/test_static.py`: Guard tests validating PSA baseline labels, ephemeral storage class on previews, and reaper annotations.[8](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/tests/test_static.py?display=source#L1) - **Dependencies & Tools:** - `fluxcd` (v2.8.x)[9](https://github.com/NixOS/nixpkgs/blob/nixpkgs-unstable/pkgs/by-name/fl/fluxcd/package.nix), `kubeconform` (v0.7.x)[10](https://github.com/NixOS/nixpkgs/blob/nixpkgs-unstable/pkgs/by-name/ku/kubeconform/package.nix), `conftest` (v0.63.x)[11](https://github.com/NixOS/nixpkgs/blob/nixpkgs-unstable/pkgs/by-name/co/conftest/package.nix), Datree CRD catalog schema registry.[7](https://github.com/datreeio/CRDs-catalog) - **Constraints & Edge Cases:** - CI ServiceAccount `preview-deployer` is confined to `woodpecker-runtime` with strict namespace-scoped Roles and cluster-scoped namespace CRUD only.[4](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/woodpecker/values.yaml?display=source#L41) - Previews must strictly use `storageClassName: ephemeral` to prevent orphaned local-path volumes on `/var/mnt/data/persistent`.[12](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/storage/values.yaml?display=source#L1) - Previews must strip Tailscale Ingress annotations to avoid generating stray tailnet nodes. - **Out-of-scope notes:** - In-cluster container registry provisioning is deferred; image builds in L3 use mock/pre-built images or validate manifest reconciliation only. - Multi-tenant NetworkPolicy enforcement is out of scope as Flannel CNI does not enforce NetworkPolicies.[13](https://github.com/flannel-io/flannel) ## Acceptance Criteria - [ ] `flake.nix` provides `fluxcd`, `kubeconform`, and `conftest` in the dev shell. - [ ] `woodpecker/values.yaml` configures the agent with `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP: "true"`. - [ ] RBAC manifests define `preview-deployer` in `woodpecker-runtime` scoped to `preview-system` and namespace CRUD. - [ ] `preview-system` manifests include a TTL reaper CronJob deleting expired preview resources by `milner.dev/preview-expires`. - [ ] `apps/base`, `apps/prod`, and `apps/preview` overlays isolate env variance and enforce `storageClassName: ephemeral` and no Tailscale Ingress on previews. - [ ] Conftest policy `policy/k8s.rego` enforces no `:latest` tags, defined resource limits, and non-privileged containers. - [ ] `.woodpecker.yaml` implements L0 offline render, L1 kubeconform/conftest validation, L2 dry-run diffing, and L3 live preview wait. - [ ] Static guard tests in `tests/test_static.py` validate preview storage tier and PSA labels. ## Testing Plan - `pytest tests/test_static.py`: Verify new yamllint, PSA baseline labeling, and preview storage class guard tests. - `conftest test /tmp/*.yaml --policy policy/`: Verify pass on valid overlays and failure when inserting a `:latest` tag or privileged container. - `kubeconform -strict -ignore-missing-schemas -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json'`: Verify schema validation against Flux and Kubernetes CRDs. - Local dry-run verification: Validate `kustomize build apps/prod` and `kustomize build apps/preview` render valid manifests without undefined substitution variables. ## Related Issues/PRs - Closes #49 ## Research Findings - `pkgs.fluxcd`, `pkgs.kubeconform`, and `pkgs.conftest` are available in nixpkgs for `aarch64-linux` and installable via `flake.nix`.[1](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/flake.nix?display=source#L20) - Offline validation of Flux custom resources requires pointing `kubeconform` to the Datree CRDs catalog schema location.[7](https://github.com/datreeio/CRDs-catalog) - Woodpecker's Kubernetes backend requires `WOODPECKER_BACKEND_K8S_SERVICE_ACCOUNT_NAME_ALLOW_FROM_STEP=true` on agents to permit pipeline steps to bind `preview-deployer`.[3](https://woodpecker-ci.org/docs/administration/configuration/backends/kubernetes) - Flux v2 `postBuild.substituteFrom` requires substitution `ConfigMap` resources to reside in the same namespace as the `Kustomization` object (`preview-system`).[14](https://fluxcd.io/flux/components/kustomize/kustomizations/) - Previews require lower `PriorityClass` and `ResourceQuota` limits to avoid resource starvation on the single 20GB shared worker node.[15](https://git.gloom.garden/tepichord/milner/src/commit/7507e21fb6808c2e61a31ec95d4f206dcfc73acc/worker/main.tf?display=source#L52) ## Pre-Mortem 1. **Resource exhaustion on the shared worker node** — Concurrent live preview environments could exceed the single 20GB worker's memory and ephemeral disk capacity if ResourceQuotas or PriorityClasses are misconfigured. This would trigger pod evictions or OOM kills that disrupt co-located production workloads. 2. **RBAC permission or step ServiceAccount assignment failures** — The `preview-deployer` ServiceAccount may lack cluster-level namespace permissions, or the Woodpecker agent may reject step-level ServiceAccount overrides. This would cause L2 server-side dry-run and L3 preview lifecycle stages to fail immediately in CI. 3. **Reaper CronJob failures causing preview resource leaks** — If the TTL reaper fails to parse expiration annotations or lacks permissions across preview namespaces, abandoned preview environments will persist indefinitely. This would gradually leak ephemeral storage volumes and consume cluster resources when PR close webhooks fail.

tepichord added

and removed

labels

2026-09-11 11:50:05 -07:00

tepichord referenced this issue from a commit

2026-09-11 18:34:55 -07:00

feat(woodpecker): let pipeline steps select a service account

tepichord referenced this issue from a commit