tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Integrate KEDA into milner #85

New issue
Closed
opened 2026-09-22 09:37:25 -07:00 by tepichord · 2 comments
tepichord commented 2026-09-22 09:37:25 -07:00
Owner
Copy link

Here's the KEDA change spelled out against this tree's actual conventions. All paths are relative to gitops/clusters/milner/.

Files to add

  • infrastructure/namespaces/keda.yaml — kind: Namespace, name keda, labels: pod-security.kubernetes.io/enforce: restricted. KEDA's chart defaults satisfy restricted
    (pods runAsNonRoot: true; containers drop ALL capabilities, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, seccompProfile: RuntimeDefault), so
    take the tightest level that works rather than baseline — postgres already sets the precedent. If you'd rather not bet on chart defaults for the namespace's
    admission level, that's the one place to decide it.
  • infrastructure/keda/helmrepository.yaml — kind: HelmRepository, ns keda, url: https://kedacore.github.io/charts, interval: 24h (the convention across all four
    existing HelmRepositories). Not OCI: ghcr.io/kedacore/charts/keda answers 403 anonymously, so HelmRepository is the route — and it's the pattern metrics-server,
    cloudnative-pg, vault-secrets-operator and vault already use.
  • infrastructure/keda/helmrelease.yaml — helm.toolkit.fluxcd.io/v2, ns keda, chart.spec.chart: keda, version: "2.20.2" (quoted, per cnpg), sourceRef.kind:
    HelmRepository, interval: 10m, valuesFrom → the ConfigMap below.
  • infrastructure/keda/values.yaml — pin explicitly rather than inherit: the tree's own doctrine is that a value the tree leaves out hands the decision to the chart
    default, and that default is nowhere in this repository (tests/test_chart_sources.py)`. So:
    • nodeSelector: {node-role: worker} — top-level applies to operator, metrics server and webhooks; the control plane is tainted, so without it two of three pods
      stay Pending.
    • watchNamespace: "" — watch all namespaces, so a ScaledJob in annia is visible. Left at the default this is implicit; state it.
    • crds.install: true — the chart default, but write it down.
    • resources.{operator,metricServer,webhooks} — the chart defaults (requests 100m/100Mi, limits 1/1000Mi) already carry both halves, so policy/k8s.rego is
      satisfied either way; pin them if you want the numbers reviewable in git.
  • infrastructure/keda/kustomization.yaml — mirror metrics-server: namespace: keda, generatorOptions.disableNameSuffixHash: true, configMapGenerator for keda-values
    from values.yaml, resources helmrepository.yaml then helmrelease.yaml.

Files to modify

  • infrastructure/kustomization.yaml — add - namespaces/keda.yaml in the namespaces block and - keda in the component list (after rabbitmq-operator is fine; order
    among components is stylistic, the layer is applied atomically).
  • No layer change. KEDA belongs in infrastructure, which precedes apps — that is exactly what
    tests/test_gitops.py::test_crd_providers_are_not_in_the_layer_that_uses_their_crds asserts once a keda.sh object exists.
  • tests/test_gitops.py — one trap, and it's the reason to think about sequencing. CRD_PROVIDERS needs a row ("keda.sh", "keda", "KEDA"), but that test also does
    assert users, f"nothing applies {group} resources; drop this case" — so the row cannot land in this change alone. Adding it with no keda.sh resource anywhere
    makes the suite red with "nothing applies keda.sh resources". Two honest options:
    • land KEDA now and add the row in the change that adds the first ScaledJob; or
    • include a minimal consumer now (a placeholder ScaledObject/ScaledJob) so the row is meaningful.
      Whichever you pick, say so in the HelmRelease comment, because the next reader will wonder why the row is missing.
  • docs/keda.md — new file, next free section is §17 (§16 is RabbitMQ). Cover: what KEDA is for here, the keda namespace's PSA level and why, how to read a
    ScaledJob's state, and how to pause/unpause scaling for maintenance (autoscaling.keda.sh/paused — the browser-reachable lever the annia remediation design wants).
  • OPERATIONS.md — a row in the "State | Where it is managed" table (KEDA namespace, Helm release → gitops/clusters/milner/infrastructure/keda/), and a row in the
    docs index table for docs/keda.md.
  • AGENTS.md — nothing required; its gitops/ row describes layers, not components, and KEDA adds neither a layer nor a top-level directory.

Ordering and CRD notes to write into the comments

  • CRD upgrades are not automatic. The tree sets no crds: policy anywhere, so this release inherits helm-controller's defaults: install.crds: Create (CRDs land on
    first install) and upgrade.crds: Skip (later chart bumps do not update them). Since a KEDA minor bump can change its CRD schemas, state the policy deliberately in
    helmrelease.yaml rather than inheriting it — either upgrade.crds: CreateReplace or an explicit "CRDs are updated by hand, here is where" note.
  • KEDA's metrics server registers the aggregated APIService v1beta1.external.metrics.k8s.io; that is what makes ScaledObject/ScaledJob work at all, and it's the
    thing to check when scaling silently does nothing.

Verification

  • python -m pytest tests/ -v — test_all_namespaces_have_psa_enforce picks up the new namespace automatically; test_helmreleases_use_v2_api and test_chart_sources.py
    (network-backed: index served, keda 2.20.2 listed) cover the rest.
  • kustomize build gitops/clusters/milner/infrastructure renders, and flux build kustomization infrastructure --path=gitops/clusters/milner/infrastructure
    --kustomization-file=gitops/clusters/milner/infrastructure.yaml --dry-run is what the L0 gate runs.
  • Live after Flux reconciles: kubectl -n keda get pods → three Ready pods on the worker; kubectl get crd | grep keda.sh → four CRDs; kubectl get apiservice
    v1beta1.external.metrics.k8s.io → True; kubectl -n keda get deploy -o wide → all on talos-qpo-xhw.
Here's the KEDA change spelled out against this tree's actual conventions. All paths are relative to gitops/clusters/milner/. Files to add - infrastructure/namespaces/keda.yaml — kind: Namespace, name keda, labels: pod-security.kubernetes.io/enforce: restricted. KEDA's chart defaults satisfy restricted (pods runAsNonRoot: true; containers drop ALL capabilities, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, seccompProfile: RuntimeDefault), so take the tightest level that works rather than baseline — postgres already sets the precedent. If you'd rather not bet on chart defaults for the namespace's admission level, that's the one place to decide it. - infrastructure/keda/helmrepository.yaml — kind: HelmRepository, ns keda, url: https://kedacore.github.io/charts, interval: 24h (the convention across all four existing HelmRepositories). Not OCI: ghcr.io/kedacore/charts/keda answers 403 anonymously, so HelmRepository is the route — and it's the pattern metrics-server, cloudnative-pg, vault-secrets-operator and vault already use. - infrastructure/keda/helmrelease.yaml — helm.toolkit.fluxcd.io/v2, ns keda, chart.spec.chart: keda, version: "2.20.2" (quoted, per cnpg), sourceRef.kind: HelmRepository, interval: 10m, valuesFrom → the ConfigMap below. - infrastructure/keda/values.yaml — pin explicitly rather than inherit: the tree's own doctrine is that a value the tree leaves out hands the decision to the chart default, and that default is nowhere in this repository (tests/test_chart_sources.py)`. So: - nodeSelector: {node-role: worker} — top-level applies to operator, metrics server and webhooks; the control plane is tainted, so without it two of three pods stay Pending. - watchNamespace: "" — watch all namespaces, so a ScaledJob in annia is visible. Left at the default this is implicit; state it. - crds.install: true — the chart default, but write it down. - resources.{operator,metricServer,webhooks} — the chart defaults (requests 100m/100Mi, limits 1/1000Mi) already carry both halves, so policy/k8s.rego is satisfied either way; pin them if you want the numbers reviewable in git. - infrastructure/keda/kustomization.yaml — mirror metrics-server: namespace: keda, generatorOptions.disableNameSuffixHash: true, configMapGenerator for keda-values from values.yaml, resources helmrepository.yaml then helmrelease.yaml. Files to modify - infrastructure/kustomization.yaml — add - namespaces/keda.yaml in the namespaces block and - keda in the component list (after rabbitmq-operator is fine; order among components is stylistic, the layer is applied atomically). - No layer change. KEDA belongs in infrastructure, which precedes apps — that is exactly what tests/test_gitops.py::test_crd_providers_are_not_in_the_layer_that_uses_their_crds asserts once a keda.sh object exists. - tests/test_gitops.py — one trap, and it's the reason to think about sequencing. CRD_PROVIDERS needs a row ("keda.sh", "keda", "KEDA"), but that test also does assert users, f"nothing applies {group} resources; drop this case" — so the row cannot land in this change alone. Adding it with no keda.sh resource anywhere makes the suite red with "nothing applies keda.sh resources". Two honest options: - land KEDA now and add the row in the change that adds the first ScaledJob; or - include a minimal consumer now (a placeholder ScaledObject/ScaledJob) so the row is meaningful. Whichever you pick, say so in the HelmRelease comment, because the next reader will wonder why the row is missing. - docs/keda.md — new file, next free section is §17 (§16 is RabbitMQ). Cover: what KEDA is for here, the keda namespace's PSA level and why, how to read a ScaledJob's state, and how to pause/unpause scaling for maintenance (autoscaling.keda.sh/paused — the browser-reachable lever the annia remediation design wants). - OPERATIONS.md — a row in the "State | Where it is managed" table (KEDA namespace, Helm release → gitops/clusters/milner/infrastructure/keda/), and a row in the docs index table for docs/keda.md. - AGENTS.md — nothing required; its gitops/ row describes layers, not components, and KEDA adds neither a layer nor a top-level directory. Ordering and CRD notes to write into the comments - CRD upgrades are not automatic. The tree sets no crds: policy anywhere, so this release inherits helm-controller's defaults: install.crds: Create (CRDs land on first install) and upgrade.crds: Skip (later chart bumps do not update them). Since a KEDA minor bump can change its CRD schemas, state the policy deliberately in helmrelease.yaml rather than inheriting it — either upgrade.crds: CreateReplace or an explicit "CRDs are updated by hand, here is where" note. - KEDA's metrics server registers the aggregated APIService v1beta1.external.metrics.k8s.io; that is what makes ScaledObject/ScaledJob work at all, and it's the thing to check when scaling silently does nothing. Verification - python -m pytest tests/ -v — test_all_namespaces_have_psa_enforce picks up the new namespace automatically; test_helmreleases_use_v2_api and test_chart_sources.py (network-backed: index served, keda 2.20.2 listed) cover the rest. - kustomize build gitops/clusters/milner/infrastructure renders, and flux build kustomization infrastructure --path=gitops/clusters/milner/infrastructure --kustomization-file=gitops/clusters/milner/infrastructure.yaml --dry-run is what the L0 gate runs. - Live after Flux reconciles: kubectl -n keda get pods → three Ready pods on the worker; kubectl get crd | grep keda.sh → four CRDs; kubectl get apiservice v1beta1.external.metrics.k8s.io → True; kubectl -n keda get deploy -o wide → all on talos-qpo-xhw.
tepichord added this to the MVP project 2026-09-22 09:37:25 -07:00
tepichord commented 2026-09-22 10:29:12 -07:00
Author
Owner
Copy link

Naina:
Updated on 2026-09-22 based on maintainer feedback.

Overview

Install KEDA 2.20.2 as an infrastructure-layer Flux component in a new keda namespace, so a future ScaledJob/ScaledObject in apps can scale workload pods from RabbitMQ queue depth (ARCHITECTURE.md §2). The change adds namespace + HelmRepository + HelmRelease + pinned values + docs/keda.md §17, and deliberately defers the CRD_PROVIDERS row until annia's ScaledJob (#25, step 10) lands.

Technical Approach

  1. Add the namespace — restricted, same precedent as postgres.yaml1:

    apiVersion: v1
kind: Namespace
metadata:
  name: keda
  labels:
    pod-security.kubernetes.io/enforce: restricted

  2. Add the HelmRepository — non-OCI (anonymous OCI pull from ghcr.io/kedacore/charts/keda returns 401/4032), 24h like every other source:

    apiVersion: source.toolkit.fluxcd.io/v1
kind: HelmRepository
metadata: {name: keda, namespace: keda}
spec:
  interval: 24h
  url: https://kedacore.github.io/charts

  3. Add the HelmRelease — v2, quoted version, values from the ConfigMap:

    apiVersion: helm.toolkit.fluxcd.io/v2
kind: HelmRelease
metadata: {name: keda, namespace: keda}
spec:
  interval: 10m
  chart:
    spec:
      chart: keda
      version: "2.20.2"
      sourceRef: {kind: HelmRepository, name: keda}
  valuesFrom:
    - {kind: ConfigMap, name: keda-values}

  4. Add values.yaml — pin all three components for reviewability (the chart's own values.yaml already sets identical blocks, and Helm deep-merges, so a partial pin is safe34):

    crds: {install: true}
watchNamespace: ""
rbac: {controlPlaneServiceAccountsNamespace: kube-system}  # chart default, pinned to be explicit
nodeSelector: {node-role: worker}   # fallback for operator, metricServer, webhooks; merged with kubernetes.io/os: linux

# The chart sets these identically for all three components and Helm deep-merges,
# so a partial pin does NOT delete sibling keys. Pin the full block per component.
# Do NOT set an empty map (securityContext: {} or securityContext.operator: {}):
# the template's `{{- if .Values.securityContext.<component> }} ... {{- else }}`
# else-branch then renders the whole three-component map under one key.
securityContext:
  operator: {capabilities: {drop: [ALL]}, allowPrivilegeEscalation: false,
             readOnlyRootFilesystem: true, seccompProfile: {type: RuntimeDefault}}
  # repeat for metricServer and webhooks (identical)
podSecurityContext:
  operator: {runAsNonRoot: true}
  # repeat for metricServer and webhooks (identical)
resources:
  operator: {requests: {cpu: 100m, memory: 100Mi}, limits: {cpu: "1", memory: 1000Mi}}
  # repeat for metricServer and webhooks (identical)

  5. Add kustomization.yaml — mirror metrics-server/5:

    namespace: keda
generatorOptions: {disableNameSuffixHash: true}
configMapGenerator:
  - {name: keda-values, files: [values.yaml]}
resources: [helmrepository.yaml, helmrelease.yaml]

  6. Wire into infrastructure/kustomization.yaml6:

    resources:
  - namespaces/keda.yaml   # in the namespaces block
  - keda                   # in the component list, after rabbitmq-operator

  7. State the real CRD lifecycle in the HelmRelease comment — do not inherit it silently, and do not repeat the stale "upgrade.crds defaults to Skip" claim78:

    # KEDA ships its CRDs in templates/crds/, not in the chart's crds/ directory,
# so helm-controller's install.crds/upgrade.crds policy never runs for them:
# they ride the release manifest and are replaced on every KEDA bump (upstream
# intent since chart 2.2.1). Read the release notes before bumping — a minor
# bump can change a CRD schema.

  8. Defer the CRD_PROVIDERS row. Adding ("keda.sh", "keda", "KEDA") now fails test_crd_providers_are_not_in_the_layer_that_uses_their_crds with nothing applies keda.sh resources; drop this case9, and a placeholder ScaledObject needs a real scale target to be honest. The consumer is annia's ScaledJob (#25, step 10); add the row there and say so in the HelmRelease comment.

  9. Document it — new docs/keda.md §17 (§16 is RabbitMQ): what KEDA is for here, the namespace PSA level and why, reading ScaledJob state, and the pause lever autoscaling.keda.sh/paused used by the oz console (#90). Add the managed-state row and the docs-index row in OPERATIONS.md.

Implementation Details

Create

File Note
gitops/clusters/milner/infrastructure/namespaces/keda.yaml restricted PSA
gitops/clusters/milner/infrastructure/keda/helmrepository.yaml non-OCI source
gitops/clusters/milner/infrastructure/keda/helmrelease.yaml 2.20.2, CRD-lifecycle + CRD_PROVIDERS-deferral comments
gitops/clusters/milner/infrastructure/keda/values.yaml fully pinned, no empty maps
gitops/clusters/milner/infrastructure/keda/kustomization.yaml metrics-server mirror
docs/keda.md §17

Modify

File Change
gitops/clusters/milner/infrastructure/kustomization.yaml namespaces/keda.yaml + keda
OPERATIONS.md managed-state row + docs-index row

Constraints

  • No dependency additions: KEDA self-manages kedaorg-certs (certificates.autoGenerated: true, certificates.certManager.enabled: false), so no cert-manager is needed.
  • All three images (keda, keda-metrics-apiserver, keda-admission-webhooks) are multi-arch including linux/arm6410; no :latest.
  • Do not add a pod-level seccompProfile — restricted allows container-level-only seccomp, and the chart sets it there.
  • Do not use an empty securityContext/podSecurityContext map: the chart template's else branch renders the three-component map under a single component key, producing a malformed manifest. unverified

docs/keda.md pause guidance (corrected)

Action Command
Pause kubectl -n annia annotate scaledjob <name> autoscaling.keda.sh/paused=true
Resume remove the annotation: kubectl -n annia annotate scaledjob <name> autoscaling.keda.sh/paused-
  • "false" also resumes on 2.20.2 — the bug where it did not (#5215) was fixed in v2.13.011. Any value outside Go's ParseBool set stays paused12.
  • Read the annotation, not the ScaledJob Paused condition: the condition lags and was historically unreliable (#5251 closed stale; the pause reconciler was reworked in v2.20.0 by #7663/#766413).

Out-of-scope notes

  • No TriggerAuthentication/RabbitMQ trigger — that lands with the first ScaledJob and needs Vault.
  • rbac.controlPlaneServiceAccountsNamespace only moves the keda-operator-auth-reader RoleBinding; pinning kube-system matches the chart default and is documentation-only.
  • The admission webhooks register cluster-wide regardless of watchNamespace (which only scopes the cache and RBAC); failurePolicy stays Ignore.
  • KEDA reads Secrets cluster-wide and grants apiGroups: * for scaleTargetRef by default (permissions.operator.restrict.secret, rbac.scaledRefKinds); narrowing both is a separate security decision.

Acceptance Criteria

  • kustomize build gitops/clusters/milner/infrastructure renders the keda Namespace, HelmRepository, HelmRelease, and keda-values ConfigMap.
  • python -m pytest tests/ -v passes with CRD_PROVIDERS unchanged.
  • values.yaml pins nodeSelector, watchNamespace, crds.install, rbac.controlPlaneServiceAccountsNamespace, and all three components' resources + securityContext/podSecurityContext, with no empty maps.
  • helmrelease.yaml carries the CRD-lifecycle comment and the CRD_PROVIDERS-deferral note.
  • docs/keda.md exists as §17 and OPERATIONS.md links it and lists the KEDA release in the managed-state table.

Testing Plan

Check Command
Full suite (PSA, HelmRelease v2, chart-source index) python -m pytest tests/ -v
Render + schema kustomize build gitops/clusters/milner/infrastructure | kubeconform -strict
L0 gate (as the pipeline runs it) flux build kustomization infrastructure --path=gitops/clusters/milner/infrastructure --kustomization-file=gitops/clusters/milner/infrastructure.yaml --dry-run

Post-reconcile, manual: kubectl -n keda get pods → three Ready pods on a worker; kubectl get crd -o name \| grep -c keda.sh → six; kubectl get apiservice v1beta1.external.metrics.k8s.io → True.

Related Issues/PRs

  • Closes #85
  • #25 Create Annia pod — step 10 adds annia's ScaledJob and the CRD_PROVIDERS row.
  • #90 Add the annia pause lever to the oz console — writes autoscaling.keda.sh/paused on annia's ScaledJob.
  • #89 Create the annia enqueuer — produces the queue depth the ScaledJob scales on.

Research Findings

  • Helm deep-merges values over chart defaults: a partial per-component securityContext/podSecurityContext pin does not delete sibling keys, and the chart's defaults already satisfy restricted for all three components. The only real template footgun is an empty map, whose else branch renders the three-component map under one key.
  • #5215 (setting paused: "false" sticking) was fixed in v2.13.0 and is included in 2.20.2; only values outside Go's ParseBool set stay paused, so both removing the annotation and setting false resume.
  • #5251 (ScaledJob Paused condition not flipping back) was closed stale, but the pause reconciler was reworked in v2.20.0 (#7663/#7664); on 2.20.2 the condition does flip back. Read the annotation because it is the source of truth, not because the condition is broken.
  • rbac.controlPlaneServiceAccountsNamespace only moves the keda-operator-auth-reader RoleBinding; pinning kube-system is a documentation no-op (same as the chart default).
  • The admission webhooks register cluster-wide regardless of watchNamespace, which only scopes the cache and RBAC; failurePolicy is Ignore by default.
  • KEDA's six CRDs live in templates/crds/ gated by crds.install, so Flux's install.crds/upgrade.crds (both default Create) never apply and every bump replaces them; Helm never deletes CRDs.
  • Anonymous OCI pull from ghcr.io/kedacore/charts/keda returns 401/403, confirming the non-OCI HelmRepository; https://kedacore.github.io/charts/index.yaml lists 2.20.2.
  • KEDA's v1beta1.external.metrics.k8s.io coexists with metrics-server's v1beta1.metrics.k8s.io; KEDA uses the standard metrics.k8s.io HPA path for cpu/memory triggers.

Pre-Mortem

Assume this plan shipped and then failed. Ranked by likelihood:

  1. A values-wiring typo ships green because nothing renders the chart. kustomize build, flux build --dry-run, and pytest all stop at the generated ConfigMap, so a mis-nested values.yaml key or a keda-values/valuesFrom name mismatch only surfaces when helm-controller reconciles on the cluster. Keep the valuesFrom name byte-identical to the configMapGenerator name and treat the first post-reconcile kubectl -n keda get pods as the real gate.
  2. An empty securityContext/podSecurityContext map malforms the release. Writing securityContext: {} or securityContext.operator: {} trips the chart template's else branch and renders all three components' blocks under one key, producing a manifest that offline validation accepts. Never emit an empty map; pin the full block for each of the three components.
  3. The deferred CRD_PROVIDERS row leaves the layer-ordering invariant unenforced for keda.sh. Since the row lands only with annia's ScaledJob (#25, step 10), nothing fails if that ScaledJob is placed in infrastructure instead of apps, which is exactly the mistake the test exists to catch. Add the row in that follow-up change and keep the consumer in apps.
Naina: *Updated on 2026-09-22 based on maintainer feedback.* ## Overview Install KEDA `2.20.2` as an `infrastructure`-layer Flux component in a new `keda` namespace, so a future ScaledJob/ScaledObject in `apps` can scale workload pods from RabbitMQ queue depth (ARCHITECTURE.md §2). The change adds namespace + HelmRepository + HelmRelease + pinned values + `docs/keda.md §17`, and deliberately **defers** the `CRD_PROVIDERS` row until annia's ScaledJob (#25, step 10) lands. ## Technical Approach 1. **Add the namespace** — `restricted`, same precedent as `postgres.yaml`<sup>[1](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/infrastructure/namespaces/postgres.yaml?display=source#L6)</sup>: ```yaml apiVersion: v1 kind: Namespace metadata: name: keda labels: pod-security.kubernetes.io/enforce: restricted ``` 2. **Add the HelmRepository** — non-OCI (anonymous OCI pull from `ghcr.io/kedacore/charts/keda` returns 401/403<sup>[2](https://kedacore.github.io/charts/keda-2.20.2.tgz)</sup>), 24h like every other source: ```yaml apiVersion: source.toolkit.fluxcd.io/v1 kind: HelmRepository metadata: {name: keda, namespace: keda} spec: interval: 24h url: https://kedacore.github.io/charts ``` 3. **Add the HelmRelease** — v2, quoted version, values from the ConfigMap: ```yaml apiVersion: helm.toolkit.fluxcd.io/v2 kind: HelmRelease metadata: {name: keda, namespace: keda} spec: interval: 10m chart: spec: chart: keda version: "2.20.2" sourceRef: {kind: HelmRepository, name: keda} valuesFrom: - {kind: ConfigMap, name: keda-values} ``` 4. **Add `values.yaml`** — pin all three components for reviewability (the chart's own `values.yaml` already sets identical blocks, and Helm deep-merges, so a partial pin is safe<sup>[3](https://github.com/kedacore/charts/blob/v2.20.2/keda/values.yaml#L467-L511)</sup><sup>[4](https://helm.sh/docs/chart_template_guide/values_files/)</sup>): ```yaml crds: {install: true} watchNamespace: "" rbac: {controlPlaneServiceAccountsNamespace: kube-system} # chart default, pinned to be explicit nodeSelector: {node-role: worker} # fallback for operator, metricServer, webhooks; merged with kubernetes.io/os: linux # The chart sets these identically for all three components and Helm deep-merges, # so a partial pin does NOT delete sibling keys. Pin the full block per component. # Do NOT set an empty map (securityContext: {} or securityContext.operator: {}): # the template's `{{- if .Values.securityContext.<component> }} ... {{- else }}` # else-branch then renders the whole three-component map under one key. securityContext: operator: {capabilities: {drop: [ALL]}, allowPrivilegeEscalation: false, readOnlyRootFilesystem: true, seccompProfile: {type: RuntimeDefault}} # repeat for metricServer and webhooks (identical) podSecurityContext: operator: {runAsNonRoot: true} # repeat for metricServer and webhooks (identical) resources: operator: {requests: {cpu: 100m, memory: 100Mi}, limits: {cpu: "1", memory: 1000Mi}} # repeat for metricServer and webhooks (identical) ``` 5. **Add `kustomization.yaml`** — mirror `metrics-server/`<sup>[5](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/infrastructure/metrics-server/kustomization.yaml?display=source#L1)</sup>: ```yaml namespace: keda generatorOptions: {disableNameSuffixHash: true} configMapGenerator: - {name: keda-values, files: [values.yaml]} resources: [helmrepository.yaml, helmrelease.yaml] ``` 6. **Wire into `infrastructure/kustomization.yaml`**<sup>[6](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/infrastructure/kustomization.yaml?display=source#L29)</sup>: ```yaml resources: - namespaces/keda.yaml # in the namespaces block - keda # in the component list, after rabbitmq-operator ``` 7. **State the real CRD lifecycle in the HelmRelease comment** — do not inherit it silently, and do not repeat the stale "upgrade.crds defaults to Skip" claim<sup>[7](https://github.com/kedacore/charts/blob/v2.20.2/keda/templates/crds/crd-scaledobjects.yaml#L1)</sup><sup>[8](https://github.com/kedacore/charts/blob/v2.2.1/keda/templates/04-crd-scaledobjects.keda.sh.yaml)</sup>: ```yaml # KEDA ships its CRDs in templates/crds/, not in the chart's crds/ directory, # so helm-controller's install.crds/upgrade.crds policy never runs for them: # they ride the release manifest and are replaced on every KEDA bump (upstream # intent since chart 2.2.1). Read the release notes before bumping — a minor # bump can change a CRD schema. ``` 8. **Defer the `CRD_PROVIDERS` row.** Adding `("keda.sh", "keda", "KEDA")` now fails `test_crd_providers_are_not_in_the_layer_that_uses_their_crds` with `nothing applies keda.sh resources; drop this case`<sup>[9](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/tests/test_gitops.py?display=source#L387)</sup>, and a placeholder ScaledObject needs a real scale target to be honest. The consumer is annia's ScaledJob (#25, step 10); add the row there and say so in the HelmRelease comment. 9. **Document it** — new `docs/keda.md` §17 (§16 is RabbitMQ): what KEDA is for here, the namespace PSA level and why, reading ScaledJob state, and the pause lever `autoscaling.keda.sh/paused` used by the oz console (#90). Add the managed-state row and the docs-index row in `OPERATIONS.md`. ## Implementation Details **Create** | File | Note | | --- | --- | | `gitops/clusters/milner/infrastructure/namespaces/keda.yaml` | `restricted` PSA | | `gitops/clusters/milner/infrastructure/keda/helmrepository.yaml` | non-OCI source | | `gitops/clusters/milner/infrastructure/keda/helmrelease.yaml` | `2.20.2`, CRD-lifecycle + CRD_PROVIDERS-deferral comments | | `gitops/clusters/milner/infrastructure/keda/values.yaml` | fully pinned, no empty maps | | `gitops/clusters/milner/infrastructure/keda/kustomization.yaml` | metrics-server mirror | | `docs/keda.md` | §17 | **Modify** | File | Change | | --- | --- | | `gitops/clusters/milner/infrastructure/kustomization.yaml` | `namespaces/keda.yaml` + `keda` | | `OPERATIONS.md` | managed-state row + docs-index row | **Constraints** - No dependency additions: KEDA self-manages `kedaorg-certs` (`certificates.autoGenerated: true`, `certificates.certManager.enabled: false`), so no cert-manager is needed. - All three images (`keda`, `keda-metrics-apiserver`, `keda-admission-webhooks`) are multi-arch including `linux/arm64`<sup>[10](https://github.com/kedacore/keda/blob/v2.20.2/Makefile#L20)</sup>; no `:latest`. - Do **not** add a pod-level `seccompProfile` — `restricted` allows container-level-only seccomp, and the chart sets it there. - Do **not** use an empty `securityContext`/`podSecurityContext` map: the chart template's `else` branch renders the three-component map under a single component key, producing a malformed manifest. _<sup>unverified</sup>_ **docs/keda.md pause guidance (corrected)** | Action | Command | | --- | --- | | Pause | `kubectl -n annia annotate scaledjob <name> autoscaling.keda.sh/paused=true` | | Resume | remove the annotation: `kubectl -n annia annotate scaledjob <name> autoscaling.keda.sh/paused-` | - `"false"` also resumes on 2.20.2 — the bug where it did not (#5215) was fixed in v2.13.0<sup>[11](https://github.com/kedacore/keda/blob/v2.13.0/CHANGELOG.md#L108)</sup>. Any value outside Go's `ParseBool` set stays paused<sup>[12](https://github.com/kedacore/keda/blob/v2.20.2/apis/keda/v1alpha1/scaledobject_types.go#L265-L276)</sup>. - Read the annotation, not the ScaledJob `Paused` condition: the condition lags and was historically unreliable (#5251 closed stale; the pause reconciler was reworked in v2.20.0 by #7663/#7664<sup>[13](https://github.com/kedacore/keda/pull/7664)</sup>). **Out-of-scope notes** - No `TriggerAuthentication`/RabbitMQ trigger — that lands with the first ScaledJob and needs Vault. - `rbac.controlPlaneServiceAccountsNamespace` only moves the `keda-operator-auth-reader` RoleBinding; pinning `kube-system` matches the chart default and is documentation-only. - The admission webhooks register cluster-wide regardless of `watchNamespace` (which only scopes the cache and RBAC); `failurePolicy` stays `Ignore`. - KEDA reads Secrets cluster-wide and grants `apiGroups: *` for `scaleTargetRef` by default (`permissions.operator.restrict.secret`, `rbac.scaledRefKinds`); narrowing both is a separate security decision. ## Acceptance Criteria - [ ] `kustomize build gitops/clusters/milner/infrastructure` renders the `keda` Namespace, HelmRepository, HelmRelease, and `keda-values` ConfigMap. - [ ] `python -m pytest tests/ -v` passes with `CRD_PROVIDERS` unchanged. - [ ] `values.yaml` pins `nodeSelector`, `watchNamespace`, `crds.install`, `rbac.controlPlaneServiceAccountsNamespace`, and all three components' `resources` + `securityContext`/`podSecurityContext`, with no empty maps. - [ ] `helmrelease.yaml` carries the CRD-lifecycle comment and the CRD_PROVIDERS-deferral note. - [ ] `docs/keda.md` exists as §17 and `OPERATIONS.md` links it and lists the KEDA release in the managed-state table. ## Testing Plan | Check | Command | | --- | --- | | Full suite (PSA, HelmRelease v2, chart-source index) | `python -m pytest tests/ -v` | | Render + schema | `kustomize build gitops/clusters/milner/infrastructure \| kubeconform -strict` | | L0 gate (as the pipeline runs it) | `flux build kustomization infrastructure --path=gitops/clusters/milner/infrastructure --kustomization-file=gitops/clusters/milner/infrastructure.yaml --dry-run` | Post-reconcile, manual: `kubectl -n keda get pods` → three Ready pods on a worker; `kubectl get crd -o name \| grep -c keda.sh` → six; `kubectl get apiservice v1beta1.external.metrics.k8s.io` → `True`. ## Related Issues/PRs - Closes [#85](https://git.gloom.garden/tepichord/milner/issues/85) - [#25](https://git.gloom.garden/tepichord/milner/issues/25) Create Annia pod — step 10 adds annia's ScaledJob and the `CRD_PROVIDERS` row. - [#90](https://git.gloom.garden/tepichord/milner/issues/90) Add the annia pause lever to the oz console — writes `autoscaling.keda.sh/paused` on annia's ScaledJob. - [#89](https://git.gloom.garden/tepichord/milner/issues/89) Create the annia enqueuer — produces the queue depth the ScaledJob scales on. ## Research Findings - Helm deep-merges values over chart defaults: a partial per-component `securityContext`/`podSecurityContext` pin does not delete sibling keys, and the chart's defaults already satisfy `restricted` for all three components. The only real template footgun is an empty map, whose `else` branch renders the three-component map under one key. - `#5215` (setting `paused: "false"` sticking) was fixed in v2.13.0 and is included in 2.20.2; only values outside Go's `ParseBool` set stay paused, so both removing the annotation and setting `false` resume. - `#5251` (ScaledJob `Paused` condition not flipping back) was closed stale, but the pause reconciler was reworked in v2.20.0 (#7663/#7664); on 2.20.2 the condition does flip back. Read the annotation because it is the source of truth, not because the condition is broken. - `rbac.controlPlaneServiceAccountsNamespace` only moves the `keda-operator-auth-reader` RoleBinding; pinning `kube-system` is a documentation no-op (same as the chart default). - The admission webhooks register cluster-wide regardless of `watchNamespace`, which only scopes the cache and RBAC; `failurePolicy` is `Ignore` by default. - KEDA's six CRDs live in `templates/crds/` gated by `crds.install`, so Flux's `install.crds`/`upgrade.crds` (both default `Create`) never apply and every bump replaces them; Helm never deletes CRDs. - Anonymous OCI pull from `ghcr.io/kedacore/charts/keda` returns 401/403, confirming the non-OCI `HelmRepository`; `https://kedacore.github.io/charts/index.yaml` lists `2.20.2`. - KEDA's `v1beta1.external.metrics.k8s.io` coexists with metrics-server's `v1beta1.metrics.k8s.io`; KEDA uses the standard `metrics.k8s.io` HPA path for `cpu`/`memory` triggers. ## Pre-Mortem Assume this plan shipped and then failed. Ranked by likelihood: 1. **A values-wiring typo ships green because nothing renders the chart.** `kustomize build`, `flux build --dry-run`, and pytest all stop at the generated ConfigMap, so a mis-nested `values.yaml` key or a `keda-values`/`valuesFrom` name mismatch only surfaces when helm-controller reconciles on the cluster. Keep the `valuesFrom` name byte-identical to the `configMapGenerator` name and treat the first post-reconcile `kubectl -n keda get pods` as the real gate. 2. **An empty `securityContext`/`podSecurityContext` map malforms the release.** Writing `securityContext: {}` or `securityContext.operator: {}` trips the chart template's `else` branch and renders all three components' blocks under one key, producing a manifest that offline validation accepts. Never emit an empty map; pin the full block for each of the three components. 3. **The deferred `CRD_PROVIDERS` row leaves the layer-ordering invariant unenforced for `keda.sh`.** Since the row lands only with annia's ScaledJob (#25, step 10), nothing fails if that ScaledJob is placed in `infrastructure` instead of `apps`, which is exactly the mistake the test exists to catch. Add the row in that follow-up change and keep the consumer in `apps`.
tepichord commented 2026-09-22 14:18:09 -07:00
Author
Owner
Copy link

One linkage and two coordination facts, all small.

The consumer that resolves the open sequencing question. This plan records that the
("keda.sh", "keda", "KEDA") row in tests/test_gitops.py's CRD_PROVIDERS cannot land
with no consumer, so KEDA lands with the first ScaledJob. That consumer is annia's
ScaledJob, owned by the annia pod issue as its step 10, now explicitly after this one.
Linking the two issues is enough; nothing in this plan needs rewriting.

The browser lever has an owner now. This plan names autoscaling.keda.sh/paused as the
browser-reachable lever for annia's crash loop. That is now a console change — the issue
Add the annia pause lever to the oz console adds a control in oz that writes that
annotation on annia's ScaledJob, backed by a Role in annia's namespace for get/patch
on scaledjobs and nothing else. Two details worth carrying into docs/keda.md when it is
written, because both are easy to get wrong:

  • Resume by removing the annotation. Setting it to "false" had a documented history
    of leaving the object paused (kedacore/keda#5215, fixed by kedacore/keda#5257).
  • Read the annotation, not the ScaledJob's Paused condition, which has been reported
    not to flip back to False when the annotation is removed (kedacore/keda#5251).

Documentation numbering. This plan takes §17 for docs/keda.md, and that is yours: the
egress/relay work adds a section to docs/tailscale.md and deliberately took "the next free
number at the time it is written" instead of pinning one, so there is no collision to
resolve.

One linkage and two coordination facts, all small. **The consumer that resolves the open sequencing question.** This plan records that the `("keda.sh", "keda", "KEDA")` row in `tests/test_gitops.py`'s `CRD_PROVIDERS` cannot land with no consumer, so KEDA lands with the first ScaledJob. That consumer is annia's ScaledJob, owned by the annia pod issue as its step 10, now explicitly after this one. Linking the two issues is enough; nothing in this plan needs rewriting. **The browser lever has an owner now.** This plan names `autoscaling.keda.sh/paused` as the browser-reachable lever for annia's crash loop. That is now a console change — the issue `Add the annia pause lever to the oz console` adds a control in `oz` that writes that annotation on annia's ScaledJob, backed by a Role in annia's namespace for `get`/`patch` on `scaledjobs` and nothing else. Two details worth carrying into `docs/keda.md` when it is written, because both are easy to get wrong: - **Resume by removing the annotation.** Setting it to `"false"` had a documented history of leaving the object paused (kedacore/keda#5215, fixed by kedacore/keda#5257). - **Read the annotation, not the ScaledJob's `Paused` condition**, which has been reported not to flip back to `False` when the annotation is removed (kedacore/keda#5251). **Documentation numbering.** This plan takes §17 for `docs/keda.md`, and that is yours: the egress/relay work adds a section to `docs/tailscale.md` and deliberately took "the next free number at the time it is written" instead of pinning one, so there is no collision to resolve.
tepichord 2026-09-25 21:19:58 -07:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/109-nix-base
feat/109-layered-nix-store
feat/110-zot-push-credential
feat/90-annia-pause-lever-oz-console
feat/87-tailscale-egress-from-cluster
chore/ci-step-resources
docs/vault-secret-prompt
feat/42-support-backups-persistent-data
feat/41-integrate-restic-tfstate-backup
feat/3-setting-up-agent-development
kvm-kubernetes-setup
No results found.
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner#85
No description provided.