tepichord/milner

Fork 0

Create a RabbitMQ Service on Milner #55

New issue

Closed

opened 2026-09-11 18:47:51 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-11 18:47:51 -07:00

Owner

No description provided.

tepichord added the

labels

2026-09-11 18:47:51 -07:00

tepichord added this to the MVP project

2026-09-11 18:47:51 -07:00

tepichord commented

2026-09-11 19:58:52 -07:00

Author

Owner

Naina:

Overview

Deploy a single-node RabbitMQ broker on Milner as the cluster-internal message bus for the evolved-npcs orchestrator/simulation pipeline, following the allocation and topology already fixed in ARCHITECTURE.md §4 and §7. This issue covers the RabbitMQ Cluster Operator, the RabbitmqCluster instance, Vault-backed credentials, and the offline gates that keep them honest; KEDA is deliberately out of scope.

Technical Approach

Add the rabbitmq namespace — infrastructure/namespaces/rabbitmq.yaml, then list it in infrastructure/kustomization.yaml.¹²

apiVersion: v1
kind: Namespace
metadata:
  name: rabbitmq
  labels:
    pod-security.kubernetes.io/enforce: baseline

Install the operator in infrastructure/rabbitmq-operator/ using the cloudpirates OCI chart — it wraps the official ghcr.io/rabbitmq/cluster-operator:2.23.0 arm64 image and generates no cert-manager objects.³⁴⁵⁶

# ocirepository.yaml
spec:
  url: oci://registry-1.docker.io/cloudpirates/rabbitmq-cluster-operator
  ref: { tag: "0.6.16" }          # chart 0.6.16 -> appVersion 2.23.0
  layerSelector: { mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip" }

# helmrelease.yaml
spec:
  interval: 10m
  chartRef: { kind: OCIRepository, name: rabbitmq-cluster-operator }
  valuesFrom:
    - { kind: ConfigMap, name: rabbitmq-cluster-operator-values }

# values.yaml — webhook.enabled defaults to false, so no cert-manager and no Issuer/Certificate
clusterOperator:
  nodeSelector: { node-role: worker }
  resources:                       # chart default is {} -> upstream 200m/500Mi
    requests: { cpu: 100m, memory: 256Mi }
    limits:   { cpu: 200m, memory: 500Mi }
msgTopologyOperator:
  enabled: false

Create the broker in apps/rabbitmq/rabbitmqcluster.yaml — one CR is all the operator needs; it then creates Service rabbitmq (5672/15672/15692), StatefulSet rabbitmq-server, and the PVC.⁷⁸

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata: { name: rabbitmq, namespace: rabbitmq }
spec:
  replicas: 1
  secretBackend:
    externalSecret: { name: rabbitmq-credentials }
  resources:
    requests: { cpu: 100m, memory: 128Mi }
    limits:   { cpu: 250m, memory: 512Mi }   # ARCHITECTURE says 256Mi; see Implementation Details
  persistence: { storageClassName: persistent, storage: 10Gi }
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - { key: node-role, operator: In, values: [worker] }

Render the default-user Secret from Vault — secrets/vaultauth-rabbitmq.yaml + secrets/rabbitmq-credentials.yaml. externalSecret makes the operator skip its own rabbitmq-default-user Secret and wait for this one.⁹¹⁰¹¹

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: { name: rabbitmq-credentials, namespace: rabbitmq }
spec:
  type: kv-v2
  mount: secret
  path: rabbitmq/config
  refreshAfter: 1h
  vaultAuthRef: default
  destination:
    name: rabbitmq-credentials
    create: true
    overwrite: true
    transformation:
      templates:
        "default_user.conf":
          text: |
            {{- printf "default_user = %s\ndefault_pass = %s" (get .Secrets "username") (get .Secrets "password") -}}

Teach the CRD-layer guard about chartRef before adding the RabbitMQ row — _helmrelease_chart returns None for chartRef, so the row would silently find no provider.¹²¹³

CRD_PROVIDERS = [
    ("secrets.hashicorp.com", "vault-secrets-operator", "Vault Secrets Operator"),
    ("rabbitmq.com", "rabbitmq-cluster-operator", "RabbitMQ Cluster Operator"),
]
# resolve chartRef -> OCIRepository url basename (".../rabbitmq-cluster-operator")

Move the VSO tripwire and document the Vault bootstrap — test_vault_crs_match_the_vso_crd_shape asserts exact counts, and Vault must hold the broker's credentials before the StatefulSet can start.¹⁴¹⁵

assert (len(auths), len(statics)) == (4, 4)

v policy write rabbitmq-read - <<'HCL'
path "secret/data/rabbitmq/*" { capabilities = ["read"] }
HCL
v write auth/kubernetes/role/rabbitmq \
  bound_service_account_names=default bound_service_account_namespaces=rabbitmq \
  policies=rabbitmq-read ttl=1h
v kv put secret/rabbitmq/config username=evolved password="$(openssl rand -base64 24)"

Fix the guest:guest example in ARCHITECTURE.md §4 — RabbitMQ refuses remote logins for guest with "user 'guest' can only connect via localhost", so the KEDA trigger example must reference a TriggerAuthentication instead.¹⁶¹⁷¹⁸

Implementation Details

New files

Path	Purpose
`gitops/clusters/milner/infrastructure/namespaces/rabbitmq.yaml`	namespace + PSA label
`gitops/clusters/milner/infrastructure/rabbitmq-operator/{kustomization,ocirepository,helmrelease,values}.yaml`	operator, `namespace: rabbitmq`, `configMapGenerator` for values
`gitops/clusters/milner/apps/rabbitmq/{kustomization,rabbitmqcluster}.yaml`	broker CR only
`gitops/clusters/milner/secrets/{vaultauth-rabbitmq,rabbitmq-credentials}.yaml`	Vault → Secret

Modified files

Path	Change
`gitops/clusters/milner/infrastructure/kustomization.yaml`	add `namespaces/rabbitmq.yaml` + `rabbitmq-operator`
`gitops/clusters/milner/apps/kustomization.yaml`	add `rabbitmq`
`gitops/clusters/milner/secrets/kustomization.yaml`	add the two VSO CRs
`tests/test_gitops.py`	`chartRef`-aware `_helmrelease_chart`, `rabbitmq.com` row, VSO count `(3,3)` → `(4,4)`
`OPERATIONS.md`	§4c-ter policy + role, §7 new `vk` path, RabbitMQ section¹⁹²⁰
`ARCHITECTURE.md`	`guest:guest` fix; RabbitMQ rows `planned` → `deployed`²¹²²
`gitops/clusters/milner/apps/evolved-npcs/README.md`	drop the "no KEDA or RabbitMQ yet" claim²³

Dependencies / constraints

Layering is load-bearing: the operator's CRDs land in infrastructure, the RabbitmqCluster CR in apps, and the credentials Secret (via secrets) must beat the apps layer — already guaranteed by dependsOn.²⁴²⁵²⁶
Operator image is arm64; chart pins every image by digest.²⁷²⁸
cert-manager is not added: clusterOperator.webhook.enabled stays false, so the operator runs with ENABLE_WEBHOOKS=false. Trade-off: RabbitmqCluster CRs get no admission validation.²⁹³⁰
The chart installs all crds/ files unconditionally, including the messaging-topology CRDs, even with msgTopologyOperator.enabled: false; they are inert with no controller.³¹³²
Memory limit raised from ARCHITECTURE's 256Mi to 512Mi: the broker's high-water mark is 0.6× the limit, so 256Mi alarms at ~123Mi — below a comfortable Erlang baseline. Requests stay at the documented 100m/128Mi.^33343536
Vault credentials must exist before Flux applies apps; a missing path leaves the broker Pending forever.³⁷³⁸³⁹

Out-of-scope notes

KEDA (ScaledObject, TriggerAuthentication, queue sim-tasks) is not part of this issue — no scaling target pods exist yet. Its Vault credentials can reuse secret/rabbitmq/config.
msgTopologyOperator is left disabled, so sim-tasks and per-service users are not declared declaratively here; the orchestrator's idempotent queue.declare covers it.
rabbitmq.com has no entry in the datreeio CRD catalog, so kubeconform -ignore-missing-schemas skips the CR offline; L2 only dry-runs the preview overlay, so the CR's real schema check happens at L3.⁴⁰⁴¹⁴² ^unverified — the catalog does carry rabbitmq.com/rabbitmqcluster_v1beta1.json (verified since 2022), and L1 validates it against that catalog; L2/L3 only ever render apps/preview, so neither is the gate this bullet claims.

Acceptance Criteria

python -m pytest tests/ -v passes, including the new rabbitmq.com row in test_crd_providers_are_not_in_the_layer_that_uses_their_crds and the (4, 4) VSO count.
flux build renders infrastructure, secrets, and apps; grep finds no cert-manager HelmRelease, Certificate, or Issuer anywhere in gitops/.
On Milner, kubectl -n rabbitmq get rabbitmqcluster/rabbitmq -o jsonpath='{.status.conditions[?(@.type=="AllReplicasReady")].status}' returns True.
rabbitmq.rabbitmq.svc.cluster.local:5672 and :15672 both answer from a throwaway in-cluster pod authenticated with the credentials in secret/rabbitmq-credentials.
secret/rabbitmq-default-user does not exist (proves secretBackend.externalSecret took effect) and secret/rabbitmq-credentials carries a non-empty default_user.conf.

Testing Plan

Offline: run the full suite (kustomize build + kubeconform + conftest + namespace/PSA/CRD-layer/VSO-shape guards) — the CRD-layer row must fail if the operator and the RabbitmqCluster CR are ever put in one layer.
Chart reachability: tests/test_chart_sources.py proves the pinned 0.6.16 OCI tag exists.
Live broker: kubectl -n rabbitmq get pods,pvc,svc — one rabbitmq-server-0 Running/Ready with a bound persistent PVC.
Live auth: kubectl -n rabbitmq port-forward svc/rabbitmq 15672:15672, then log in with the Vault username/password; /api/overview returns 200.
Live AMQP: publish and consume one message on sim-tasks from a throwaway pod using the same Secret.
Pipeline: open the PR and confirm L0–L3 are green (L3 exercises the CR against real CRDs).

ARCHITECTURE.md §4 (RabbitMQ allocation) and §6/§7 (KEDA, network topology) — the pre-agreed constraints.
#49 — the four-tier validation pipeline this change must pass.
#38 — Pocket ID, the last app added; its OCIRepository/HelmRelease/VSO shape is reused here.
#47 — Flux GitOps install and the layer-atomicity rule.
Follow-up to file: KEDA deployment + ScaledObject for sim-tasks (ARCHITECTURE.md §4, §6).

Research Findings

There is no official Helm chart for the RabbitMQ Cluster Operator; the Bitnami chart is unusable (bitnami/rabbitmq-cluster-operator images returned 404), leaving the cloudpirates OCI chart as the only chart using the official arm64 ghcr.io/rabbitmq/cluster-operator:2.23.0 image.
Operator v2.20+ genuinely requires cert-manager for its webhooks; disabling webhooks (chart default) or using the chart's self-signed certs removes that dependency.
The operator creates Service rabbitmq with 5672, 15672, and 15692 as ClusterIP, so KEDA can reach the management API without extra configuration.
The operator's spec.secretBackend.externalSecret lets VSO own the default-user Secret; its native Vault integration needs the Vault Agent Injector, which Milner disables.
KEDA's TriggerAuthentication can only reference a Secret in the same namespace as the ScaledObject, and ARCHITECTURE.md's guest:guest trigger can never work because guest is loopback-only.

Pre-Mortem

The third-party cloudpirates chart does not behave as assumed. The plan rests on clusterOperator.webhook.enabled defaulting to false and on exact value keys/digest pins that were never rendered locally, so a schema mismatch fails the HelmRelease or, worse, silently deploys cert-manager Issuer/Certificate objects. With no official chart and no maintainer fallback, an upstream value rename leaves Milner without a broker until the chart is re-pinned.
The Vault-to-Secret handoff blocks the broker indefinitely. secretBackend.externalSecret makes the StatefulSet wait for rabbitmq-credentials, so if the Vault path is not bootstrapped first or the VSO destination.transformation.templates shape differs from the installed VSO version, rabbitmq-server-0 stays Pending with no obvious error. Because the plan treats the template as known-good rather than testing the rendered default_user.conf, this surfaces only on the live cluster.
The offline gates cannot catch a malformed RabbitmqCluster CR. kubeconform runs with -ignore-missing-schemas and rabbitmq.com has no catalog entry, so wrong field names such as secretBackend or persistence pass CI and fail at apply time; meanwhile the brittle exact-count assertion ((4, 4) VSO objects, the new chartRef branch) can fail the suite for unrelated reasons. Either failure mode stops the PR without proving the broker actually works.

Naina: ## Overview Deploy a single-node RabbitMQ broker on Milner as the cluster-internal message bus for the evolved-npcs orchestrator/simulation pipeline, following the allocation and topology already fixed in `ARCHITECTURE.md` §4 and §7. This issue covers the RabbitMQ Cluster Operator, the `RabbitmqCluster` instance, Vault-backed credentials, and the offline gates that keep them honest; KEDA is deliberately out of scope. ## Technical Approach 1. **Add the `rabbitmq` namespace** — `infrastructure/namespaces/rabbitmq.yaml`, then list it in `infrastructure/kustomization.yaml`.[1](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/gitops/clusters/milner/infrastructure/kustomization.yaml?display=source#L4)[2](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/gitops/clusters/milner/infrastructure/namespaces/pocket-id.yaml?display=source#L6) ```yaml apiVersion: v1 kind: Namespace metadata: name: rabbitmq labels: pod-security.kubernetes.io/enforce: baseline ``` 2. **Install the operator in `infrastructure/rabbitmq-operator/`** using the cloudpirates OCI chart — it wraps the official `ghcr.io/rabbitmq/cluster-operator:2.23.0` arm64 image and generates no cert-manager objects.[3](https://raw.githubusercontent.com/CloudPirates-io/helm-charts/rabbitmq-cluster-operator-0.6.16/charts/rabbitmq-cluster-operator/Chart.yaml)[4](https://raw.githubusercontent.com/CloudPirates-io/helm-charts/rabbitmq-cluster-operator-0.6.16/charts/rabbitmq-cluster-operator/values.yaml)[5](https://github.com/rabbitmq/cluster-operator/pkgs/container/cluster-operator)[6](https://github.com/rabbitmq/cluster-operator/blob/v2.23.0/config/manager/manager.yaml#L62) ```yaml # ocirepository.yaml spec: url: oci://registry-1.docker.io/cloudpirates/rabbitmq-cluster-operator ref: { tag: "0.6.16" } # chart 0.6.16 -> appVersion 2.23.0 layerSelector: { mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip" } ``` ```yaml # helmrelease.yaml spec: interval: 10m chartRef: { kind: OCIRepository, name: rabbitmq-cluster-operator } valuesFrom: - { kind: ConfigMap, name: rabbitmq-cluster-operator-values } ``` ```yaml # values.yaml — webhook.enabled defaults to false, so no cert-manager and no Issuer/Certificate clusterOperator: nodeSelector: { node-role: worker } resources: # chart default is {} -> upstream 200m/500Mi requests: { cpu: 100m, memory: 256Mi } limits: { cpu: 200m, memory: 500Mi } msgTopologyOperator: enabled: false ``` 3. **Create the broker in `apps/rabbitmq/rabbitmqcluster.yaml`** — one CR is all the operator needs; it then creates Service `rabbitmq` (5672/15672/15692), StatefulSet `rabbitmq-server`, and the PVC.[7](https://github.com/rabbitmq/cluster-operator/blob/v2.23.0/internal/resource/service.go#L126)[8](https://github.com/rabbitmq/cluster-operator/blob/v2.23.0/internal/resource/statefulset.go#L40) ```yaml apiVersion: rabbitmq.com/v1beta1 kind: RabbitmqCluster metadata: { name: rabbitmq, namespace: rabbitmq } spec: replicas: 1 secretBackend: externalSecret: { name: rabbitmq-credentials } resources: requests: { cpu: 100m, memory: 128Mi } limits: { cpu: 250m, memory: 512Mi } # ARCHITECTURE says 256Mi; see Implementation Details persistence: { storageClassName: persistent, storage: 10Gi } affinity: nodeAffinity: requiredDuringSchedulingIgnoredDuringExecution: nodeSelectorTerms: - matchExpressions: - { key: node-role, operator: In, values: [worker] } ``` 4. **Render the default-user Secret from Vault** — `secrets/vaultauth-rabbitmq.yaml` + `secrets/rabbitmq-credentials.yaml`. `externalSecret` makes the operator skip its own `rabbitmq-default-user` Secret and wait for this one.[9](https://github.com/rabbitmq/cluster-operator/blob/v2.23.0/internal/resource/rabbitmq_resource_builder.go#L89)[10](https://github.com/rabbitmq/cluster-operator/blob/v2.23.0/internal/resource/statefulset.go#L528)[11](https://developer.hashicorp.com/vault/docs/deploy/kubernetes/vso/secret-transformation) ```yaml apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: { name: rabbitmq-credentials, namespace: rabbitmq } spec: type: kv-v2 mount: secret path: rabbitmq/config refreshAfter: 1h vaultAuthRef: default destination: name: rabbitmq-credentials create: true overwrite: true transformation: templates: "default_user.conf": text: | {{- printf "default_user = %s\ndefault_pass = %s" (get .Secrets "username") (get .Secrets "password") -}} ``` 5. **Teach the CRD-layer guard about `chartRef` before adding the RabbitMQ row** — `_helmrelease_chart` returns `None` for `chartRef`, so the row would silently find no provider.[12](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/tests/test_gitops.py?display=source#L259)[13](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/gitops/clusters/milner/apps/pocket-id/helmrelease.yaml?display=source#L8) ```python CRD_PROVIDERS = [ ("secrets.hashicorp.com", "vault-secrets-operator", "Vault Secrets Operator"), ("rabbitmq.com", "rabbitmq-cluster-operator", "RabbitMQ Cluster Operator"), ] # resolve chartRef -> OCIRepository url basename (".../rabbitmq-cluster-operator") ``` 6. **Move the VSO tripwire and document the Vault bootstrap** — `test_vault_crs_match_the_vso_crd_shape` asserts exact counts, and Vault must hold the broker's credentials before the StatefulSet can start.[14](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/tests/test_gitops.py?display=source#L360)[15](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/OPERATIONS.md?display=source#L239) ```python assert (len(auths), len(statics)) == (4, 4) ``` ```bash v policy write rabbitmq-read - <<'HCL' path "secret/data/rabbitmq/*" { capabilities = ["read"] } HCL v write auth/kubernetes/role/rabbitmq \ bound_service_account_names=default bound_service_account_namespaces=rabbitmq \ policies=rabbitmq-read ttl=1h v kv put secret/rabbitmq/config username=evolved password="$(openssl rand -base64 24)" ``` 7. **Fix the `guest:guest` example in `ARCHITECTURE.md` §4** — RabbitMQ refuses remote logins for `guest` with "user 'guest' can only connect via localhost", so the KEDA trigger example must reference a `TriggerAuthentication` instead.[16](https://www.rabbitmq.com/docs/access-control#loopback-users)[17](https://keda.sh/docs/latest/concepts/authentication/)[18](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/ARCHITECTURE.md?display=source#L56) ## Implementation Details **New files** | Path | Purpose | |---|---| | `gitops/clusters/milner/infrastructure/namespaces/rabbitmq.yaml` | namespace + PSA label | | `gitops/clusters/milner/infrastructure/rabbitmq-operator/{kustomization,ocirepository,helmrelease,values}.yaml` | operator, `namespace: rabbitmq`, `configMapGenerator` for values | | `gitops/clusters/milner/apps/rabbitmq/{kustomization,rabbitmqcluster}.yaml` | broker CR only | | `gitops/clusters/milner/secrets/{vaultauth-rabbitmq,rabbitmq-credentials}.yaml` | Vault → Secret | **Modified files** | Path | Change | |---|---| | `gitops/clusters/milner/infrastructure/kustomization.yaml` | add `namespaces/rabbitmq.yaml` + `rabbitmq-operator` | | `gitops/clusters/milner/apps/kustomization.yaml` | add `rabbitmq` | | `gitops/clusters/milner/secrets/kustomization.yaml` | add the two VSO CRs | | `tests/test_gitops.py` | `chartRef`-aware `_helmrelease_chart`, `rabbitmq.com` row, VSO count `(3,3)` → `(4,4)` | | `OPERATIONS.md` | §4c-ter policy + role, §7 new `vk` path, RabbitMQ section[19](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/OPERATIONS.md?display=source#L158)[20](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/OPERATIONS.md?display=source#L167) | | `ARCHITECTURE.md` | `guest:guest` fix; RabbitMQ rows `planned` → `deployed`[21](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/ARCHITECTURE.md?display=source#L283)[22](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/ARCHITECTURE.md?display=source#L56) | | `gitops/clusters/milner/apps/evolved-npcs/README.md` | drop the "no KEDA or RabbitMQ yet" claim[23](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/gitops/clusters/milner/apps/evolved-npcs/README.md?display=source#L5) | **Dependencies / constraints** - Layering is load-bearing: the operator's CRDs land in `infrastructure`, the `RabbitmqCluster` CR in `apps`, and the credentials Secret (via `secrets`) must beat the `apps` layer — already guaranteed by `dependsOn`.[24](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/gitops/clusters/milner/apps.yaml?display=source#L11)[25](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/gitops/clusters/milner/secrets.yaml?display=source#L11)[26](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/tests/test_gitops.py?display=source#L254) - Operator image is arm64; chart pins every image by digest.[27](https://github.com/rabbitmq/cluster-operator/pkgs/container/cluster-operator)[28](https://raw.githubusercontent.com/CloudPirates-io/helm-charts/rabbitmq-cluster-operator-0.6.16/charts/rabbitmq-cluster-operator/values.yaml) - `cert-manager` is **not** added: `clusterOperator.webhook.enabled` stays `false`, so the operator runs with `ENABLE_WEBHOOKS=false`. Trade-off: `RabbitmqCluster` CRs get no admission validation.[29](https://raw.githubusercontent.com/CloudPirates-io/helm-charts/rabbitmq-cluster-operator-0.6.16/charts/rabbitmq-cluster-operator/values.yaml)[30](https://raw.githubusercontent.com/CloudPirates-io/helm-charts/rabbitmq-cluster-operator-0.6.16/charts/rabbitmq-cluster-operator/README.md) - The chart installs *all* `crds/` files unconditionally, including the messaging-topology CRDs, even with `msgTopologyOperator.enabled: false`; they are inert with no controller.[31](https://github.com/CloudPirates-io/helm-charts/tree/rabbitmq-cluster-operator-0.6.16/charts/rabbitmq-cluster-operator/crds)[32](https://helm.sh/docs/chart_best_practices/custom_resource_definitions/) - Memory limit raised from ARCHITECTURE's 256Mi to 512Mi: the broker's high-water mark is 0.6× the limit, so 256Mi alarms at ~123Mi — below a comfortable Erlang baseline. Requests stay at the documented 100m/128Mi.[33](https://github.com/rabbitmq/cluster-operator/blob/v2.23.0/internal/resource/configmap.go#L275)[34](https://github.com/rabbitmq/cluster-operator/blob/v2.23.0/internal/resource/configmap.go#L430)[35](https://www.rabbitmq.com/docs/memory)[36](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/ARCHITECTURE.md?display=source#L179) - Vault credentials must exist before Flux applies `apps`; a missing path leaves the broker Pending forever.[37](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/gitops/clusters/milner/apps.yaml?display=source#L11)[38](https://github.com/rabbitmq/cluster-operator/blob/v2.23.0/internal/resource/statefulset.go#L528)[39](https://www.rabbitmq.com/kubernetes/operator/using-operator) **Out-of-scope notes** - KEDA (`ScaledObject`, `TriggerAuthentication`, queue `sim-tasks`) is not part of this issue — no scaling target pods exist yet. Its Vault credentials can reuse `secret/rabbitmq/config`. - `msgTopologyOperator` is left disabled, so `sim-tasks` and per-service users are not declared declaratively here; the orchestrator's idempotent `queue.declare` covers it. - `rabbitmq.com` has no entry in the datreeio CRD catalog, so `kubeconform -ignore-missing-schemas` skips the CR offline; L2 only dry-runs the preview overlay, so the CR's real schema check happens at L3.[40](https://github.com/datreeio/CRDs-catalog/tree/main/rabbitmq.com)[41](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/.woodpecker.yaml?display=source#L101)[42](https://git.gloom.garden/tepichord/milner/src/commit/3b98d837fbde3a3adfc44e32135634a3596641be/.woodpecker.yaml?display=source#L207) _unverified_ — the catalog **does** carry `rabbitmq.com/rabbitmqcluster_v1beta1.json` (verified since 2022), and L1 validates it against that catalog; L2/L3 only ever render `apps/preview`, so neither is the gate this bullet claims. ## Acceptance Criteria - [ ] `python -m pytest tests/ -v` passes, including the new `rabbitmq.com` row in `test_crd_providers_are_not_in_the_layer_that_uses_their_crds` and the `(4, 4)` VSO count. - [ ] `flux build` renders `infrastructure`, `secrets`, and `apps`; `grep` finds no cert-manager HelmRelease, `Certificate`, or `Issuer` anywhere in `gitops/`. - [ ] On Milner, `kubectl -n rabbitmq get rabbitmqcluster/rabbitmq -o jsonpath='{.status.conditions[?(@.type=="AllReplicasReady")].status}'` returns `True`. - [ ] `rabbitmq.rabbitmq.svc.cluster.local:5672` and `:15672` both answer from a throwaway in-cluster pod authenticated with the credentials in `secret/rabbitmq-credentials`. - [ ] `secret/rabbitmq-default-user` does not exist (proves `secretBackend.externalSecret` took effect) and `secret/rabbitmq-credentials` carries a non-empty `default_user.conf`. ## Testing Plan - Offline: run the full suite (`kustomize build` + kubeconform + conftest + namespace/PSA/CRD-layer/VSO-shape guards) — the CRD-layer row must fail if the operator and the `RabbitmqCluster` CR are ever put in one layer. - Chart reachability: `tests/test_chart_sources.py` proves the pinned `0.6.16` OCI tag exists. - Live broker: `kubectl -n rabbitmq get pods,pvc,svc` — one `rabbitmq-server-0` `Running/Ready` with a bound `persistent` PVC. - Live auth: `kubectl -n rabbitmq port-forward svc/rabbitmq 15672:15672`, then log in with the Vault username/password; `/api/overview` returns 200. - Live AMQP: publish and consume one message on `sim-tasks` from a throwaway pod using the same Secret. - Pipeline: open the PR and confirm L0–L3 are green (L3 exercises the CR against real CRDs). ## Related Issues/PRs - `ARCHITECTURE.md` §4 (RabbitMQ allocation) and §6/§7 (KEDA, network topology) — the pre-agreed constraints. - #49 — the four-tier validation pipeline this change must pass. - #38 — Pocket ID, the last app added; its `OCIRepository`/`HelmRelease`/VSO shape is reused here. - #47 — Flux GitOps install and the layer-atomicity rule. - Follow-up to file: KEDA deployment + `ScaledObject` for `sim-tasks` (ARCHITECTURE.md §4, §6). ## Research Findings - There is **no official Helm chart** for the RabbitMQ Cluster Operator; the Bitnami chart is unusable (`bitnami/rabbitmq-cluster-operator` images returned 404), leaving the cloudpirates OCI chart as the only chart using the official arm64 `ghcr.io/rabbitmq/cluster-operator:2.23.0` image. - Operator v2.20+ genuinely requires cert-manager for its webhooks; disabling webhooks (chart default) or using the chart's self-signed certs removes that dependency. - The operator creates Service `rabbitmq` with 5672, 15672, and 15692 as ClusterIP, so KEDA can reach the management API without extra configuration. - The operator's `spec.secretBackend.externalSecret` lets VSO own the default-user Secret; its native Vault integration needs the Vault Agent Injector, which Milner disables. - KEDA's `TriggerAuthentication` can only reference a Secret in the same namespace as the `ScaledObject`, and ARCHITECTURE.md's `guest:guest` trigger can never work because `guest` is loopback-only. ## Pre-Mortem 1. **The third-party cloudpirates chart does not behave as assumed.** The plan rests on `clusterOperator.webhook.enabled` defaulting to `false` and on exact value keys/digest pins that were never rendered locally, so a schema mismatch fails the HelmRelease or, worse, silently deploys cert-manager `Issuer`/`Certificate` objects. With no official chart and no maintainer fallback, an upstream value rename leaves Milner without a broker until the chart is re-pinned. 2. **The Vault-to-Secret handoff blocks the broker indefinitely.** `secretBackend.externalSecret` makes the StatefulSet wait for `rabbitmq-credentials`, so if the Vault path is not bootstrapped first or the VSO `destination.transformation.templates` shape differs from the installed VSO version, `rabbitmq-server-0` stays Pending with no obvious error. Because the plan treats the template as known-good rather than testing the rendered `default_user.conf`, this surfaces only on the live cluster. 3. **The offline gates cannot catch a malformed `RabbitmqCluster` CR.** `kubeconform` runs with `-ignore-missing-schemas` and `rabbitmq.com` has no catalog entry, so wrong field names such as `secretBackend` or `persistence` pass CI and fail at apply time; meanwhile the brittle exact-count assertion (`(4, 4)` VSO objects, the new `chartRef` branch) can fail the suite for unrelated reasons. Either failure mode stops the PR without proving the broker actually works.

tepichord added

and removed

labels

2026-09-11 20:05:19 -07:00

tepichord referenced this issue from a commit

2026-09-12 08:23:49 -07:00

feat(rabbitmq): add Vault-backed credentials for broker default user

tepichord referenced this issue from a commit