tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Create a RabbitMQ Service on Milner #57

Merged
tepichord merged 29 commits from feat/55-create-rabbitmq-service into main 2026-09-20 20:25:07 -07:00
Conversation 2 Commits 29 Files changed 26 +714 -24
tepichord commented 2026-09-12 08:25:06 -07:00
Owner
Copy link

Closes #55

Automated PR created by forgejo-done adapter.

Closes #55 Automated PR created by forgejo-done adapter.
Refs: #55
CloudPirates chart 0.6.16 wraps the official rabbitmq/cluster-operator
v2.23.0 (arm64). webhook.enabled defaults to false so no cert-manager
dependency. msgTopologyOperator disabled — queue.declare covers it.

Refs: #55
Single replica, Vault-backed credentials via secretBackend.externalSecret,
persistent storage, worker node affinity. Memory limit raised to 512Mi
(Erlang high-water mark is 0.6x limit).

Refs: #55
VaultStaticSecret renders default_user.conf from secret/rabbitmq/config
via VSO transformation template. VaultAuth uses cross-namespace
vaultConnectionRef per VSO >=1.5 schema.

Refs: #55
Add namespaces/rabbitmq.yaml and rabbitmq-operator to infrastructure,
rabbitmq to apps, vaultauth-rabbitmq and rabbitmq-credentials to secrets.

Refs: #55
Add rabbitmq.com to CRD_PROVIDERS, make _helmrelease_chart resolve
chartRef-based releases via OCIRepository name, bump VSO count
assertion from (3,3) to (4,4).

Refs: #55
Fix guest:guest example in ARCHITECTURE §4 (RabbitMQ rejects remote
login for guest). Update RabbitMQ status from planned to deployed.
Add rabbitmq Vault policy/role to §4c-ter. Add §12 for Vault
credentials bootstrap. Drop 'no RabbitMQ' claim from evolved-npcs.

Refs: #55
Docker Hub's token endpoint lives at auth.docker.io, not
registry-1.docker.io. The test was using the registry host as the
token endpoint, which returns 404 for all Docker Hub repos. This was
previously hidden because all OCI sources were on ghcr.io; the new
RabbitMQ chart (on registry-1.docker.io) exposed the bug.
authenticationRef is a field of the trigger, not of metadata. As written
the ScaledObject would be rejected, and the example is the only KEDA
reference in the tree.
The operator values comment claimed a 512Mi limit while the value was
500Mi, and the rationale it carried was about the Erlang broker, whose
limits live on the RabbitmqCluster CR. State the broker's §4 deviation
where the value is, and describe the operator values accurately.
The function returns the OCIRepository name, not a name parsed from its
URL; the docstring described a resolution that was never implemented.
Every other values.yaml is covered by the yamllint guard.
test(gitops): drop stale chartRef resolution comment
All checks were successful
ci/woodpecker/pr/woodpecker Pipeline was successful
Details
1aa9a0d04a 
The comment repeated the URL-basename claim the docstring just corrected.
tepichord added this to the MVP project 2026-09-15 10:07:23 -07:00
tepichord commented 2026-09-16 13:58:56 -07:00
Author
Owner
Copy link

Significant architectural changes were made to milner, please merge the changes into this branch and integrate them.

Significant architectural changes were made to milner, please merge the changes into this branch and integrate them.
The merge brought in main's §12 (Logging) while the branch's RabbitMQ
section was also §12, leaving two §12s and, worse, two §12b's — Logging's
own "Read the real store with §12b" then named an ambiguous section.
Renumber the RabbitMQ section to §13 and move it ahead of the bring-up
order, which the runbook says is last; the bring-up order becomes §14 and
its "assumes §2–§12 are readable" becomes §2–§13. Its step 11 also gains
§13a, since a wiped cluster otherwise comes back with rabbitmq-server-0
Pending forever on a Vault path nobody created.
§13a passed the generated broker password as a `v kv put` argument, where
`ps` exposes it, and §13b ran `base64 -d` over `default_user.conf`, which
echoes the password into the terminal. Both contradict §5a — whose recipe
sends the token and the generated value on stdin and verifies with
`wc -c` — and the credential rule in AGENTS.md ("never in argv, never
echoed"). Use §5a's shape, and verify the synced file by printing only
the username line and counting the bytes.
The merge combines main's preview-system VaultAuth/VaultStaticSecret with
the branch's rabbitmq pair, so the secrets + preview-system scan now sees
5 of each, not 4. The assertion was the one hunk neither side's edit
covered: each had independently written (4, 4) for a different reason, so
git took one of them and the suite went red — invisible until now because
the gate runs `kustomize build` and kustomize was not installed, skipping
the test rather than failing it.
ARCHITECTURE §4 said the operator "Requires cert-manager for Operator
v2.20+" while the chart is installed with `clusterOperator.webhook.enabled
false` — chosen, per infrastructure/rabbitmq-operator/values.yaml, so that
no cert-manager is needed at all. State the webhook decision, so a reader
does not go and install cert-manager for a dependency this cluster does
not have.
The chart nests its knobs under `clusterOperator`, and Helm drops unknown
values without a word; the operator's pod never passes through kustomize
either, so a mis-nested key is an operator that schedules anywhere and
runs unbounded with every offline gate green — `policy/k8s.rego` only ever
sees the HelmRelease, never the pod spec. Same class of silent failure as
test_victoria_logs_values_survive_the_chart_shape.

The second half pairs the CR's `secretBackend.externalSecret.name` with
the VaultStaticSecret's destination. The two live in different layers, and
if they drift, rabbitmq-server-0 waits forever on a Secret nobody creates.
RabbitMQ claims 10Gi on the `persistent` class, so the shared data disk's
durable budget is 61 GiB of 100 GiB, not 51. Both figures (ARCHITECTURE's
storage-tiers note and OPERATIONS §12d's resize blocker, which exists because
the disk is full) were written before this claim arrived, and the resize
blocker's number is the whole point of the section.
test(gitops): assert the key the RabbitMQ operator mounts
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
2cd7070d26 
The existing tripwire checked only that the CR and the VaultStaticSecret
agree on the Secret's *name*. The operator's StatefulSet projection selects
one item out of that Secret — `default_user.conf` — and RabbitMQ reads the
two settings inside it, so a Secret that appears under the right name
without that key still leaves the broker unable to start, with a mount
error rather than a missing object. Assert the key and the settings it
carries, which is the half of the contract no offline gate was checking.

Also correct `_helmrelease_chart`'s docstring while here: it claimed every
OCIRepository in the tree is named after the chart it carries, but
`victoria-logs` carries `victoria-logs-single`, so the shorthand holds only
for the CRD providers the function is used on — a naming assumption, not a
reading of the artifact.
merge: resolve conflicts with main (#57)
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
21059a6fe0
tepichord commented 2026-09-18 08:48:38 -07:00
Author
Owner
Copy link

Merge changes from main.

Merge changes from main.
The comment said the high-water mark is 0.6x the limit, which does not
produce the 123Mi it cites: 0.6 x 256Mi is 154Mi. The operator sets
total_memory_available_override_value to 0.8 of the limit (it keeps 20%
as Erlang headroom, `removeHeadroom` in the operator's configmap.go) and
RabbitMQ's vm_memory_high_watermark.relative default is 0.6 of that --
0.48 x the limit, ~123Mi. The number was right, the mechanism was not,
and the next person sizing this limit is the one who has to compute it.
docs(rabbitmq): explain the OCIRepository url contract like its siblings
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
fef34f3c4e 
Every other OCIRepository in the tree carries the note that Flux pulls
`url:tag` verbatim, so the URL has to name the chart artifact itself --
and that tests/test_chart_sources.py is the only gate that can see the
difference, because kubeconform validates the schema and nothing compares
the URL with the registry. This one had no comment at all, and that trap
is the one the module's own docstring says cost three live failures.
merge: Postgres data layer, Nix images and the nix class into RabbitMQ (#55)
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
a19084cb13 
Keeps main's architectural decisions whole — the CloudNativePG `data` layer,
the Nix-built console/router with its empty privileged tier, and the three
StorageClasses — and adds the RabbitMQ broker to them rather than the reverse.

Conflicts, resolved:
- apps/infrastructure/secrets kustomizations: union. RabbitMQ keeps its
  namespace, operator and Secrets, and main keeps `data`, `cnpg-system`,
  `postgres` and the `../data/` comment.
- docs/vault.md §4c-ter: both policies, one loop. `rabbitmq-read` beside
  main's `postgres-read` glob, and `postgres` + `rabbitmq` both in the
  per-namespace role list.
- tests/test_gitops.py: CRD_PROVIDERS gains both operators (three rows), and
  the VaultAuth/VaultStaticSecret tripwire moves (6,7)/(6,10) -> (7,11),
  read off the rendered layers.
- ARCHITECTURE.md storage tiers: main's paragraph kept — three classes, the
  `nix` store, Postgres, Woodpecker keeping nothing durable — with RabbitMQ's
  10 GiB claim folded into its accounting: 70 GiB on `persistent` plus the
  retired 10 GiB sqlite and 200 GiB of `nix`, 280 GiB of the 2 TiB disk.

Collisions the merge introduced but did not flag:
- docs/rabbitmq.md is §16, not §13: main's Postgres runbook already owns §13,
  and it is referenced from eleven places. OPERATIONS.md follows.
- docs/logging.md §12d's budget re-stated on main's accounting, which the old
  sentence contradicted.
- ARCHITECTURE.md's `/var/mnt/data` row names the `nix` class its own
  paragraph above describes.

Verified: 189 offline tests, control-plane/worker/storage terraform tests,
and kustomize + kubeconform -strict + conftest over all six layers.
fix(qos): Vault, Postgres and Pocket ID are Guaranteed, not BestEffort
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
8ed7c39981 
All three declared no requests and no limits, which is BestEffort — the class
the kubelet evicts first under node memory pressure, ahead of every Burstable
pod on the node. Each is required for operation, and none of them is in the
shape the eviction ranking assumes:

* `vault-0` comes back *sealed*, needing three of five unseal keys by hand
  (docs/vault.md §4c).
* `postgres-1` holds the state of every app in the cluster (Woodpecker,
  Pocket ID).
* `pocket-id-0` is the OIDC provider every login goes through.

All three become Guaranteed (requests == limits):

* Vault — `server.resources` (the `vault.resources` helper reads nothing else)
  and its own block on the `chown-vault-data` init container: Kubernetes asks
  request == limit of every container, init containers included, so a
  Guaranteed Vault container beside an empty init container is a Burstable
  pod. Memory limit ~10x the ~54Mi observed, so the OOM killer cannot be what
  triggers the unseal either.
* Postgres — `spec.resources`, which the operator applies to the
  `bootstrap-controller` init container as well (pkg/specs/containers.go).
  Guaranteed is the class CloudNativePG asks for: at Guaranteed it sets
  PG_OOM_ADJUST_VALUE=0 on the postmaster, so a reached cgroup limit kills a
  backend and the instance survives instead of the postmaster (OOM score
  -997). `shared_buffers` is now written down as a quarter of the limit, the
  ratio CNPG's resource guidance gives, in PostgreSQL's own units — "Mi" is
  not one it accepts and the instance fails to start with one.
* Pocket ID — top-level `resources`, which its chart's StatefulSet renders.

Zot is already Burstable and Woodpecker stays BestEffort, both on purpose.

The pods are chart-generated or operator-generated, so kustomize and
`policy/k8s.rego` never see their specifications and no offline gate would
notice a block landing a level away from where it is read. Each pod therefore
gets a tripwire in tests/test_gitops.py that asserts the QoS class Kubernetes
will assign — the property that decides eviction order — rather than a figure;
each was checked by mutating the manifest into the mistake it guards.
fix(qos): the claim provisioner and the Tailscale operator too
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
ci/woodpecker/pull_request_closed/cleanup Pipeline was successful
Details
837dbfc03c 
The last two BestEffort pods that are required for operation, and the two
hardest to notice: neither is a workload anybody looks at.

* local-path-provisioner — the only thing that can bind a new claim. While it
  is evicted a PVC stays Pending and the pod that needs it never starts, and
  the failure is quiet by construction: the volume directories are on the data
  disk rather than in the pod, so existing claims keep working and only the
  next one hangs. `resources` in its values.yaml — where the chart reads it
  (templates/deployment.yaml) and what Flux renders into
  `local-path-provisioner-values`.

* tailscale-operator — the ingress path for every hostname in the cluster.
  This release has no HelmRelease (its OAuth credentials must never reach
  git), so docs/tailscale.md §3 installs it by hand and *that command is its
  manifest*; it now carries `operatorConfig.resources`. The key matters as
  much as the numbers: the chart reads `operatorConfig.resources`, and
  `operator.resources` one level up is stored by helm, never rendered, and the
  command reports success either way (checked against the chart's deployment
  template, which emits no resources for it). §3 also gains the
  `--reuse-values` form for the install that predates the block, which needs
  no credential.

Both become Guaranteed (requests == limits): ~12Mi and ~26Mi observed against
128Mi and 256Mi declared.

Two tripwires: one asserts the provisioner's QoS class; the other parses the
hand-run command out of the runbook and asserts the exact key path, requests
== limits, and the absence of an `operator.resources` flag — that runbook is
the only description of that pod anywhere, so a doc edit could put it back to
BestEffort with every gate still green. Both were checked by mutating the
file into the mistake each guards.

The operator is hand-run by design, so it is the one change here that does not
reach the cluster on merge: that same §3 command applies it.
tepichord deleted branch feat/55-create-rabbitmq-service 2026-09-20 20:25:07 -07:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner!57
No description provided.