tepichord/milner

Fork 0

Create the annia enqueuer #89

New issue

Closed

opened 2026-09-22 14:10:20 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-22 14:10:20 -07:00

Owner

   Annia's enqueuer is the always-on service that turns one Forgejo delivery into one message
   on the broker: it receives the webhook, verifies the HMAC over the raw body, and publishes
   a pointer. It is the producer end of the contract with `annia-infra` #308.

   Nothing owns it today. `annia-infra` mentions no `enqueuer`, `rabbitmq` or `amqp`, and
   this board has no issue for it. It is also the one piece of annia that nothing blocks: the
   broker landed in `apps/rabbitmq/` (#55), and this service needs neither the LLM path
   (which serves the worker pod) nor KEDA (which serves the ScaledJob).

   ## What it must be

   - **Namespace `annia`**, `pod-security.kubernetes.io/enforce: baseline`, declared in
     `infrastructure/namespaces/annia.yaml` and listed in that layer's kustomization.
     Whichever of this issue, the egress issue and #25 lands first creates it; never twice.
   - **Service `annia-enqueuer` on port 80**, in annia's namespace. The name and the port are
     a contract with the Cloudflare route issue, whose tunnel route is
     `http://annia-enqueuer.annia.svc.cluster.local:80`.
   - **Path `/api/hook`** — the path that route publishes, so the zone's WAF rules cover it
     with no change to `cloudflare/main.tf`.
   - **HMAC verification over the raw request body.** The tunnel gives POST-only and rate
     limiting; neither is authentication, and an unverified receiver is an open door to
     annia's queue.
   - **One pointer message per delivery** — `{repo, kind, number, event_id}` — which is the
     contract with `annia-infra` #308. The pointer, not the payload: annia fetches what it
     needs, and a webhook body is not a payload format to build on.
   - **Its own image**, built from annia's repository, with the tag pinned in the manifest
     here and moved by Flux image automation. Which pipeline publishes it is
     `annia-infra` #310/#311's question, and the answer needs to be one of them rather than
     none.

   ## Credentials — Vault-first, two paths

   | Vault path | What it is | Read by |
   |---|---|---|
   | `secret/annia/forgejo-webhook` | The HMAC secret Forgejo signs with | The enqueuer |
   | `secret/annia/rabbitmq` | The broker user and password | The enqueuer, and KEDA's trigger later |

   Each path needs what docs/vault.md §4c-ter describes for a namespace: a read-only policy
   for `secret/annia/*`, a kubernetes-auth role bound to annia's service account, and then a
   `VaultAuth` plus `VaultStaticSecret` in the `secrets` layer. **Never a hand-run Secret.**

   The tripwire this moves: `tests/test_gitops.py` currently expects "7 VaultAuths and 11
   VaultStaticSecrets" — annia's additions change both numbers, which is the same class of
   tripwire #25's plan already carries for KEDA.

   ## The broker objects — declared in git

   **Enable the Messaging Topology Operator:** `msgTopologyOperator.enabled: true` in
   `infrastructure/rabbitmq-operator/values.yaml`, where the line is currently a deliberate
   `false`. It ships in the chart this cluster already installs and pins by tag
   (`ocirepository.yaml`), and its webhooks use the chart's self-signed certificate default
   (`useCertManager: false`) — so no cert-manager, which is the thing #25's stale step 1 was
   for and the reason the cluster operator itself runs with `ENABLE_WEBHOOKS=false`.

   **Declare annia's topology as CRs in the `data` layer**, `data/annia/rabbitmq.yaml`,
   following `data/postgres/{roles,databases}.yaml` — a data-plane object belongs in `data`,
   and `apps` is the layer that names it:

   - `Vhost` `annia`
   - `Exchange` `annia.events`, topic, durable
   - `Queue` `annia.dispatch` — **quorum**, with `x-delivery-limit: 2` and
     `x-dead-letter-exchange: annia.dlx`
   - `Queue` `annia.dispatch.dead` and its binding to `annia.dlx`
   - `Binding` from `annia.events` to `annia.dispatch` on annia's routing key
   - `Permission` for annia's user on the `annia` vhost (configure, write, read)

   **Why a quorum queue with a delivery limit.** `x-delivery-limit` is a quorum-queue
   argument — a classic queue ignores it silently — and it *is* the 2-failure rule: after two
   failed deliveries the broker dead-letters the message. That is what stops the crash loop
   before a human is involved, with no failure bookkeeping inside annia, and it is why the
   dead-letter queue has to exist before the first failure rather than after.

   **Why the topology is in git and not in annia's code.** This repo's answer to "an app needs
   a data-plane object" is a CR in `data`, resolved before `apps` names it — AGENTS.md's layer
   rule, with `data/postgres/` as the precedent. The queue and its arguments are the cross-repo
   contract: in git they are reviewable and gate-checked, in another repo's runtime code they
   are inferred.

   Layering and gates:

   - `CRD_PROVIDERS` in `tests/test_gitops.py` already carries a `rabbitmq.com` row. The
     topology kinds are in that same API group and are installed by that same chart, so the
     row covers them — and the test it feeds is what guarantees the HelmRelease lands in an
     earlier layer than these CRs. If the group ever splits, that row is the thing that
     notices.
   - Enabling the sub-operator adds CRDs and its own pods; it does not change the chart pin.

   **The alternative**, if the cluster should not carry a second operator: let the enqueuer
   declare the exchange, queue and binding at startup and keep no CRs, documenting the names
   and arguments here and in `docs/rabbitmq.md` as the contract. That is the standard AMQP
   declare-on-publish shape and it is less machinery — but the topology then exists only as
   annia's code, and the queue and its dead-letter queue appear only after the first delivery,
   which is also when KEDA needs them.

   ## The broker identity

   annia's enqueuer does not publish as the broker's default user. `secret/annia/rabbitmq`
   holds the username and password, written to Vault first; the broker-side user is created
   once with that password, the same bootstrap shape `docs/rabbitmq.md` §16a already uses for
   the broker's own default user, and its permissions are the `Permission` CR above.

   Stated cost: that creation is a documented bootstrap step with a row in `OPERATIONS.md` §1,
   because no CR can carry a Vault-held password — the topology operator's `User` kind
   generates its own password into a Secret in the `rabbitmq` namespace, which no consumer in
   another namespace may read, and copying that Secret by hand is exactly the pattern this
   repo forbids.

   The cheaper alternative is publishing as the broker's default user, whose credential is
   already at `secret/rabbitmq/config`. That makes annia's Vault role read another namespace's
   path — the tree's first cross-namespace read — and puts every consumer on one credential.

   ## Verification

   In-cluster, with no tunnel involved:

   ```bash
   # The receiver accepts a correctly signed delivery and publishes one pointer:
   kubectl -n annia exec deploy/annia-enqueuer -- ...
   # -> 2xx, and one message on the queue (management UI at rabbitmq.faun-snapper.ts.net)

   # The same body with a bad signature is rejected and publishes nothing:
   # -> the app's own rejection, and no change in queue depth

   # The queue exists with its arguments (type and delivery limit, not just the name):
   kubectl -n rabbitmq exec rabbitmq-server-0 -- rabbitmqctl list_queues name type arguments
   ```

   ## Done when

   - A signed delivery on the Service reaches the enqueuer and publishes one pointer message;
     an unsigned or wrongly signed one is rejected and publishes nothing.
   - `annia.dispatch` exists as a quorum queue with `x-delivery-limit 2` and a working
     dead-letter target, verified by a message that fails twice and lands in the DLQ.
   - The queue, exchange, routing-key and pointer names are written in the comment on
     `annia-infra` #308 as the cross-repo contract.
   - `docs/rabbitmq.md` gains annia's vhost, user and topology; `OPERATIONS.md` §1 gains the
     broker-side bootstrap row.
   - The offline gates are green: `python3 -m pytest tests/ -k gitops`, `kustomize build` for
     the `infrastructure` and `data` layers, `conftest test --policy policy/`.

   The end-to-end check through the edge — a delivery arriving from the internet at
   `annia.vando.dev/api/hook` — belongs to the Cloudflare route issue, so these two do not
   each wait on the other.

   ## Out of scope

   - annia's worker pod, its `/nix` claim, and the ScaledJob that consumes this queue — that
     is #25, after #85.
   - The console's pause lever for the crash loop — that is its own issue.

````markdown Annia's enqueuer is the always-on service that turns one Forgejo delivery into one message on the broker: it receives the webhook, verifies the HMAC over the raw body, and publishes a pointer. It is the producer end of the contract with `annia-infra` #308. Nothing owns it today. `annia-infra` mentions no `enqueuer`, `rabbitmq` or `amqp`, and this board has no issue for it. It is also the one piece of annia that nothing blocks: the broker landed in `apps/rabbitmq/` (#55), and this service needs neither the LLM path (which serves the worker pod) nor KEDA (which serves the ScaledJob). ## What it must be - **Namespace `annia`**, `pod-security.kubernetes.io/enforce: baseline`, declared in `infrastructure/namespaces/annia.yaml` and listed in that layer's kustomization. Whichever of this issue, the egress issue and #25 lands first creates it; never twice. - **Service `annia-enqueuer` on port 80**, in annia's namespace. The name and the port are a contract with the Cloudflare route issue, whose tunnel route is `http://annia-enqueuer.annia.svc.cluster.local:80`. - **Path `/api/hook`** — the path that route publishes, so the zone's WAF rules cover it with no change to `cloudflare/main.tf`. - **HMAC verification over the raw request body.** The tunnel gives POST-only and rate limiting; neither is authentication, and an unverified receiver is an open door to annia's queue. - **One pointer message per delivery** — `{repo, kind, number, event_id}` — which is the contract with `annia-infra` #308. The pointer, not the payload: annia fetches what it needs, and a webhook body is not a payload format to build on. - **Its own image**, built from annia's repository, with the tag pinned in the manifest here and moved by Flux image automation. Which pipeline publishes it is `annia-infra` #310/#311's question, and the answer needs to be one of them rather than none. ## Credentials — Vault-first, two paths | Vault path | What it is | Read by | |---|---|---| | `secret/annia/forgejo-webhook` | The HMAC secret Forgejo signs with | The enqueuer | | `secret/annia/rabbitmq` | The broker user and password | The enqueuer, and KEDA's trigger later | Each path needs what docs/vault.md §4c-ter describes for a namespace: a read-only policy for `secret/annia/*`, a kubernetes-auth role bound to annia's service account, and then a `VaultAuth` plus `VaultStaticSecret` in the `secrets` layer. **Never a hand-run Secret.** The tripwire this moves: `tests/test_gitops.py` currently expects "7 VaultAuths and 11 VaultStaticSecrets" — annia's additions change both numbers, which is the same class of tripwire #25's plan already carries for KEDA. ## The broker objects — declared in git **Enable the Messaging Topology Operator:** `msgTopologyOperator.enabled: true` in `infrastructure/rabbitmq-operator/values.yaml`, where the line is currently a deliberate `false`. It ships in the chart this cluster already installs and pins by tag (`ocirepository.yaml`), and its webhooks use the chart's self-signed certificate default (`useCertManager: false`) — so no cert-manager, which is the thing #25's stale step 1 was for and the reason the cluster operator itself runs with `ENABLE_WEBHOOKS=false`. **Declare annia's topology as CRs in the `data` layer**, `data/annia/rabbitmq.yaml`, following `data/postgres/{roles,databases}.yaml` — a data-plane object belongs in `data`, and `apps` is the layer that names it: - `Vhost` `annia` - `Exchange` `annia.events`, topic, durable - `Queue` `annia.dispatch` — **quorum**, with `x-delivery-limit: 2` and `x-dead-letter-exchange: annia.dlx` - `Queue` `annia.dispatch.dead` and its binding to `annia.dlx` - `Binding` from `annia.events` to `annia.dispatch` on annia's routing key - `Permission` for annia's user on the `annia` vhost (configure, write, read) **Why a quorum queue with a delivery limit.** `x-delivery-limit` is a quorum-queue argument — a classic queue ignores it silently — and it *is* the 2-failure rule: after two failed deliveries the broker dead-letters the message. That is what stops the crash loop before a human is involved, with no failure bookkeeping inside annia, and it is why the dead-letter queue has to exist before the first failure rather than after. **Why the topology is in git and not in annia's code.** This repo's answer to "an app needs a data-plane object" is a CR in `data`, resolved before `apps` names it — AGENTS.md's layer rule, with `data/postgres/` as the precedent. The queue and its arguments are the cross-repo contract: in git they are reviewable and gate-checked, in another repo's runtime code they are inferred. Layering and gates: - `CRD_PROVIDERS` in `tests/test_gitops.py` already carries a `rabbitmq.com` row. The topology kinds are in that same API group and are installed by that same chart, so the row covers them — and the test it feeds is what guarantees the HelmRelease lands in an earlier layer than these CRs. If the group ever splits, that row is the thing that notices. - Enabling the sub-operator adds CRDs and its own pods; it does not change the chart pin. **The alternative**, if the cluster should not carry a second operator: let the enqueuer declare the exchange, queue and binding at startup and keep no CRs, documenting the names and arguments here and in `docs/rabbitmq.md` as the contract. That is the standard AMQP declare-on-publish shape and it is less machinery — but the topology then exists only as annia's code, and the queue and its dead-letter queue appear only after the first delivery, which is also when KEDA needs them. ## The broker identity annia's enqueuer does not publish as the broker's default user. `secret/annia/rabbitmq` holds the username and password, written to Vault first; the broker-side user is created once with that password, the same bootstrap shape `docs/rabbitmq.md` §16a already uses for the broker's own default user, and its permissions are the `Permission` CR above. Stated cost: that creation is a documented bootstrap step with a row in `OPERATIONS.md` §1, because no CR can carry a Vault-held password — the topology operator's `User` kind generates its own password into a Secret in the `rabbitmq` namespace, which no consumer in another namespace may read, and copying that Secret by hand is exactly the pattern this repo forbids. The cheaper alternative is publishing as the broker's default user, whose credential is already at `secret/rabbitmq/config`. That makes annia's Vault role read another namespace's path — the tree's first cross-namespace read — and puts every consumer on one credential. ## Verification In-cluster, with no tunnel involved: ```bash # The receiver accepts a correctly signed delivery and publishes one pointer: kubectl -n annia exec deploy/annia-enqueuer -- ... # -> 2xx, and one message on the queue (management UI at rabbitmq.faun-snapper.ts.net) # The same body with a bad signature is rejected and publishes nothing: # -> the app's own rejection, and no change in queue depth # The queue exists with its arguments (type and delivery limit, not just the name): kubectl -n rabbitmq exec rabbitmq-server-0 -- rabbitmqctl list_queues name type arguments ``` ## Done when - A signed delivery on the Service reaches the enqueuer and publishes one pointer message; an unsigned or wrongly signed one is rejected and publishes nothing. - `annia.dispatch` exists as a quorum queue with `x-delivery-limit 2` and a working dead-letter target, verified by a message that fails twice and lands in the DLQ. - The queue, exchange, routing-key and pointer names are written in the comment on `annia-infra` #308 as the cross-repo contract. - `docs/rabbitmq.md` gains annia's vhost, user and topology; `OPERATIONS.md` §1 gains the broker-side bootstrap row. - The offline gates are green: `python3 -m pytest tests/ -k gitops`, `kustomize build` for the `infrastructure` and `data` layers, `conftest test --policy policy/`. The end-to-end check through the edge — a delivery arriving from the internet at `annia.vando.dev/api/hook` — belongs to the Cloudflare route issue, so these two do not each wait on the other. ## Out of scope - annia's worker pod, its `/nix` claim, and the ScaledJob that consumes this queue — that is #25, after #85. - The console's pause lever for the crash loop — that is its own issue. ````

tepichord added the

labels

2026-09-22 14:10:20 -07:00

tepichord added this to the MVP project

2026-09-22 14:10:20 -07:00

tepichord referenced this issue

2026-09-22 14:12:02 -07:00

Publish the annia webhook through the Cloudflare Tunnel #88

~~tepichord referenced this issue 2026-09-22 14:12:03 -07:00~~

Publish the annia webhook through the Cloudflare Tunnel #88

tepichord commented

2026-09-22 15:35:22 -07:00

Author

Owner

Naina:

Overview

Create annia's cluster-side enqueuer: the annia namespace, its two Vault credentials, the RabbitMQ Messaging Topology Operator and the topology CRs that are the cross-repo contract with annia-infra #308, and the annia-enqueuer Service (port 80, /api/hook) plus the Deployment that pins the enqueuer image. The enqueuer's application source and image are annia-infra's (#308/#310) — this issue owns the deployment contract and the broker topology. The annia namespace is created here; #25 and #87 must not create it again.

Technical Approach

Add the namespace — infrastructure/namespaces/annia.yaml, and list it in infrastructure/kustomization.yaml after namespaces/rabbitmq.yaml.

apiVersion: v1
kind: Namespace
metadata:
  name: annia
  labels:
    pod-security.kubernetes.io/enforce: baseline

Enable the Messaging Topology Operator and pin its knobs — infrastructure/rabbitmq-operator/values.yaml (it is enabled: false today⁵, and test_rabbitmq_knobs_land_where_they_are_read pins that⁶). The chart generates its own webhook certs (useCertManager: false)⁴, so no cert-manager.

msgTopologyOperator:
  enabled: true
  nodeSelector:
    node-role: worker
  resources:
    requests: {cpu: 100m, memory: 128Mi}
    limits: {cpu: 200m, memory: 256Mi}

Declare the topology — data/annia/rabbitmq.yaml (all CRs namespace: annia, rabbitmqClusterReference: {name: rabbitmq, namespace: rabbitmq}¹⁰). The DLX and x-dead-letter-routing-key are deliberate: without the latter, dead-lettering reuses the publisher's routing key and nothing catches it¹²¹³.

- kind: Vhost
  metadata: {name: annia}
  spec: {defaultQueueType: quorum, deletionPolicy: retain}
- kind: Exchange
  metadata: {name: annia.events}
  spec: {vhost: annia, type: topic, durable: true}
- kind: Exchange
  metadata: {name: annia.dlx}
  spec: {vhost: annia, type: topic, durable: true}
- kind: Queue
  metadata: {name: annia.dispatch}
  spec:
    vhost: annia
    type: quorum
    durable: true
    arguments:
      x-delivery-limit: 2
      x-dead-letter-exchange: annia.dlx
      x-dead-letter-routing-key: annia.dead
- kind: Queue
  metadata: {name: annia.dispatch.dead}
  spec: {vhost: annia, type: quorum, durable: true}
- kind: Binding   # annia.events -> annia.dispatch, key "*": one word per repo
  spec: {vhost: annia, source: annia.events, destination: annia.dispatch, destinationType: queue, routingKey: "*"}
- kind: Binding   # annia.dlx -> annia.dispatch.dead
  spec: {vhost: annia, source: annia.dlx, destination: annia.dispatch.dead, destinationType: queue, routingKey: annia.dead}
- kind: Permission
  spec: {vhost: annia, user: annia, permissions: {configure: ".*", write: ".*", read: ".*"}}

Queue.spec.durable must be written explicitly — the operator defaults it to false¹¹ (the chart installs only a validating webhook, so the upstream mutating defaulter that would set it to true never runs).

Wire the data layer — data/annia/kustomization.yaml (namespace: annia, resource rabbitmq.yaml) and add - annia to data/kustomization.yaml. No healthCheckExprs change: kstatus reads these CRs' own Ready condition, and before the broker exists they have none (→ Current)⁸⁹, so the expression list stays ["Cluster", "DatabaseRole", "Database"].
Add the Vault policy and role — docs/vault.md §4c-ter: add annia to the for ns in … loop and this policy above it.

v policy write annia-read - <<'HCL'
path "secret/data/annia/*" { capabilities = ["read"] }
HCL

Sync the two credentials — secrets/vaultauth-annia.yaml (ns annia, role annia, SA default, vaultConnectionRef: vault-secrets-operator/vault-connection)¹⁴ and secrets/annia-secrets.yaml (the kv-v2 / refreshAfter: 1h / create: true / overwrite: true shape matches the existing VaultStaticSecrets¹⁵).

- kind: VaultStaticSecret   # name: annia-hmac-secret,  path: annia/forgejo-webhook
  destination: {name: annia-hmac-secret}
- kind: VaultStaticSecret   # name: annia-rabbitmq-credentials, path: annia/rabbitmq
  destination: {name: annia-rabbitmq-credentials}

Both kv-v2, refreshAfter: 1h, destination.create: true, destination.overwrite: true. List both files in secrets/kustomization.yaml.

Add the Service and Deployment — apps/annia/{service.yaml,deployment.yaml,kustomization.yaml} (namespace: annia), and - annia in apps/kustomization.yaml. TCP probes, so the app needs no /healthz.

kind: Service
metadata: {name: annia-enqueuer}
spec:
  ports: [{name: http, port: 80, targetPort: http}]
---
kind: Deployment
spec:
  replicas: 1
  template:
    spec:
      nodeSelector: {node-role: worker}
      containers:
        - name: enqueuer
          image: zot.faun-snapper.ts.net/milner/annia-enqueuer:<annia-infra-pin>
          ports: [{name: http, containerPort: 8080}]
          env:
            - {name: FORGEJO_WEBHOOK_SECRET, valueFrom: {secretKeyRef: {name: annia-hmac-secret, key: hmac-secret}}}
            - {name: RABBITMQ_USERNAME, valueFrom: {secretKeyRef: {name: annia-rabbitmq-credentials, key: username}}}
            - {name: RABBITMQ_PASSWORD, valueFrom: {secretKeyRef: {name: annia-rabbitmq-credentials, key: password}}}
            - {name: RABBITMQ_URL, value: "amqp://$(RABBITMQ_USERNAME):$(RABBITMQ_PASSWORD)@rabbitmq.rabbitmq.svc.cluster.local:5672/annia"}
          readinessProbe: {tcpSocket: {port: http}}
          resources:
            requests: {cpu: 10m, memory: 32Mi}
            limits: {cpu: 100m, memory: 128Mi}

Move the two tripwires and add the contract test — tests/test_gitops.py (the VSO count tripwire is (7, 11) today⁷, and 7 + 1 = 8, 11 + 2 = 13):

# test_vault_crs_match_the_vso_crd_shape
assert (len(auths), len(statics)) == (8, 13)
# test_rabbitmq_knobs_land_where_they_are_read
assert values["msgTopologyOperator"]["enabled"] is True

Add test_annia_topology_matches_the_cross_repo_contract asserting the queue's type/durable/arguments, the DLQ binding key, the annia.events → annia.dispatch binding, the Permission, and the Service's name/port.

Document and post the contract — docs/rabbitmq.md §16e (vhost, user, topology, DLX inspection, the imperative user creation) and an OPERATIONS.md §1 row. Then comment on annia-infra #308:

{"repo": "tepichord/milner", "kind": "issue", "number": 89, "event_id": "<X-Forgejo-Delivery>"}

annia.events (topic) → annia.dispatch (quorum, x-delivery-limit: 2, DLX annia.dlx → annia.dispatch.dead); publish routing key = repo short name (milner), binding key *; persistent (delivery_mode=2) with publisher confirms, 2xx only after the confirm; HMAC-SHA256 over the raw body, X-Forgejo-Signature (bare hex) or X-Hub-Signature-256: sha256=<hex>¹⁶.

Implementation Details

Files to create:

Path	What
`gitops/clusters/milner/infrastructure/namespaces/annia.yaml`	Namespace, PSA `baseline`
`gitops/clusters/milner/data/annia/rabbitmq.yaml`	2 exchanges, 2 queues, 2 bindings, vhost, permission
`gitops/clusters/milner/data/annia/kustomization.yaml`	`namespace: annia`
`gitops/clusters/milner/secrets/vaultauth-annia.yaml`	VaultAuth `default` in `annia`
`gitops/clusters/milner/secrets/annia-secrets.yaml`	Two VaultStaticSecrets
`gitops/clusters/milner/apps/annia/{service,deployment,kustomization}.yaml`	Service port 80 → 8080; Deployment

Files to modify: infrastructure/kustomization.yaml, infrastructure/rabbitmq-operator/values.yaml, data/kustomization.yaml, secrets/kustomization.yaml, apps/kustomization.yaml, tests/test_gitops.py (3 places), docs/vault.md §4c-ter, docs/rabbitmq.md (§16e), OPERATIONS.md §1.

Vault keys the manifests depend on (hand-run, docs/rabbitmq.md §16e): secret/annia/forgejo-webhook → hmac-secret; secret/annia/rabbitmq → username, password. Broker user annia is created imperatively with that password (rabbitmqctl add_user), because no CR can carry a Vault-held password¹⁷ ^unverified — the operator's User CR does expose spec.importCredentialsSecret, which imports username/password from a Secret, so the premise that no CR can carry a Vault-held password needs re-checking.

Constraints: layer ordering is infrastructure (operator) → secrets → data (topology) → apps (workload)¹²; the topology operator's webhooks are failurePolicy: Fail³, so a broken operator surfaces as a data-layer failure; policy/k8s.rego needs cpu+memory limits on the Deployment container.

Coordination / out-of-scope notes:

The namespace's single declaration lives here; #25 and #87 must not add infrastructure/namespaces/annia.yaml.
The enqueuer source and image are annia-infra's (#308 source, #310/#311 pipeline), per #89. Until #310 publishes the pinned tag the pod is ImagePullBackOff and the apps layer reports not-Ready — a stated cross-repo dependency, not a misconfiguration. #308's plan comment claims the source is Milner's; confirm which holds before implementing step 7.
Flux image automation (ImageRepository/ImagePolicy/ImageUpdateAutomation) does not exist in this cluster; the tag is a literal pin, as #25 records. Adding image automation is #25's, not this issue's.
The end-to-end edge delivery at annia.vando.dev/api/hook is #88.

Acceptance Criteria

kustomize build succeeds for the infrastructure, data, and apps layers; python3 -m pytest tests/ -k gitops is green.
data/annia/rabbitmq.yaml declares vhost annia, exchanges annia.events/annia.dlx (topic, durable), quorum+durable queues annia.dispatch (x-delivery-limit: 2, DLX annia.dlx, DLX routing key annia.dead) and annia.dispatch.dead, both bindings, and annia's Permission; a test asserts these arguments.
msgTopologyOperator.enabled: true with pinned nodeSelector/resources, and the test_rabbitmq_knobs_land_where_they_are_read assertion flipped to True.
secrets/vaultauth-annia.yaml + two VaultStaticSecrets exist, the VSO count tripwire is (8, 13), and docs/vault.md §4c-ter carries annia-read and annia in the role loop.
annia-enqueuer Service (port 80 → http) and its Deployment (image pinned, both VSO Secrets wired, resources set) exist in apps/annia/ and are listed in apps/kustomization.yaml.
docs/rabbitmq.md §16e + the OPERATIONS.md §1 row document annia's vhost, user, topology, and the hand-run broker user creation.
The exchange, queue, arguments, routing-key and pointer contract is posted on annia-infra #308.

Testing Plan

python3 -m pytest tests/ -k gitops — render, kubeconform and the new contract test.
kustomize build gitops/clusters/milner/{infrastructure,data,apps} — the three touched layers.
conftest test --policy policy/ on the rendered apps layer — Deployment limits, no :latest.
Post-merge, in-cluster (the checks only prose can carry, from #89):

kubectl -n rabbitmq exec rabbitmq-server-0 -- rabbitmqctl list_queues name type arguments
# -> annia.dispatch  quorum  [{"x-delivery-limit",2},{"x-dead-letter-exchange","annia.dlx"}]

A correctly signed delivery through the Service → 2xx and queue depth +1; the same body with a bad signature → rejected and no change.
Reject a message twice (basic.nack with requeue, or a crashing consumer) → it lands in annia.dispatch.dead with x-death reason: delivery_limit.

#88 — publishes this Service through the Cloudflare Tunnel (http://annia-enqueuer.annia.svc.cluster.local:80/api/hook).
#25 — worker pod, /nix claim, ScaledJob, and Flux image automation for this tag; owns the other apps/annia/ workloads.
#85 — KEDA (consumes annia.dispatch); #87 — Tailscale egress (also creates namespaces).
annia-infra #308 — the pointer contract this topology is the other half of; #310/#311 — the image build pipeline.

Research Findings

Forgejo signs the raw body with HMAC-SHA256 and sends X-Forgejo-Signature (bare hex), X-Hub-Signature-256 (sha256=<hex>), plus X-Gitea/X-Gogs-Signature; there is no timestamp or replay protection. (codeberg.org/forgejo/forgejo services/webhook/shared/payloader.go; forgejo.org/docs/next/user/webhooks)
The chart's useCertManager: false path generates its own 365-day webhook cert Secret via genCA/genSignedCert, so flipping msgTopologyOperator.enabled: true needs no cert-manager. (github.com/CloudPirates-io/helm-charts rabbitmq-cluster-operator)
Topology CRs do have a Ready condition, but when the RabbitmqCluster is absent the operator writes no status, so kstatus reports the data layer Current rather than deadlocking. (rabbitmq/messaging-topology-operator topology_controller.go; kubernetes-sigs/cli-utils pkg/kstatus)
x-delivery-limit is a quorum-queue argument; on RabbitMQ 4.3 it counts genuine failures, and without x-dead-letter-routing-key a dead-lettered message keeps its original routing key — hence the annia.dead key above. (rabbitmq.com/docs/quorum-queues, /docs/dlx)

Pre-Mortem

The cross-repo contract with annia-infra #308 drifts — the image tag pinned in step 7 or the pointer schema (exchange/queue/routing key/{repo, kind, number, event_id}) does not match what the enqueuer actually publishes or what #310 builds, leaving the pod in ImagePullBackOff or the apps layer not-Ready. The plan itself flags that #308 claims the source is Milner's, so the two repos can land different contracts unless step 9's comment is reconciled first.
Enabling msgTopologyOperator with fail-closed webhooks breaks the data layer — the operator's validating webhooks use failurePolicy: Fail, and if its pod/webhook service is not Ready when data/annia/rabbitmq.yaml is applied, every topology CR is rejected and the layer never reconciles. The useCertManager: false self-signed-cert premise is asserted from chart templates rather than verified in-cluster, so a missing cert path would compound this.
The broker-side credential bootstrap is skipped or drifts — Vault's secret/annia/rabbitmq password must be applied out-of-band with rabbitmqctl add_user before the operator's Permission CR and the enqueuer can authenticate, and the plan's claim that no CR can carry a Vault-held password was flagged unverified. If that hand-run step is missed or the password changes, the Permission CR fails and the enqueuer crash-loops on AMQP auth errors that the static tests cannot catch.

Naina: ## Overview Create annia's cluster-side enqueuer: the `annia` namespace, its two Vault credentials, the RabbitMQ Messaging Topology Operator and the topology CRs that are the cross-repo contract with `annia-infra` #308, and the `annia-enqueuer` Service (port 80, `/api/hook`) plus the Deployment that pins the enqueuer image. The enqueuer's application source and image are `annia-infra`'s (#308/#310) — this issue owns the deployment contract and the broker topology. The `annia` namespace is created here; #25 and #87 must not create it again. ## Technical Approach 1. **Add the namespace** — `infrastructure/namespaces/annia.yaml`, and list it in `infrastructure/kustomization.yaml` after `namespaces/rabbitmq.yaml`. ```yaml apiVersion: v1 kind: Namespace metadata: name: annia labels: pod-security.kubernetes.io/enforce: baseline ``` 2. **Enable the Messaging Topology Operator and pin its knobs** — `infrastructure/rabbitmq-operator/values.yaml` (it is `enabled: false` today[5](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/infrastructure/rabbitmq-operator/values.yaml?display=source#L23), and `test_rabbitmq_knobs_land_where_they_are_read` pins that[6](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/tests/test_gitops.py?display=source#L1043)). The chart generates its own webhook certs (`useCertManager: false`)[4](https://github.com/CloudPirates-io/helm-charts/blob/rabbitmq-cluster-operator-0.6.16/charts/rabbitmq-cluster-operator/templates/messaging-topology-operator/validating-webhook-configuration.yaml#L6), so no cert-manager. ```yaml msgTopologyOperator: enabled: true nodeSelector: node-role: worker resources: requests: {cpu: 100m, memory: 128Mi} limits: {cpu: 200m, memory: 256Mi} ``` 3. **Declare the topology** — `data/annia/rabbitmq.yaml` (all CRs `namespace: annia`, `rabbitmqClusterReference: {name: rabbitmq, namespace: rabbitmq}`[10](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/apps/rabbitmq/rabbitmqcluster.yaml?display=source#L4)). The DLX and `x-dead-letter-routing-key` are deliberate: without the latter, dead-lettering reuses the publisher's routing key and nothing catches it[12](https://www.rabbitmq.com/docs/dlx)[13](https://www.rabbitmq.com/docs/quorum-queues). ```yaml - kind: Vhost metadata: {name: annia} spec: {defaultQueueType: quorum, deletionPolicy: retain} - kind: Exchange metadata: {name: annia.events} spec: {vhost: annia, type: topic, durable: true} - kind: Exchange metadata: {name: annia.dlx} spec: {vhost: annia, type: topic, durable: true} - kind: Queue metadata: {name: annia.dispatch} spec: vhost: annia type: quorum durable: true arguments: x-delivery-limit: 2 x-dead-letter-exchange: annia.dlx x-dead-letter-routing-key: annia.dead - kind: Queue metadata: {name: annia.dispatch.dead} spec: {vhost: annia, type: quorum, durable: true} - kind: Binding # annia.events -> annia.dispatch, key "*": one word per repo spec: {vhost: annia, source: annia.events, destination: annia.dispatch, destinationType: queue, routingKey: "*"} - kind: Binding # annia.dlx -> annia.dispatch.dead spec: {vhost: annia, source: annia.dlx, destination: annia.dispatch.dead, destinationType: queue, routingKey: annia.dead} - kind: Permission spec: {vhost: annia, user: annia, permissions: {configure: ".*", write: ".*", read: ".*"}} ``` `Queue.spec.durable` must be written explicitly — the operator defaults it to `false`[11](https://github.com/rabbitmq/messaging-topology-operator/blob/0c9a4d54d92e095aa01595dd83aacda80908292b/internal/queue_settings.go#L35) (the chart installs only a validating webhook, so the upstream mutating defaulter that would set it to `true` never runs). 4. **Wire the data layer** — `data/annia/kustomization.yaml` (`namespace: annia`, resource `rabbitmq.yaml`) and add `- annia` to `data/kustomization.yaml`. No `healthCheckExprs` change: kstatus reads these CRs' own `Ready` condition, and before the broker exists they have none (→ Current)[8](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/data.yaml?display=source#L15)[9](https://github.com/kubernetes-sigs/cli-utils/blob/23cb9a58d34b1779c904b6386ef9982d2d6a7344/pkg/kstatus/README.md#L70), so the expression list stays `["Cluster", "DatabaseRole", "Database"]`. 5. **Add the Vault policy and role** — `docs/vault.md` §4c-ter: add `annia` to the `for ns in …` loop and this policy above it. ```hcl v policy write annia-read - <<'HCL' path "secret/data/annia/*" { capabilities = ["read"] } HCL ``` 6. **Sync the two credentials** — `secrets/vaultauth-annia.yaml` (ns `annia`, role `annia`, SA `default`, `vaultConnectionRef: vault-secrets-operator/vault-connection`)[14](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/secrets/vaultauth-rabbitmq.yaml?display=source#L10) and `secrets/annia-secrets.yaml` (the `kv-v2` / `refreshAfter: 1h` / `create: true` / `overwrite: true` shape matches the existing `VaultStaticSecret`s[15](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/secrets/rabbitmq-credentials.yaml?display=source#L7)). ```yaml - kind: VaultStaticSecret # name: annia-hmac-secret, path: annia/forgejo-webhook destination: {name: annia-hmac-secret} - kind: VaultStaticSecret # name: annia-rabbitmq-credentials, path: annia/rabbitmq destination: {name: annia-rabbitmq-credentials} ``` Both `kv-v2`, `refreshAfter: 1h`, `destination.create: true`, `destination.overwrite: true`. List both files in `secrets/kustomization.yaml`. 7. **Add the Service and Deployment** — `apps/annia/{service.yaml,deployment.yaml,kustomization.yaml}` (`namespace: annia`), and `- annia` in `apps/kustomization.yaml`. TCP probes, so the app needs no `/healthz`. ```yaml kind: Service metadata: {name: annia-enqueuer} spec: ports: [{name: http, port: 80, targetPort: http}] --- kind: Deployment spec: replicas: 1 template: spec: nodeSelector: {node-role: worker} containers: - name: enqueuer image: zot.faun-snapper.ts.net/milner/annia-enqueuer:<annia-infra-pin> ports: [{name: http, containerPort: 8080}] env: - {name: FORGEJO_WEBHOOK_SECRET, valueFrom: {secretKeyRef: {name: annia-hmac-secret, key: hmac-secret}}} - {name: RABBITMQ_USERNAME, valueFrom: {secretKeyRef: {name: annia-rabbitmq-credentials, key: username}}} - {name: RABBITMQ_PASSWORD, valueFrom: {secretKeyRef: {name: annia-rabbitmq-credentials, key: password}}} - {name: RABBITMQ_URL, value: "amqp://$(RABBITMQ_USERNAME):$(RABBITMQ_PASSWORD)@rabbitmq.rabbitmq.svc.cluster.local:5672/annia"} readinessProbe: {tcpSocket: {port: http}} resources: requests: {cpu: 10m, memory: 32Mi} limits: {cpu: 100m, memory: 128Mi} ``` 8. **Move the two tripwires and add the contract test** — `tests/test_gitops.py` (the VSO count tripwire is `(7, 11)` today[7](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/tests/test_gitops.py?display=source#L448), and `7 + 1 = 8`, `11 + 2 = 13`): ```python # test_vault_crs_match_the_vso_crd_shape assert (len(auths), len(statics)) == (8, 13) # test_rabbitmq_knobs_land_where_they_are_read assert values["msgTopologyOperator"]["enabled"] is True ``` Add `test_annia_topology_matches_the_cross_repo_contract` asserting the queue's `type`/`durable`/`arguments`, the DLQ binding key, the `annia.events → annia.dispatch` binding, the `Permission`, and the Service's name/port. 9. **Document and post the contract** — `docs/rabbitmq.md` §16e (vhost, user, topology, DLX inspection, the imperative user creation) and an `OPERATIONS.md` §1 row. Then comment on `annia-infra` #308: ```json {"repo": "tepichord/milner", "kind": "issue", "number": 89, "event_id": "<X-Forgejo-Delivery>"} ``` `annia.events` (topic) → `annia.dispatch` (quorum, `x-delivery-limit: 2`, DLX `annia.dlx` → `annia.dispatch.dead`); publish routing key = repo short name (`milner`), binding key `*`; persistent (`delivery_mode=2`) with publisher confirms, 2xx only after the confirm; HMAC-SHA256 over the raw body, `X-Forgejo-Signature` (bare hex) or `X-Hub-Signature-256: sha256=<hex>`[16](https://codeberg.org/forgejo/forgejo/src/commit/43266cb27cae98db7e4f31569fb31b79ee61bec0/services/webhook/shared/payloader.go#L161). ## Implementation Details Files to create: | Path | What | |---|---| | `gitops/clusters/milner/infrastructure/namespaces/annia.yaml` | Namespace, PSA `baseline` | | `gitops/clusters/milner/data/annia/rabbitmq.yaml` | 2 exchanges, 2 queues, 2 bindings, vhost, permission | | `gitops/clusters/milner/data/annia/kustomization.yaml` | `namespace: annia` | | `gitops/clusters/milner/secrets/vaultauth-annia.yaml` | VaultAuth `default` in `annia` | | `gitops/clusters/milner/secrets/annia-secrets.yaml` | Two VaultStaticSecrets | | `gitops/clusters/milner/apps/annia/{service,deployment,kustomization}.yaml` | Service port 80 → 8080; Deployment | Files to modify: `infrastructure/kustomization.yaml`, `infrastructure/rabbitmq-operator/values.yaml`, `data/kustomization.yaml`, `secrets/kustomization.yaml`, `apps/kustomization.yaml`, `tests/test_gitops.py` (3 places), `docs/vault.md` §4c-ter, `docs/rabbitmq.md` (§16e), `OPERATIONS.md` §1. Vault keys the manifests depend on (hand-run, docs/rabbitmq.md §16e): `secret/annia/forgejo-webhook` → `hmac-secret`; `secret/annia/rabbitmq` → `username`, `password`. Broker user `annia` is created imperatively with that password (`rabbitmqctl add_user`), because no CR can carry a Vault-held password[17](https://github.com/rabbitmq/messaging-topology-operator/blob/8272c2c341538a78e7ccadc4ea2720d9eea77288/api/v1beta1/user_types.go#L40) _unverified_ — the operator's `User` CR does expose `spec.importCredentialsSecret`, which imports `username`/`password` from a Secret, so the premise that no CR can carry a Vault-held password needs re-checking. Constraints: layer ordering is `infrastructure` (operator) → `secrets` → `data` (topology) → `apps` (workload)[1](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/data.yaml?display=source#L51)[2](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/apps.yaml?display=source#L15); the topology operator's webhooks are `failurePolicy: Fail`[3](https://github.com/CloudPirates-io/helm-charts/blob/rabbitmq-cluster-operator-0.6.16/charts/rabbitmq-cluster-operator/templates/messaging-topology-operator/validating-webhook-configuration.yaml#L53), so a broken operator surfaces as a `data`-layer failure; policy/k8s.rego needs cpu+memory limits on the Deployment container. Coordination / out-of-scope notes: - The namespace's single declaration lives here; #25 and #87 must not add `infrastructure/namespaces/annia.yaml`. - The enqueuer source and image are `annia-infra`'s (#308 source, #310/#311 pipeline), per #89. Until #310 publishes the pinned tag the pod is `ImagePullBackOff` and the `apps` layer reports not-Ready — a stated cross-repo dependency, not a misconfiguration. #308's plan comment claims the source is Milner's; confirm which holds before implementing step 7. - Flux image automation (`ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation`) does not exist in this cluster; the tag is a literal pin, as #25 records. Adding image automation is #25's, not this issue's. - The end-to-end edge delivery at `annia.vando.dev/api/hook` is #88. ## Acceptance Criteria - [ ] `kustomize build` succeeds for the `infrastructure`, `data`, and `apps` layers; `python3 -m pytest tests/ -k gitops` is green. - [ ] `data/annia/rabbitmq.yaml` declares vhost `annia`, exchanges `annia.events`/`annia.dlx` (topic, durable), quorum+durable queues `annia.dispatch` (`x-delivery-limit: 2`, DLX `annia.dlx`, DLX routing key `annia.dead`) and `annia.dispatch.dead`, both bindings, and annia's `Permission`; a test asserts these arguments. - [ ] `msgTopologyOperator.enabled: true` with pinned `nodeSelector`/`resources`, and the `test_rabbitmq_knobs_land_where_they_are_read` assertion flipped to `True`. - [ ] `secrets/vaultauth-annia.yaml` + two VaultStaticSecrets exist, the VSO count tripwire is `(8, 13)`, and `docs/vault.md` §4c-ter carries `annia-read` and `annia` in the role loop. - [ ] `annia-enqueuer` Service (port 80 → `http`) and its Deployment (image pinned, both VSO Secrets wired, resources set) exist in `apps/annia/` and are listed in `apps/kustomization.yaml`. - [ ] `docs/rabbitmq.md` §16e + the `OPERATIONS.md` §1 row document annia's vhost, user, topology, and the hand-run broker user creation. - [ ] The exchange, queue, arguments, routing-key and pointer contract is posted on `annia-infra` #308. ## Testing Plan - `python3 -m pytest tests/ -k gitops` — render, kubeconform and the new contract test. - `kustomize build gitops/clusters/milner/{infrastructure,data,apps}` — the three touched layers. - `conftest test --policy policy/` on the rendered `apps` layer — Deployment limits, no `:latest`. - Post-merge, in-cluster (the checks only prose can carry, from #89): ```bash kubectl -n rabbitmq exec rabbitmq-server-0 -- rabbitmqctl list_queues name type arguments # -> annia.dispatch quorum [{"x-delivery-limit",2},{"x-dead-letter-exchange","annia.dlx"}] ``` - A correctly signed delivery through the Service → 2xx and queue depth +1; the same body with a bad signature → rejected and no change. - Reject a message twice (`basic.nack` with requeue, or a crashing consumer) → it lands in `annia.dispatch.dead` with `x-death` `reason: delivery_limit`. ## Related Issues/PRs - #88 — publishes this Service through the Cloudflare Tunnel (`http://annia-enqueuer.annia.svc.cluster.local:80/api/hook`). - #25 — worker pod, `/nix` claim, ScaledJob, and Flux image automation for this tag; owns the other `apps/annia/` workloads. - #85 — KEDA (consumes `annia.dispatch`); #87 — Tailscale egress (also creates namespaces). - `annia-infra` #308 — the pointer contract this topology is the other half of; #310/#311 — the image build pipeline. ## Research Findings - Forgejo signs the raw body with HMAC-SHA256 and sends `X-Forgejo-Signature` (bare hex), `X-Hub-Signature-256` (`sha256=<hex>`), plus `X-Gitea/X-Gogs-Signature`; there is no timestamp or replay protection. (codeberg.org/forgejo/forgejo `services/webhook/shared/payloader.go`; forgejo.org/docs/next/user/webhooks) - The chart's `useCertManager: false` path generates its own 365-day webhook cert Secret via `genCA`/`genSignedCert`, so flipping `msgTopologyOperator.enabled: true` needs no cert-manager. (github.com/CloudPirates-io/helm-charts `rabbitmq-cluster-operator`) - Topology CRs do have a `Ready` condition, but when the `RabbitmqCluster` is absent the operator writes no status, so kstatus reports the `data` layer Current rather than deadlocking. (rabbitmq/messaging-topology-operator `topology_controller.go`; kubernetes-sigs/cli-utils `pkg/kstatus`) - `x-delivery-limit` is a quorum-queue argument; on RabbitMQ 4.3 it counts genuine failures, and without `x-dead-letter-routing-key` a dead-lettered message keeps its original routing key — hence the `annia.dead` key above. (rabbitmq.com/docs/quorum-queues, /docs/dlx) ## Pre-Mortem 1. **The cross-repo contract with `annia-infra` #308 drifts** — the image tag pinned in step 7 or the pointer schema (exchange/queue/routing key/`{repo, kind, number, event_id}`) does not match what the enqueuer actually publishes or what #310 builds, leaving the pod in `ImagePullBackOff` or the `apps` layer not-Ready. The plan itself flags that #308 claims the source is Milner's, so the two repos can land different contracts unless step 9's comment is reconciled first. 2. **Enabling `msgTopologyOperator` with fail-closed webhooks breaks the `data` layer** — the operator's validating webhooks use `failurePolicy: Fail`, and if its pod/webhook service is not Ready when `data/annia/rabbitmq.yaml` is applied, every topology CR is rejected and the layer never reconciles. The `useCertManager: false` self-signed-cert premise is asserted from chart templates rather than verified in-cluster, so a missing cert path would compound this. 3. **The broker-side credential bootstrap is skipped or drifts** — Vault's `secret/annia/rabbitmq` password must be applied out-of-band with `rabbitmqctl add_user` before the operator's `Permission` CR and the enqueuer can authenticate, and the plan's claim that no CR can carry a Vault-held password was flagged unverified. If that hand-run step is missed or the password changes, the `Permission` CR fails and the enqueuer crash-loops on AMQP auth errors that the static tests cannot catch.

tepichord added

and removed

labels

2026-09-22 15:40:59 -07:00

tepichord referenced this issue

2026-09-22 15:48:56 -07:00

Integrate KEDA into milner #85

tepichord added