tepichord/milner

Fork 0

Give the annia enqueuer a source tree and an image #100

New issue

Closed

opened 2026-09-24 17:10:13 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-24 17:10:13 -07:00

Owner

#89 closed with the cluster side of annia's enqueuer — the namespace, the two Vault
credentials, the broker topology, the annia-enqueuer Service — and landed no source:
there is no enqueuer tree, no derivation, no build step. The Deployment it left behind pins
zot.faun-snapper.ts.net/milner/annia-enqueuer:<annia-infra-pin> and is deliberately not
listed in its kustomization, because the registry has no such repository (zot answers
NAME_UNKNOWN). So nothing turns a Forgejo delivery into a message on annia.dispatch, and
what makes that visible is a Deployment nobody applies.

The premise written into the merged tree — that annia-infra builds this image — is not
true. annia-infra #310 ("Create build pipeline") builds annia's runtime/workspace
image (zot.faun-snapper.ts.net/annia:latest, the dockerTools.buildLayeredImage closure
that carries entrypoint.sh, .annia.toml and the skills); it never mentions the enqueuer.
The counterpart issue, annia-infra #308, says explicitly who owns it:

"Producer/enqueuer webhook services remain owned by Milner (data/annia/rabbitmq.yaml),
while Annia manages queue consumption, pointer deduplication, and container packaging."

Out of scope: "Webhook receiver and enqueuer daemon source code (owned and deployed by
Milner under issues #88/#89)."

This issue gives the enqueuer what #89 described and did not build: its source, its image,
and the pin that makes the Deployment runnable.

The contract it must satisfy — already merged, not a proposal

Read out of main. None of this is this issue's to choose.

Service annia-enqueuer, port 80 → targetPort: http; the Deployment declares
container port http = 8080 and a tcpSocket readiness probe on it — so listening on
8080 is the contract and no HTTP health endpoint is required.
Path /api/hook (#89). It is what #88's Cloudflare route publishes and what the
zone's WAF rules already cover.
Env, already wired: FORGEJO_WEBHOOK_SECRET (annia-hmac-secret/hmac-secret),
RABBITMQ_USERNAME and RABBITMQ_PASSWORD (annia-rabbitmq-credentials), and
RABBITMQ_URL="amqp://…@rabbitmq.rabbitmq.svc.cluster.local:5672/annia".
One pointer per delivery — {repo, kind, number, event_id} — which is the contract
with #308. The pointer, not the payload.
Signature: Forgejo sets X-Hub-Signature-256: sha256=<hex> and the bare-hex
X-Forgejo-Signature / X-Gitea-Signature on every delivery (docs/woodpecker-ci.md §9h),
so this is HMAC-SHA256 over the raw body.
The broker names are this repo's, in data/annia/rabbitmq.yaml: vhost annia,
exchange annia.events (topic, durable), queue annia.dispatch (quorum,
x-delivery-limit: 2, DLX annia.dlx). The binding's key is * — one word per repo — so
the enqueuer publishes with routing key milner, which is exactly what #308's consumer
reads (RABBITMQ_QUEUE="annia.dispatch", RABBITMQ_ROUTING_KEY="milner"). #308's own
words: "take the names, the routing key and the pointer shape from that manifest rather
than declaring your own: two declarers of one queue fail with PRECONDITION_FAILED" —
publish, never declare.

Implementation steps

Source tree — enqueuer/, shaped like router/: __init__.py, app.py,
__main__.py, with main() behind __name__ == "__main__" so the image check can import
the module graph without starting a server (routerCheck's trick). It must:
- serve POST /api/hook and nothing else;
- read the raw body once, bounded — the receiver sits on the public edge, so an
 unbounded read is the cheapest way to take it down;
- verify HMAC-SHA256 over those raw bytes against FORGEJO_WEBHOOK_SECRET with
 hmac.compare_digest, before any parse, accepting X-Hub-Signature-256 (sha256=<hex>)
 and the bare-hex variants; no secret, no publish, and 401 + no message on mismatch;
- then parse, and publish one persistent pointer to exchange annia.events with routing
 key milner — {repo, kind, number, event_id}, where kind is X-GitHub-Event and
 event_id is the delivery id the forge sets (it is #308's dedup key, so it has to be the
 same value on a redelivery) — with publisher confirms on;
- reconnect rather than crash when the broker restarts, and declare nothing.
pika from the flake's nixpkgs pin. Verified available as
python313Packages.pika = 1.3.2 in this flake's nixpkgs. router/'s stdlib-only comment
does not transfer — there is no AMQP client in the standard library — so the image carries
it and the app imports it. It must come from the pin, never a pip install in a step (a
fetch and a container builder, which story 20 forbids). Say so where the image family is
described (docs/nix.md §15), so the difference from router/ does not read as an
inconsistency someone later tidies away.
Flake: enqueuerApp, enqueuerCheck, enqueuerImage, beside
routerApp/routerCheck/routerImage (flake.nix ~190-270). Same shape —
enqueuer/ copied under /app; dockerTools.buildLayeredImage { name = "milner/annia-enqueuer"; contents = [ enqueuerApp pkgs.python313 pkgs.python313Packages.pika pkgs.pkgsStatic.busybox ]; config = { Cmd = [ "/bin/python" "-m" "enqueuer" ]; … User = "65535:65535"; ExposedPorts = { "8080/tcp" = {}; }; }; }.
enqueuerCheck is routerCheck's assertions asked of the image's own bytes: /bin/python
resolves inside the layers, /app/enqueuer/*.py are links that resolve, the image's own
config.Cmd is python -m enqueuer, and the module graph imports from the extracted root.
ci/enqueuer-image.sh — the reader of the image: line, symmetric with
ci/router-image.sh / ci/oz-image.sh (including the single-quoted ${…} for
shellcheck SC2016, which is the reason it is a script and not a step's own commands).
Callers: the build step and the guard.
ci/enqueuer-tag-guard.sh — the fourth pair, holding the invariant that the tag moves
if and only if the image's inputs do. It is ci/router-tag-guard.sh's shape with the
compared path set widened by the flake:
git diff --name-only "$REF" HEAD -- enqueuer flake.nix flake.lock. Both directions then
cover everything the image is built from — any of those moved and the tag did not fails
(the main-push build re-pushes a tag the kubelet will not re-pull), and the tag moved with
none of them fails (the path-filtered build never fires). No third condition is needed,
because the pin is in the set. Compared against the base branch's tip, since the
workspace is a shallow clone and git merge-base finds no ancestor — the failure pipeline
#67 hit one step after passing the tag check. The reaper's guard is one-sided only because
its image has no source directory; this one has both inputs, so it keeps the router's
both-directions form.
.woodpecker/gates.yaml: build-enqueuer, shaped like build-reaper (the image no
preview deploys), with the path filter widened to the same set the guard compares —
when: push branch main path include [enqueuer/**, flake.nix, flake.lock] plus manual;
build ./#enqueuerImage and ./#enqueuerCheck, cat the check, skopeo copy the
docker-archive to docker://zot.faun-snapper.ts.net/milner/annia-enqueuer:<pin>, then
verify the registry serves the digest that was pushed (the curl … | sha256sum comparison
every build step ends with). The pull-request half (when: event: pull_request, same
filter) builds and checks and pushes nothing: the enqueuer is one always-on service
pinned to one literal, no preview namespace runs it, and preview/** is the preview
system's namespace — a per-PR tag would have no consumer, so this half exists for build
validation alone. Plus an enqueuer-tag-guard step with when: event: pull_request, as
all three existing guards have. tests/test_static.py holds the step-to-script mapping
(~1035), the executable list (~61) and one test_the_…_tag_guard_runs_on_pull_requests per
guard — each gains a fourth entry.
Manifest: list it and pin it. apps/annia/kustomization.yaml gains deployment.yaml;
the Deployment's image: becomes the literal
zot.faun-snapper.ts.net/milner/annia-enqueuer:0.1.0 (the bootstrap tag; the number is a
counter of the image's revisions, not a compatibility claim — x.y.z is all the reader's
grep asks for). Nothing in zot changes: milner/** already grants annia-ci
create/update and anonymous read/create/update (infrastructure/zot/values.yaml), which
is all the push and the kubelet need.
Retire the placeholder and the premise behind it.
tests/test_gitops.py::test_no_rendered_layer_pins_a_placeholder_image exists because
of this Deployment and says in its docstring that "the image belongs to annia-infra
(#310)". The test stays — a placeholder reaching any layer is still the failure it
describes — but that prose is stale the moment the tag is real, and the
<annia-infra-pin> name goes with it.
The other stale attributions. apps/annia/kustomization.yaml:15-17,
apps/annia/set-credential.sh:21,375 and docs/rabbitmq.md:291 all say annia-infra
builds this image; each becomes the truth. docs/rabbitmq.md:238's "when annia-infra
#310 registers the webhook" is a second error in the same sentence: registration is a
Forgejo setting pointing at annia.vando.dev/api/hook with the Vault-held secret, and it
is ours — and it waits on #88's route, not on annia-infra.
Docs. docs/rabbitmq.md §16e gains the source tree, the build step, the pin, the
pointer/routing-key contract and the "declare nothing" rule; docs/nix.md §15's image
family gains a fourth member, including the one sentence that the image tag is a counter
bumped whenever the image moves, not a compatibility claim; AGENTS.md's repo map describes
three tag-guard pairs in ci/, so the fourth pair and the enqueuer's own row make that
description true again.
Verify. Offline: python3 -m pytest tests/ -q, with the new guard demonstrated to
bite both ways; nix build ./#enqueuerImage ./#enqueuerCheck. In-cluster, no tunnel —
this is #89's own acceptance test: a signed POST from a pod in the namespace to
http://annia-enqueuer.annia.svc.cluster.local/api/hook → 2xx and one message; the
same body with a wrong signature → 401 and no change in queue depth;
kubectl -n rabbitmq exec rabbitmq-server-0 -- rabbitmqctl list_queues name type arguments
shows annia.dispatch as a quorum queue with x-delivery-limit 2.

Done when

A signed delivery to the Service publishes exactly one {repo, kind, number, event_id}
message to annia.dispatch with routing key milner; an unsigned or wrongly signed one is
rejected and publishes nothing — observed in-cluster, not inferred.
nix build ./#enqueuerImage ./#enqueuerCheck passes, and the image carries the app, python
and pika and nothing else (story 20).
A push to main touching enqueuer/** rebuilds the image under the tag the manifest pins,
and apps reaches Ready with the Deployment listed (stories 12 and 17).
A flake.lock bump that changes the image moves the tag too, and that build fires without
any change under enqueuer/**.
The tag guard fails a pull request whose enqueuer/** or flake files moved without a tag
bump, and one whose tag moved with neither — demonstrated, not asserted.
No attribution of this image to annia-infra #310 or #308 survives anywhere in the tree.
The offline gates are green: python3 -m pytest tests/ -q, kustomize build for apps,
conftest test --policy policy/.

Out of scope

The Cloudflare route and the public hostname (#88), and the Forgejo webhook registration
that points at it — the end-to-end path from the internet waits on the tunnel, so these two
do not each wait on the other.
annia's worker pod, its /nix claim, KEDA and the ScaledJob (#25); the console's pause
lever for the crash loop.
Everything #89 landed — namespace, credentials, broker topology, Service. The Deployment
is the only thing it left unfinished.

Decisions (settled)

pika from the pin, not hand-rolled AMQP. There is no AMQP client in the standard
library, so the router's stdlib-only rule does not transfer; pika is pure Python from the
nixpkgs revision flake.lock already pins (1.3.2, verified). The difference from router/
is written down in docs/nix.md §15 rather than left to look like an oversight.
No preview involvement — the pull-request half pushes nothing. The enqueuer is one
always-on service pinned to one literal, and there is nothing about it to preview: no
preview namespace runs it, and preview/** belongs to the preview system. So
build-enqueuer's pull-request half builds and checks the image and stops there; it exists
for build validation alone.
The tag tracks enqueuer/** and the flake pin. A flake.lock bump that changes
python or pika moves the tag without any app change — the reaper's invariant, applied to an
image that also has a source directory. It needs no third guard condition: putting
flake.nix and flake.lock in the compared path set lets the router's two directions
cover it, and the build's path filter is that same set.
0.1.0 to start, and the number is a counter. Any change that moves the image bumps the
last component, the way oz is at 0.1.7. Nothing reads the digits for meaning — the
kubelet, the manifest and the guard do not — and docs/nix.md §15 says so in a sentence,
because under decision 3 a routine pin bump moves this tag too.

`#89` closed with the cluster side of annia's enqueuer — the namespace, the two Vault credentials, the broker topology, the `annia-enqueuer` Service — and landed **no source**: there is no enqueuer tree, no derivation, no build step. The Deployment it left behind pins `zot.faun-snapper.ts.net/milner/annia-enqueuer:<annia-infra-pin>` and is deliberately not listed in its kustomization, because the registry has no such repository (zot answers `NAME_UNKNOWN`). So nothing turns a Forgejo delivery into a message on `annia.dispatch`, and what makes that visible is a Deployment nobody applies. The premise written into the merged tree — that `annia-infra` builds this image — is not true. **annia-infra #310** ("Create build pipeline") builds annia's *runtime/workspace* image (`zot.faun-snapper.ts.net/annia:latest`, the `dockerTools.buildLayeredImage` closure that carries `entrypoint.sh`, `.annia.toml` and the skills); it never mentions the enqueuer. The counterpart issue, **annia-infra #308**, says explicitly who owns it: > "**Producer/enqueuer webhook services remain owned by Milner** (`data/annia/rabbitmq.yaml`), > while Annia manages queue consumption, pointer deduplication, and container packaging." > > Out of scope: "Webhook receiver and enqueuer daemon source code (**owned and deployed by > Milner** under issues #88/#89)." This issue gives the enqueuer what #89 described and did not build: its source, its image, and the pin that makes the Deployment runnable. ## The contract it must satisfy — already merged, not a proposal Read out of `main`. None of this is this issue's to choose. - **Service `annia-enqueuer`**, port 80 → `targetPort: http`; the Deployment declares container port `http` = **8080** and a `tcpSocket` readiness probe on it — so listening on 8080 is the contract and no HTTP health endpoint is required. - **Path `/api/hook`** (#89). It is what `#88`'s Cloudflare route publishes and what the zone's WAF rules already cover. - **Env, already wired**: `FORGEJO_WEBHOOK_SECRET` (`annia-hmac-secret/hmac-secret`), `RABBITMQ_USERNAME` and `RABBITMQ_PASSWORD` (`annia-rabbitmq-credentials`), and `RABBITMQ_URL="amqp://…@rabbitmq.rabbitmq.svc.cluster.local:5672/annia"`. - **One pointer per delivery** — `{repo, kind, number, event_id}` — which is the contract with #308. The pointer, not the payload. - **Signature**: Forgejo sets `X-Hub-Signature-256: sha256=<hex>` and the bare-hex `X-Forgejo-Signature` / `X-Gitea-Signature` on every delivery (`docs/woodpecker-ci.md` §9h), so this is HMAC-SHA256 over the **raw** body. - **The broker names are this repo's**, in `data/annia/rabbitmq.yaml`: vhost `annia`, exchange `annia.events` (topic, durable), queue `annia.dispatch` (quorum, `x-delivery-limit: 2`, DLX `annia.dlx`). The binding's key is `*` — one word per repo — so the enqueuer publishes with routing key **`milner`**, which is exactly what #308's consumer reads (`RABBITMQ_QUEUE="annia.dispatch"`, `RABBITMQ_ROUTING_KEY="milner"`). #308's own words: *"take the names, the routing key and the pointer shape from that manifest rather than declaring your own: two declarers of one queue fail with `PRECONDITION_FAILED`"* — **publish, never declare**. ## Implementation steps 1. **Source tree — `enqueuer/`**, shaped like `router/`: `__init__.py`, `app.py`, `__main__.py`, with `main()` behind `__name__ == "__main__"` so the image check can import the module graph without starting a server (`routerCheck`'s trick). It must: - serve `POST /api/hook` and nothing else; - read the raw body **once, bounded** — the receiver sits on the public edge, so an unbounded read is the cheapest way to take it down; - verify HMAC-SHA256 over those raw bytes against `FORGEJO_WEBHOOK_SECRET` with `hmac.compare_digest`, before any parse, accepting `X-Hub-Signature-256` (`sha256=<hex>`) and the bare-hex variants; **no secret, no publish**, and 401 + no message on mismatch; - then parse, and publish one persistent pointer to exchange `annia.events` with routing key `milner` — `{repo, kind, number, event_id}`, where `kind` is `X-GitHub-Event` and `event_id` is the delivery id the forge sets (it is #308's dedup key, so it has to be the same value on a redelivery) — with publisher confirms on; - reconnect rather than crash when the broker restarts, and declare nothing. 2. **`pika` from the flake's nixpkgs pin.** Verified available as `python313Packages.pika` = 1.3.2 in this flake's nixpkgs. `router/`'s stdlib-only comment does not transfer — there is no AMQP client in the standard library — so the image carries it and the app imports it. It must come from the pin, never a `pip install` in a step (a fetch and a container builder, which story 20 forbids). Say so where the image family is described (`docs/nix.md` §15), so the difference from `router/` does not read as an inconsistency someone later tidies away. 3. **Flake: `enqueuerApp`, `enqueuerCheck`, `enqueuerImage`**, beside `routerApp`/`routerCheck`/`routerImage` (`flake.nix` ~190-270). Same shape — `enqueuer/` copied under `/app`; `dockerTools.buildLayeredImage { name = "milner/annia-enqueuer"; contents = [ enqueuerApp pkgs.python313 pkgs.python313Packages.pika pkgs.pkgsStatic.busybox ]; config = { Cmd = [ "/bin/python" "-m" "enqueuer" ]; … User = "65535:65535"; ExposedPorts = { "8080/tcp" = {}; }; }; }`. `enqueuerCheck` is `routerCheck`'s assertions asked of the image's own bytes: `/bin/python` resolves inside the layers, `/app/enqueuer/*.py` are links that resolve, the image's own `config.Cmd` is `python -m enqueuer`, and the module graph imports from the extracted root. 4. **`ci/enqueuer-image.sh`** — the reader of the `image:` line, symmetric with `ci/router-image.sh` / `ci/oz-image.sh` (including the single-quoted `${…}` for shellcheck SC2016, which is the reason it is a script and not a step's own commands). Callers: the build step and the guard. 5. **`ci/enqueuer-tag-guard.sh`** — the fourth pair, holding the invariant that the tag moves if and only if the image's inputs do. It is `ci/router-tag-guard.sh`'s *shape* with the compared path set widened by the flake: `git diff --name-only "$REF" HEAD -- enqueuer flake.nix flake.lock`. Both directions then cover everything the image is built from — any of those moved and the tag did not fails (the main-push build re-pushes a tag the kubelet will not re-pull), and the tag moved with none of them fails (the path-filtered build never fires). No third condition is needed, because the pin is in the set. Compared against the base branch's **tip**, since the workspace is a shallow clone and `git merge-base` finds no ancestor — the failure pipeline #67 hit one step after passing the tag check. The reaper's guard is one-sided only because its image has no source directory; this one has both inputs, so it keeps the router's both-directions form. 6. **`.woodpecker/gates.yaml`: `build-enqueuer`**, shaped like `build-reaper` (the image no preview deploys), with the path filter widened to the same set the guard compares — `when: push branch main path include [enqueuer/**, flake.nix, flake.lock]` plus `manual`; build `./#enqueuerImage` and `./#enqueuerCheck`, `cat` the check, `skopeo copy` the docker-archive to `docker://zot.faun-snapper.ts.net/milner/annia-enqueuer:<pin>`, then verify the registry serves the digest that was pushed (the `curl … | sha256sum` comparison every build step ends with). The pull-request half (`when: event: pull_request`, same filter) builds and checks and **pushes nothing**: the enqueuer is one always-on service pinned to one literal, no preview namespace runs it, and `preview/**` is the preview system's namespace — a per-PR tag would have no consumer, so this half exists for build validation alone. Plus an `enqueuer-tag-guard` step with `when: event: pull_request`, as all three existing guards have. `tests/test_static.py` holds the step-to-script mapping (~1035), the executable list (~61) and one `test_the_…_tag_guard_runs_on_pull_requests` per guard — each gains a fourth entry. 7. **Manifest: list it and pin it.** `apps/annia/kustomization.yaml` gains `deployment.yaml`; the Deployment's `image:` becomes the literal `zot.faun-snapper.ts.net/milner/annia-enqueuer:0.1.0` (the bootstrap tag; the number is a counter of the image's revisions, not a compatibility claim — `x.y.z` is all the reader's grep asks for). Nothing in zot changes: `milner/**` already grants `annia-ci` create/update and anonymous read/create/update (`infrastructure/zot/values.yaml`), which is all the push and the kubelet need. 8. **Retire the placeholder and the premise behind it.** `tests/test_gitops.py::test_no_rendered_layer_pins_a_placeholder_image` exists *because* of this Deployment and says in its docstring that "the image belongs to annia-infra (#310)". The test stays — a placeholder reaching any layer is still the failure it describes — but that prose is stale the moment the tag is real, and the `<annia-infra-pin>` name goes with it. 9. **The other stale attributions.** `apps/annia/kustomization.yaml:15-17`, `apps/annia/set-credential.sh:21,375` and `docs/rabbitmq.md:291` all say annia-infra builds this image; each becomes the truth. `docs/rabbitmq.md:238`'s "when annia-infra #310 registers the webhook" is a second error in the same sentence: registration is a Forgejo setting pointing at `annia.vando.dev/api/hook` with the Vault-held secret, and it is ours — and it waits on #88's route, not on annia-infra. 10. **Docs.** `docs/rabbitmq.md` §16e gains the source tree, the build step, the pin, the pointer/routing-key contract and the "declare nothing" rule; `docs/nix.md` §15's image family gains a fourth member, including the one sentence that the image tag is a counter bumped whenever the image moves, not a compatibility claim; AGENTS.md's repo map describes three tag-guard pairs in `ci/`, so the fourth pair and the enqueuer's own row make that description true again. 11. **Verify.** Offline: `python3 -m pytest tests/ -q`, with the new guard demonstrated to bite both ways; `nix build ./#enqueuerImage ./#enqueuerCheck`. In-cluster, no tunnel — this is #89's own acceptance test: a signed POST from a pod in the namespace to `http://annia-enqueuer.annia.svc.cluster.local/api/hook` → 2xx and **one** message; the same body with a wrong signature → 401 and no change in queue depth; `kubectl -n rabbitmq exec rabbitmq-server-0 -- rabbitmqctl list_queues name type arguments` shows `annia.dispatch` as a quorum queue with `x-delivery-limit` 2. ## Done when - A signed delivery to the Service publishes exactly one `{repo, kind, number, event_id}` message to `annia.dispatch` with routing key `milner`; an unsigned or wrongly signed one is rejected and publishes nothing — observed in-cluster, not inferred. - `nix build ./#enqueuerImage ./#enqueuerCheck` passes, and the image carries the app, python and pika and nothing else (story 20). - A push to `main` touching `enqueuer/**` rebuilds the image under the tag the manifest pins, and `apps` reaches Ready with the Deployment listed (stories 12 and 17). - A `flake.lock` bump that changes the image moves the tag too, and that build fires without any change under `enqueuer/**`. - The tag guard fails a pull request whose `enqueuer/**` or flake files moved without a tag bump, and one whose tag moved with neither — demonstrated, not asserted. - No attribution of this image to annia-infra #310 or #308 survives anywhere in the tree. - The offline gates are green: `python3 -m pytest tests/ -q`, `kustomize build` for `apps`, `conftest test --policy policy/`. ## Out of scope - The Cloudflare route and the public hostname (`#88`), and the Forgejo webhook registration that points at it — the end-to-end path from the internet waits on the tunnel, so these two do not each wait on the other. - annia's worker pod, its `/nix` claim, KEDA and the ScaledJob (`#25`); the console's pause lever for the crash loop. - Everything `#89` landed — namespace, credentials, broker topology, Service. The Deployment is the only thing it left unfinished. ## Decisions (settled) 1. **`pika` from the pin, not hand-rolled AMQP.** There is no AMQP client in the standard library, so the router's stdlib-only rule does not transfer; pika is pure Python from the nixpkgs revision `flake.lock` already pins (1.3.2, verified). The difference from `router/` is written down in `docs/nix.md` §15 rather than left to look like an oversight. 2. **No preview involvement — the pull-request half pushes nothing.** The enqueuer is one always-on service pinned to one literal, and there is nothing about it to preview: no preview namespace runs it, and `preview/**` belongs to the preview system. So `build-enqueuer`'s pull-request half builds and checks the image and stops there; it exists for build validation alone. 3. **The tag tracks `enqueuer/**` *and* the flake pin.** A `flake.lock` bump that changes python or pika moves the tag without any app change — the reaper's invariant, applied to an image that also has a source directory. It needs no third guard condition: putting `flake.nix` and `flake.lock` in the compared path set lets the router's two directions cover it, and the build's path filter is that same set. 4. **`0.1.0` to start, and the number is a counter.** Any change that moves the image bumps the last component, the way `oz` is at `0.1.7`. Nothing reads the digits for meaning — the kubelet, the manifest and the guard do not — and `docs/nix.md` §15 says so in a sentence, because under decision 3 a routine pin bump moves this tag too.

tepichord added the

labels

2026-09-24 17:10:14 -07:00

tepichord added this to the MVP project

2026-09-24 17:46:37 -07:00

tepichord commented

2026-09-24 18:08:40 -07:00

Author

Owner

Naina:

Overview

Give the annia enqueuer the source tree, Nix image, CI build and tag-guard that
#89 described but did not land, then wire the Deployment into the apps layer
with a real 0.1.0 tag and retire every stale "built by annia-infra #310" claim.
The contract is already merged: listen on 8080, POST /api/hook, HMAC-SHA256
over the raw body, publish one {repo, kind, number, event_id} pointer to
exchange annia.events with routing key milner, and declare nothing.

Technical Approach

Source tree enqueuer/, shaped like router/: __init__.py, app.py, __main__.py¹.

# enqueuer/__main__.py — import-safe, so enqueuerCheck loads the graph without a server
from enqueuer.app import main

if __name__ == "__main__":
    main()

ThreadingHTTPServer serving POST /api/hook and nothing else; every other path → 404².

Read the raw body once, bounded, before any parse.

MAX = 25 * 1024 * 1024          # GitHub caps webhook payloads at 25 MB
n = int(self.headers.get("Content-Length", ""))   # Forgejo always sends it (Go bytes.Reader)
if n <= 0:      return self._deny(411)            # no Content-Length ⇒ unbounded read
if n > MAX:     return self._deny(413)            # reject before reading
body = self.rfile.read(n)                          # exactly the bytes the HMAC covers

GitHub caps webhook payloads at 25 MB³; Go's http.NewRequest sets ContentLength to the exact length for a *bytes.Reader body⁴.

Verify HMAC-SHA256 over body, then publish; constant-time, no secret ⇒ no publish.

digests = []
if (h := self.headers.get("X-Hub-Signature-256", "")).startswith("sha256="):
    digests.append(h.removeprefix("sha256="))
for name in ("X-Forgejo-Signature", "X-Gitea-Signature"):
    if (v := self.headers.get(name)): digests.append(v)   # bare hex, never X-Hub-Signature (sha1=)
ok = secret and any(hmac.compare_digest(hmac.new(secret, body, hashlib.sha256).hexdigest(), d) for d in digests)
if not ok: return self._deny(401)

Forgejo sends X-Hub-Signature-256: sha256=<hex> and the bare-hex X-Forgejo-Signature/X-Gitea-Signature⁵⁶, all HMAC-SHA256 over the raw body⁷; X-Hub-Signature carries sha1= and must never be accepted.

Publish one persistent pointer; declare nothing.

channel.confirm_delivery()
channel.basic_publish(
    exchange="annia.events", routing_key="milner", body=json.dumps(pointer),
    properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent),
)
# pointer = {"repo": payload["repository"]["full_name"], "kind": event,
#            "number": <pull_request|issue>.number or 0 for push, "event_id": delivery_id}
# event  = self.headers["X-GitHub-Event"];  delivery_id = self.headers["X-GitHub-Delivery"]

The exchange, routing key and pointer shape are the already-merged topology⁸ and the cross-repo contract with annia-infra #308⁹, which is still open — its consumer adapter is unwritten.

pika is not thread-safe and does not auto-reconnect. ThreadingHTTPServer calls the connection from many threads; BlockingConnection allows only add_callback_threadsafe off its own thread¹⁰.
```
# one owner thread per connection; handlers enqueue work, the owner publishes
connection.add_callback_threadsafe(lambda: publish(pointer))   # pika's own long_running_publisher pattern
# owner loop: catch AMQPConnectionError, close, backoff-sleep via connection.sleep, rebuild conn+channel
```
Set blocked_connection_timeout on URLParameters so a blocked broker cannot wedge the receiver¹¹; the rebuild loop is pika's own recovery pattern¹².

Flake: enqueuerSrc, enqueuerApp, enqueuerCheck, enqueuerImage beside router* (flake.nix ~184–270)¹³ — the derivations sit at 184–270 while routerSrc is the let binding at line 134¹⁴.

enqueuerSrc = pkgs.lib.fileset.toSource { root = ./.; fileset = pkgs.lib.fileset.unions [ ./enqueuer ]; };
enqueuerApp = pkgs.runCommand "enqueuer-app" { } ''
  mkdir -p $out/app
  cp -a ${enqueuerSrc}/enqueuer $out/app/enqueuer
'';
enqueuerImage = pkgs.dockerTools.buildLayeredImage {
  name = "milner/annia-enqueuer"; tag = "prod";
  contents = [ enqueuerApp (pkgs.python313.withPackages (ps: [ ps.pika ])) pkgs.pkgsStatic.busybox ];
  config = {
    Cmd = [ "/bin/python" "-m" "enqueuer" ];
    Env = [ "PYTHONDONTWRITEBYTECODE=1" "PYTHONUNBUFFERED=1" "PYTHONPATH=/app" "PATH=/bin" ];
    User = "65535:65535"; ExposedPorts = { "8080/tcp" = { }; };
  };
};

python313.withPackages replaces the issue's literal pkgs.python313 pkgs.python313Packages.pika — the literal form lands pika in the image root but off sys.path (research §1)¹⁵. enqueuerCheck is routerCheck's assertions plus a pika-enabled interpreter in nativeBuildInputs, and it must assert the /bin/python3.13 link resolves rather than equals ${pkgs.python313} (the env path differs)¹⁶.

ci/enqueuer-image.sh — sed reader for the image: line, #!/bin/sh + set -eu, single-quoted ${...} with # shellcheck disable=SC2016 (Woodpecker pre-substitutes ${...}). Symmetric with ci/router-image.sh; shared by the build step and the guard.
ci/enqueuer-tag-guard.sh — router-tag-guard.sh's both-directions shape over the widened input set.
```
git diff --name-only "$REF" HEAD -- enqueuer flake.nix flake.lock
# REF = base branch tip (never merge-base — pipeline #67), fetched then fallen back to origin/$BASE
```
Confirmed in the existing guard, which compares against the base tip and documents the pipeline #67 merge-base failure¹⁷.

.woodpecker/gates.yaml: build-enqueuer + enqueuer-tag-guard, shaped like build-reaper.

when:
  - event: pull_request
    path: { include: [enqueuer/**, flake.nix, flake.lock] }   # builds+checks, pushes NOTHING (decision 2)
  - event: push
    branch: main
    path: { include: [enqueuer/**, flake.nix, flake.lock] }
  - event: manual
# main half: nix build ./#enqueuerImage ./#enqueuerCheck → skopeo copy → curl|sha256sum digest verify
# enqueuer-tag-guard: when: event: pull_request, unprivileged, nix develop --command sh ci/enqueuer-tag-guard.sh

List and pin the Deployment.

# gitops/clusters/milner/apps/annia/kustomization.yaml
resources:
  - service.yaml
  - llm-gateway.yaml
  - deployment.yaml
# gitops/clusters/milner/apps/annia/deployment.yaml
image: zot.faun-snapper.ts.net/milner/annia-enqueuer:0.1.0

At HEAD the Deployment still pins the <annia-infra-pin> placeholder¹⁸ and is absent from resources:¹⁹; the port, env names and vhost-annia URL the plan relies on are already declared²⁰.

Extend tests/test_static.py — shellcheck list (~60) gains both scripts; flake_steps (~1035) gains "enqueuer-tag-guard": ("ci/enqueuer-tag-guard.sh",); a fourth test_the_enqueuer_tag_guard_runs_on_pull_requests; path-filter tracked-file checks; an image-closure test parallel to test_the_router_image_carries_its_own_closure. Add both-directions cases to tests/test_ci_guards.py (demonstrated, not asserted).

Retire every stale attribution (grep-verified inventory, all must change):

gitops/clusters/milner/apps/annia/deployment.yaml:20      <annia-infra-pin> → 0.1.0
gitops/clusters/milner/apps/annia/kustomization.yaml:15,17 unlisted/annia-infra → listed, ours
gitops/clusters/milner/apps/annia/set-credential.sh:21,375 annia-infra #310 → #88's route registers the webhook
docs/rabbitmq.md:238,291,295,297                           annia-infra builds it → this repo builds it
tests/test_gitops.py:119-120                               docstring "belongs to annia-infra" → neutral
docs/nix.md §14/§15, AGENTS.md repo map                    three images / three pairs → four

Every line matches HEAD: deployment.yaml:20²¹, kustomization.yaml:15,17²², set-credential.sh:21,375²³, docs/rabbitmq.md:238,291,295,297²⁴, tests/test_gitops.py:119-120²⁵ and the ci/ row of the AGENTS.md repo map²⁶.
Keep docs/rabbitmq.md:139 and set-credential.sh:123 — both correctly name #308 as the consumer contract.

Docs. docs/rabbitmq.md §16e gains the source tree, build step, pin, pointer/routing-key contract and "declare nothing"; docs/nix.md §15 gains the fourth image, the one sentence that pika comes from the flake pin (not pip install), and that the tag is a counter not a compatibility claim.

Implementation Details

File	Change
`enqueuer/{__init__,app,__main__}.py`	new package — receiver, HMAC verify, pointer publish
`flake.nix`	`enqueuerSrc`/`enqueuerApp`/`enqueuerCheck`/`enqueuerImage`
`ci/enqueuer-image.sh`, `ci/enqueuer-tag-guard.sh`	new
`.woodpecker/gates.yaml`	`build-enqueuer`, `enqueuer-tag-guard`
`gitops/clusters/milner/apps/annia/{kustomization,deployment}.yaml`	list + pin `0.1.0`
`gitops/clusters/milner/apps/annia/set-credential.sh`	fix stale lines 21, 375
`docs/rabbitmq.md`, `docs/nix.md`, `AGENTS.md`	fourth image/pair + stale prose
`tests/test_static.py`, `tests/test_ci_guards.py`, `tests/test_gitops.py`	extend + docstring

Dependencies: pkgs.python313Packages.pika = 1.3.2, already in flake.lock's nixpkgs pin²⁷ (nix eval against the pin reports 1.3.2; the package is pkgs/development/python-modules/pika/default.nix²⁸). Nothing to add to flake.lock.
Constraints: port 8080; /api/hook; the Deployment's env names are fixed; RABBITMQ_URL already contains vhost annia. Publish to annia.events, never declare.
Out-of-scope notes:
- #308's planning comment says kind is the singular "issue" and event_id has an evt- prefix²⁹; this issue says kind = X-GitHub-Event and the raw delivery UUID. Implement #100 literally; the consumer's still-unwritten adapter normalises ^unverified (no such adapter exists on annia-infra@main). Flag for reviewer.
- pika's propagatedBuildInputs pulls gevent/tornado/twisted (~75 MiB)³⁰. Default is the honest "no pip install" reading; pika.overridePythonAttrs (o: { propagatedBuildInputs = []; }) is safe for a blocking publisher if "pika and nothing else" is read literally. Flag for reviewer.
- A manual Forgejo replay mints a new delivery UUID³¹, so event_id is stable per attempt, not across redeliveries. Flag for reviewer; do not claim stability the forge does not provide.

Acceptance Criteria

A signed POST /api/hook publishes exactly one {repo, kind, number, event_id} message to annia.dispatch with routing key milner; unsigned or wrong-signature → 401 and no message.
nix build ./#enqueuerImage ./#enqueuerCheck passes; enqueuerCheck reads the image's own layers.
python3 -m pytest tests/ -q, kustomize build for apps, and conftest test --policy policy/ are green.
A push to main touching enqueuer/** or the flake files rebuilds the pinned tag; apps reaches Ready with deployment.yaml listed.
Both tag-guard directions demonstrated in tests/test_ci_guards.py.
No attribution of this image to annia-infra #310 or #308 survives anywhere in the tree.

Testing Plan

tests/test_ci_guards.py: enqueuer/** change without a tag bump → fail; tag bump with no enqueuer/** or flake change → fail; flake.lock bump without a tag bump → fail; shallow-workspace case.
tests/test_static.py: shellcheck covers both new scripts; flake_steps maps enqueuer-tag-guard; a fourth test_the_enqueuer_tag_guard_runs_on_pull_requests; every build-enqueuer path filter matches a tracked file; image-closure test reads contents/Cmd/User/PYTHONPATH.
tests/test_gitops.py: placeholder test stays and passes once deployment.yaml is listed with 0.1.0.
In-cluster (Done-when, #89's own acceptance): a signed POST from a pod to http://annia-enqueuer.annia.svc.cluster.local/api/hook → 2xx and one message; wrong signature → 401, queue depth unchanged; rabbitmqctl list_queues name type arguments shows quorum + x-delivery-limit 2.

#89 — cluster side merged; this lands the source, image and pin it left unfinished.
#88 — Cloudflare route + hostname; out of scope here.
#67 — the shallow-clone merge-base failure the guard shape encodes.
annia-infra #308 — consumer contract: pointer shape, RABBITMQ_QUEUE=annia.dispatch, RABBITMQ_ROUTING_KEY=milner (open).
annia-infra #310 — the wrong owner being retired (open; never built this image).

Research Findings

The issue's literal image contents does not work. [ enqueuerApp pkgs.python313 pkgs.python313Packages.pika … ] leaves import pika failing under the image's bare /bin/python; use (pkgs.python313.withPackages (ps: [ ps.pika ])), verified by building and chroot-running both variants.
pika 1.3.2 is not thread-safe and does not auto-reconnect. BlockingConnection permits only add_callback_threadsafe off its own thread, so a single owner thread or callback hand-off is required; the reconnect loop catches AMQPConnectionError and rebuilds connection + channel.
Forgejo's delivery id is stable per attempt, not across redeliveries. Replay copies the hook task into a new row with a fresh UUID, and Forgejo has no automatic retry.
Bare-hex X-Forgejo-Signature/X-Gitea-Signature is hex HMAC-SHA256 over the raw body with no prefix, per services/webhook/shared/payloader.go; X-Hub-Signature (sha1=) must never be accepted.
Forgejo always sends Content-Length (Go bytes.Reader bodies), so a Content-Length-gated bounded read is the accepted http.server idiom.

Pre-Mortem

The image is built with the issue's literal contents list. pkgs.python313 plus pkgs.python313Packages.pika leaves pika off the interpreter's sys.path, so the pod starts but every signed hook dies with ModuleNotFoundError, and enqueuerCheck's exact /bin/python3.13 link assertion fails. The plan's withPackages fix is easy to skim past because the issue body still reads as the specification.
The publish path shares one pika BlockingConnection across ThreadingHTTPServer handler threads. The connection is not thread-safe, so concurrent deliveries interleave frames, hang, or lose confirms; the plan names single-owner-thread vs add_callback_threadsafe without pinning one. The app then looks correct under a single-request smoke test and only breaks under real webhook bursts.
The Deployment is wired into apps with tag 0.1.0 before the image exists in Zot. Layers use dependsOn/wait: true, so a not-Ready enqueuer holds the whole apps layer, turning one missing image into a cluster-wide stall. The tag-guard and path-filter tripwires only check the source/tag relationship, never that the tag was actually pushed.

Naina: ## Overview Give the annia enqueuer the source tree, Nix image, CI build and tag-guard that #89 described but did not land, then wire the Deployment into the `apps` layer with a real `0.1.0` tag and retire every stale "built by annia-infra #310" claim. The contract is already merged: listen on 8080, `POST /api/hook`, HMAC-SHA256 over the raw body, publish one `{repo, kind, number, event_id}` pointer to exchange `annia.events` with routing key `milner`, and declare nothing. ## Technical Approach 1. **Source tree `enqueuer/`**, shaped like `router/`: `__init__.py`, `app.py`, `__main__.py`[1](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/router/__main__.py?display=source#L1). ```python # enqueuer/__main__.py — import-safe, so enqueuerCheck loads the graph without a server from enqueuer.app import main if __name__ == "__main__": main() ``` `ThreadingHTTPServer` serving `POST /api/hook` and nothing else; every other path → 404[2](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/router/app.py?display=source#L45). 2. **Read the raw body once, bounded, before any parse.** ```python MAX = 25 * 1024 * 1024 # GitHub caps webhook payloads at 25 MB n = int(self.headers.get("Content-Length", "")) # Forgejo always sends it (Go bytes.Reader) if n <= 0: return self._deny(411) # no Content-Length ⇒ unbounded read if n > MAX: return self._deny(413) # reject before reading body = self.rfile.read(n) # exactly the bytes the HMAC covers ``` GitHub caps webhook payloads at 25 MB[3](https://docs.github.com/webhooks/webhook-events-and-payloads); Go's `http.NewRequest` sets `ContentLength` to the exact length for a `*bytes.Reader` body[4](https://pkg.go.dev/net/http#NewRequest). 3. **Verify HMAC-SHA256 over `body`, then publish; constant-time, no secret ⇒ no publish.** ```python digests = [] if (h := self.headers.get("X-Hub-Signature-256", "")).startswith("sha256="): digests.append(h.removeprefix("sha256=")) for name in ("X-Forgejo-Signature", "X-Gitea-Signature"): if (v := self.headers.get(name)): digests.append(v) # bare hex, never X-Hub-Signature (sha1=) ok = secret and any(hmac.compare_digest(hmac.new(secret, body, hashlib.sha256).hexdigest(), d) for d in digests) if not ok: return self._deny(401) ``` Forgejo sends `X-Hub-Signature-256: sha256=<hex>` and the bare-hex `X-Forgejo-Signature`/`X-Gitea-Signature`[5](https://forgejo.org/docs/v15.0/user/webhooks/)[6](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/docs/woodpecker-ci.md?display=source#L440), all HMAC-SHA256 over the raw body[7](https://git.redalder.org/magic_rb/forgejo/src/commit/8b54d38cbe85f8004161a1e40b76f30db3370642/services/webhook/shared/payloader.go); `X-Hub-Signature` carries `sha1=` and must never be accepted. 4. **Publish one persistent pointer; declare nothing.** ```python channel.confirm_delivery() channel.basic_publish( exchange="annia.events", routing_key="milner", body=json.dumps(pointer), properties=pika.BasicProperties(delivery_mode=pika.DeliveryMode.Persistent), ) # pointer = {"repo": payload["repository"]["full_name"], "kind": event, # "number": <pull_request|issue>.number or 0 for push, "event_id": delivery_id} # event = self.headers["X-GitHub-Event"]; delivery_id = self.headers["X-GitHub-Delivery"] ``` The exchange, routing key and pointer shape are the already-merged topology[8](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/gitops/clusters/milner/data/annia/rabbitmq.yaml?display=source#L38) and the cross-repo contract with annia-infra #308[9](https://git.gloom.garden/tepichord/annia-infra/issues/308#issuecomment-9144), which is still open — its consumer adapter is unwritten. 5. **pika is not thread-safe and does not auto-reconnect.** `ThreadingHTTPServer` calls the connection from many threads; `BlockingConnection` allows only `add_callback_threadsafe` off its own thread[10](https://pika.readthedocs.io/en/1.3.2/modules/adapters/blocking.html). ```python # one owner thread per connection; handlers enqueue work, the owner publishes connection.add_callback_threadsafe(lambda: publish(pointer)) # pika's own long_running_publisher pattern # owner loop: catch AMQPConnectionError, close, backoff-sleep via connection.sleep, rebuild conn+channel ``` Set `blocked_connection_timeout` on `URLParameters` so a blocked broker cannot wedge the receiver[11](https://pika.readthedocs.io/en/1.3.2/modules/parameters.html); the rebuild loop is pika's own recovery pattern[12](https://pika.readthedocs.io/en/1.3.2/examples/blocking_consume_recover_multiple_hosts.html). 6. **Flake: `enqueuerSrc`, `enqueuerApp`, `enqueuerCheck`, `enqueuerImage`** beside `router*` (`flake.nix` ~184–270)[13](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/flake.nix?display=source#L184) — the derivations sit at 184–270 while `routerSrc` is the `let` binding at line 134[14](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/flake.nix?display=source#L134). ```nix enqueuerSrc = pkgs.lib.fileset.toSource { root = ./.; fileset = pkgs.lib.fileset.unions [ ./enqueuer ]; }; enqueuerApp = pkgs.runCommand "enqueuer-app" { } '' mkdir -p $out/app cp -a ${enqueuerSrc}/enqueuer $out/app/enqueuer ''; enqueuerImage = pkgs.dockerTools.buildLayeredImage { name = "milner/annia-enqueuer"; tag = "prod"; contents = [ enqueuerApp (pkgs.python313.withPackages (ps: [ ps.pika ])) pkgs.pkgsStatic.busybox ]; config = { Cmd = [ "/bin/python" "-m" "enqueuer" ]; Env = [ "PYTHONDONTWRITEBYTECODE=1" "PYTHONUNBUFFERED=1" "PYTHONPATH=/app" "PATH=/bin" ]; User = "65535:65535"; ExposedPorts = { "8080/tcp" = { }; }; }; }; ``` `python313.withPackages` replaces the issue's literal `pkgs.python313 pkgs.python313Packages.pika` — the literal form lands pika in the image root but off `sys.path` (research §1)[15](https://github.com/NixOS/nixpkgs/blob/e5bdc4a41d4c072fe1e3787eaa0320a384741d44/pkgs/development/python-modules/pika/default.nix). `enqueuerCheck` is `routerCheck`'s assertions plus a pika-enabled interpreter in `nativeBuildInputs`, and it must assert the `/bin/python3.13` link *resolves* rather than equals `${pkgs.python313}` (the env path differs)[16](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/flake.nix?display=source#L213). 7. **`ci/enqueuer-image.sh`** — `sed` reader for the `image:` line, `#!/bin/sh` + `set -eu`, single-quoted `${...}` with `# shellcheck disable=SC2016` (Woodpecker pre-substitutes `${...}`). Symmetric with `ci/router-image.sh`; shared by the build step and the guard. 8. **`ci/enqueuer-tag-guard.sh`** — `router-tag-guard.sh`'s both-directions shape over the widened input set. ```sh git diff --name-only "$REF" HEAD -- enqueuer flake.nix flake.lock # REF = base branch tip (never merge-base — pipeline #67), fetched then fallen back to origin/$BASE ``` Confirmed in the existing guard, which compares against the base tip and documents the pipeline #67 merge-base failure[17](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/ci/router-tag-guard.sh?display=source#L67). 9. **`.woodpecker/gates.yaml`: `build-enqueuer` + `enqueuer-tag-guard`**, shaped like `build-reaper`. ```yaml when: - event: pull_request path: { include: [enqueuer/**, flake.nix, flake.lock] } # builds+checks, pushes NOTHING (decision 2) - event: push branch: main path: { include: [enqueuer/**, flake.nix, flake.lock] } - event: manual # main half: nix build ./#enqueuerImage ./#enqueuerCheck → skopeo copy → curl|sha256sum digest verify # enqueuer-tag-guard: when: event: pull_request, unprivileged, nix develop --command sh ci/enqueuer-tag-guard.sh ``` 10. **List and pin the Deployment.** ```yaml # gitops/clusters/milner/apps/annia/kustomization.yaml resources: - service.yaml - llm-gateway.yaml - deployment.yaml # gitops/clusters/milner/apps/annia/deployment.yaml image: zot.faun-snapper.ts.net/milner/annia-enqueuer:0.1.0 ``` At HEAD the Deployment still pins the `<annia-infra-pin>` placeholder[18](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/gitops/clusters/milner/apps/annia/deployment.yaml?display=source#L20) and is absent from `resources:`[19](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/gitops/clusters/milner/apps/annia/kustomization.yaml?display=source#L15); the port, env names and vhost-`annia` URL the plan relies on are already declared[20](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/gitops/clusters/milner/apps/annia/deployment.yaml?display=source#L41). 11. **Extend `tests/test_static.py`** — shellcheck list (~60) gains both scripts; `flake_steps` (~1035) gains `"enqueuer-tag-guard": ("ci/enqueuer-tag-guard.sh",)`; a fourth `test_the_enqueuer_tag_guard_runs_on_pull_requests`; path-filter tracked-file checks; an image-closure test parallel to `test_the_router_image_carries_its_own_closure`. Add both-directions cases to `tests/test_ci_guards.py` (demonstrated, not asserted). 12. **Retire every stale attribution** (grep-verified inventory, all must change): ``` gitops/clusters/milner/apps/annia/deployment.yaml:20 <annia-infra-pin> → 0.1.0 gitops/clusters/milner/apps/annia/kustomization.yaml:15,17 unlisted/annia-infra → listed, ours gitops/clusters/milner/apps/annia/set-credential.sh:21,375 annia-infra #310 → #88's route registers the webhook docs/rabbitmq.md:238,291,295,297 annia-infra builds it → this repo builds it tests/test_gitops.py:119-120 docstring "belongs to annia-infra" → neutral docs/nix.md §14/§15, AGENTS.md repo map three images / three pairs → four ``` Every line matches HEAD: `deployment.yaml:20`[21](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/gitops/clusters/milner/apps/annia/deployment.yaml?display=source#L20), `kustomization.yaml:15,17`[22](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/gitops/clusters/milner/apps/annia/kustomization.yaml?display=source#L15), `set-credential.sh:21,375`[23](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/gitops/clusters/milner/apps/annia/set-credential.sh?display=source#L21), `docs/rabbitmq.md:238,291,295,297`[24](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/docs/rabbitmq.md?display=source#L291), `tests/test_gitops.py:119-120`[25](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/tests/test_gitops.py?display=source#L119) and the `ci/` row of the AGENTS.md repo map[26](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/AGENTS.md?display=source#L44). Keep `docs/rabbitmq.md:139` and `set-credential.sh:123` — both correctly name #308 as the *consumer* contract. 13. **Docs.** `docs/rabbitmq.md` §16e gains the source tree, build step, pin, pointer/routing-key contract and "declare nothing"; `docs/nix.md` §15 gains the fourth image, the one sentence that pika comes from the flake pin (not `pip install`), and that the tag is a counter not a compatibility claim. ## Implementation Details | File | Change | |---|---| | `enqueuer/{__init__,app,__main__}.py` | new package — receiver, HMAC verify, pointer publish | | `flake.nix` | `enqueuerSrc`/`enqueuerApp`/`enqueuerCheck`/`enqueuerImage` | | `ci/enqueuer-image.sh`, `ci/enqueuer-tag-guard.sh` | new | | `.woodpecker/gates.yaml` | `build-enqueuer`, `enqueuer-tag-guard` | | `gitops/clusters/milner/apps/annia/{kustomization,deployment}.yaml` | list + pin `0.1.0` | | `gitops/clusters/milner/apps/annia/set-credential.sh` | fix stale lines 21, 375 | | `docs/rabbitmq.md`, `docs/nix.md`, `AGENTS.md` | fourth image/pair + stale prose | | `tests/test_static.py`, `tests/test_ci_guards.py`, `tests/test_gitops.py` | extend + docstring | - **Dependencies**: `pkgs.python313Packages.pika` = 1.3.2, already in `flake.lock`'s nixpkgs pin[27](https://git.gloom.garden/tepichord/milner/src/commit/bc226a127a4d4b44cf4e0acd026aee7b45dab801/flake.lock?display=source#L27) (`nix eval` against the pin reports `1.3.2`; the package is `pkgs/development/python-modules/pika/default.nix`[28](https://github.com/NixOS/nixpkgs/blob/e5bdc4a41d4c072fe1e3787eaa0320a384741d44/pkgs/development/python-modules/pika/default.nix)). Nothing to add to `flake.lock`. - **Constraints**: port 8080; `/api/hook`; the Deployment's env names are fixed; `RABBITMQ_URL` already contains vhost `annia`. Publish to `annia.events`, never declare. - **Out-of-scope notes**: - `#308`'s planning comment says `kind` is the singular `"issue"` and `event_id` has an `evt-` prefix[29](https://git.gloom.garden/tepichord/annia-infra/issues/308#issuecomment-9144); this issue says `kind = X-GitHub-Event` and the raw delivery UUID. Implement #100 literally; the consumer's still-unwritten adapter normalises _unverified_ (no such adapter exists on `annia-infra@main`). **Flag for reviewer.** - pika's `propagatedBuildInputs` pulls gevent/tornado/twisted (~75 MiB)[30](https://github.com/NixOS/nixpkgs/blob/e5bdc4a41d4c072fe1e3787eaa0320a384741d44/pkgs/development/python-modules/pika/default.nix). Default is the honest "no `pip install`" reading; `pika.overridePythonAttrs (o: { propagatedBuildInputs = []; })` is safe for a blocking publisher if "pika and nothing else" is read literally. **Flag for reviewer.** - A manual Forgejo replay mints a **new** delivery UUID[31](https://git.redalder.org/magic_rb/forgejo/src/commit/d934e0c9fbe0ddfed806e415d912933fd92500ca/models/webhook/hooktask.go), so `event_id` is stable per attempt, not across redeliveries. **Flag for reviewer; do not claim stability the forge does not provide.** ## Acceptance Criteria - [ ] A signed `POST /api/hook` publishes exactly one `{repo, kind, number, event_id}` message to `annia.dispatch` with routing key `milner`; unsigned or wrong-signature → 401 and no message. - [ ] `nix build ./#enqueuerImage ./#enqueuerCheck` passes; `enqueuerCheck` reads the image's own layers. - [ ] `python3 -m pytest tests/ -q`, `kustomize build` for `apps`, and `conftest test --policy policy/` are green. - [ ] A push to `main` touching `enqueuer/**` or the flake files rebuilds the pinned tag; `apps` reaches Ready with `deployment.yaml` listed. - [ ] Both tag-guard directions demonstrated in `tests/test_ci_guards.py`. - [ ] No attribution of this image to annia-infra #310 or #308 survives anywhere in the tree. ## Testing Plan - `tests/test_ci_guards.py`: `enqueuer/**` change without a tag bump → fail; tag bump with no `enqueuer/**` or flake change → fail; `flake.lock` bump without a tag bump → fail; shallow-workspace case. - `tests/test_static.py`: shellcheck covers both new scripts; `flake_steps` maps `enqueuer-tag-guard`; a fourth `test_the_enqueuer_tag_guard_runs_on_pull_requests`; every `build-enqueuer` path filter matches a tracked file; image-closure test reads `contents`/`Cmd`/`User`/`PYTHONPATH`. - `tests/test_gitops.py`: placeholder test stays and passes once `deployment.yaml` is listed with `0.1.0`. - In-cluster (Done-when, #89's own acceptance): a signed POST from a pod to `http://annia-enqueuer.annia.svc.cluster.local/api/hook` → 2xx and one message; wrong signature → 401, queue depth unchanged; `rabbitmqctl list_queues name type arguments` shows quorum + `x-delivery-limit 2`. ## Related Issues - #89 — cluster side merged; this lands the source, image and pin it left unfinished. - #88 — Cloudflare route + hostname; out of scope here. - #67 — the shallow-clone `merge-base` failure the guard shape encodes. - annia-infra #308 — consumer contract: pointer shape, `RABBITMQ_QUEUE=annia.dispatch`, `RABBITMQ_ROUTING_KEY=milner` (open). - annia-infra #310 — the wrong owner being retired (open; never built this image). ## Research Findings - **The issue's literal image `contents` does not work.** `[ enqueuerApp pkgs.python313 pkgs.python313Packages.pika … ]` leaves `import pika` failing under the image's bare `/bin/python`; use `(pkgs.python313.withPackages (ps: [ ps.pika ]))`, verified by building and chroot-running both variants. - **pika 1.3.2 is not thread-safe and does not auto-reconnect.** `BlockingConnection` permits only `add_callback_threadsafe` off its own thread, so a single owner thread or callback hand-off is required; the reconnect loop catches `AMQPConnectionError` and rebuilds connection + channel. - **Forgejo's delivery id is stable per attempt, not across redeliveries.** Replay copies the hook task into a new row with a fresh `UUID`, and Forgejo has no automatic retry. - **Bare-hex `X-Forgejo-Signature`/`X-Gitea-Signature` is hex HMAC-SHA256 over the raw body with no prefix**, per `services/webhook/shared/payloader.go`; `X-Hub-Signature` (`sha1=`) must never be accepted. - **Forgejo always sends `Content-Length`** (Go `bytes.Reader` bodies), so a `Content-Length`-gated bounded read is the accepted `http.server` idiom. ## Pre-Mortem 1. **The image is built with the issue's literal `contents` list.** `pkgs.python313` plus `pkgs.python313Packages.pika` leaves pika off the interpreter's `sys.path`, so the pod starts but every signed hook dies with `ModuleNotFoundError`, and `enqueuerCheck`'s exact `/bin/python3.13` link assertion fails. The plan's `withPackages` fix is easy to skim past because the issue body still reads as the specification. 2. **The publish path shares one pika `BlockingConnection` across `ThreadingHTTPServer` handler threads.** The connection is not thread-safe, so concurrent deliveries interleave frames, hang, or lose confirms; the plan names single-owner-thread vs `add_callback_threadsafe` without pinning one. The app then looks correct under a single-request smoke test and only breaks under real webhook bursts. 3. **The Deployment is wired into `apps` with tag `0.1.0` before the image exists in Zot.** Layers use `dependsOn`/`wait: true`, so a not-Ready enqueuer holds the whole `apps` layer, turning one missing image into a cluster-wide stall. The tag-guard and path-filter tripwires only check the source/tag relationship, never that the tag was actually pushed.

tepichord added

and removed

labels

2026-09-24 18:15:03 -07:00

tepichord referenced this issue from a commit

2026-09-24 21:01:09 -07:00

feat(enqueuer): verify a webhook and publish one pointer per delivery

tepichord referenced this issue from a commit