tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Create Annia pod #61

Closed
tepichord wants to merge 24 commits from feat/25-create-annia-pod into main
pull from: feat/25-create-annia-pod
merge into: tepichord:main
Conversation 0 Commits 24 Files changed 47 +8029 -67
tepichord commented 2026-09-15 10:51:05 -07:00
Owner
Copy link

Closes #25

Automated PR created by forgejo-done adapter.

Closes #25 Automated PR created by forgejo-done adapter.
Add cert-manager v1.21.2 as a new Flux layer before infrastructure.
The RabbitMQ Cluster Operator >= 2.20 requires cert-manager for its
Issuer/Certificate CRs, so cert-manager must be applied first.

Refs: #25
Vendor the official RabbitMQ Cluster Operator manifest (v2.22.5) into
infrastructure/. The RabbitMQ team publishes no Helm chart, so the
operator is installed from the pinned cluster-operator-ghcr-io.yml
manifest with a strategic merge patch adding PSA labels to the
rabbitmq-system Namespace.

Refs: #25
Add KEDA 2.20.2 via HelmRepository + HelmRelease in a labeled keda
Namespace. KEDA installs the keda.sh CRDs and defaults resource limits
on all three components. The ScaledJob CRD is used by the Annia
wake-up serialization.

Refs: #25
Add Namespace resources for rabbitmq and annia with baseline PSA labels.
Add a cluster-scoped ProxyGroup for Tailscale egress (type: egress,
replicas: 1) to route traffic to the LLM gateway off-cluster.

Refs: #25
Add VaultAuth for the annia namespace, and two VaultStaticSecrets:
annia-rabbitmq (RabbitMQ credentials) and annia-forgejo-webhook
(HMAC for Forgejo webhook verification). Both use create/overwrite
true for VSO adoption.

Refs: #25
Add a RabbitmqCluster CR in apps/rabbitmq/ with 1 replica, pinned
resources (100m/128Mi request, 250m/256Mi limit), and 2Gi persistent
storage. The operator generates one admin user whose credentials are
recorded into Vault imperatively.

Refs: #25
Add the Annia workloads: a persistent Deployment (recreate strategy,
/node-role: worker, /nix PVC), an enqueuer Deployment that verifies
Forgejo HMAC and publishes to RabbitMQ, a Service for the enqueuer,
an ExternalName Service for the LLM gateway with Tailscale egress
annotations, and a 2Gi persistent PVC.

Refs: #25
Add a KEDA ScaledJob (maxReplicaCount: 1) that serializes wake-up
jobs triggered by RabbitMQ queue length. Add Flux ImageRepository,
ImagePolicy (semver >=0.0.0), and ImageUpdateAutomation to
automatically bump the annia tag on both the Deployment and ScaledJob.

Refs: #25
Add flux-system-write GitRepository with a secretRef to a hand-
provisioned deploy key. The ImageUpdateAutomation pushes tag bumps
to the flux-updates branch via Forgejo AGit.

Refs: #25
Add a Cloudflare Tunnel route for annia.vando.dev/enqueue pointing
to the Annia enqueuer service. Update plan tests: route count 3->4,
add assertions for the /enqueue route, and update the WAF expression
to include the new path. The path avoids the /hook substring so the
WAF contains clauses stay disjoint.

Refs: #25
Update LAYERS to include cert-manager before infrastructure. Add
cert-manager.io, rabbitmq.com, and keda.sh to CRD_PROVIDERS with
raw-manifest provider detection for RabbitMQ (vendored manifest).
Update VSO tripwire from (3, 3) to (4, 5) for the new Annia secrets.

Refs: #25
Add cert-manager.yaml to root kustomization before infrastructure.
Add dependsOn: [cert-manager] to infrastructure.yaml. Add rabbitmq-
operator, keda, tailscale-egress, and new namespaces to
infrastructure/kustomization. Add rabbitmq and annia to
apps/kustomization. Add Annia Vault secrets to secrets/kustomization.
Add flux-system-write GitRepository to flux-system/kustomization.

Refs: #25
Update the L0 render loop in .woodpecker.yaml to include cert-manager
in the layer list so it is rendered and validated in CI.

Refs: #25
Update AGENTS.md repo map to include cert-manager as the sixth Flux
layer. Add OPERATIONS.md section 10 covering Annia Vault bootstrap
(RabbitMQ creds, Forgejo webhook HMAC), Vault auth role, and Flux
write deploy key for image automation. Renumber subsequent sections.

Refs: #25
Refs: #25
`infrastructure/kustomization.yaml` referenced `namespaces/keda.yaml`,
which was never created: the file landed at `keda/namespace.yaml`
instead. `flux build kustomization infrastructure` therefore failed with

    lstat .../infrastructure/namespaces/keda.yaml: no such file or directory

and the layer could not reconcile at all.

Move it to `namespaces/keda.yaml`, matching where every other
infrastructure namespace lives, and drop it from the KEDA sub-kustomization
so the object is not emitted twice.
The chart's admission-webhooks section is `webhooks`, not
`admissionWebhooks`. Helm ignores unknown values, so the key was a no-op
and the webhook pod had no node-role: worker selector, unlike the operator
and metrics server.
Three problems in one container block:

- `stdin`/`tty` sat on the PodSpec, where they are not valid fields
  (kubeconform -strict rejects them and the API server ignores them).
  They belong on the container.
- No container declared CPU/memory limits, so `policy/k8s.rego` denied
  both Deployments and the L1 conftest gate failed. Every other raw
  workload in the repo (preview-system/reaper.yaml) declares
  requests + limits.
- The ScaledJob's rabbitmq trigger used `hostFromEnv: RABBITMQ_HOST`,
  but the scaler's `host` must be `scheme://host:port/vhost` for
  `protocol: amqp`. RABBITMQ_HOST holds a bare service name, so the
  trigger could never build an AMQP connection. Pin the literal URI and
  document the Service as <cluster>.<namespace>.

Also add the `{"$imagepolicy": "annia:annia:tag"}` setter markers the
`Setters` ImageUpdateAutomation reads; without them the automation has
nothing to rewrite.
Four defects in one object:

- `apiVersion: image.toolkit.fluxcd.io/v1beta1` is not served by the
  v2.9.4 CRD, which exposes only `v1`; the apps layer would be rejected
  for an unknown version.
- `commit.author: fluxbot` is a string where the CRD requires an object
  with an `email`; `commit.message` is not a field either (it is
  `messageTemplate`).
- `sourceRef` defaulted to the `annia` namespace, but the write-capable
  GitRepository lives in `flux-system`. Cross-namespace refs are
  supported and must be stated.
- Without `push.refspec`, the commit lands on a plain `flux-updates`
  branch and no pull request is ever opened. Forgejo AGit needs
  `refs/heads/flux-updates:refs/for/main` plus the push options.
`flux install --export --namespace flux-system` emits only source,
kustomize, helm and notification. The ImageRepository/ImagePolicy/
ImageUpdateAutomation objects in apps/annia were applied against CRDs
that were never installed, so the whole apps layer would be rejected with

    no matches for kind "ImageRepository" in version
    "image.toolkit.fluxcd.io/v1"

and never recover, because the fix is one of the rejected objects.

Regenerate with
`--components-extra=image-reflector-controller,image-automation-controller`
(Flux v2.9.4, the version install.sh names). Every pre-existing document
is byte-identical to the previous export; the delta is the three
image.toolkit CRDs, the two controller Deployments and their
ServiceAccounts.
The Secret holds an SSH deploy key (`identity`), but the URL was the
anonymous `https://` clone the sync source uses. Flux selects the auth
method from the URL scheme: an `identity` key on an HTTPS source is
ignored and the GitRepository stays NotReady with an authentication
error, so ImageUpdateAutomation would have nothing to push to.

Point it at the SSH remote and add the `known_hosts` half of the
credential to the runbook, which SSH sources also require.
Inserting Annia as §10 renumbered the section headers but not their
subsections, leaving two `### 10a.` headings, two `### 11c.`, and
cross-references pointing a section off. Renumber Storage to 11a–11f and
Durable volumes to 12a–12g, and update every §11/§12 reference in
OPERATIONS.md, AGENTS.md, the zot README and the §12d doc guard.

Also correct the Annia runbook: RabbitMQ's Service is
`<cluster>.<namespace>` (`annia.rabbitmq`), not
`annia-annia.annia`, and the write deploy key needs `known_hosts`
alongside `identity` for the SSH GitRepository.
gotk-components.yaml is the one manifest L0 does not render, so a missing
component is invisible to every other gate: kubeconform skips the unknown
group under -ignore-missing-schemas and only the API server rejects the
layer. Assert that every image.toolkit.fluxcd.io CR applied by a layer
has a served CRD version and an installed controller Deployment.
test(chart-sources): send a User-Agent when probing a Helm index
All checks were successful
ci/woodpecker/pr/woodpecker Pipeline was successful
Details
ca15a54677 
charts.jetstack.io sits behind Cloudflare, which answers the default
Python-urllib User-Agent with 403. The module reads 403 as an
authoritative negative, so the new cert-manager HelmRepository failed the
reachability probe even though the index serves the pinned v1.21.2. A
bot filter is not the DENIED/404/missing-tag case this test is about.
tepichord 2026-09-15 10:51:06 -07:00
All checks were successful
ci/woodpecker/pr/woodpecker Pipeline was successful
Required
Details

Pull request closed

This pull request cannot be reopened because the branch was deleted.
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner!61
No description provided.