tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Create Annia pod #25

New issue
Open
opened 2026-08-20 21:08:38 -07:00 by tepichord · 5 comments
tepichord commented 2026-08-20 21:08:38 -07:00
Owner
Copy link
  1. Add cert-manager under gitops/clusters/milner/infrastructure/, because RabbitMQ Cluster Operator ≥ 2.20 requires it and milner has none today.
  2. Add the RabbitMQ Cluster Operator plus a rabbitmq Namespace carrying a PSA enforce label, in that same infrastructure layer.
  3. Add the RabbitMQ Cluster CR in apps/ — an earlier layer than its controller would violate the CRD-ordering rule — with pinned resources and a dedicated user.
  4. Add KEDA to infrastructure/ and add its CRD row to CRD_PROVIDERS in tests/test_gitops.py, or the existing tripwire won't know about it.
  5. Write the RabbitMQ credentials and the Forgejo webhook HMAC into Vault, then add the matching VaultStaticSecret CRs to the secrets layer with create/overwrite both true.
  6. Add the enqueuer Deployment and Service in the apps layer: it verifies the Forgejo HMAC and publishes the pointer message.
  7. Publish the enqueuer's route in cloudflare/main.tf's webhook_routes on its own hostname, with a path that does not nest inside /hook/ or /api/hook.
  8. Add a Tailscale egress ProxyGroup and an ExternalName Service annotated tailscale.com/tailnet-fqdn for the LLM gateway, which entrypoint.sh currently reaches off-cluster.
  9. Add apps/annia/ with a Namespace, a replicas: 1 / strategy: Recreate Deployment on node-role: worker using the VSO secrets and a /nix PVC, and list it in
    apps/kustomization.yaml.
  10. Add a KEDA ScaledJob with maxReplicaCount: 1 so wake-ups are serialized, plus Flux ImageRepository/ImagePolicy/ImageUpdateAutomation to bump the tag on a schedule.
1. Add cert-manager under gitops/clusters/milner/infrastructure/, because RabbitMQ Cluster Operator ≥ 2.20 requires it and milner has none today. 2. Add the RabbitMQ Cluster Operator plus a rabbitmq Namespace carrying a PSA enforce label, in that same infrastructure layer. 3. Add the RabbitMQ Cluster CR in apps/ — an earlier layer than its controller would violate the CRD-ordering rule — with pinned resources and a dedicated user. 4. Add KEDA to infrastructure/ and add its CRD row to CRD_PROVIDERS in tests/test_gitops.py, or the existing tripwire won't know about it. 5. Write the RabbitMQ credentials and the Forgejo webhook HMAC into Vault, then add the matching VaultStaticSecret CRs to the secrets layer with create/overwrite both true. 6. Add the enqueuer Deployment and Service in the apps layer: it verifies the Forgejo HMAC and publishes the pointer message. 7. Publish the enqueuer's route in cloudflare/main.tf's webhook_routes on its own hostname, with a path that does not nest inside /hook/ or /api/hook. 8. Add a Tailscale egress ProxyGroup and an ExternalName Service annotated tailscale.com/tailnet-fqdn for the LLM gateway, which entrypoint.sh currently reaches off-cluster. 9. Add apps/annia/ with a Namespace, a replicas: 1 / strategy: Recreate Deployment on node-role: worker using the VSO secrets and a /nix PVC, and list it in apps/kustomization.yaml. 10. Add a KEDA ScaledJob with maxReplicaCount: 1 so wake-ups are serialized, plus Flux ImageRepository/ImagePolicy/ImageUpdateAutomation to bump the tag on a schedule.
tepichord added this to the MVP project 2026-08-20 21:08:38 -07:00
tepichord commented 2026-09-11 20:17:34 -07:00
Author
Owner
Copy link

Naina:
Updated on 2026-09-30 based on maintainer feedback.

Overview

Run Annia on milner: the persistent Deployment, the serialized KEDA ScaledJob watching annia.dispatch, the three Vault reads it needs (clone key, repo env, Forgejo token), the shared /nix and /root/.pi mounts, Flux image automation, and the tripwires and docs that move with them. Body items 1–4, 6, 7 and 8 have landed (RabbitMQ, KEDA 2.20.2, the annia.dispatch topology, the enqueuer, its Cloudflare route, and the LLM egress Service), so this plan is items 5, 9 and 10 only. Every live criterion is blocked by annia-infra#310 (no milner/annia image is published today); /root/.pi additionally depends on milner#21, and the annia-nix claim's shape on the layered-store issue.

Technical Approach

  1. Add the clone-key read — one VaultStaticSecret for ci/milner, plus the named policy path. Note the nesting: transformation goes under spec.destination, and excludeRaw: true is required so VSO's synthetic _raw does not land in the Secret1.
# gitops/clusters/milner/secrets/annia-secrets.yaml (append)
kind: VaultStaticSecret
metadata: { name: annia-forgejo-key, namespace: annia }
spec:
  type: kv-v2
  mount: secret
  path: ci/milner
  refreshAfter: 1h
  vaultAuthRef: default
  destination:
    name: annia-forgejo-key
    create: true
    overwrite: true
    transformation:
      excludeRaw: true
      excludes: [".*"]
      templates:
        id_ed25519: { text: '{{ get .Secrets "SSH_KEY" }}' }

# apps/annia/set-credential.sh — POLICY_HCL gains two named paths
path "secret/data/ci/milner"  { capabilities = ["read"] }
path "secret/data/ci/forgejo" { capabilities = ["read"] }

# apps/annia/set-credential.sh — constants the producer tripwire scans for
CLONE_KEY_PATH=secret/ci/milner
REPO_ENV_PATH=secret/annia/env      # written by the OPERATIONS.md runbook
FORGEJO_TOKEN_PATH=secret/ci/forgejo
  1. Add the two remaining Vault reads — annia/env (one field per environments/*.env file) and ci/forgejo; same kv-v2/vaultAuthRef: default shape as above, no transformation.
# annia-secrets.yaml (append)
kind: VaultStaticSecret
metadata: { name: annia-repo-env, namespace: annia }
spec: { path: annia/env, ... destination: { name: annia-repo-env } }
---
kind: VaultStaticSecret
metadata: { name: annia-forgejo-token, namespace: annia }
spec: { path: ci/forgejo, ... destination: { name: annia-forgejo-token } }

annia-read already covers annia/env; the two ci/ paths are the new grants.

  1. Two claims — /nix is the shared store (story 19), /root/.pi is the Syncthing-synced folder (milner#21). The /workspace claim is deleted: the container layer is the workspace.
# apps/annia/annia-nix-pvc.yaml   (provisional — the layered-store issue re-derives it)
spec: { accessModes: [ReadWriteOnce], storageClassName: nix, resources: { requests: { storage: 200Gi } } }
# apps/annia/annia-pi-pvc.yaml    (whole volume, no subPath; Syncthing mounts the same claim)
spec: { accessModes: [ReadWriteOnce], storageClassName: persistent, resources: { requests: { storage: 20Gi } } }

RWO means the annia pod and the Syncthing pod must be in one namespace and on one node; nodeSelector: node-role: worker plus the single worker makes that free today.

  1. The Deployment — new file, because deployment.yaml is the enqueuer's and ci/enqueuer-tag-guard.sh pins its image line2. args: only: command: would replace the image's ENTRYPOINT3, so the clone, key wiring and git config never run4.
# apps/annia/annia-deployment.yaml
spec:
  replicas: 1
  strategy: { type: Recreate }
  template:
    spec:
      nodeSelector: { node-role: worker }
      containers:
        - name: annia
          image: zot.faun-snapper.ts.net/milner/annia:0.1.0 # {"$imagepolicy": "annia:annia"}
          args: ["start"]            # persistent loop; the Job below runs the one-shot "run"
          env:
            - { name: FORGEJO_URL, value: "https://git.gloom.garden" }
            - { name: FORGEJO_CLONE_URL, value: "clone.gloom.garden" }
            - { name: FORGEJO_PORT, value: "22" }
            - name: FORGEJO_TOKEN
              valueFrom: { secretKeyRef: { name: annia-forgejo-token, key: FORGEJO_TOKEN } }
          resources: { requests: { cpu: 500m, memory: 1Gi }, limits: { cpu: "2", memory: 4Gi } }
          volumeMounts:
            - { name: nix,        mountPath: /nix }
            - { name: pi,         mountPath: /root/.pi }   # whole volume, no subPath
            - { name: repo-env,   mountPath: /workspace/environments, readOnly: true }
            - { name: forgejo-key, mountPath: /root/.ssh/id_ed25519, subPath: id_ed25519, readOnly: true }
      volumes:
        - { name: nix,  persistentVolumeClaim: { claimName: annia-nix } }
        - { name: pi,   persistentVolumeClaim: { claimName: annia-pi } }
        - { name: repo-env, secret: { secretName: annia-repo-env } }
        - { name: forgejo-key, secret: { secretName: annia-forgejo-key, defaultMode: 0400 } }

securityContext stays unset: annia is baseline and the image is root. There is no /workspace volume; /workspace/environments is the only thing mounted under it.

  1. The ScaledJob — one Job per queued pointer; mounts the shared store and the key, and carries the same tag marker so image automation bumps both.
# apps/annia/scaledjob.yaml
apiVersion: keda.sh/v1alpha1
kind: ScaledJob
metadata:
  annotations:
    kustomize.toolkit.fluxcd.io/ssa: Merge   # keep a hand-set pause annotation across a reconcile
spec:
  maxReplicaCount: 1
  pollingInterval: 15
  successfulJobsHistoryLimit: 3
  failedJobsHistoryLimit: 3
  jobTargetRef:
    backoffLimit: 0            # one attempt per Job; the queue's x-delivery-limit is the crash-loop bound
    template:
      spec:
        restartPolicy: Never
        nodeSelector: { node-role: worker }
        containers:
          - name: annia
            image: zot.faun-snapper.ts.net/milner/annia:0.1.0 # {"$imagepolicy": "annia:annia"}
            args: ["run"]      # single sync -> schedule -> dispatch cycle
            env:
              - { name: RABBITMQ_HOST, value: "amqp://rabbitmq.rabbitmq.svc.cluster.local:5672/annia" }
              - { name: RABBITMQ_USERNAME, valueFrom: { secretKeyRef: { name: annia-rabbitmq-credentials, key: username } } }
              - { name: RABBITMQ_PASSWORD, valueFrom: { secretKeyRef: { name: annia-rabbitmq-credentials, key: password } } }
            resources: { requests: { cpu: 500m, memory: 1Gi }, limits: { cpu: "2", memory: 4Gi } }
            volumeMounts:
              - { name: nix, mountPath: /nix }
              - { name: forgejo-key, mountPath: /root/.ssh/id_ed25519, subPath: id_ed25519, readOnly: true }
        volumes:
          - { name: nix, persistentVolumeClaim: { claimName: annia-nix } }
          - { name: forgejo-key, secret: { secretName: annia-forgejo-key, defaultMode: 0400 } }
  triggers:
    - type: rabbitmq
      metadata:
        protocol: amqp
        mode: QueueLength
        value: "1"
        queueName: annia.dispatch      # never the obsolete annia-wakeups
        hostFromEnv: RABBITMQ_HOST     # the scaler reads the Job template's env
        usernameFromEnv: RABBITMQ_USERNAME
        passwordFromEnv: RABBITMQ_PASSWORD

hostFromEnv resolves from the Job container's env, so there is no TriggerAuthentication and no Secret in keda5. The URI's /annia supplies the vhost, so vhostName is omitted.

  1. The pause lever's RBAC — a Role/RoleBinding over keda.sh/scaledjobs for the default ServiceAccount, which is what the landed VaultAuth binds6.
# apps/annia/rbac.yaml
kind: Role          # rules: [{ apiGroups: [keda.sh], resources: [scaledjobs], verbs: [get, patch] }]
kind: RoleBinding   # -> ServiceAccount annia/default
  1. Move the tripwires — the ScaledJob's containers live at a path neither the policy nor the test walker knows today78910.
CRD_PROVIDERS = CRD_PROVIDERS + [("keda.sh", "keda", "KEDA")]
assert len(paths) == 17                               # 14 + clone key + repo env + forgejo token
assert (len(auths), len(statics)) == (9, 17)          # both count lines move together

# tests/test_gitops.py — extend _pod_specs() so the placeholder/image walks see the Job template
if kind == "ScaledJob":
    return [("spec.jobTargetRef.template.spec", doc["spec"]["jobTargetRef"]["template"]["spec"])]

# policy/k8s.rego — L1 is blind to a ScaledJob's containers today
_containers contains container if {
	input.kind == "ScaledJob"
	container := input.spec.jobTargetRef.template.spec.containers[_]
}
  1. Image automation — regenerate the vendored install to add the two missing controllers, then the three image objects and the write source.
flux install --export --version=v2.9.4 --namespace flux-system \
  --components-extra=image-reflector-controller,image-automation-controller \
  > gitops/clusters/milner/flux-system/gotk-components.yaml

# apps/annia/image-automation.yaml  (image.toolkit.fluxcd.io/v1)
kind: ImageRepository          # image: zot.faun-snapper.ts.net/milner/annia, interval: 10m — zot reads are anonymous
kind: ImagePolicy              # policy: { semver: { range: ">=0.0.0" } }  # skips :sha-* / :latest
kind: ImageUpdateAutomation
spec:
  sourceRef: { kind: GitRepository, name: flux-system-write }   # second source; the Receiver-backed flux-system is untouched
  update: { path: ./gitops/clusters/milner/apps/annia }
  git:
    checkout: { ref: { branch: main } }
    push:
      branch: flux-updates
      refspec: refs/heads/flux-updates:refs/for/main   # the local half equals push.branch
      options: { topic: flux-updates, force-push: "true" }   # force-push is what updates the existing PR

# apps/annia/gitrepository-write.yaml — the credential Secret is a hand-run bootstrap step (OPERATIONS.md)
kind: GitRepository
metadata: { name: flux-system-write, namespace: annia }
spec: { url: ssh://git@clone.gloom.garden/tepichord/milner.git, secretRef: { name: flux-write-deploy-key } }

The flux-write-deploy-key must be write-enabled; Forgejo reserves force-push/rewrite history for Write and above11 12, and the "second bump updates the same PR" criterion depends on it.

  1. Docs and stories, in the same change (AGENTS.md Hard Rule). docs/nix.md §14b/§14c/§14d (add a builder-pod row; reframe "one writer, ever" as the GC rule and record the append-only interim); docs/vault.md §4c-ter (the two ci/ policy paths); OPERATIONS.md (clone-key/annia/env runbook, flux-write-deploy-key, image automation, pause runbook); AGENTS.md (repo map row, and the /root/.pi Vault-first standing exception); USER_STORIES.md story 20 scoped to apps whose source is fixed at build time.

Implementation Details

Path Purpose
gitops/clusters/milner/apps/annia/annia-deployment.yaml the pod; the enqueuer's deployment.yaml is untouched
gitops/clusters/milner/apps/annia/annia-nix-pvc.yaml shared nix-class claim, provisional
gitops/clusters/milner/apps/annia/annia-pi-pvc.yaml /root/.pi, whole volume, shared with Syncthing
gitops/clusters/milner/apps/annia/scaledjob.yaml serialized Job, ssa: Merge13, queueName: annia.dispatch
gitops/clusters/milner/apps/annia/rbac.yaml Role/RoleBinding over keda.sh/scaledjobs for SA default
gitops/clusters/milner/apps/annia/{image-automation,gitrepository-write}.yaml image objects + write source
gitops/clusters/milner/apps/annia/kustomization.yaml list every new resource
gitops/clusters/milner/secrets/annia-secrets.yaml append the three VaultStaticSecrets (file already listed)
gitops/clusters/milner/apps/annia/set-credential.sh widen POLICY_HCL; add the three <NAME>_PATH= constants
gitops/clusters/milner/flux-system/gotk-components.yaml regenerated with the two image controllers
tests/test_gitops.py, policy/k8s.rego the tripwires and the _containers arm

Constraints: the ScaledJob is a keda.sh CR, so it stays in apps (a layer later than the infrastructure HelmRelease); the KEDA row lands now or test_crd_providers_are_not_in_the_layer_that_uses_their_crds fails with "nothing applies keda.sh resources"1415; RABBITMQ_HOST must carry an amqp:// scheme and the /annia vhost; transformation nests under spec.destination; the SSH key mount needs defaultMode: 0400 with readOnly: true; PVC sizes are nominal under local-path.

Dependencies: landed KEDA 2.20.2, the annia.dispatch topology, and Flux v2.9.4; annia-infra#310 for the image; milner#21 for the /root/.pi folder; the layered-store issue for the /nix claim's final shape.

Out-of-scope notes

  • annia-infra#310 owns: publishing milner/annia:<semver>, making the image runnable non-interactively (args passthrough in entrypoint.sh), staging pi in the image, and failing on a missing environments/*.env instead of skipping it.
  • The reject-not-nack client change makes x-delivery-limit: 2 actually fire; it is annia-infra's.
  • Selecting the lain-milner-external provider in .annia.toml is a later test, not this change; the LLM egress Service already landed.
  • apps/evolved-npcs/, the enqueuer manifest and its tag guard, and the six-layer list are untouched.

Acceptance Criteria

  • python -m pytest tests/ -v is green with the VSO tripwires matching the three new reads (ci/milner, annia/env, ci/forgejo): len(paths) == 17 and (9, 17), the keda.sh row, and the extended _pod_specs().
  • conftest test on the rendered apps layer reaches spec.jobTargetRef.template; reverting the Rego _containers arm makes the case fail.
  • The rendered ScaledJob carries queueName: annia.dispatch, hostFromEnv: RABBITMQ_HOST, maxReplicaCount: 1, backoffLimit: 0, the shared nix claim, and ssa: Merge.
  • The rendered Deployment uses args: ["start"] (no command:), has no /workspace volume, and mounts /nix, the whole-volume /root/.pi, the 0400 key, and /workspace/environments.
  • Live, blocked by annia-infra#310/milner#21: the pod is Running with the mounts and env | grep -E 'FORGEJO_TOKEN|RABBITMQ_' populated; one message on annia.dispatch mints exactly one Job.
  • Live, blocked: autoscaling.keda.sh/paused: "true" set in Headlamp stops job creation and survives a Flux reconcile; an image bump opens one PR via refs/for/main, and a second bump updates it.

Testing Plan

  • python -m pytest tests/ -v — render, kubeconform, conftest and the tripwires above.
  • conftest test --policy policy/ on a rendered ScaledJob; revert the _containers arm and observe the failure.
  • flux build kustomization for secrets, apps and the regenerated flux-system.
  • Live (#310): kubectl -n annia exec deploy/annia -- env | grep RABBITMQ_; publish one message and watch kubectl -n annia get jobs -w; flux get images all -A.
  • Headlamp: add and remove the pause annotation and watch kubectl -n annia get jobs answer each way.

Related Issues/PRs

  • Implements #25; answers review comments #9210, #9790, #10103 and folds in the #10143 revision.
  • Blocked by tepichord/annia-infra#310 (publish milner/annia); /root/.pi blocked by milner#21; /nix coupled to the layered-store issue.
  • Patterns reused: infrastructure/nix-store/persistentvolumeclaim.yaml (the nix claim), apps/pocket-id/persistentvolumeclaim.yaml (Flux-owned persistent claim), apps/annia/deployment.yaml (the enqueuer's env mapping), secrets/pocket-id-database.yaml (destination.transformation + excludeRaw).

Research Findings

  • KEDA 2.20's rabbitmq scaler wants mode: QueueLength + value (queueLength is deprecated); vhostName is optional and the AMQP URI's vhost wins when omitted.
  • hostFromEnv/usernameFromEnv/passwordFromEnv resolve from the Job container's env, so no TriggerAuthentication is needed; maxReplicaCount: 1 serialises Jobs, but a Failed Job stops counting — the queue's x-delivery-limit: 2 is the real crash-loop bound.
  • VSO's transformation belongs under spec.destination.transformation, and excludeRaw: true is needed alongside excludes: [".*"] to drop the synthetic _raw field.
  • Flux's default Override SSA policy strips a hand-added annotation on reconcile; kustomize.toolkit.fluxcd.io/ssa: Merge preserves it for other field managers while Git-declared fields stay enforced.
  • Forgejo AGit opens a PR on a push to refs/for/main and needs topic + force-push: "true" to update the same PR on a second bump; push.refspec's local half must equal push.branch.
  • flux install --export should pass --version=v2.9.4; the extra components add their Deployments, RBAC and the image-toolkit CRDs, and the change is not byte-stable if the CLI version drifts.
  • Kubernetes ReadWriteOnce permits multiple pods on one node, so the Syncthing/annia shared /root/.pi claim is sound while both are pinned to the worker.
  • Nix serialises concurrent builds with a global lock; the documented hazard is nix-collect-garbage against a live store, not concurrent population, so "one writer, ever" should be reframed as the GC rule until the layered store lands.

Pre-Mortem

  1. The milner/annia image never lands (annia-infra#310) — Every live acceptance criterion and the ImagePolicy semver depend on a published image tag, so only manifest skeletons can merge while the apps layer's wait: true stays red. Because placeholders are banned, the work cannot satisfy its own live criteria and stalls at the offline gate.
  2. The Flux image automation is never verified and the AGit PR never updates — Regenerating gotk-components.yaml with flux install --export --version=v2.9.4 can drift with the CLI's embedded version and mismatch the image-toolkit CRDs with its controllers. Even when installed correctly, "a second bump updates the same PR" needs a write-enabled deploy key plus topic and force-push, and that registration is an unconfirmed external step the plan itself flags.
  3. A tripwire is missed and the ScaledJob either breaks CI or bypasses policy — The plan must move the two count assertions together (17 and (9, 17)), add the ("keda.sh", "keda", "KEDA") row, and extend both _pod_specs() and policy/k8s.rego's _containers arm. Missing or mis-pathing any one of these leaves pytest/conftest red or silently lets the Job container escape the CPU/memory limits rule.
Naina: *Updated on 2026-09-30 based on maintainer feedback.* ## Overview Run Annia on milner: the persistent Deployment, the serialized KEDA ScaledJob watching `annia.dispatch`, the three Vault reads it needs (clone key, repo env, Forgejo token), the shared `/nix` and `/root/.pi` mounts, Flux image automation, and the tripwires and docs that move with them. Body items **1–4, 6, 7 and 8 have landed** (RabbitMQ, KEDA 2.20.2, the `annia.dispatch` topology, the enqueuer, its Cloudflare route, and the LLM egress Service), so this plan is **items 5, 9 and 10 only**. Every live criterion is blocked by `annia-infra#310` (no `milner/annia` image is published today); `/root/.pi` additionally depends on `milner#21`, and the `annia-nix` claim's shape on the layered-store issue. ## Technical Approach 1. **Add the clone-key read** — one VaultStaticSecret for `ci/milner`, plus the named policy path. Note the nesting: `transformation` goes under `spec.destination`, and `excludeRaw: true` is required so VSO's synthetic `_raw` does not land in the Secret<sup>[1](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/gitops/clusters/milner/secrets/pocket-id-database.yaml?display=source#L71)</sup>. ```yaml # gitops/clusters/milner/secrets/annia-secrets.yaml (append) kind: VaultStaticSecret metadata: { name: annia-forgejo-key, namespace: annia } spec: type: kv-v2 mount: secret path: ci/milner refreshAfter: 1h vaultAuthRef: default destination: name: annia-forgejo-key create: true overwrite: true transformation: excludeRaw: true excludes: [".*"] templates: id_ed25519: { text: '{{ get .Secrets "SSH_KEY" }}' } ``` ```hcl # apps/annia/set-credential.sh — POLICY_HCL gains two named paths path "secret/data/ci/milner" { capabilities = ["read"] } path "secret/data/ci/forgejo" { capabilities = ["read"] } ``` ```bash # apps/annia/set-credential.sh — constants the producer tripwire scans for CLONE_KEY_PATH=secret/ci/milner REPO_ENV_PATH=secret/annia/env # written by the OPERATIONS.md runbook FORGEJO_TOKEN_PATH=secret/ci/forgejo ``` 2. **Add the two remaining Vault reads** — `annia/env` (one field per `environments/*.env` file) and `ci/forgejo`; same `kv-v2`/`vaultAuthRef: default` shape as above, no transformation. ```yaml # annia-secrets.yaml (append) kind: VaultStaticSecret metadata: { name: annia-repo-env, namespace: annia } spec: { path: annia/env, ... destination: { name: annia-repo-env } } --- kind: VaultStaticSecret metadata: { name: annia-forgejo-token, namespace: annia } spec: { path: ci/forgejo, ... destination: { name: annia-forgejo-token } } ``` `annia-read` already covers `annia/env`; the two `ci/` paths are the new grants. 3. **Two claims** — `/nix` is the shared store (story 19), `/root/.pi` is the Syncthing-synced folder (milner#21). The `/workspace` claim is **deleted**: the container layer is the workspace. ```yaml # apps/annia/annia-nix-pvc.yaml (provisional — the layered-store issue re-derives it) spec: { accessModes: [ReadWriteOnce], storageClassName: nix, resources: { requests: { storage: 200Gi } } } # apps/annia/annia-pi-pvc.yaml (whole volume, no subPath; Syncthing mounts the same claim) spec: { accessModes: [ReadWriteOnce], storageClassName: persistent, resources: { requests: { storage: 20Gi } } } ``` RWO means the annia pod and the Syncthing pod must be in one namespace and on one node; `nodeSelector: node-role: worker` plus the single worker makes that free today. 4. **The Deployment** — new file, because `deployment.yaml` is the enqueuer's and `ci/enqueuer-tag-guard.sh` pins its image line<sup>[2](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/ci/enqueuer-tag-guard.sh?display=source#L34)</sup>. `args:` only: `command:` would replace the image's ENTRYPOINT<sup>[3](https://git.gloom.garden/tepichord/annia-infra/src/commit/b17806678d1d90fcbc9821b85ca5eae82d08335b/Dockerfile?display=source#L24)</sup>, so the clone, key wiring and git config never run<sup>[4](https://git.gloom.garden/tepichord/annia-infra/src/commit/b17806678d1d90fcbc9821b85ca5eae82d08335b/entrypoint.sh?display=source#L8)</sup>. ```yaml # apps/annia/annia-deployment.yaml spec: replicas: 1 strategy: { type: Recreate } template: spec: nodeSelector: { node-role: worker } containers: - name: annia image: zot.faun-snapper.ts.net/milner/annia:0.1.0 # {"$imagepolicy": "annia:annia"} args: ["start"] # persistent loop; the Job below runs the one-shot "run" env: - { name: FORGEJO_URL, value: "https://git.gloom.garden" } - { name: FORGEJO_CLONE_URL, value: "clone.gloom.garden" } - { name: FORGEJO_PORT, value: "22" } - name: FORGEJO_TOKEN valueFrom: { secretKeyRef: { name: annia-forgejo-token, key: FORGEJO_TOKEN } } resources: { requests: { cpu: 500m, memory: 1Gi }, limits: { cpu: "2", memory: 4Gi } } volumeMounts: - { name: nix, mountPath: /nix } - { name: pi, mountPath: /root/.pi } # whole volume, no subPath - { name: repo-env, mountPath: /workspace/environments, readOnly: true } - { name: forgejo-key, mountPath: /root/.ssh/id_ed25519, subPath: id_ed25519, readOnly: true } volumes: - { name: nix, persistentVolumeClaim: { claimName: annia-nix } } - { name: pi, persistentVolumeClaim: { claimName: annia-pi } } - { name: repo-env, secret: { secretName: annia-repo-env } } - { name: forgejo-key, secret: { secretName: annia-forgejo-key, defaultMode: 0400 } } ``` `securityContext` stays unset: `annia` is `baseline` and the image is root. There is **no** `/workspace` volume; `/workspace/environments` is the only thing mounted under it. 5. **The ScaledJob** — one Job per queued pointer; mounts the shared store and the key, and carries the same tag marker so image automation bumps both. ```yaml # apps/annia/scaledjob.yaml apiVersion: keda.sh/v1alpha1 kind: ScaledJob metadata: annotations: kustomize.toolkit.fluxcd.io/ssa: Merge # keep a hand-set pause annotation across a reconcile spec: maxReplicaCount: 1 pollingInterval: 15 successfulJobsHistoryLimit: 3 failedJobsHistoryLimit: 3 jobTargetRef: backoffLimit: 0 # one attempt per Job; the queue's x-delivery-limit is the crash-loop bound template: spec: restartPolicy: Never nodeSelector: { node-role: worker } containers: - name: annia image: zot.faun-snapper.ts.net/milner/annia:0.1.0 # {"$imagepolicy": "annia:annia"} args: ["run"] # single sync -> schedule -> dispatch cycle env: - { name: RABBITMQ_HOST, value: "amqp://rabbitmq.rabbitmq.svc.cluster.local:5672/annia" } - { name: RABBITMQ_USERNAME, valueFrom: { secretKeyRef: { name: annia-rabbitmq-credentials, key: username } } } - { name: RABBITMQ_PASSWORD, valueFrom: { secretKeyRef: { name: annia-rabbitmq-credentials, key: password } } } resources: { requests: { cpu: 500m, memory: 1Gi }, limits: { cpu: "2", memory: 4Gi } } volumeMounts: - { name: nix, mountPath: /nix } - { name: forgejo-key, mountPath: /root/.ssh/id_ed25519, subPath: id_ed25519, readOnly: true } volumes: - { name: nix, persistentVolumeClaim: { claimName: annia-nix } } - { name: forgejo-key, secret: { secretName: annia-forgejo-key, defaultMode: 0400 } } triggers: - type: rabbitmq metadata: protocol: amqp mode: QueueLength value: "1" queueName: annia.dispatch # never the obsolete annia-wakeups hostFromEnv: RABBITMQ_HOST # the scaler reads the Job template's env usernameFromEnv: RABBITMQ_USERNAME passwordFromEnv: RABBITMQ_PASSWORD ``` `hostFromEnv` resolves from the Job container's env, so there is no TriggerAuthentication and no Secret in `keda`<sup>[5](https://keda.sh/docs/2.20/scalers/rabbitmq-queue/)</sup>. The URI's `/annia` supplies the vhost, so `vhostName` is omitted. 6. **The pause lever's RBAC** — a Role/RoleBinding over `keda.sh/scaledjobs` for the `default` ServiceAccount, which is what the landed VaultAuth binds<sup>[6](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/gitops/clusters/milner/secrets/vaultauth-annia.yaml?display=source#L11)</sup>. ```yaml # apps/annia/rbac.yaml kind: Role # rules: [{ apiGroups: [keda.sh], resources: [scaledjobs], verbs: [get, patch] }] kind: RoleBinding # -> ServiceAccount annia/default ``` 7. **Move the tripwires** — the ScaledJob's containers live at a path neither the policy nor the test walker knows today<sup>[7](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/policy/k8s.rego?display=source#L21)</sup><sup>[8](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/tests/test_gitops.py?display=source#L48)</sup><sup>[9](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/tests/test_gitops.py?display=source#L527)</sup><sup>[10](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/tests/test_gitops.py?display=source#L782)</sup>. ```python CRD_PROVIDERS = CRD_PROVIDERS + [("keda.sh", "keda", "KEDA")] assert len(paths) == 17 # 14 + clone key + repo env + forgejo token assert (len(auths), len(statics)) == (9, 17) # both count lines move together ``` ```python # tests/test_gitops.py — extend _pod_specs() so the placeholder/image walks see the Job template if kind == "ScaledJob": return [("spec.jobTargetRef.template.spec", doc["spec"]["jobTargetRef"]["template"]["spec"])] ``` ```rego # policy/k8s.rego — L1 is blind to a ScaledJob's containers today _containers contains container if { input.kind == "ScaledJob" container := input.spec.jobTargetRef.template.spec.containers[_] } ``` 8. **Image automation** — regenerate the vendored install to add the two missing controllers, then the three image objects and the write source. ```bash flux install --export --version=v2.9.4 --namespace flux-system \ --components-extra=image-reflector-controller,image-automation-controller \ > gitops/clusters/milner/flux-system/gotk-components.yaml ``` ```yaml # apps/annia/image-automation.yaml (image.toolkit.fluxcd.io/v1) kind: ImageRepository # image: zot.faun-snapper.ts.net/milner/annia, interval: 10m — zot reads are anonymous kind: ImagePolicy # policy: { semver: { range: ">=0.0.0" } } # skips :sha-* / :latest kind: ImageUpdateAutomation spec: sourceRef: { kind: GitRepository, name: flux-system-write } # second source; the Receiver-backed flux-system is untouched update: { path: ./gitops/clusters/milner/apps/annia } git: checkout: { ref: { branch: main } } push: branch: flux-updates refspec: refs/heads/flux-updates:refs/for/main # the local half equals push.branch options: { topic: flux-updates, force-push: "true" } # force-push is what updates the existing PR ``` ```yaml # apps/annia/gitrepository-write.yaml — the credential Secret is a hand-run bootstrap step (OPERATIONS.md) kind: GitRepository metadata: { name: flux-system-write, namespace: annia } spec: { url: ssh://git@clone.gloom.garden/tepichord/milner.git, secretRef: { name: flux-write-deploy-key } } ``` The `flux-write-deploy-key` must be **write-enabled**; Forgejo reserves force-push/rewrite history for Write and above<sup>[11](https://forgejo.org/docs/latest/user/git-cli/agit-support/) [12](https://forgejo.org/docs/latest/user/collaboration/repo-permissions/)</sup>, and the "second bump updates the same PR" criterion depends on it. 9. **Docs and stories, in the same change** (AGENTS.md Hard Rule). `docs/nix.md` §14b/§14c/§14d (add a builder-pod row; reframe "one writer, ever" as the GC rule and record the append-only interim); `docs/vault.md` §4c-ter (the two `ci/` policy paths); `OPERATIONS.md` (clone-key/`annia/env` runbook, `flux-write-deploy-key`, image automation, pause runbook); `AGENTS.md` (repo map row, and the `/root/.pi` Vault-first standing exception); `USER_STORIES.md` story 20 scoped to apps whose source is fixed at build time. ## Implementation Details | Path | Purpose | |---|---| | `gitops/clusters/milner/apps/annia/annia-deployment.yaml` | the pod; the enqueuer's `deployment.yaml` is untouched | | `gitops/clusters/milner/apps/annia/annia-nix-pvc.yaml` | shared `nix`-class claim, provisional | | `gitops/clusters/milner/apps/annia/annia-pi-pvc.yaml` | `/root/.pi`, whole volume, shared with Syncthing | | `gitops/clusters/milner/apps/annia/scaledjob.yaml` | serialized Job, `ssa: Merge`<sup>[13](https://fluxcd.io/flux/components/kustomize/kustomizations/#controlling-the-apply-behavior-of-resources)</sup>, `queueName: annia.dispatch` | | `gitops/clusters/milner/apps/annia/rbac.yaml` | Role/RoleBinding over `keda.sh/scaledjobs` for SA `default` | | `gitops/clusters/milner/apps/annia/{image-automation,gitrepository-write}.yaml` | image objects + write source | | `gitops/clusters/milner/apps/annia/kustomization.yaml` | list every new resource | | `gitops/clusters/milner/secrets/annia-secrets.yaml` | append the three VaultStaticSecrets (file already listed) | | `gitops/clusters/milner/apps/annia/set-credential.sh` | widen `POLICY_HCL`; add the three `<NAME>_PATH=` constants | | `gitops/clusters/milner/flux-system/gotk-components.yaml` | regenerated with the two image controllers | | `tests/test_gitops.py`, `policy/k8s.rego` | the tripwires and the `_containers` arm | **Constraints:** the ScaledJob is a `keda.sh` CR, so it stays in `apps` (a layer later than the `infrastructure` HelmRelease); the KEDA row lands now or `test_crd_providers_are_not_in_the_layer_that_uses_their_crds` fails with "nothing applies keda.sh resources"<sup>[14](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/gitops/clusters/milner/infrastructure/keda/helmrelease.yaml?display=source#L21)</sup><sup>[15](https://git.gloom.garden/tepichord/milner/src/commit/d3a5ed016c5ddf8b8319ef82dfbf3e1da2fee038/tests/test_gitops.py?display=source#L472)</sup>; `RABBITMQ_HOST` must carry an `amqp://` scheme and the `/annia` vhost; `transformation` nests under `spec.destination`; the SSH key mount needs `defaultMode: 0400` with `readOnly: true`; PVC sizes are nominal under local-path. **Dependencies:** landed KEDA 2.20.2, the `annia.dispatch` topology, and Flux v2.9.4; `annia-infra#310` for the image; `milner#21` for the `/root/.pi` folder; the layered-store issue for the `/nix` claim's final shape. **Out-of-scope notes** - `annia-infra#310` owns: publishing `milner/annia:<semver>`, making the image runnable non-interactively (`args` passthrough in `entrypoint.sh`), staging pi in the image, and failing on a missing `environments/*.env` instead of skipping it. - The `reject`-not-`nack` client change makes `x-delivery-limit: 2` actually fire; it is annia-infra's. - Selecting the `lain-milner-external` provider in `.annia.toml` is a later test, not this change; the LLM egress Service already landed. - `apps/evolved-npcs/`, the enqueuer manifest and its tag guard, and the six-layer list are untouched. ## Acceptance Criteria - [ ] `python -m pytest tests/ -v` is green with the VSO tripwires matching the three new reads (`ci/milner`, `annia/env`, `ci/forgejo`): `len(paths) == 17` and `(9, 17)`, the `keda.sh` row, and the extended `_pod_specs()`. - [ ] `conftest test` on the rendered `apps` layer reaches `spec.jobTargetRef.template`; reverting the Rego `_containers` arm makes the case fail. - [ ] The rendered ScaledJob carries `queueName: annia.dispatch`, `hostFromEnv: RABBITMQ_HOST`, `maxReplicaCount: 1`, `backoffLimit: 0`, the shared `nix` claim, and `ssa: Merge`. - [ ] The rendered Deployment uses `args: ["start"]` (no `command:`), has no `/workspace` volume, and mounts `/nix`, the whole-volume `/root/.pi`, the `0400` key, and `/workspace/environments`. - [ ] *Live, blocked by `annia-infra#310`/`milner#21`:* the pod is `Running` with the mounts and `env | grep -E 'FORGEJO_TOKEN|RABBITMQ_'` populated; one message on `annia.dispatch` mints exactly one Job. - [ ] *Live, blocked:* `autoscaling.keda.sh/paused: "true"` set in Headlamp stops job creation and survives a Flux reconcile; an image bump opens one PR via `refs/for/main`, and a second bump updates it. ## Testing Plan - `python -m pytest tests/ -v` — render, kubeconform, conftest and the tripwires above. - `conftest test --policy policy/` on a rendered ScaledJob; revert the `_containers` arm and observe the failure. - `flux build kustomization` for `secrets`, `apps` and the regenerated `flux-system`. - Live (`#310`): `kubectl -n annia exec deploy/annia -- env | grep RABBITMQ_`; publish one message and watch `kubectl -n annia get jobs -w`; `flux get images all -A`. - Headlamp: add and remove the pause annotation and watch `kubectl -n annia get jobs` answer each way. ## Related Issues/PRs - Implements [#25](https://git.gloom.garden/tepichord/milner/issues/25); answers review comments [#9210](https://git.gloom.garden/tepichord/milner/issues/25#issuecomment-9210), [#9790](https://git.gloom.garden/tepichord/milner/issues/25#issuecomment-9790), [#10103](https://git.gloom.garden/tepichord/milner/issues/25#issuecomment-10103) and folds in the [#10143](https://git.gloom.garden/tepichord/milner/issues/25#issuecomment-10143) revision. - Blocked by `tepichord/annia-infra#310` (publish `milner/annia`); `/root/.pi` blocked by `milner#21`; `/nix` coupled to the layered-store issue. - Patterns reused: `infrastructure/nix-store/persistentvolumeclaim.yaml` (the `nix` claim), `apps/pocket-id/persistentvolumeclaim.yaml` (Flux-owned `persistent` claim), `apps/annia/deployment.yaml` (the enqueuer's env mapping), `secrets/pocket-id-database.yaml` (`destination.transformation` + `excludeRaw`). ## Research Findings - KEDA 2.20's rabbitmq scaler wants `mode: QueueLength` + `value` (`queueLength` is deprecated); `vhostName` is optional and the AMQP URI's vhost wins when omitted. - `hostFromEnv`/`usernameFromEnv`/`passwordFromEnv` resolve from the Job container's env, so no TriggerAuthentication is needed; `maxReplicaCount: 1` serialises Jobs, but a `Failed` Job stops counting — the queue's `x-delivery-limit: 2` is the real crash-loop bound. - VSO's `transformation` belongs under `spec.destination.transformation`, and `excludeRaw: true` is needed alongside `excludes: [".*"]` to drop the synthetic `_raw` field. - Flux's default `Override` SSA policy strips a hand-added annotation on reconcile; `kustomize.toolkit.fluxcd.io/ssa: Merge` preserves it for other field managers while Git-declared fields stay enforced. - Forgejo AGit opens a PR on a push to `refs/for/main` and needs `topic` + `force-push: "true"` to update the same PR on a second bump; `push.refspec`'s local half must equal `push.branch`. - `flux install --export` should pass `--version=v2.9.4`; the extra components add their Deployments, RBAC and the image-toolkit CRDs, and the change is not byte-stable if the CLI version drifts. - Kubernetes `ReadWriteOnce` permits multiple pods on one node, so the Syncthing/annia shared `/root/.pi` claim is sound while both are pinned to the worker. - Nix serialises concurrent builds with a global lock; the documented hazard is `nix-collect-garbage` against a live store, not concurrent population, so "one writer, ever" should be reframed as the GC rule until the layered store lands. ## Pre-Mortem 1. **The `milner/annia` image never lands (`annia-infra#310`)** — Every live acceptance criterion and the `ImagePolicy` semver depend on a published image tag, so only manifest skeletons can merge while the `apps` layer's `wait: true` stays red. Because placeholders are banned, the work cannot satisfy its own live criteria and stalls at the offline gate. 2. **The Flux image automation is never verified and the AGit PR never updates** — Regenerating `gotk-components.yaml` with `flux install --export --version=v2.9.4` can drift with the CLI's embedded version and mismatch the image-toolkit CRDs with its controllers. Even when installed correctly, "a second bump updates the same PR" needs a write-enabled deploy key plus `topic` and `force-push`, and that registration is an unconfirmed external step the plan itself flags. 3. **A tripwire is missed and the ScaledJob either breaks CI or bypasses policy** — The plan must move the two count assertions together (`17` and `(9, 17)`), add the `("keda.sh", "keda", "KEDA")` row, and extend both `_pod_specs()` and `policy/k8s.rego`'s `_containers` arm. Missing or mis-pathing any one of these leaves `pytest`/`conftest` red or silently lets the Job container escape the CPU/memory limits rule.
tepichord commented 2026-09-15 15:48:26 -07:00
Author
Owner
Copy link
  • RabbitMQ service has landed in milner, please look at RabbitMQ's implementation and update the plan accordingly
  • Also, importantly crash loops are possible with annia, and there needs to be a way to prevent endlessly spinning up the pod when it occurs
    • It is very likely that annia will enter a broken state in which case spinning up an annia pod is pointless and it will just keep crashing and erroring out
    • There needs to be a way for annia to determine if it's in a truly broken state or if an error is transient, I would say 2 failures on the same RabbitMQ message is a good indicator. Once that happens, annia will be put into a state where manual remediation is necessary
    • Maintainer needs to investigate the errors and make necessary adjustments
    • Then there needs to be a way for the maintainer to unflag the annia service so that it can start accepting RabbitMQ messages again, preferably a method that works from a browser. Headlamp is already set up but also a new admin page in oz would be acceptable. Investigate both of these options and pick one, (or suggest an easier third option).
- RabbitMQ service has landed in milner, please look at RabbitMQ's implementation and update the plan accordingly - Also, importantly crash loops are possible with annia, and there needs to be a way to prevent endlessly spinning up the pod when it occurs - It is very likely that annia will enter a broken state in which case spinning up an annia pod is pointless and it will just keep crashing and erroring out - There needs to be a way for annia to determine if it's in a truly broken state or if an error is transient, I would say 2 failures on the same RabbitMQ message is a good indicator. Once that happens, annia will be put into a state where manual remediation is necessary - Maintainer needs to investigate the errors and make necessary adjustments - Then there needs to be a way for the maintainer to unflag the annia service so that it can start accepting RabbitMQ messages again, preferably a method that works from a browser. Headlamp is already set up but also a new admin page in oz would be acceptable. Investigate both of these options and pick one, (or suggest an easier third option).
tepichord commented 2026-09-22 14:14:51 -07:00
Author
Owner
Copy link
  Facts the plan needs, from a planning pass over `main` and this board. Short names below
  map to the new issues this work was split into:

  | Title | Short name |
  |---|---|
  | `Reach the tailnet-shared LLM gateway: relay on the libvirt host` | the relay issue |
  | `Tailscale egress from the cluster: an ExternalName Service for the relay` | the egress issue |
  | `Publish the annia webhook through the Cloudflare Tunnel` | the route issue |
  | `Run the annia enqueuer: HMAC-verifying webhook receiver publishing pointer messages` | the enqueuer issue |
  | `Add the annia pause lever to the oz console` | the console issue |

  ## Corrections to the rest of the body

  - **`apps/annia/` stands.** The committed `apps/evolved-npcs/` stub is *another project's* — `evolved-npcs`, a genetic-algorithm research repository. It is
`README.md` plus `resources: []`, it is not listed in `apps/kustomization.yaml`, and Flux never builds it. Leave it exactly as it is.
  - **`ghcr.io/tepichord/annia{,-enqueuer}` is stale.** The cluster's registry is zot, and the `annia-ci` identity already holds push rights there.
  - **The enqueuer is no longer this issue's.** It is the enqueuer issue, and it is the one annia piece nothing blocks today — the broker already landed. It needs
neither the LLM path (which serves this pod) nor KEDA (which serves the ScaledJob).
  - **The VSO tripwire has moved.** `tests/test_gitops.py` now expects *"expected 7 VaultAuths and 11 VaultStaticSecrets"*. annia's VaultAuth and VaultStaticSecrets
change both numbers — the same class of tripwire this plan already carries for KEDA.
  - **The name the pod dials.** `http://llm-gateway.annia.svc.cluster.local:4000`, an `ExternalName` Service in annia's own namespace rather than a shared one in
`infrastructure`: story 14 says a preview holds no real credential and no real power and must not inherit a live path to the inference box. The cluster resolves
nothing in the gateway's own tailnet (NXDOMAIN for the whole zone), so a Service name is the only route.
```markdown Facts the plan needs, from a planning pass over `main` and this board. Short names below map to the new issues this work was split into: | Title | Short name | |---|---| | `Reach the tailnet-shared LLM gateway: relay on the libvirt host` | the relay issue | | `Tailscale egress from the cluster: an ExternalName Service for the relay` | the egress issue | | `Publish the annia webhook through the Cloudflare Tunnel` | the route issue | | `Run the annia enqueuer: HMAC-verifying webhook receiver publishing pointer messages` | the enqueuer issue | | `Add the annia pause lever to the oz console` | the console issue | ## Corrections to the rest of the body - **`apps/annia/` stands.** The committed `apps/evolved-npcs/` stub is *another project's* — `evolved-npcs`, a genetic-algorithm research repository. It is `README.md` plus `resources: []`, it is not listed in `apps/kustomization.yaml`, and Flux never builds it. Leave it exactly as it is. - **`ghcr.io/tepichord/annia{,-enqueuer}` is stale.** The cluster's registry is zot, and the `annia-ci` identity already holds push rights there. - **The enqueuer is no longer this issue's.** It is the enqueuer issue, and it is the one annia piece nothing blocks today — the broker already landed. It needs neither the LLM path (which serves this pod) nor KEDA (which serves the ScaledJob). - **The VSO tripwire has moved.** `tests/test_gitops.py` now expects *"expected 7 VaultAuths and 11 VaultStaticSecrets"*. annia's VaultAuth and VaultStaticSecrets change both numbers — the same class of tripwire this plan already carries for KEDA. - **The name the pod dials.** `http://llm-gateway.annia.svc.cluster.local:4000`, an `ExternalName` Service in annia's own namespace rather than a shared one in `infrastructure`: story 14 says a preview holds no real credential and no real power and must not inherit a live path to the inference box. The cluster resolves nothing in the gateway's own tailnet (NXDOMAIN for the whole zone), so a Service name is the only route. ```
tepichord commented 2026-09-25 23:57:41 -07:00
Author
Owner
Copy link

Decisions from review, and the plan against what has landed

The plan above still holds in outline. This records what has merged since it was written, one decision that changes the store design, and the
corrections that follow from both.

1. Landed since the plan was written

Plan step Status
1. cert-manager for the RabbitMQ operator obsolete — no cert-manager in the cluster (0 CRDs, no namespace). The operator landed without it,
and KEDA self-signs its webhook
2. RabbitMQ operator + namespace landed
3. RabbitMQ Cluster CR + the annia vhost topology landed as data/annia/rabbitmq.yaml
4. KEDA in infrastructure/ landed — 2.20.2, 6 CRDs (keda.sh ×4, eventing.keda.sh ×2), namespace restricted, HelmRelease Ready, 3 pods
on the worker, v1beta1.external.metrics.k8s.io Available
6, 7. the enqueuer and its public route landed (separate issues)
— infrastructure/namespaces/annia.yaml, the annia VaultAuth landed — secrets/vaultauth-annia.yaml (role annia, serviceAccount
default)

So the remaining work is the body's items 5, 9 and 10 — the pod, its secrets, and the ScaledJob — not items 1–4.

2. Decision: annia is a builder, so it mounts the shared store

Annia clones its own sources and runs them; its environment is the flake of the repo and branch under work (today annia-infra and milner, in
future dozens). It builds. So story 19 governs it — "I mount a nix volume at /nix in whatever builds in this cluster — a pipeline step, any
namespace" — and story 20 governs apps whose source is fixed at build time, not this.

That makes apps/annia/nix-pvc.yaml wrong as written. It is the ephemeral class: a private, empty volume, sized for one environment. It must be
the nix class, whose constant pathPattern: shared resolves every claim of that class, in any namespace, to the one directory
/var/mnt/data/nix/shared (docs/nix.md §14b). A private store cannot work here: it starts empty, and a mount over /nix hides the image's own store,
so the pod cannot exec its entrypoint (§14c, §14d) — and it would share nothing with the builds that already realised both shells.

Two consequences:

  • The size question leaves the claim. The shared store lives on the worker's 2 TiB data disk, so 20Gi nominal is moot.
  • The ScaledJob should mount it too. "The wake-up Job mounts nothing, on purpose" was correct while the store was private and empty. With a
    shared warm store, a Job that mounts nothing re-realises every environment it touches on every wake-up.

/workspace and /root/.pi stay as they are — the persistent class, opaque state the pod writes and reads back. The /nix mount is the only one
that changes.

3. Corrections to the plan as written

  • The ScaledJob names a queue that does not exist. The plan says queueName: annia-wakeups; the landed topology declares annia.dispatch. The
    enqueuer publishes to annia.events with routing key milner (enqueuer/app.py:61–62). The crash-loop bound the maintainer asked for has landed:
    annia.dispatch is a quorum queue with x-delivery-limit: 2 and x-dead-letter-exchange: annia.dlx, asserted in tests/test_gitops.py:1505–1556. The
    reject-not-nack change is the only remaining half of that defence.
  • KEDA's CRD row is owed by this change. KEDA landed deliberately without CRD_PROVIDERS gaining ("keda.sh", "keda", "KEDA"): the tripwire
    asserts nothing consumes keda.sh until something does. The first ScaledJob is that something, so the row must be added here or the suite fails.
  • The VSO tripwire is stale. The plan says "was (7, 11)"; the current assertion is expected 9 VaultAuths and 14 VaultStaticSecrets
    (tests/test_gitops.py:783). The annia VaultAuth already exists, and annia-rabbitmq-credentials (path annia/rabbitmq) already exists in the
    namespace — so this change adds one VaultStaticSecret, the clone key: (9, 15).
  • The forge credential is secret/ci/milner → SSH_KEY — the existing key, no new keypair (tepichord/annia-infra#310 settles where it lives, and
    the container there uses the same one). Two consequences here: the annia Vault role's read-only policy must include that path, and the Secret-mounted
    private key needs defaultMode: 0400 with readOnly: true, because ssh refuses a key that is group- or world-readable.
  • The sandbox payload is hidden by the /workspace claim. The image stages .annia.toml, .agents/skills and diagnostics at /workspace/…,
    which worked under Docker's VOLUME and does not under Kubernetes — the PVC hides it, exactly as a store mount hides /nix. The image must stage them
    elsewhere and entrypoint.sh must seed an empty /workspace from there. That change belongs to annia-infra#310; this issue depends on it.
  • policy/k8s.rego still needs the _containers arm for ScaledJob, whose containers live at spec.jobTargetRef.template.
  • The pause lever stands as decided — autoscaling.keda.sh/paused applied by hand in Headlamp — as does maxReplicaCount: 1.

4. Open before implementing

  • Writer discipline. docs/nix.md §14b and §14d: one writer at a time, because the store's database is a single SQLite file. With the pod on the
    shared store the writers are CI's serial steps, the Deployment, and Jobs — and maxReplicaCount: 1 serialises Jobs against each other and nothing else.
    Decide between accepting that and serialising it.
  • Where annia's commits go. If it pushes to milner as well as annia-infra, that is a second grant on the same key's account; if it opens pull
    requests, it needs a token, and entrypoint.sh reads none today.
  • The image it runs, from annia-infra#310: the repository path and tag convention (milner/annia:<semver> would match the rest of the
    registry), and whether the tag-tracking automation earns its place while a bootstrap image changes rarely.
  • The docs and the story go with it. §14c's "Nothing mounts it at runtime, and that is the design" is true of running a closure and false of a
    pod that builds; its table wants a row for "a builder, any namespace". Story 20 arguably wants scoping to apps whose source is fixed at build time. Per
    AGENTS.md the correction lands in the change that contradicts it — this one.

5. Sequencing

tepichord/annia-infra#310 first. There is no annia image to run today — the registry holds milner/annia-enqueuer and nothing else in that family —
so the pod's live criteria are unreachable until that pipeline publishes one, and #310 is not blocked by this issue.

## Decisions from review, and the plan against what has landed The plan above still holds in outline. This records what has merged since it was written, one decision that changes the store design, and the corrections that follow from both. ### 1. Landed since the plan was written | Plan step | Status | |---|---| | 1. cert-manager for the RabbitMQ operator | **obsolete** — no cert-manager in the cluster (0 CRDs, no namespace). The operator landed without it, and KEDA self-signs its webhook | | 2. RabbitMQ operator + namespace | **landed** | | 3. RabbitMQ Cluster CR + the `annia` vhost topology | **landed** as `data/annia/rabbitmq.yaml` | | 4. KEDA in `infrastructure/` | **landed** — 2.20.2, 6 CRDs (`keda.sh` ×4, `eventing.keda.sh` ×2), namespace `restricted`, HelmRelease Ready, 3 pods on the worker, `v1beta1.external.metrics.k8s.io` Available | | 6, 7. the enqueuer and its public route | **landed** (separate issues) | | — `infrastructure/namespaces/annia.yaml`, the `annia` VaultAuth | **landed** — `secrets/vaultauth-annia.yaml` (role `annia`, serviceAccount `default`) | So the remaining work is the body's items 5, 9 and 10 — the pod, its secrets, and the ScaledJob — not items 1–4. ### 2. Decision: annia is a builder, so it mounts the shared store Annia clones its own sources and runs them; its environment is the flake of the repo and branch under work (today `annia-infra` and `milner`, in future dozens). It **builds**. So story 19 governs it — *"I mount a `nix` volume at `/nix` in whatever builds in this cluster — a pipeline step, **any namespace**"* — and story 20 governs apps whose source is fixed at build time, not this. That makes `apps/annia/nix-pvc.yaml` wrong as written. It is the **`ephemeral`** class: a private, empty volume, sized for one environment. It must be the **`nix`** class, whose constant `pathPattern: shared` resolves every claim of that class, in any namespace, to the one directory `/var/mnt/data/nix/shared` (docs/nix.md §14b). A private store cannot work here: it starts empty, and a mount over `/nix` hides the image's own store, so the pod cannot exec its entrypoint (§14c, §14d) — and it would share nothing with the builds that already realised both shells. Two consequences: - **The size question leaves the claim.** The shared store lives on the worker's 2 TiB data disk, so `20Gi` nominal is moot. - **The ScaledJob should mount it too.** *"The wake-up Job mounts nothing, on purpose"* was correct while the store was private and empty. With a shared warm store, a Job that mounts nothing re-realises every environment it touches on every wake-up. `/workspace` and `/root/.pi` stay as they are — the `persistent` class, opaque state the pod writes and reads back. The `/nix` mount is the only one that changes. ### 3. Corrections to the plan as written - **The ScaledJob names a queue that does not exist.** The plan says `queueName: annia-wakeups`; the landed topology declares `annia.dispatch`. The enqueuer publishes to `annia.events` with routing key `milner` (`enqueuer/app.py:61–62`). The crash-loop bound the maintainer asked for **has landed**: `annia.dispatch` is a quorum queue with `x-delivery-limit: 2` and `x-dead-letter-exchange: annia.dlx`, asserted in `tests/test_gitops.py:1505–1556`. The `reject`-not-`nack` change is the only remaining half of that defence. - **KEDA's CRD row is owed by this change.** KEDA landed deliberately without `CRD_PROVIDERS` gaining `("keda.sh", "keda", "KEDA")`: the tripwire asserts nothing consumes `keda.sh` until something does. The first ScaledJob is that something, so the row must be added here or the suite fails. - **The VSO tripwire is stale.** The plan says *"was (7, 11)"*; the current assertion is `expected 9 VaultAuths and 14 VaultStaticSecrets` (`tests/test_gitops.py:783`). The `annia` VaultAuth already exists, and `annia-rabbitmq-credentials` (path `annia/rabbitmq`) already exists in the namespace — so this change adds **one** VaultStaticSecret, the clone key: `(9, 15)`. - **The forge credential is `secret/ci/milner → SSH_KEY`** — the existing key, no new keypair (`tepichord/annia-infra#310` settles where it lives, and the container there uses the same one). Two consequences here: the `annia` Vault role's read-only policy must include that path, and the Secret-mounted private key needs `defaultMode: 0400` with `readOnly: true`, because `ssh` refuses a key that is group- or world-readable. - **The sandbox payload is hidden by the `/workspace` claim.** The image stages `.annia.toml`, `.agents/skills` and `diagnostics` at `/workspace/…`, which worked under Docker's `VOLUME` and does not under Kubernetes — the PVC hides it, exactly as a store mount hides `/nix`. The image must stage them elsewhere and `entrypoint.sh` must seed an empty `/workspace` from there. That change belongs to `annia-infra#310`; this issue depends on it. - **`policy/k8s.rego`** still needs the `_containers` arm for `ScaledJob`, whose containers live at `spec.jobTargetRef.template`. - **The pause lever stands as decided** — `autoscaling.keda.sh/paused` applied by hand in Headlamp — as does `maxReplicaCount: 1`. ### 4. Open before implementing - **Writer discipline.** docs/nix.md §14b and §14d: one writer at a time, because the store's database is a single SQLite file. With the pod on the shared store the writers are CI's serial steps, the Deployment, and Jobs — and `maxReplicaCount: 1` serialises Jobs against each other and nothing else. Decide between accepting that and serialising it. - **Where annia's commits go.** If it pushes to `milner` as well as `annia-infra`, that is a second grant on the same key's account; if it opens pull requests, it needs a token, and `entrypoint.sh` reads none today. - **The image it runs**, from `annia-infra#310`: the repository path and tag convention (`milner/annia:<semver>` would match the rest of the registry), and whether the tag-tracking automation earns its place while a bootstrap image changes rarely. - **The docs and the story go with it.** §14c's *"Nothing mounts it at runtime, and that is the design"* is true of running a closure and false of a pod that builds; its table wants a row for "a builder, any namespace". Story 20 arguably wants scoping to apps whose source is fixed at build time. Per AGENTS.md the correction lands in the change that contradicts it — this one. ### 5. Sequencing `tepichord/annia-infra#310` first. There is no annia image to run today — the registry holds `milner/annia-enqueuer` and nothing else in that family — so the pod's live criteria are unreachable until that pipeline publishes one, and #310 is not blocked by this issue.
tepichord commented 2026-09-29 16:27:29 -07:00
Author
Owner
Copy link

Naina:

Plan revision — the 2026-09-28 decision session

Body items 5, 9 and 10 still stand, and the technical approach is otherwise unchanged. The pod's shape changes as follows; the parts not named here hold as written.

1. Drop command: — the entrypoint must stay

Kubernetes command: replaces the image's ENTRYPOINT. With command: ["annia"] (in both the Deployment and the ScaledJob) /entrypoint.sh never runs, so the clone, the SSH key wiring and the git config do not happen. Pass args: ["start"] and args: ["run"] only and let the ENTRYPOINT do its job — the image's entrypoint.sh gains if [ "$#" -gt 0 ]; then exec "$@"; fi for exactly this (annia-infra#310).

2. Delete the /workspace claim and its mount

There is no /workspace volume in-cluster: the container's own writable layer is the workspace, exactly as it already is locally. Delete the annia-workspace PVC, the /workspace mount, and the pi subPath with it. This is one path on both machines, the image's payload (.annia.toml, .agents/skills, diagnostics/http-tap.py) is simply visible, the clone is unconditional on every start, and nothing durable is lost — the durable parts are the nix store and /root/.pi, the backlog lives in Forgejo, .annia/ is scratch, and the artefact of a run is a commit or a pull request.

3. /root/.pi is the shared Syncthing PVC (milner#21)

/root/.pi becomes a whole-volume mount (no subPath) from a persistent-class PVC that the cluster's always-on Syncthing pod (milner#21) also mounts. That pod owns the folder naccs-5znau ("pi" = the operator's ~/.pi) and keeps it send/receive in sync with the operator's machines, so the pod's /root/.pi is the operator's folder — credentials, skills, extensions, session history, all of it — and what annia writes lands back on the operator's host.

  • RWO: the claim is ReadWriteOnce, so the Syncthing pod and the annia pod must land on the same node. Both select node-role: worker and there is one worker today, so this is free; it is a real constraint the day a second worker exists.
  • The pi extensions ride the folder, so .annia.toml's hardcoded -e /root/.pi/agent/npm/node_modules/pi-web-access is correct in the pod with nothing baked into the image. Nothing installs extensions at pod start.
  • pi itself is in the image (annia-infra#310) — the plan's runtime npm install -g goes away; only the state and extensions come from the folder.
  • This makes /root/.pi depend on milner#21 shipping, so #21 moves onto this issue's critical path.

4. No Vault path for pi's credentials — a documented standing exception

The folder carries pi's provider keys (agent/auth.json, models.json's lain.apiKey), so there is no secret/annia/pi* path, no VaultStaticSecret for pi, and no template rendering. The ~/.pi mount is the source of truth. AGENTS.md's "Secrets are Vault-first" therefore gains a standing exception, written down where it is used (the way the Tailscale operator's OAuth client is) — the operator has confirmed this, and the exception sentence is part of this change.

The in-cluster provider is a named provider in the operator's own models.json — lain-milner-external → http://llm-gateway.annia.svc.cluster.local:4000, same shape as the existing lain, pointing at the already-landed llm-gateway Service. lain-llm.tailc23d6.ts.net still never appears in-cluster. Selecting that provider in .annia.toml is a later test, not part of this change (today's default is deepseek).

5. environments/*.env come from Vault

secret/annia/env (one field per file, the field name is the file name) synced by VaultStaticSecret annia-repo-env into a Secret, mounted read-only at /workspace/environments — so the two files arrive at exactly the paths .annia.toml names, with no templates and no policy change (annia-read already covers secret/data/annia/*). A missing file now fails instead of being skipped (annia-infra#310 flips the crate's behaviour and its test).

6. FORGEJO_TOKEN access is missing today and is owed here

secret/ci/forgejo → FORGEJO_TOKEN exists and is byte-identical to secret/forgejo/token, but the pod has no access: no VaultStaticSecret, no env entry, and the annia Vault role covers only secret/data/annia/*. Add the access — secretKeyRef from a synced Secret (env is what the skills read) — with the same named-path policy argument the clone key already carries.

7. The annia-nix claim is provisional

Under the layered store (/nix/store = image lower + shared volume upper), the consumer's claim and its /nix mount stay, but the backing directory becomes the merge point the node-side assembler owns rather than the whole store. The A-shaped claim (storageClassName: nix at /nix) must not be copied forward — it is re-derived by the new issue Convert the shared nix store to a layered store, which also deletes the hand-run seed Job and the docs/storage.md §10g section that papered over the empty store.

8. Unchanged

The Flux ImageRepository/ImagePolicy/ImageUpdateAutomation and the write source; ImagePolicy semver: { range: ">=0.0.0" } (the tag the pipeline mints, <major>.<minor>.<count>, compares as semver — :sha-* tags are not semver and are correctly ignored, :latest is never published); the clone-key VaultStaticSecret and the named path "secret/data/ci/milner" policy widen with CLONE_KEY_PATH=secret/ci/milner; the KEDA ScaledJob (maxReplicaCount: 1, backoffLimit: 0, queueName: annia.dispatch, the ssa: Merge annotation, the pause-lever criteria); securityContext unset (annia is baseline, the image is root); the hand-run flux-write-deploy-key prerequisite in OPERATIONS.md. The verification list's /nix, /workspace, /root/.pi line becomes /nix (the merged store) and /root/.pi (the synced folder), with /workspace carrying no claim.

Dependencies

  • Blocked by annia-infra#310 for every live criterion (no annia image is published today) — unchanged.
  • Now also blocked by milner#21 for /root/.pi, and by the layered-store issue for what /nix is.
Naina: ## Plan revision — the 2026-09-28 decision session Body items **5, 9 and 10** still stand, and the technical approach is otherwise unchanged. The pod's shape changes as follows; the parts not named here hold as written. --- ### 1. Drop `command:` — the entrypoint must stay Kubernetes `command:` **replaces** the image's ENTRYPOINT. With `command: ["annia"]` (in both the Deployment and the ScaledJob) `/entrypoint.sh` never runs, so the clone, the SSH key wiring and the git config do not happen. Pass **`args: ["start"]`** and **`args: ["run"]`** only and let the ENTRYPOINT do its job — the image's `entrypoint.sh` gains `if [ "$#" -gt 0 ]; then exec "$@"; fi` for exactly this (annia-infra#310). ### 2. Delete the `/workspace` claim and its mount There is **no `/workspace` volume in-cluster**: the container's own writable layer *is* the workspace, exactly as it already is locally. Delete the `annia-workspace` PVC, the `/workspace` mount, and the `pi` subPath with it. This is one path on both machines, the image's payload (`.annia.toml`, `.agents/skills`, `diagnostics/http-tap.py`) is simply visible, the clone is unconditional on every start, and nothing durable is lost — the durable parts are the nix store and `/root/.pi`, the backlog lives in Forgejo, `.annia/` is scratch, and the artefact of a run is a commit or a pull request. ### 3. `/root/.pi` is the shared Syncthing PVC (milner#21) `/root/.pi` becomes a **whole-volume mount** (no `subPath`) from a `persistent`-class PVC that the cluster's always-on Syncthing pod (milner#21) also mounts. That pod owns the folder `naccs-5znau` ("pi" = the operator's `~/.pi`) and keeps it send/receive in sync with the operator's machines, so the pod's `/root/.pi` *is* the operator's folder — credentials, skills, extensions, session history, all of it — and what annia writes lands back on the operator's host. - **RWO**: the claim is `ReadWriteOnce`, so the Syncthing pod and the annia pod must land on the same node. Both select `node-role: worker` and there is one worker today, so this is free; it is a real constraint the day a second worker exists. - **The pi extensions ride the folder**, so `.annia.toml`'s hardcoded `-e /root/.pi/agent/npm/node_modules/pi-web-access` is correct in the pod with nothing baked into the image. Nothing installs extensions at pod start. - **pi itself is in the image** (annia-infra#310) — the plan's runtime `npm install -g` goes away; only the *state and extensions* come from the folder. - This makes `/root/.pi` depend on **milner#21 shipping**, so #21 moves onto this issue's critical path. ### 4. No Vault path for pi's credentials — a documented standing exception The folder carries pi's provider keys (`agent/auth.json`, `models.json`'s `lain.apiKey`), so there is **no `secret/annia/pi*` path, no `VaultStaticSecret` for pi, and no template rendering**. The `~/.pi` mount is the source of truth. `AGENTS.md`'s "Secrets are Vault-first" therefore gains a **standing exception**, written down where it is used (the way the Tailscale operator's OAuth client is) — the operator has confirmed this, and the exception sentence is part of this change. The in-cluster provider is a **named provider** in the operator's own `models.json` — `lain-milner-external` → `http://llm-gateway.annia.svc.cluster.local:4000`, same shape as the existing `lain`, pointing at the already-landed `llm-gateway` Service. `lain-llm.tailc23d6.ts.net` still never appears in-cluster. Selecting that provider in `.annia.toml` is a later test, not part of this change (today's default is deepseek). ### 5. `environments/*.env` come from Vault `secret/annia/env` (one field per file, the field name *is* the file name) synced by `VaultStaticSecret` `annia-repo-env` into a Secret, mounted **read-only at `/workspace/environments`** — so the two files arrive at exactly the paths `.annia.toml` names, with no templates and no policy change (`annia-read` already covers `secret/data/annia/*`). A missing file now **fails** instead of being skipped (annia-infra#310 flips the crate's behaviour and its test). ### 6. `FORGEJO_TOKEN` access is missing today and is owed here `secret/ci/forgejo` → `FORGEJO_TOKEN` exists and is byte-identical to `secret/forgejo/token`, but the pod has **no access**: no `VaultStaticSecret`, no env entry, and the `annia` Vault role covers only `secret/data/annia/*`. Add the access — `secretKeyRef` from a synced Secret (env is what the skills read) — with the same named-path policy argument the clone key already carries. ### 7. The `annia-nix` claim is provisional Under the **layered store** (`/nix/store` = image lower + shared volume upper), the consumer's claim and its `/nix` mount stay, but the backing directory becomes the *merge point* the node-side assembler owns rather than the whole store. The A-shaped claim (`storageClassName: nix` at `/nix`) must **not** be copied forward — it is re-derived by the new issue *Convert the shared nix store to a layered store*, which also deletes the hand-run seed Job and the `docs/storage.md` §10g section that papered over the empty store. ### 8. Unchanged The Flux `ImageRepository`/`ImagePolicy`/`ImageUpdateAutomation` and the write source; `ImagePolicy` `semver: { range: ">=0.0.0" }` (the tag the pipeline mints, `<major>.<minor>.<count>`, compares as semver — `:sha-*` tags are not semver and are correctly ignored, `:latest` is never published); the clone-key `VaultStaticSecret` and the named `path "secret/data/ci/milner"` policy widen with `CLONE_KEY_PATH=secret/ci/milner`; the KEDA `ScaledJob` (`maxReplicaCount: 1`, `backoffLimit: 0`, `queueName: annia.dispatch`, the `ssa: Merge` annotation, the pause-lever criteria); `securityContext` unset (`annia` is `baseline`, the image is root); the hand-run `flux-write-deploy-key` prerequisite in `OPERATIONS.md`. The verification list's `/nix`, `/workspace`, `/root/.pi` line becomes `/nix` (the merged store) and `/root/.pi` (the synced folder), with `/workspace` carrying no claim. ### Dependencies - **Blocked by annia-infra#310** for every live criterion (no annia image is published today) — unchanged. - **Now also blocked by milner#21** for `/root/.pi`, and by the layered-store issue for what `/nix` is.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/109-nix-base
feat/109-layered-nix-store
feat/110-zot-push-credential
feat/90-annia-pause-lever-oz-console
feat/87-tailscale-egress-from-cluster
chore/ci-step-resources
docs/vault-secret-prompt
feat/42-support-backups-persistent-data
feat/41-integrate-restic-tfstate-backup
feat/3-setting-up-agent-development
kvm-kubernetes-setup
No results found.
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner#25
No description provided.