tepichord/milner

Fork 0

Add the annia pause lever to the oz console #90

New issue

Open

opened 2026-09-22 14:12:20 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-22 14:12:20 -07:00

Owner

   This answers the 2026-09-15 ask on #25: a crash-looping annia pod must not spin up
   forever, and there has to be a way to clear it **from a browser**. Two halves answer it,
   and this issue is the second one.

   - **The loop stops by itself.** annia's queue is a quorum queue with
     `x-delivery-limit: 2` and a dead-letter exchange (declared in the `data` layer by the
     enqueuer issue), so two failed deliveries move the message to the dead-letter queue and
     the trigger goes quiet. No failure bookkeeping in annia, and no human needed to stop the
     spinning.
   - **The human gets a lever.** This issue: a control in `oz` that writes KEDA's
     `autoscaling.keda.sh/paused` annotation on annia's ScaledJob, so the remaining decision
     — "should this run again?" — is one press from a browser.

   ## Why the console and not Headlamp

   Headlamp is deployed and can patch the annotation, but the press is not one press there:
   it is opening Headlamp, finding annia's namespace, opening the ScaledJob, editing YAML to
   add an annotation, saving, deleting the still-running Job, and later editing that YAML
   again to remove it. The ask was a method that works from a browser, and that is a manifest
   edit, not one. `oz` is already milner's browser surface — behind Pocket ID, tailnet-served,
   with its own service account — and this is one more control on it. Headlamp stays deployed
   as the general fallback for anything unusual.

   ## The mechanism, precisely

   - **Pause:** set `metadata.annotations["autoscaling.keda.sh/paused"] = "true"` on annia's
     ScaledJob. KEDA then stops creating new Jobs for it.
   - **Resume: remove the annotation.** KEDA's own ScaledJob documentation allows removing it
     *or* setting it to `"false"`, but setting `"false"` had a documented history of leaving
     the object paused — it is the subject of kedacore/keda#5215 and was fixed by
     kedacore/keda#5257, whose title says it: *"paused annotation being set to false still
     leads to scaled objects/jobs being paused"*. Removing the key works in every version, so
     that is what the control does.
   - **Read the annotation, not the status.** The ScaledJob's `Paused` *condition* has been
     reported not to flip back to `False` when the annotation is removed (kedacore/keda#5251,
     "ScaledJob Paused Condition doesn't change to False when removing annotation"). The page
     shows the annotation's presence, and the press toggles that annotation. Nothing else is
     stateful, so a stale condition cannot make the page lie about what it will do.
   - **A running Job is not killed by pausing.** For a KEDA Job the useful default is
     `jobTargetRef.backoffLimit: 0` with `restartPolicy: Never`, so a crashing pod ends its
     own Job rather than retrying inside it, and pausing is enough to stop the loop. **Check
     what #25's ScaledJob actually declares** and say so in this issue's plan: if the
     `backoffLimit` is not 0, a crash loop continues inside the current Job and pausing must
     be paired with something that stops it — and that would need a second grant, which is
     worth knowing before it is granted.
   - **One constant for the ScaledJob's name and namespace.** #25 names the ScaledJob; this
     page reads it from a single constant so a rename is one edit, not a search.

   ## Where the code goes

   The console's existing shape is one page plus POST endpoints behind CSRF and an auth
   check, each with a preview-build branch:

   | File | Change |
   |---|---|
   | `oz/src/routes/api/pause/+server.ts` | New POST handler, modelled on `api/delete/+server.ts`: preview-build branch first, then auth + CSRF, then the patch |
   | `oz/src/routes/+page.server.ts`, `+page.svelte` | A new section, following the existing rows/notice pattern: annia's ScaledJob, its state, and the press |
   | `oz/src/lib/k8s.ts` | A read of the one ScaledJob and an annotation patch — `get`, `patch` and `request` already exist; `request` takes any resource path, so
 this is likely no new method at all |
   | `oz/src/lib/preview-mode.ts` | The simulated state gains the annia row, and the simulated action log gains `pause` |

   **The preview-build branch is not optional.** A preview namespace holds no cluster grant
   at all, and the previewed console is the fixture build (`/app/PREVIEW-BUILD`): the press
   must be logged and simulated there, never performed. `isPreviewBuild()` returns early
   before the auth check in every existing endpoint for the same reason.

   ## The grant

   A Role in annia's namespace:

   ```yaml
   apiVersion: rbac.authorization.k8s.io/v1
   kind: Role
   metadata:
     name: preview-console-annia-pause
     namespace: annia
   rules:
     - apiGroups: ["keda.sh"]
       resources: ["scaledjobs"]
       verbs: ["get", "patch"]
   ---
   apiVersion: rbac.authorization.k8s.io/v1
   kind: RoleBinding
   metadata:
     name: preview-console-annia-pause
     namespace: annia
   roleRef:
     apiGroup: rbac.authorization.k8s.io
     kind: Role
     name: preview-console-annia-pause
   subjects:
     - kind: ServiceAccount
       name: preview-console
       namespace: preview-system
   ```

   - **It goes in `gitops/clusters/milner/preview-system/rbac.yaml`**, beside the console's
     existing identity — that file is where the console's boundary is argued, and its comment
     already makes this exact point: *"`patch` and not `delete` … makes the boundary a
     property of the grant rather than of the handler."* A second grant filed elsewhere would
     split the boundary across two files. The layer order is fine: `infrastructure` creates
     annia's namespace before `preview-system` runs.
   - **`get` and `patch` only.** No `delete` (so the console cannot delete a ScaledJob or its
     Jobs), no `list` or `watch` (the page reads one object by name), and no verbs on
     `jobs` or `pods` — the console stops the loop by pausing, not by killing.
   - The console's cluster-wide `preview-console` ClusterRole is untouched. `scaledjobs` is
     namespaced, so this cannot be widened by accident.

   ## Tests and gates

   - `oz/src/lib/k8s.test.ts` and a new test for the endpoint's preview-build branch: the
     simulated press logs and patches nothing.
   - The console's unit tests run inside the Nix build (`oz/flake.nix`'s `oz-test`,
     `deno task test`), so `nix build` is the local check.
   - `ci/oz-tag-guard.sh`: the console's pinned tag must move with its source, and the
     main-push build re-publishes `milner/oz:<pin>` — a console change that does not move the
     tag ships a console that does not have the page.
   - The RBAC files are covered by `kustomize build` + `kubeconform` for the
     `preview-system` layer and by `conftest test --policy policy/`.

   ## Done when

   ```bash
   # The platform can do exactly this and nothing more:
   kubectl auth can-i patch scaledjobs.keda.sh -n annia \
     --as=system:serviceaccount:preview-system:preview-console     # yes
   kubectl auth can-i delete scaledjobs.keda.sh -n annia \
     --as=system:serviceaccount:preview-system:preview-console     # no
   kubectl auth can-i patch scaledjobs.keda.sh -n rabbitmq \
     --as=system:serviceaccount:preview-system:preview-console     # no
   ```

   And by hand, from `oz.*`: annia's ScaledJob shows as running; one press adds the
   annotation and the page shows it paused; a second press removes the annotation and the
   page shows it running. In a preview of the console, the press is logged and the annotation
   is unchanged.

   Documentation: `docs/preview-environment.md` §11h's console paragraph gains this second
   write and its grant, in the same breath as the expiry patch it already describes.
   `OPERATIONS.md` needs no row — this is Flux-owned cluster state.

````markdown This answers the 2026-09-15 ask on #25: a crash-looping annia pod must not spin up forever, and there has to be a way to clear it **from a browser**. Two halves answer it, and this issue is the second one. - **The loop stops by itself.** annia's queue is a quorum queue with `x-delivery-limit: 2` and a dead-letter exchange (declared in the `data` layer by the enqueuer issue), so two failed deliveries move the message to the dead-letter queue and the trigger goes quiet. No failure bookkeeping in annia, and no human needed to stop the spinning. - **The human gets a lever.** This issue: a control in `oz` that writes KEDA's `autoscaling.keda.sh/paused` annotation on annia's ScaledJob, so the remaining decision — "should this run again?" — is one press from a browser. ## Why the console and not Headlamp Headlamp is deployed and can patch the annotation, but the press is not one press there: it is opening Headlamp, finding annia's namespace, opening the ScaledJob, editing YAML to add an annotation, saving, deleting the still-running Job, and later editing that YAML again to remove it. The ask was a method that works from a browser, and that is a manifest edit, not one. `oz` is already milner's browser surface — behind Pocket ID, tailnet-served, with its own service account — and this is one more control on it. Headlamp stays deployed as the general fallback for anything unusual. ## The mechanism, precisely - **Pause:** set `metadata.annotations["autoscaling.keda.sh/paused"] = "true"` on annia's ScaledJob. KEDA then stops creating new Jobs for it. - **Resume: remove the annotation.** KEDA's own ScaledJob documentation allows removing it *or* setting it to `"false"`, but setting `"false"` had a documented history of leaving the object paused — it is the subject of kedacore/keda#5215 and was fixed by kedacore/keda#5257, whose title says it: *"paused annotation being set to false still leads to scaled objects/jobs being paused"*. Removing the key works in every version, so that is what the control does. - **Read the annotation, not the status.** The ScaledJob's `Paused` *condition* has been reported not to flip back to `False` when the annotation is removed (kedacore/keda#5251, "ScaledJob Paused Condition doesn't change to False when removing annotation"). The page shows the annotation's presence, and the press toggles that annotation. Nothing else is stateful, so a stale condition cannot make the page lie about what it will do. - **A running Job is not killed by pausing.** For a KEDA Job the useful default is `jobTargetRef.backoffLimit: 0` with `restartPolicy: Never`, so a crashing pod ends its own Job rather than retrying inside it, and pausing is enough to stop the loop. **Check what #25's ScaledJob actually declares** and say so in this issue's plan: if the `backoffLimit` is not 0, a crash loop continues inside the current Job and pausing must be paired with something that stops it — and that would need a second grant, which is worth knowing before it is granted. - **One constant for the ScaledJob's name and namespace.** #25 names the ScaledJob; this page reads it from a single constant so a rename is one edit, not a search. ## Where the code goes The console's existing shape is one page plus POST endpoints behind CSRF and an auth check, each with a preview-build branch: | File | Change | |---|---| | `oz/src/routes/api/pause/+server.ts` | New POST handler, modelled on `api/delete/+server.ts`: preview-build branch first, then auth + CSRF, then the patch | | `oz/src/routes/+page.server.ts`, `+page.svelte` | A new section, following the existing rows/notice pattern: annia's ScaledJob, its state, and the press | | `oz/src/lib/k8s.ts` | A read of the one ScaledJob and an annotation patch — `get`, `patch` and `request` already exist; `request` takes any resource path, so this is likely no new method at all | | `oz/src/lib/preview-mode.ts` | The simulated state gains the annia row, and the simulated action log gains `pause` | **The preview-build branch is not optional.** A preview namespace holds no cluster grant at all, and the previewed console is the fixture build (`/app/PREVIEW-BUILD`): the press must be logged and simulated there, never performed. `isPreviewBuild()` returns early before the auth check in every existing endpoint for the same reason. ## The grant A Role in annia's namespace: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: preview-console-annia-pause namespace: annia rules: - apiGroups: ["keda.sh"] resources: ["scaledjobs"] verbs: ["get", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: preview-console-annia-pause namespace: annia roleRef: apiGroup: rbac.authorization.k8s.io kind: Role name: preview-console-annia-pause subjects: - kind: ServiceAccount name: preview-console namespace: preview-system ``` - **It goes in `gitops/clusters/milner/preview-system/rbac.yaml`**, beside the console's existing identity — that file is where the console's boundary is argued, and its comment already makes this exact point: *"`patch` and not `delete` … makes the boundary a property of the grant rather than of the handler."* A second grant filed elsewhere would split the boundary across two files. The layer order is fine: `infrastructure` creates annia's namespace before `preview-system` runs. - **`get` and `patch` only.** No `delete` (so the console cannot delete a ScaledJob or its Jobs), no `list` or `watch` (the page reads one object by name), and no verbs on `jobs` or `pods` — the console stops the loop by pausing, not by killing. - The console's cluster-wide `preview-console` ClusterRole is untouched. `scaledjobs` is namespaced, so this cannot be widened by accident. ## Tests and gates - `oz/src/lib/k8s.test.ts` and a new test for the endpoint's preview-build branch: the simulated press logs and patches nothing. - The console's unit tests run inside the Nix build (`oz/flake.nix`'s `oz-test`, `deno task test`), so `nix build` is the local check. - `ci/oz-tag-guard.sh`: the console's pinned tag must move with its source, and the main-push build re-publishes `milner/oz:<pin>` — a console change that does not move the tag ships a console that does not have the page. - The RBAC files are covered by `kustomize build` + `kubeconform` for the `preview-system` layer and by `conftest test --policy policy/`. ## Done when ```bash # The platform can do exactly this and nothing more: kubectl auth can-i patch scaledjobs.keda.sh -n annia \ --as=system:serviceaccount:preview-system:preview-console # yes kubectl auth can-i delete scaledjobs.keda.sh -n annia \ --as=system:serviceaccount:preview-system:preview-console # no kubectl auth can-i patch scaledjobs.keda.sh -n rabbitmq \ --as=system:serviceaccount:preview-system:preview-console # no ``` And by hand, from `oz.*`: annia's ScaledJob shows as running; one press adds the annotation and the page shows it paused; a second press removes the annotation and the page shows it running. In a preview of the console, the press is logged and the annotation is unchanged. Documentation: `docs/preview-environment.md` §11h's console paragraph gains this second write and its grant, in the same breath as the expiry patch it already describes. `OPERATIONS.md` needs no row — this is Flux-owned cluster state. ````

tepichord added the

labels

2026-09-22 14:12:20 -07:00

tepichord added this to the MVP project

2026-09-22 14:12:20 -07:00

tepichord commented

2026-09-22 15:12:52 -07:00

Author

Owner

Naina:

Overview

Annia's annia-wakeup ScaledJob can crash-loop, and the only lever today is opening Headlamp and editing YAML. This adds a one-press pause/resume to the oz console: a section that reads autoscaling.keda.sh/paused on the ScaledJob, and a POST button that sets it with "true" or removes it. The console's identity gains get/patch on scaledjobs.keda.sh in annia — and nothing else. Existing identity verified: the console runs as ServiceAccount preview-console in preview-system and today holds read verbs plus one patch verb (on namespaces) and no delete⁹.

Technical Approach

New module oz/src/lib/annia.ts — one constant for the target, one for the annotation.

export const ANNIA_NAMESPACE = 'annia';
export const ANNIA_SCALEDJOB = 'annia-wakeup';
export const PAUSED_ANNOTATION = 'autoscaling.keda.sh/paused';
const SCALEDJOB_PATH = `/apis/keda.sh/v1alpha1/namespaces/${ANNIA_NAMESPACE}/scaledjobs/${ANNIA_SCALEDJOB}`;

autoscaling.keda.sh/paused is KEDA's own PausedAnnotation constant¹.

Read with request() — get() hard-codes the /api/v1 prefix and cannot reach a CRD^2,2 (get() at #L134, request() at #L103, same file).

export async function readPaused(client: KubernetesClient): Promise<boolean> {
	return isPausedByAnnotation(await client.request('GET', SCALEDJOB_PATH));
}

Mirror KEDA's own parse, not a truthiness test — NeedToBePausedByAnnotation treats a present-but-unparseable value as paused³.

const RESUME_VALUES = new Set(['0', 'f', 'false']);
export function isPausedByAnnotation(sj: Record<string, unknown>): boolean {
	const value = (sj.metadata as any)?.annotations?.[PAUSED_ANNOTATION];
	if (value === undefined || value === null) return false;
	return !RESUME_VALUES.has(String(value).toLowerCase());
}

Patch to the opposite of what the server just read — string "true" to pause, JSON null to remove the key⁴.

const patch = paused
	? { metadata: { annotations: { [PAUSED_ANNOTATION]: null } } }
	: { metadata: { annotations: { [PAUSED_ANNOTATION]: 'true' } } };
await client.request('PATCH', SCALEDJOB_PATH, new TextEncoder().encode(JSON.stringify(patch)));

New POST handler oz/src/routes/api/pause/+server.ts, same order as api/delete/+server.ts⁵:

if (isPreviewBuild()) {
	applySimulated(simulatedState(), 'pause', '');
	console.log('preview mode: would toggle the annia pause annotation');
	throw redirect(303, '/?preview=pause');
}
if (!locals.user || !csrfMatches(locals.user, String(form.get('csrf') || ''))) {
	return new Response(null, { status: 403 });
}
const client = new KubernetesClient();
try {
	const paused = await readPaused(client);
	await setPaused(client, !paused);
	return new Response(null, { status: 302, headers: { Location: `/?paused=${paused ? '0' : '1'}` } });
} catch (error: unknown) {
	return htmlError('Rejected', error, 400);
}

Page loader — preview mode supplies the fixture; real mode reads the cluster and never 500s on a missing CRD or a grant that has not reconciled yet⁶.

// every return in +page.server.ts gains this field, including the `!user` early return
let annia: { paused: boolean | null } = { paused: null };
try {
	annia = { paused: await readPaused(client) };
} catch {
	// CRD absent, or the Role has not reconciled: the section says "unknown"
}
return { rows, hostname, csrf: user.csrf, preview: false, annia };

// preview branch
notice: action === 'pause'
	? 'Simulated pause toggle: the page reflects it, the cluster does not, and the action was logged to the pod.'
	: ['activate', 'delete', 'build', 'front-door'].includes(action) ? `Simulated ${action} on PR ${pr}: …` : ''

Simulated state gains paused, handled before applySimulated's numeric-PR guard⁷:

export interface SimulatedState { /* … */ paused: boolean; }
export function applySimulated(state: SimulatedState, action: string, pr: string, app = ''): void {
	if (action === 'pause') { state.paused = !state.paused; return; }
	if (!/^\d+$/.test(pr)) return;
	// …
}
// resetSimulatedState(): _simulated.paused = false;

The page section — a <section> outside {#if data.rows}, rendered in both modes⁸ (the template's guard is {#if data.rows && data.rows.length > 0}):

<section class="annia">
	<h2>annia</h2>
	<p>ScaledJob <code>annia-wakeup</code>:
		{data.annia.paused === null ? 'unknown' : data.annia.paused ? 'paused' : 'running'}</p>
	<form method="post" action="/api/pause">
		<input type="hidden" name="csrf" value={data.csrf ?? ''} />
		<Button.Root type="submit" data-button-root>{data.annia.paused ? 'Resume' : 'Pause'}</Button.Root>
	</form>
</section>

The grant, in gitops/clusters/milner/preview-system/rbac.yaml beside the existing console Role:

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: preview-console-annia-pause
  namespace: annia
rules:
  - apiGroups: ["keda.sh"]
    resources: ["scaledjobs"]
    verbs: ["get", "patch"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: preview-console-annia-pause
  namespace: annia
roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: preview-console-annia-pause }
subjects:
  - { kind: ServiceAccount, name: preview-console, namespace: preview-system }

The preview-console ServiceAccount and its namespace are as the plan states⁹. The annia namespace this Role targets does not exist in this repo yet — there is no apps/annia/ directory and no namespace: annia manifest — so the Role can only reconcile once #25 lands¹².

Move the console tag — preview-system/oz.yaml 0.1.7 → 0.1.8, and its comment, or ci/oz-tag-guard.sh fails the pull request^10,11.

Implementation Details

File	Change
`oz/src/lib/annia.ts`	new — path + annotation constants, `isPausedByAnnotation`, `readPaused`, `setPaused`
`oz/src/lib/annia.test.ts`	new — parse cases and patch bodies against a fake `Transport`
`oz/src/routes/api/pause/+server.ts`	new POST handler
`oz/src/routes/+page.server.ts`	`annia` field in every return; `pause` in the preview notice
`oz/src/routes/+page.svelte`	the annia section, above the table
`oz/src/lib/preview-mode.ts`	`paused` in `SimulatedState`, `pause` branch, reset
`gitops/clusters/milner/preview-system/rbac.yaml`	Role + RoleBinding `preview-console-annia-pause`
`gitops/clusters/milner/preview-system/oz.yaml`	tag `0.1.7` → `0.1.8`
`docs/preview-environment.md` §11h	the console paragraph gains this second write and its grant¹⁵
`tests/test_static.py`	`test_preview_console_rbac_is_bounded` covers the annia Role¹⁴

Dependencies: none new.

Constraints:

get/patch only — no delete, list, or watch, and no verbs on jobs or pods.
Preview mode never constructs a KubernetesClient; the press logs and simulates.
Annotation values are strings; resume is null under application/merge-patch+json (RFC 7386).
The page reads the annotation, never status.conditions[Paused].
Tests run inside the Nix build (nix build ./oz#test)¹⁶.

Out-of-scope notes:

Flagged for the human reviewer: backoffLimit cannot be verified. annia-wakeup does not exist in this repo — #25 is unmerged¹² — and as #25 is planned it declares restartPolicy: Never but omits backoffLimit, which then defaults to 6¹³.

Verified at check-assumptions time: annia-wakeup appears nowhere in this tree, #25 is open, and #25's planning comment (id 9029) declares restartPolicy: Never and omits backoffLimit under jobTargetRef.template.spec — so the default of 6 applies^12,13. Pausing stops KEDA's scale loop but never kills a running Job, so a crash loop keeps retrying inside the current Job until backoffLimit is reached. The issue's fix is one line in #25's scaledjob.yaml:

jobTargetRef:
  backoffLimit: 0

If #25 ships without it, stopping the in-flight Job needs a second grant (delete on jobs.batch in annia) and UI this issue deliberately does not add. annia.ts carries a comment recording both.

The Paused condition is not shown: it needs no extra RBAC but must never decide the button's target state.

Acceptance Criteria

By hand from oz.*: the section reads running; one press adds the annotation and shows paused; a second press removes it and shows running.
In a preview build the press logs and patches nothing, and the fixture state flips instead.
The grant is exactly this and no more:

kubectl auth can-i patch scaledjobs.keda.sh -n annia \
  --as=system:serviceaccount:preview-system:preview-console     # yes
kubectl auth can-i delete scaledjobs.keda.sh -n annia \
  --as=system:serviceaccount:preview-system:preview-console     # no
kubectl auth can-i patch scaledjobs.keda.sh -n rabbitmq \
  --as=system:serviceaccount:preview-system:preview-console     # no

nix build ./oz#test, python3 -m pytest tests/test_static.py, and ci/oz-tag-guard.sh all pass with the tag at 0.1.8.
docs §11h names the annia grant in the same paragraph as the expiry patch.

Testing Plan

annia.test.ts — isPausedByAnnotation: absent → false; "true"/"1" → true; "false"/"0"/"f" → false; ""/"yes" → true (KEDA's fail-safe).
annia.test.ts — fake transport asserts the path /apis/keda.sh/v1alpha1/namespaces/annia/scaledjobs/annia-wakeup and method GET for the read, PATCH for the write.
annia.test.ts — pause body carries "autoscaling.keda.sh/paused": "true" (a string); resume body carries null, and the client sends application/merge-patch+json.
preview-mode.test.ts — applySimulated(state, 'pause', '') flips paused and leaves active/expired/built/frontDoor untouched; resetSimulatedState() clears it.
tests/test_static.py — the annia Role's rules are exactly keda.sh/scaledjobs with {get, patch}, and preview-console still holds no delete anywhere.
Manual drill: press in the preview build, then press against the cluster with kubectl get scaledjob -n annia annia-wakeup -o jsonpath='{.metadata.annotations}' before and after.

#25 — the annia-wakeup ScaledJob this page reads, and the backoffLimit: 0 line it needs.
kedacore/keda#5215, kedacore/keda#5257 — "false" resume was broken; this removes the key instead.
kedacore/keda#5251 — the stale Paused condition; why the annotation is the source of truth.
#49 — the preview-console identity and the no-delete boundary this grant joins.

Research Findings

The CRD path is /apis/keda.sh/v1alpha1/namespaces/annia/scaledjobs/annia-wakeup; KubernetesClient.get()/patch() cannot reach it (they prefix /api/v1), while request() can.
In KEDA 2.20.x both removing the annotation and setting "false" resume (fixed by #5257, shipped 2.13.0); removal is chosen because it works on every version.
A Kubernetes annotation value must be a string, so pause is "true"; resume is null in a merge patch, which removes the key (RFC 7386).
KEDA's Paused condition does flip back to False in 2.20.x, but drift reports persisted through March 2025, so the page treats the condition as untrusted decoration.
Pausing stops only KEDA's scale loop — checkIfPaused returns before deletePreviousVersionScaleJobs — so a running Job is never killed, and with the default backoffLimit: 6 a crash loop continues inside it.

Pre-Mortem

The pause lever does not actually stop the crash loop it was built for — #25 is still unmerged and its planned ScaledJob omits backoffLimit (default 6), so pressing Pause halts KEDA's scale loop while the in-flight Job keeps retrying. Unless backoffLimit: 0 lands in #25 or this issue adds the second jobs.batch grant it deliberately excludes, the feature ships green and useless.
The RBAC grant cannot reconcile or be verified, so it is never exercised — the annia namespace itself does not exist in this repo yet, so the Role/RoleBinding have nothing to bind to and Flux either errors or silently applies a dead grant. The three kubectl auth can-i acceptance checks then cannot pass either, so nobody notices the console's get/patch path was never proven.
One +page.server.ts return path is missed and the page 500s in a state no test covers — the plan requires the annia field on every return including the !user early return, and the page dereferences data.annia.paused unconditionally. A missed branch, or forgetting the applySimulated numeric-PR carve-out for preview mode, breaks the console page while the new unit tests still pass.

Naina: ## Overview Annia's `annia-wakeup` ScaledJob can crash-loop, and the only lever today is opening Headlamp and editing YAML. This adds a one-press pause/resume to the oz console: a section that reads `autoscaling.keda.sh/paused` on the ScaledJob, and a POST button that sets it with `"true"` or removes it. The console's identity gains `get`/`patch` on `scaledjobs.keda.sh` in `annia` — and nothing else. Existing identity verified: the console runs as ServiceAccount `preview-console` in `preview-system` and today holds read verbs plus one `patch` verb (on `namespaces`) and no `delete`<sup>[9](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/preview-system/rbac.yaml?display=source#L217)</sup>. ## Technical Approach 1. **New module `oz/src/lib/annia.ts`** — one constant for the target, one for the annotation. ```ts export const ANNIA_NAMESPACE = 'annia'; export const ANNIA_SCALEDJOB = 'annia-wakeup'; export const PAUSED_ANNOTATION = 'autoscaling.keda.sh/paused'; const SCALEDJOB_PATH = `/apis/keda.sh/v1alpha1/namespaces/${ANNIA_NAMESPACE}/scaledjobs/${ANNIA_SCALEDJOB}`; ``` `autoscaling.keda.sh/paused` is KEDA's own `PausedAnnotation` constant<sup>[1](https://github.com/kedacore/keda/blob/v2.20.0/apis/keda/v1alpha1/scaledobject_types.go#L59)</sup>. 2. **Read with `request()`** — `get()` hard-codes the `/api/v1` prefix and cannot reach a CRD<sup>[2](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/oz/src/lib/k8s.ts?display=source#L134),[2](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/oz/src/lib/k8s.ts?display=source#L103)</sup> (`get()` at `#L134`, `request()` at `#L103`, same file). ```ts export async function readPaused(client: KubernetesClient): Promise<boolean> { return isPausedByAnnotation(await client.request('GET', SCALEDJOB_PATH)); } ``` 3. **Mirror KEDA's own parse, not a truthiness test** — `NeedToBePausedByAnnotation` treats a present-but-unparseable value as paused<sup>[3](https://github.com/kedacore/keda/blob/v2.20.0/apis/keda/v1alpha1/scaledjob_types.go#L174-L184)</sup>. ```ts const RESUME_VALUES = new Set(['0', 'f', 'false']); export function isPausedByAnnotation(sj: Record<string, unknown>): boolean { const value = (sj.metadata as any)?.annotations?.[PAUSED_ANNOTATION]; if (value === undefined || value === null) return false; return !RESUME_VALUES.has(String(value).toLowerCase()); } ``` 4. **Patch to the opposite of what the server just read** — string `"true"` to pause, JSON `null` to remove the key<sup>[4](https://www.rfc-editor.org/rfc/rfc7386.html#section-2)</sup>. ```ts const patch = paused ? { metadata: { annotations: { [PAUSED_ANNOTATION]: null } } } : { metadata: { annotations: { [PAUSED_ANNOTATION]: 'true' } } }; await client.request('PATCH', SCALEDJOB_PATH, new TextEncoder().encode(JSON.stringify(patch))); ``` 5. **New POST handler `oz/src/routes/api/pause/+server.ts`**, same order as `api/delete/+server.ts`<sup>[5](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/oz/src/routes/api/delete/+server.ts?display=source#L20)</sup>: ```ts if (isPreviewBuild()) { applySimulated(simulatedState(), 'pause', ''); console.log('preview mode: would toggle the annia pause annotation'); throw redirect(303, '/?preview=pause'); } if (!locals.user || !csrfMatches(locals.user, String(form.get('csrf') || ''))) { return new Response(null, { status: 403 }); } const client = new KubernetesClient(); try { const paused = await readPaused(client); await setPaused(client, !paused); return new Response(null, { status: 302, headers: { Location: `/?paused=${paused ? '0' : '1'}` } }); } catch (error: unknown) { return htmlError('Rejected', error, 400); } ``` 6. **Page loader** — preview mode supplies the fixture; real mode reads the cluster and never 500s on a missing CRD or a grant that has not reconciled yet<sup>[6](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/oz/src/routes/+page.server.ts?display=source#L31)</sup>. ```ts // every return in +page.server.ts gains this field, including the `!user` early return let annia: { paused: boolean | null } = { paused: null }; try { annia = { paused: await readPaused(client) }; } catch { // CRD absent, or the Role has not reconciled: the section says "unknown" } return { rows, hostname, csrf: user.csrf, preview: false, annia }; ``` ```ts // preview branch notice: action === 'pause' ? 'Simulated pause toggle: the page reflects it, the cluster does not, and the action was logged to the pod.' : ['activate', 'delete', 'build', 'front-door'].includes(action) ? `Simulated ${action} on PR ${pr}: …` : '' ``` 7. **Simulated state gains `paused`**, handled before `applySimulated`'s numeric-PR guard<sup>[7](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/oz/src/lib/preview-mode.ts?display=source#L178)</sup>: ```ts export interface SimulatedState { /* … */ paused: boolean; } export function applySimulated(state: SimulatedState, action: string, pr: string, app = ''): void { if (action === 'pause') { state.paused = !state.paused; return; } if (!/^\d+$/.test(pr)) return; // … } // resetSimulatedState(): _simulated.paused = false; ``` 8. **The page section** — a `<section>` outside `{#if data.rows}`, rendered in both modes<sup>[8](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/oz/src/routes/+page.svelte?display=source#L40)</sup> (the template's guard is `{#if data.rows && data.rows.length > 0}`): ```svelte <section class="annia"> <h2>annia</h2> <p>ScaledJob <code>annia-wakeup</code>: {data.annia.paused === null ? 'unknown' : data.annia.paused ? 'paused' : 'running'}</p> <form method="post" action="/api/pause"> <input type="hidden" name="csrf" value={data.csrf ?? ''} /> <Button.Root type="submit" data-button-root>{data.annia.paused ? 'Resume' : 'Pause'}</Button.Root> </form> </section> ``` 9. **The grant**, in `gitops/clusters/milner/preview-system/rbac.yaml` beside the existing console Role: ```yaml apiVersion: rbac.authorization.k8s.io/v1 kind: Role metadata: name: preview-console-annia-pause namespace: annia rules: - apiGroups: ["keda.sh"] resources: ["scaledjobs"] verbs: ["get", "patch"] --- apiVersion: rbac.authorization.k8s.io/v1 kind: RoleBinding metadata: name: preview-console-annia-pause namespace: annia roleRef: { apiGroup: rbac.authorization.k8s.io, kind: Role, name: preview-console-annia-pause } subjects: - { kind: ServiceAccount, name: preview-console, namespace: preview-system } ``` The `preview-console` ServiceAccount and its namespace are as the plan states<sup>[9](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/preview-system/rbac.yaml?display=source#L217)</sup>. The `annia` namespace this Role targets does not exist in this repo yet — there is no `apps/annia/` directory and no `namespace: annia` manifest — so the Role can only reconcile once #25 lands<sup>[12](https://git.gloom.garden/tepichord/milner/issues/25#issuecomment-9029)</sup>. 10. **Move the console tag** — `preview-system/oz.yaml` `0.1.7` → `0.1.8`, and its comment, or `ci/oz-tag-guard.sh` fails the pull request<sup>[10](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/preview-system/oz.yaml?display=source#L75),[11](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/ci/oz-tag-guard.sh?display=source#L104)</sup>. ## Implementation Details | File | Change | |---|---| | `oz/src/lib/annia.ts` | new — path + annotation constants, `isPausedByAnnotation`, `readPaused`, `setPaused` | | `oz/src/lib/annia.test.ts` | new — parse cases and patch bodies against a fake `Transport` | | `oz/src/routes/api/pause/+server.ts` | new POST handler | | `oz/src/routes/+page.server.ts` | `annia` field in every return; `pause` in the preview notice | | `oz/src/routes/+page.svelte` | the annia section, above the table | | `oz/src/lib/preview-mode.ts` | `paused` in `SimulatedState`, `pause` branch, reset | | `gitops/clusters/milner/preview-system/rbac.yaml` | Role + RoleBinding `preview-console-annia-pause` | | `gitops/clusters/milner/preview-system/oz.yaml` | tag `0.1.7` → `0.1.8` | | `docs/preview-environment.md` §11h | the console paragraph gains this second write and its grant<sup>[15](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/docs/preview-environment.md?display=source#L446)</sup> | | `tests/test_static.py` | `test_preview_console_rbac_is_bounded` covers the annia Role<sup>[14](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/tests/test_static.py?display=source#L2055)</sup> | **Dependencies:** none new. **Constraints:** - `get`/`patch` only — no `delete`, `list`, or `watch`, and no verbs on `jobs` or `pods`. - Preview mode never constructs a `KubernetesClient`; the press logs and simulates. - Annotation values are strings; resume is `null` under `application/merge-patch+json` (RFC 7386). - The page reads the annotation, never `status.conditions[Paused]`. - Tests run inside the Nix build (`nix build ./oz#test`)<sup>[16](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/oz/flake.nix?display=source#L126)</sup>. **Out-of-scope notes:** - **Flagged for the human reviewer: `backoffLimit` cannot be verified.** `annia-wakeup` does not exist in this repo — #25 is unmerged<sup>[12](https://git.gloom.garden/tepichord/milner/issues/25#issuecomment-9029)</sup> — and as #25 is planned it declares `restartPolicy: Never` but omits `backoffLimit`, which then defaults to **6**<sup>[13](https://kubernetes.io/docs/concepts/workloads/controllers/job/)</sup>. Verified at check-assumptions time: `annia-wakeup` appears nowhere in this tree, #25 is open, and #25's planning comment (id 9029) declares `restartPolicy: Never` and omits `backoffLimit` under `jobTargetRef.template.spec` — so the default of 6 applies<sup>[12](https://git.gloom.garden/tepichord/milner/issues/25#issuecomment-9029),[13](https://kubernetes.io/docs/concepts/workloads/controllers/job/)</sup>. Pausing stops KEDA's scale loop but never kills a running Job, so a crash loop keeps retrying inside the current Job until `backoffLimit` is reached. The issue's fix is one line in #25's `scaledjob.yaml`: ```yaml jobTargetRef: backoffLimit: 0 ``` If #25 ships without it, stopping the in-flight Job needs a second grant (`delete` on `jobs.batch` in `annia`) and UI this issue deliberately does not add. `annia.ts` carries a comment recording both. - The `Paused` condition is not shown: it needs no extra RBAC but must never decide the button's target state. ## Acceptance Criteria - [ ] By hand from `oz.*`: the section reads *running*; one press adds the annotation and shows *paused*; a second press removes it and shows *running*. - [ ] In a preview build the press logs and patches nothing, and the fixture state flips instead. - [ ] The grant is exactly this and no more: ```bash kubectl auth can-i patch scaledjobs.keda.sh -n annia \ --as=system:serviceaccount:preview-system:preview-console # yes kubectl auth can-i delete scaledjobs.keda.sh -n annia \ --as=system:serviceaccount:preview-system:preview-console # no kubectl auth can-i patch scaledjobs.keda.sh -n rabbitmq \ --as=system:serviceaccount:preview-system:preview-console # no ``` - [ ] `nix build ./oz#test`, `python3 -m pytest tests/test_static.py`, and `ci/oz-tag-guard.sh` all pass with the tag at `0.1.8`. - [ ] docs §11h names the annia grant in the same paragraph as the expiry patch. ## Testing Plan - `annia.test.ts` — `isPausedByAnnotation`: absent → `false`; `"true"`/`"1"` → `true`; `"false"`/`"0"`/`"f"` → `false`; `""`/`"yes"` → `true` (KEDA's fail-safe). - `annia.test.ts` — fake transport asserts the path `/apis/keda.sh/v1alpha1/namespaces/annia/scaledjobs/annia-wakeup` and method `GET` for the read, `PATCH` for the write. - `annia.test.ts` — pause body carries `"autoscaling.keda.sh/paused": "true"` (a string); resume body carries `null`, and the client sends `application/merge-patch+json`. - `preview-mode.test.ts` — `applySimulated(state, 'pause', '')` flips `paused` and leaves `active`/`expired`/`built`/`frontDoor` untouched; `resetSimulatedState()` clears it. - `tests/test_static.py` — the annia Role's rules are exactly `keda.sh/scaledjobs` with `{get, patch}`, and `preview-console` still holds no `delete` anywhere. - Manual drill: press in the preview build, then press against the cluster with `kubectl get scaledjob -n annia annia-wakeup -o jsonpath='{.metadata.annotations}'` before and after. ## Related Issues/PRs - #25 — the `annia-wakeup` ScaledJob this page reads, and the `backoffLimit: 0` line it needs. - kedacore/keda#5215, kedacore/keda#5257 — `"false"` resume was broken; this removes the key instead. - kedacore/keda#5251 — the stale `Paused` condition; why the annotation is the source of truth. - #49 — the preview-console identity and the no-delete boundary this grant joins. ## Research Findings - The CRD path is `/apis/keda.sh/v1alpha1/namespaces/annia/scaledjobs/annia-wakeup`; `KubernetesClient.get()`/`patch()` cannot reach it (they prefix `/api/v1`), while `request()` can. - In KEDA 2.20.x both removing the annotation and setting `"false"` resume (fixed by #5257, shipped 2.13.0); removal is chosen because it works on every version. - A Kubernetes annotation value must be a string, so pause is `"true"`; resume is `null` in a merge patch, which removes the key (RFC 7386). - KEDA's `Paused` condition does flip back to `False` in 2.20.x, but drift reports persisted through March 2025, so the page treats the condition as untrusted decoration. - Pausing stops only KEDA's scale loop — `checkIfPaused` returns before `deletePreviousVersionScaleJobs` — so a running Job is never killed, and with the default `backoffLimit: 6` a crash loop continues inside it. ## Pre-Mortem 1. **The pause lever does not actually stop the crash loop it was built for** — #25 is still unmerged and its planned ScaledJob omits `backoffLimit` (default 6), so pressing Pause halts KEDA's scale loop while the in-flight Job keeps retrying. Unless `backoffLimit: 0` lands in #25 or this issue adds the second `jobs.batch` grant it deliberately excludes, the feature ships green and useless. 2. **The RBAC grant cannot reconcile or be verified, so it is never exercised** — the `annia` namespace itself does not exist in this repo yet, so the Role/RoleBinding have nothing to bind to and Flux either errors or silently applies a dead grant. The three `kubectl auth can-i` acceptance checks then cannot pass either, so nobody notices the console's `get`/`patch` path was never proven. 3. **One `+page.server.ts` return path is missed and the page 500s in a state no test covers** — the plan requires the `annia` field on *every* return including the `!user` early return, and the page dereferences `data.annia.paused` unconditionally. A missed branch, or forgetting the `applySimulated` numeric-PR carve-out for preview mode, breaks the console page while the new unit tests still pass.

tepichord added

and removed

labels

2026-09-22 15:17:46 -07:00

tepichord referenced this issue

2026-09-22 15:48:56 -07:00

Integrate KEDA into milner #85

tepichord referenced this issue from a commit