feat(preview): add a one-month catch-all to the reaper #64

Merged

tepichord merged 9 commits from fix/reaper-collects-any-preview into main 2026-09-16 22:20:24 -07:00

Conversation 0 Commits 9 Files changed 21 +1044 -452

tepichord commented 2026-09-16 14:38:10 -07:00

Owner

Copy link

The reaper's existing pass is untouched: a namespace selected by
milner.dev/preview-pr whose milner.dev/preview-expires has passed is
collected, and every object is deleted by that label. A non-numeric or missing
expiry is still skipped. That pass is byte-identical to main apart from a
section comment.

What it could never see is a preview with no label. CI runs the branch's
pipeline, so a pull request that never rebased onto the split created its
namespace with the step that predates the label, and no step revisits a
namespace that already exists to add it. L3's create-only rule (the
preview-deployer SA holds create and delete, not patch) meant there was no
backfill path either. Nothing then moved such a namespace towards collection:
preview-61 sat past its four-hour stamp until someone looked, and eleven
branches still carry the pipeline that mints more like it.

So a second pass collects any preview-<n> that is a month old, whatever else
is true of it — label or no label, stamp or no stamp. It is found by name
(preview-<n>, digits required, which is also what keeps preview-system out)
and its four objects are deleted by the name derived from that, because the
ones it collects may carry no label to select. A month is ttl_seconds from
preview-config, read and never retyped, which makes the reaper that
ConfigMap's third reader; a run that cannot read it refuses and exits 1 rather
than guessing, having already done the first pass.

This is an addition, not a replacement: the console's Delete and Activate still
work through preview-expires, a closed PR still tears down by label, and
nothing is collected before it has had its month — preview-61 is a day old
and stays.

Story 15 said a preview cleans itself up, and this is what makes it true for
every preview rather than for the ones a later pipeline revision happened to
create; the story now says so. OPERATIONS.md's teardown section, the TTL
section, and drill steps 5-6 follow: an unlabelled preview with a stamp in the
past is reached by nothing and must survive a tick, and the same stamp on a
labelled one is the console's Delete and must be collected.

Nothing in the preview-system layer may contain a ${...} — it is applied
with no postBuild.substituteFrom — so the catch-all's expansions are
brace-free, and the test guarding that layer is what caught the first draft.

The reaper's existing pass is untouched: a namespace selected by `milner.dev/preview-pr` whose `milner.dev/preview-expires` has passed is collected, and every object is deleted by that label. A non-numeric or missing expiry is still skipped. That pass is byte-identical to `main` apart from a section comment. What it could never see is a preview with no label. CI runs the *branch's* pipeline, so a pull request that never rebased onto the split created its namespace with the step that predates the label, and no step revisits a namespace that already exists to add it. L3's create-only rule (the `preview-deployer` SA holds create and delete, not patch) meant there was no backfill path either. Nothing then moved such a namespace towards collection: `preview-61` sat past its four-hour stamp until someone looked, and eleven branches still carry the pipeline that mints more like it. So a second pass collects any `preview-<n>` that is a month old, whatever else is true of it — label or no label, stamp or no stamp. It is found by name (`preview-<n>`, digits required, which is also what keeps `preview-system` out) and its four objects are deleted by the name derived from that, because the ones it collects may carry no label to select. A month is `ttl_seconds` from `preview-config`, read and never retyped, which makes the reaper that ConfigMap's third reader; a run that cannot read it refuses and exits 1 rather than guessing, having already done the first pass. This is an addition, not a replacement: the console's Delete and Activate still work through `preview-expires`, a closed PR still tears down by label, and nothing is collected before it has had its month — `preview-61` is a day old and stays. Story 15 said a preview cleans itself up, and this is what makes it true for every preview rather than for the ones a later pipeline revision happened to create; the story now says so. OPERATIONS.md's teardown section, the TTL section, and drill steps 5-6 follow: an unlabelled preview with a stamp in the past is reached by nothing and must survive a tick, and the same stamp on a labelled one is the console's Delete and must be collected. Nothing in the `preview-system` layer may contain a `${...}` — it is applied with no `postBuild.substituteFrom` — so the catch-all's expansions are brace-free, and the test guarding that layer is what caught the first draft.

tepichord added 1 commit 2026-09-16 14:38:10 -07:00

feat(preview): add a one-month catch-all to the reaper ... 46253ec870

The reaper's existing pass is untouched: a namespace selected by
`milner.dev/preview-pr` whose `milner.dev/preview-expires` has passed is
collected, and every object is deleted by that label. A non-numeric or missing
expiry is still skipped. That pass is byte-identical to `main` apart from a
section comment.

What it could never see is a preview with no label. CI runs the *branch's*
pipeline, so a pull request that never rebased onto the split created its
namespace with the step that predates the label, and no step revisits a
namespace that already exists to add it. L3's create-only rule (the
`preview-deployer` SA holds create and delete, not patch) meant there was no
backfill path either. Nothing then moved such a namespace towards collection:
`preview-61` sat past its four-hour stamp until someone looked, and eleven
branches still carry the pipeline that mints more like it.

So a second pass collects any `preview-<n>` that is a month old, whatever else
is true of it — label or no label, stamp or no stamp. It is found by name
(`preview-<n>`, digits required, which is also what keeps `preview-system` out)
and its four objects are deleted by the name derived from that, because the
ones it collects may carry no label to select. A month is `ttl_seconds` from
`preview-config`, read and never retyped, which makes the reaper that
ConfigMap's third reader; a run that cannot read it refuses and exits 1 rather
than guessing, having already done the first pass.

This is an addition, not a replacement: the console's Delete and Activate still
work through `preview-expires`, a closed PR still tears down by label, and
nothing is collected before it has had its month — `preview-61` is a day old
and stays.

Story 15 said a preview cleans itself up, and this is what makes it true for
every preview rather than for the ones a later pipeline revision happened to
create; the story now says so. OPERATIONS.md's teardown section, the TTL
section, and drill steps 5-6 follow: an unlabelled preview with a stamp in the
past is reached by nothing and must survive a tick, and the same stamp on a
labelled one is the console's Delete and must be collected.

Nothing in the `preview-system` layer may contain a `${...}` — it is applied
with no `postBuild.substituteFrom` — so the catch-all's expansions are
brace-free, and the test guarding that layer is what caught the first draft.

tepichord added 1 commit 2026-09-16 15:21:52 -07:00

fix(ci): L2 must not submit a RoleBinding whose Role it creates itself ... 5e14c89bd5

L2 diffs the rendered preview overlay server-side. A dry run persists
nothing, so the Role `preview-router` that the same render creates does not
exist when the binding that references it is submitted — and
`preview-deployer` holds no `bind` verb, so the server resolves the roleRef
to run its escalation check and reports that resolution failure as a missing
*binding*:

    Error from server (NotFound): rolebindings.rbac.authorization.k8s.io "preview-router" not found

Measured as that identity (a roleRef naming a Role that exists creates
fine), not inferred. It went unnoticed because the grant that authorizes the
RBAC kinds only takes effect when Flux next reconciles preview-system, and
that took a run to happen: pipeline 74 diffed at 20:34 with the old Role
still live and dropped the three kinds, Flux configured the grant at 20:39,
and every run since submits the binding and fails.

L2 now names this class separately from the kinds the identity may not
submit — a gap that closes versus a limit of the check — and leaves every
other object, and every other RoleBinding, in the diff. L1 still schema- and
policy-checks it; L3 applies it for real, Role first, which is why the
preview itself was never broken.

tepichord added 1 commit 2026-09-16 15:40:59 -07:00

fix(ci): build the preview console for every change set, not only console ones ... cbb6656c6b

L3 writes `preview/oz:pr-<n>-<sha>` into the preview's values ConfigMap
unconditionally — `apps/preview/oz.yaml` is in the overlay for every change
set, as the preview's default front door and the page the disambiguation
happens on (story 7) — but `build-oz` was path-filtered on `oz/**` and
`previewlib/**` on its pull-request half too. A change set that touched
neither was handed a tag nothing had built:

    oz  0/1  ImagePullBackOff  zot.faun-snapper.ts.net/preview/oz:pr-64-5e14c89

The registry held only `pr-62-*` tags, the other three workloads in
preview-64 were Running, and L3 died ten minutes later on its health-check
timeout saying only "timed out waiting for the condition".

The router has this gap handled — L3 asks the registry and falls back to the
image the front door runs — and that fallback is right for the router, which
a preview only has when the change set is about one. It is wrong for the
console, which every preview has: the fallback is the *real* console (login,
real cluster reads, real actions), and a preview namespace deliberately
holds no credential and no cluster grant for it.

So the console's pull-request half loses its path filter: 42 seconds of
buildah on the change sets that do not touch `oz/**`, against a ten-minute
timeout that reads like a pod problem. The main-push half keeps its filter
and its pinned tag, and the docs and tests that described the old symmetry
now describe the asymmetry and why it is the right shape.

tepichord referenced this pull request from a commit 2026-09-16 18:12:01 -07:00

feat(preview): story 5 — a change set about no app is told so

tepichord added 2 commits 2026-09-16 18:12:01 -07:00

feat(preview): story 5 — a change set about no app is told so ... 60beac4d57

The story said a change set about nothing still opens on the console, the
marked fallback. It now says that nothing to preview is an answer: shown as
such from the console and at the address alike, and never filled in with
options that have no effect. So three mechanisms go, not one — the
`milner.dev/preview-default` annotation, the `DEFAULT` reason in
`previewlib/selection.py`, and the compatibility path that served a lone
candidate which declared nothing (a preview nothing can be matched against is
one nothing can be said about, so it says nothing to preview too).

Story 11 is removed: a broken change of mine shows its error rather than being
worked around. Story 10 gains what that leaves standing — the router changes a
change set brings are in effect for the preview marked active, and with no
preview to mark active the story disqualifies itself. Stories renumber to 1..16.

The states now, from the rendered overlay and the rule's own code:

  the reaper+pipeline change set (PR #64's files)
      decide unclaimed -> row "nothing to preview from this change set", no
      buttons -> address 200 "Nothing to preview", naming no app
  oz/** or previewlib/**  decide selected -> row "auto: oz", buttons
  two apps claim it       decide contended -> row "pick one", buttons
  router/app.py only      decide unclaimed -> row names the change set's own
      router while the preview is active (front_door: router), and says nothing
      to preview while it is not
  no candidate at all     decide unresolved -> row "no previewable app yet"

`oz/cluster.py` reports the change set's own router only for the preview marked
active, which is what makes story 10's clause true rather than implied: a router
named against a preview nobody is served would be a claim that something runs
when nothing does. The router gets its own page for the new state, and it names
nobody — an app list there would be exactly the options story 5 removes. The
console renders the sentence without the form, so the buttons are withheld by
the state rather than by an empty list.

oz 0.1.4 and router 0.1.2: both sources changed, and the tag guards read the
pins. 197 tests; kubeconform clean on the preview and preview-system layers,
conftest 56/56, ruff and shellcheck clean.

feat(stories): a broken change of mine shows me the break (story 11) ... 1aec3fdc46

Story 11 was removed because it was too strict — a broken change must show its
error rather than be worked around — and the reason it was too strict is itself
a need, so it is now stated: when what I changed is what the preview runs and it
does not come up, I see that failure at the address and in the console instead of
being handed something else to look at. A change served by a workaround is a
change I cannot review.

It sits beside story 10 because it is the same reader — the author of a change
set that touches the shared code — which moves the operator's stories down one,
back to a gap-free 1..17, and the console-never-drifts references back to 17.

The story has a check, since a story with no check is a gap worth naming: the
proxy test dials a change set's own router that answers nothing and asserts the
reader gets the failure page for that preview, naming no app and offering nothing
in its place. ARCHITECTURE's router-layer paragraph says the same thing where the
layer is described.

tepichord added 2 commits 2026-09-16 18:34:17 -07:00

fix(tests): a dialling check must dial loopback, not a Service name ... e1862580ba

Pipeline 83 failed `test` on the new story-11 check alone. The check passed
here and failed in CI because of the network, not the router: it built a bare
`Target(namespace="preview-64", service="preview-router", port=80)`, whose
upstream is `preview-router.preview-64.svc.cluster.local:80` — silence from a
workstation, and from a Woodpecker agent (inside the cluster) a live answer
from the preview deployed there. The router proxied for real, and the page it
read was that preview's own contention page: 503 and `text/html`, so the first
two asserts passed, and the body was a stranger's.

It was also the only dialling test in the file that did not use the file's
`_LocalTarget`, which every proxy test already uses for exactly this.

Dial loopback instead: a socket the check binds and never listens on, so the
upstream is silent for a reason the check makes rather than one the runner's
network gives, and the refusal is immediate rather than a 30s timeout.

`_serve_router` now refuses a target with no reason that is not a loopback one,
so the mistake fails at the seam on any machine — 0.08s here, where before it
looked green — and AGENTS.md's Testing notes name the network as the suite's
second local-only assumption, checked the same way the `CI_*` one is.

feat(ci): the suite gates the deployment (story 18) ... 0b1dba7884

The `test` step ran last, after L3 had already created the preview namespace and
Flux had reconciled the change set into it. Pipeline 83 is what that costs: the
suite failed, and the preview for that commit had been deployed anyway — a
staging deployment nobody asked for, for a change set the tests had already
rejected.

Steps run serially and definition order *is* the dependency graph here
(`depends_on` is deliberately absent and stays absent), so position is the only
way to say "the tests come first". The suite moves from last to just behind L1:
a failed step aborts the workflow, so nothing of a rejected change set is
submitted, pushed or deployed.

Behind L0 rather than first, because it reads L0's renders: `tests/test_gitops.py`
validates `rendered-*.yaml` when they are on disk and skips when they are not
(kustomize is not in the suite's image), so putting the suite ahead of the render
would quietly delete that layer's validation from CI.

Ahead of L2 and the image builds too, for the same reason: a red suite should
cost nothing that touches the cluster — no server-side dry run submitting the
change set's objects for admission, no privileged build pushing an image in its
name.

`tests/test_static.py::test_a_change_that_fails_its_tests_is_not_deployed` holds
it: the suite's step precedes l2-dry-run, build-oz, build-router and l3-preview,
and follows l0-render-apps. Reverted to the old order it fails in 0.05s naming
the gate that would have deployed a rejected change set.

tepichord added 1 commit 2026-09-16 18:58:00 -07:00

fix(ci): retry a dropped apk fetch instead of failing the gate ... fabef7aa0c

Pipeline 84 died in L1 after 8s, on a commit that only moved a step's position:

    WARNING: fetching https://dl-cdn.alpinelinux.org/alpine/v3.21/main:
    temporary error (try again later)
    ERROR: unable to select packages:  curl (no such package)

`apk` treats one unreachable mirror as final, so a step whose real work needs no
network at all after that line is red for the CDN's mood — the mirror's own words
are "temporary" and "try again later", which is exactly what a pipeline should
not make a person do. Measured either side of it: pipeline 83 installed the same
`curl` in the same step in 12s and 43 packages in its `test` step; pipeline 84,
one commit later, failed here in 8s.

The GitHub downloads in L1 already retry for the same class of flake
(`--retry 5 --retry-all-errors`, with the earlier "Could not resolve host" red
guard recorded above them). This is the package install's half of it, and it goes
in `ci/` — glue more than one step needs — because all four `apk add` sites in
the pipeline have the same exposure: L1, the suite, and both tag guards.

Three attempts, five and ten seconds apart: enough for a dropped fetch or a
mirror being rotated, short enough that a real outage still fails the step inside
the minute. A stub apk drops two fetches and succeeds on the third (exit 0); a
stub that never answers gives up by name on the third (exit 1). Against the real
apk in the CI image it installs `git` and exits 0.

`tests/test_static.py::test_the_pipeline_installs_packages_through_the_retry`
holds that no step writes its own `apk add` — one that did would inherit the
flake instead of the retry — and the shellcheck list gains the helper plus
`ci/oz-image.sh` and `ci/oz-tag-guard.sh`, which it had been missing.

tepichord added 1 commit 2026-09-16 22:15:24 -07:00

feat(oz): the front door is a dropdown, at the operator's preference ... afc60961dd

The row's Front door control was one submit button per choice, with the serving
app inert. The operator asked for a dropdown, so it is one: `<select name="app">`
listing `auto` and every contender, plus a `Show` button. No script — the
console renders plain HTML, and the preview build runs the same markup.

Two things it keeps, on purpose: the serving app is the *selected* option, so
"which app is this preview showing" is legible in the control itself (story 8,
which the hint under it also states), and story 7's "one press per app" still
reads true — you pick the app, you press Show. If the press is the problem, the
fix is `onchange` on the select, and that is the point at which story 7's phrase
would need the operator's words rather than mine.

The rest is the tag bump the guard requires (`milner/oz:0.1.4` → `0.1.5`, source
moved with it: `oz-tag-guard: console source and tag agree`), the CSS rule for the
removed `.on` class, and the docs that said "buttons": AGENTS.md twice,
OPERATIONS.md's front-door section and its drill step 9.

`tests/test_preview_console.py` follows the markup, and the story-5 absence check
is now written against `name="app"` rather than a `<button>`, so the control can
change shape again without the absence ceasing to be checked.

tepichord merged commit f6d4322785 into main 2026-09-16 22:20:24 -07:00

tepichord deleted branch fix/reaper-collects-any-preview 2026-09-16 22:20:24 -07:00

tepichord referenced this pull request from a commit 2026-09-16 22:20:25 -07:00

feat(preview): add a one-month catch-all to the reaper (#64)

Sign in to join this conversation.

Reviewers

No reviewers

Labels

Clear labels   

blocked

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

effort

high

  

effort

low

  

effort

medium

  

effort

minimal

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

pr

feedback

Used for PRs, indicates that feedback has been given and agent needs to make changes

  

pr

reviewing

Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  

question

More information is needed

  

status

backlog

Ignore, no work needed yet, human will assign this work by changing to status/planning

  

status

document

Record findings, create flashcards, and document knowledge in knowledgebase for the future

  

status

done

Completed when PR is merged, no more work needed.

  

status

implement

Make the code changes. Should be done by a faster and cheaper model

  

status

pending

Give human a chance to review planning details

  

status

planning

Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  

status

reviewing

PR is created and feedback loop begins between human and agent

  

waiting

  

wontfix

This won't be fixed

No labels

blocked

bug

duplicate

effort

high

effort

low

effort

medium

effort

minimal

enhancement

help wanted

invalid

pr

feedback

pr

reviewing

question

status

backlog

status

document

status

done

status

implement

status

pending

status

planning

status

reviewing

waiting

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

tepichord/milner!64

No description provided.