tepichord/milner

Fork 0

Create oz and preview service #56

New issue

Closed

opened 2026-09-12 07:19:37 -07:00 by tepichord · 0 comments

tepichord commented

2026-09-12 07:19:37 -07:00

Owner

preview-system: a stable preview URL with an admin console (`oz`)

Why

L3 already deploys every PR into its own preview-<n> namespace and reconciles it
with Flux. What it does not give anyone is a place to look. The preview overlay
deletes the Tailscale Ingress, and the workload runs registry.k8s.io/pause:3.10,
which serves nothing — so the gate can go green while no human can see anything.

Goal: one stable, bookmarkable URL — preview.faun-snapper.ts.net — that always
serves the preview a cluster_admin has selected, plus a small OIDC-gated console
at oz.faun-snapper.ts.net to select, rebuild, and delete previews.

Acceptance criteria

https://preview.faun-snapper.ts.net/ serves the selected preview at the root
path (no path prefix), so absolute asset paths, cookies, and client-side routers
behave as they would in prod.
Switching the active preview requires no admin-console action, changes nothing on
the Tailscale side, and re-approves nothing.
oz refuses anyone not in Pocket ID's cluster_admins group.
An unauthenticated tailnet member can view the active preview but cannot change,
rebuild, or tear down anything.
A preview can be deleted from oz (collected on the next reaper pass, shown as
pending until then), and is deleted automatically when its PR is closed — merged or
abandoned — without an admin-console action.
A deleted or expired preview can be rebuilt in one action — the console's Build
button, or a manual pipeline run on the PR's branch — without pushing a commit.
Previews expire after 30 days by default; when one goes away the selection clears
itself, and preview.* serves a "no preview selected" page rather than pointing at
a namespace that no longer exists.
python -m pytest tests/ -v passes offline, including new guards that the preview
namespace's render contains no Ingress and that the console/router RBAC is
bounded (no secrets, no wildcards, no Ingress create).

Design

Two hostnames, both permanent

hostname	what
`oz.faun-snapper.ts.net`	the console: lists previews, selects one, rebuilds, deletes
`preview.faun-snapper.ts.net`	the router: proxies everything to the active preview

Both Ingresses use the shared ingress-proxies ProxyGroup, so no new tailnet device
is created. Each is a Tailscale Service, so each needs one approval at setup (or
one autoApprovers.services entry each, then zero). After that neither is ever
touched again — which is the entire point of this design.

Data path

client → ingress-proxies-0 (TLS) → preview-router:8080 → orchestrator.preview-<n>.svc:80 → pod

The router resolves its target from the API, not from a mounted file: a projected
ConfigMap can lag by the kubelet sync period plus cache TTL (a minute or more), which
would make switching slower than doing nothing. The router watches namespaces and
Services and swaps the upstream in memory. In-flight requests finish against the old
target; there is no dark window.

State: a label on the selected preview, not a ConfigMap

milner.dev/preview-active: "true" as a label on the selected preview-<n>
namespace.

Rejected alternative: a preview-active ConfigMap in preview-system. It survives
its target's teardown, so a reaped preview leaves the router pointing at nothing until
someone re-selects. A label on the namespace cannot outlive the namespace, which is
why criterion 7 falls out for free.

Preview namespaces keep having no Ingress

apps/preview/kustomization.yaml deletes the Ingress and must keep doing so. The
reason currently written in that file — "a preview must not mint a tailnet node" —
is wrong: with a shared ProxyGroup an Ingress creates a Tailscale Service, not a
device. The real reasons are:

the operator allows only one Ingress per hostname per cluster (ingress-for-pg.go:
"multiple Ingresses for the same hostname in the same cluster are not allowed"), so
preview.* can only ever point at one namespace at a time; and
previews are not individually reachable — exposure is an explicit admin action, not a
side effect of opening a PR.

Fix that comment and the docstring of
tests/test_static.py::test_preview_overlay_strips_the_tailscale_ingress to say so.

Prerequisites (critical path first)

P0.1 — publish the images to zot.

oz is hand-written, so it needs an image — and this repo has never built one (no
Dockerfile, no build step anywhere). That is the critical path for this issue and is
probably worth its own ticket. Blocked by the zot work landing on main
(feat/48-zot-oci-service, unmerged at the time of writing).

How the kubelet pulls it — verify before building, because it decides whether a
Terraform change is needed. The repo already proves a node can reach a *.ts.net
name: control-plane/main.tf sets
cluster.apiServer.extraArgs."oidc-issuer-url" = "https://id.faun-snapper.ts.net", and
dashboard login works — so the kube-apiserver fetches discovery and JWKS over the
tailnet from the control plane. If the same holds for the worker, the kubelet can pull
zot.faun-snapper.ts.net/<repo> over HTTPS with a real cert and no Talos change at
all. Test it (debug pod on the worker: curl -sv https://zot.faun-snapper.ts.net/v2/).

If it does not hold, the fallback is zot.zot.svc:5000 over plain HTTP, which needs
machine.registries in the worker's Talos config — a change in worker/ plus a
terraform apply you run — and it makes the worker trust an unauthenticated registry.

Where it is built and how it is pinned. Build in Woodpecker (woodpecker-runtime is
already privileged, so buildah/kaniko is viable) and push to zot. Tag with the
commit SHA (sha-<sha>) and reference that tag in the manifest: no digest write-back
loop, so CI still never commits to the repo, and the tag names its own commit. Watch
the interaction with zot's catch-all retention tier (20 most recent, or pushed within
90 days) — an old sha-* tag ages out, so bump the manifest rather than letting a stale
pin turn into ImagePullBackOff.

The alternative is :main + imagePullPolicy: Always + a pod annotation carrying the
SHA to force a rollout. That is mutable by design, which is exactly what
policy/k8s.rego rejects :latest for — so it is a deliberate exception, not a free
win. Either way the first push has to precede the manifest that references it, or Flux
reports not-ready against an image that does not exist yet.

P0.2 — verify pod-reachable OIDC. curl https://id.faun-snapper.ts.net/.well-known/openid-configuration from a pod. The
node-side path is proven (P0.1), but pods resolve through CoreDNS, not necessarily the
node's resolver, and the same class of problem already cost time with Pocket ID's
internalAppUrl. If it fails, oz needs an internal issuer URL while its redirect URI
stays external.

P0.3 — Pocket ID client. Create an OIDC client for oz with redirect URI
https://oz.faun-snapper.ts.net/callback, and confirm the ID token carries a groups
claim containing cluster_admins.

P0.4 — teardown: three triggers, one recipe.

trigger	who	when
delete button	`oz` (console)	immediately, on the reviewer's say-so
PR closed	Woodpecker on `pull_request_closed`	after merge — and after an abandoned PR, which strands a preview today
expiry	`preview-reaper` CronJob	the only mechanism for a preview no event will ever collect

The expiry row is not redundant with the other two: the close-triggered pipeline is
driven by the same Forgejo delivery as everything else, so both automatic paths fail
together — and neither covers a PR that is simply left open. Expiry is the only path
that works with the forge unreachable, Woodpecker down, and no event ever arriving.

The recipe, in this order (the Kustomization first, so its pruner collects the
workloads):

delete Kustomization milner-pr-<n> in preview-system
delete ConfigMap preview-<n>-values in preview-system
delete namespace preview-<n>
delete GitRepository milner-pr-<n> in preview-system

So that one recipe is addressable from three callers, label all four objects with
milner.dev/preview-pr: "<n>". Every teardown then reduces to
kubectl delete <kind> -l milner.dev/preview-pr=<n>, and the namespace label is also
what oz lists previews from.

That lets the reaper stop judging four objects independently: it should find expired
previews from the namespace annotation alone (milner.dev/preview-expires), then
delete all four by label. Today each object is judged on its own annotation, so
extending or deleting one of them leaves a window where the Kustomization is reaped and
the namespace is not — which prunes the workloads out from under whoever is looking at
the preview. This changes shipped behaviour, so
tests/test_static.py::test_preview_reaper_deletes_expired_resources_by_annotation
(and its ≥4-occurrence assertion, which now reads .woodpecker/gates.yaml) must change
with it.

TTL: 30 days, up from 4 hours. A preview is an artifact you come back to, not a
four-hour window: at 4h a preview activated today for a demo tomorrow is already gone,
and rebuildable only by spending a pipeline run (P0.5). A month is long enough that
expiry is pure hygiene and never interrupts a review — which is why the issue does
not need an "exempt the active preview" rule. Rejected for the record: exempting
the selected namespace from expiry; unnecessary at a 30-day TTL, and it would let one
namespace hold a PVC indefinitely.

Budget note, since this is the change with a resource consequence: the preview
overlay's 1 Gi PVC uses the ephemeral class, which lives at
/var/mnt/data/ephemeral on the same 100 GiB shared data disk as persistent/.
The per-namespace ResourceQuota does not bound the total, so the live-preview count is
bounded only by "PRs opened in the last 30 days" — roughly 10 GiB if you open ten a
month. Acceptable, but it is the number to watch, and it is an argument for using the
delete button rather than waiting for expiry.

Keep the TTL in one place — a Flux-owned preview-config ConfigMap in
preview-system with a ttl_seconds key, read by L3 (kubectl get -o jsonpath) and
by oz. Two literals in two languages is the same drift bug as the stand-in image
duplicated across l2-dry-run and l3-preview.

Identity for each path — decided.

CI already has it. preview-deployer holds namespace create and delete, plus
CRUD on kustomizations/gitrepositories/configmaps in preview-system
(preview-system/rbac.yaml). No new grant.
oz gets patch, not delete. Its Delete button patches the namespace's
milner.dev/preview-expires to now — a verb it already needs for activation — and the
reaper collects the preview on its next pass. Three things to build around:
- the console must show a pending state ("deleting — collected by HH:MM") instead
  of removing the row, or the button looks broken and gets clicked twice;
- the lag is the reaper's schedule, so shorten it. */15 was chosen when the only
  consumer was a 4-hour expiry; with a user-facing delete in front of it, */5 is the
  right cadence, and it costs a 10m-CPU/32Mi pod five times an hour instead of three;
- a push during the window re-runs L3 and re-creates the preview. That is correct —
  a new commit legitimately rebuilds it — but a run overlapping the reaper's delete
  pass can leave the Kustomization not-ready until the next L3 run re-applies it. The
  race exists today, just narrower.
Why this over the alternatives: the console can then expire but never destroy,
which is a stronger statement than any grant list. rbac.yaml gives the reaper its own
identity precisely because "deleting namespaces cluster-wide is strictly more
dangerous than creating them, and the CronJob runs unattended" — (b) leaves that
boundary intact and gives teardown one implementation, shared with expiry. Rejected:
(a) namespaces delete on oz — a compromised console could delete vault or
flux-system; (c) a one-shot Job under the reaper's SA — immediate and
identity-preserving, but it adds a Job-spawning path and a second way to say "delete
this" for a latency nobody is waiting on.

Woodpecker mechanics — decided: split into a directory. pull_request_closed is
currently filtered out by the root when in .woodpecker.yaml, and Woodpecker loads
either .woodpecker.yaml or the .woodpecker/ directory, not both. So:

.woodpecker.yaml → .woodpecker/gates.yaml: the four validation gates, unchanged
apart from the items below;
.woodpecker/cleanup.yaml: the close-triggered teardown, with
when: [{event: pull_request_closed}] on the step.

The required status check therefore changes from ci/woodpecker/pr/woodpecker to
ci/woodpecker/pr/gates. Confirm the branch-protection setting before merging this.
It is documented as requiring the pattern ci/woodpecker/pr/*, which still matches — but
if Forgejo is holding a literal check name rather than the pattern, the rename blocks
every merge until it is updated, and that is the one step here whose mistake is not
self-announcing. The cleanup pipeline reports under the pull_request_closed event, so
it never contributes a check to an open PR.

Rejected alternative: - event: pull_request_closed on the root when plus per-step
when blocks on all four gate steps. It keeps one file and the check name, at the cost
of every gate step carrying an event list for a concern that is not theirs.

P0.5 — rebuild a deleted preview in one action.

With a 30-day TTL, expiry is rare; but a preview deleted from oz (or expired, or
collected after a merge) has to come back cheaply. Today the only route is pushing a
commit to the PR, which re-runs L3 with CI_COMMIT_PULL_REQUEST set. Requiring an empty
commit for a preview you deleted by mistake is the wrong ergonomics.

Three routes, cheapest first:

Push any commit to the PR — works today, no changes, but pollutes history and
needs a dirty-tree dance. Keep as the fallback.
Run the gates pipeline on the PR's branch from the Woodpecker UI (or restart a
previous PR pipeline). Needs one change: L3's when currently allows only
event: pull_request, so a manual or branch run never reaches it.
Build button in oz — POST /api/repos/{repo_id}/pipelines with a Bearer PAT and
body {branch, variables}, which Woodpecker documents as "Trigger a manual
pipeline" and injects into the step environment. Needs the same when change plus a
token.

The change that makes 2 and 3 both work is PR-number resolution in L3, since a manual
run has no CI_COMMIT_PULL_REQUEST:

PR="${CI_COMMIT_PULL_REQUEST:-${PREVIEW_PR:-}}"
# last resort: ask the forge which open PR has this branch as its head
[ -n "$PR" ] || PR=$(…/api/v1/repos/tepichord/milner/pulls?state=open | match head.ref == $CI_COMMIT_BRANCH)
case "$PR" in ''|*[!0-9]*) echo "no PR for this run — nothing to preview"; exit 0 ;; esac

The exit 0 guard matters: a manual run of the gates on main must not try to create a
preview. The forge lookup is what makes route 2 work with no variables at all, and it is
the only tier that needs the network — the repo is public, so it can be anonymous.

For route 3, oz needs:

a Woodpecker PAT (Vault → VSO → env), and the in-cluster endpoint
http://woodpecker-server.woodpecker.svc.cluster.local rather than the tailnet
hostname, which sidesteps P0.2's resolution question for this call;
the branch to trigger on. For a preview whose namespace still exists this is a label,
but for a deleted one it is gone — so oz should list open PRs from the forge
(GET /api/v1/repos/tepichord/milner/pulls?state=open) and correlate them with live
preview namespaces. That also fixes the console's discoverability problem: without it,
a deleted preview vanishes from a namespace-derived list and the Build button has
nothing to attach to.

Rejected: oz recreating the preview objects itself. That duplicates L3's recipe
(constructing a GitRepository pinned to the branch, a Kustomization with the right
target namespace and substitutions) inside the console, and widens its grants to
kustomizations, gitrepositories, and configmaps. Triggering the pipeline that already
knows how keeps one implementation.

Platform work

preview-system/router.yaml — router Deployment + Service + Ingress
(tls.hosts: [preview], tailscale.com/proxy-group: ingress-proxies), listening on
:8080, image pinned by tag, resources + limits, runAsNonRoot,
readOnlyRootFilesystem, all capabilities dropped. Target resolution: the namespace
labelled milner.dev/preview-active, then the Service in it labelled
milner.dev/preview-service: "true" (label-selected rather than hard-coding
orchestrator, so a second app works later). Refuse any target that is not
*.preview-<n>.svc.cluster.local for a namespace that exists and carries the preview
labels. Pass Upgrade/Connection through for WebSockets and set X-Forwarded-*.
Serve a "no preview selected" page when the label is absent, and a distinct page when
the namespace exists but has no ready endpoints.
preview-system/oz.yaml — console Deployment + Service + Ingress
(tls.hosts: [oz]), plus its SA and RBAC in this layer, so the app, its SA, and
its grants land atomically under wait: true and the preview identities stay in one
place.
preview-system/config.yaml — the preview-config ConfigMap holding ttl_seconds.
preview-system/rbac.yaml — add a preview-console SA with namespaces
get/list/watch, namespaces patch, and services get/list/watch. No delete verb
of any kind, and no Ingress create/delete: the console asks the reaper to collect, it
never destroys anything itself (P0.4). Extend tests/test_static.py so the existing
"no secrets, no wildcard" ClusterRole check covers this identity too, and assert it
holds no delete verb.
preview-system/reaper.yaml — expiry from the namespace, deletion by label (P0.4),
and the schedule shortened to */5 * * * *.
Register all of it in preview-system/kustomization.yaml.

Where it lives: recommended preview-system/, keeping all preview machinery in one
layer. apps/oz/ also works but splits the RBAC from the other preview identities and
applies in an earlier layer.

Console (`oz`) behaviour

OIDC authorization-code flow against Pocket ID; require cluster_admins in the
groups claim; signed, httpOnly, Secure, SameSite session cookie; session key
from Vault via VSO; CSRF token on every state-changing POST.
Lists open PRs (forge API) alongside live preview namespaces, so each row shows one of:
live (activate / delete), expired or never built (build), plus the active
marker, the source branch, and the expiry countdown.
Activate: clears milner.dev/preview-active from any other preview, sets it on the
chosen one, and extends that namespace's expiry to the full TTL.
Build: triggers the gates pipeline on that PR's branch (P0.5) and reports the
pipeline URL so the operator can watch it.
Delete: behind a confirmation, patches the namespace's expiry to now (P0.4) and
moves the row to a pending state naming when collection will happen — the console never
deletes anything itself.
No free-form input reaches an object spec or the pipeline trigger — the only accepted
values are a validated PR number and a validated preview identifier. Renders no
user-controlled HTML.

CI work (`.woodpecker/`)

l3-preview: replace the registry.k8s.io/pause:3.10 stand-in with an HTTP server,
so an activated preview shows something. Constraints from apps/base/deployment.yaml:
non-root (uid 65535), readOnlyRootFilesystem, and the Service's targetPort: http
is 8080. hashicorp/http-echo fits (static binary, -listen=:8080); nginx-family
images do not run under that context. Add matching args in the preview overlay so
the page names the PR.
l3-preview: add milner.dev/preview-pr: "<n>" (and the source branch) as a
label at namespace creation, and put the same PR label on the values ConfigMap,
the GitRepository, and the Kustomization (P0.4's recipe depends on all four).
l3-preview: read the TTL from preview-config instead of the literal
4 * 3600, and rewrite the comment above it — the current one ("short enough that a
forgotten PR does not hold a namespace overnight") argues for the behaviour that is
being replaced.
l3-preview: PR-number resolution per P0.5 (CI_COMMIT_PULL_REQUEST → PREVIEW_PR
→ forge lookup by branch), with the non-numeric/absent guard, and add
{event: manual} to L3's own when.
l2-dry-run: keep the stand-in literal in sync with L3 and add a test asserting the
two match — they are two copies of one fact today.
l3-preview: drop the hard-coded healthChecks: Deployment/orchestrator and rely on
wait: true. The current check names a Deployment regardless of what was applied, so
renaming or adding a workload is never health-checked.
.woodpecker/cleanup.yaml (P0.4): a pull_request_closed step implementing the four
deletes by label. It must be idempotent — the preview may already be gone (expired, or
deleted from oz) — and must not fail the run when there is nothing to delete. Move
the existing file to .woodpecker/gates.yaml in the same commit, and update its header
comment: it currently spells out the context derivation as ".woodpecker.yaml gives
woodpecker", which is now wrong — and that paragraph is the documentation of the
exact string branch protection depends on.
Leave path: ./apps/preview alone. Previewing "what the PR changed" is a separate
issue: with one app in the repo the mapping is near-degenerate, and the convention
that scales (apps/<name>/{base,prod,preview}, changed paths → that overlay) is a
restructure, not a CI tweak.

Note the preview overlay already needs no ${domain}: it deletes the Ingress and
patches storage_class to ephemeral, so ${image} is the only placeholder left and
L2's substitution needs no change. Add a guard test anyway — every ${...} in
rendered-apps-preview.yaml must have a value in the L3 values ConfigMap, so the next
placeholder added to the base cannot silently break the dry run.

Docs

OPERATIONS.md §2: the two Ingresses and their one-time approval, plus optional
autoApprovers.services entries for svc:oz and svc:preview; §11d: a preview drill
(activate → verify → delete → rebuild → verify the same namespace returned →
expire → verify the selection cleared); the teardown paths and which identity
performs each; the TTL and where it is configured.
AGENTS.md repo map: the preview-system/ row, and the apps/preview/ row (which
currently repeats the "must not mint a tailnet node" claim).
ARCHITECTURE.md and PLAN.md: the console, the router, and the image build.
README.md if it describes the preview pipeline.

Verification

Offline: python -m pytest tests/ -v; kustomize build apps/preview (must contain no
Ingress); flux build kustomization preview-system --path=... --kustomization-file=...;
kubeconform -strict -ignore-missing-schemas + conftest test --policy policy/ over the
render.

Live, in order:

Two approvals in the admin console (or the two ACL entries), once.
Activate PR A → preview.* serves A at /, and the page identifies A.
Activate PR B → B within a second or two, with no admin-console prompt and no
change to the ProxyGroup's serve config or the tailnet's service state (capture
before/after).
Delete from oz → the row moves to pending, then within one reaper tick all four
objects are gone, the console no longer lists it as live, and if it was active
preview.* serves the "no preview selected" page.
Rebuild it from oz → the same preview-<n> namespace returns and reconciles to
Ready, with the expiry reset to the full TTL. Repeat via a manual pipeline run on the
branch, then once on main to confirm the run logs "nothing to preview" and exits 0.
Close a PR → the cleanup pipeline deletes all four objects; re-running it on an
already-deleted preview succeeds without error.
A non-admin Pocket ID user gets 403 from oz; an unauthenticated tailnet member can
view preview.* but can neither select, rebuild, nor delete.

Non-goals

Per-PR hostnames or slot pools; path-prefixed previews (preview.*/45); previewing what
a PR changed rather than a fixed overlay; building the previewed app's own image (only
oz/router images are in scope); Postgres — the selection lives in the cluster
deliberately, so there is one source of truth.

Evidence for the claims above

cmd/k8s-operator/ingress-for-pg.go — one Ingress per hostname per cluster,
cluster-wide and path-blind: // It is invalid to have multiple Ingress resources for the same Tailscale Service in one cluster. → found duplicate Ingress %q for hostname %q - multiple Ingresses for the same hostname in the same cluster are not allowed.
cmd/k8s-operator/ingress.go::handlersForIngress — the proxy backend is the Service
ClusterIP, and the Ingress path is appended: Proxy: proto + net.JoinHostPort(svc.Spec.ClusterIP, fmt.Sprint(port)) + path.
Woodpecker API — POST /repos/{repo_id}/pipelines, "Trigger a manual pipeline",
Bearer PAT, PipelineOptions { branch: string; variables: Record<string, string> };
the CLI documents the same custom parameters as "custom parameters to inject into the
step environment". This is what makes P0.5's routes 2 and 3 possible.
control-plane/main.tf — cluster.apiServer.extraArgs."oidc-issuer-url" = "https://id.faun-snapper.ts.net" with working OIDC login: a node reaching a *.ts.net
name is already load-bearing in this cluster, which is why P0.1 tests the tailnet path
before falling back to machine.registries.
data/main.tf — the shared data disk is 100 GiB; infrastructure/local-path/values.yaml
puts the ephemeral class (Delete) at /var/mnt/data/ephemeral on that same disk,
which is the budget the 30-day TTL spends.
preview-system/rbac.yaml — preview-deployer already holds namespace delete and the
three preview-system kinds, so the cleanup pipeline needs no new grant; and the reaper
is a separate identity because cluster-wide namespace deletion is the dangerous verb,
which is what makes P0.4(a) a real decision.
Tailscale docs, ProxyGroup ingress — "Creates a Tailscale Service for the Ingress
ProxyGroup", and each unique hostname costs a Let's Encrypt certificate (50 unique
hostnames per week) — why per-PR hostnames were rejected.
Tailscale docs, Tags — only device tags are configurable
(operatorConfig.defaultTags, proxyConfig.defaultTags); there is no supported way to
tag an operator-created Service, which is why per-PR auto-approval is unavailable and
a constant hostname is the only zero-touch option.

# preview-system: a stable preview URL with an admin console (`oz`) ## Why L3 already deploys every PR into its own `preview-<n>` namespace and reconciles it with Flux. What it does not give anyone is **a place to look**. The preview overlay deletes the Tailscale Ingress, and the workload runs `registry.k8s.io/pause:3.10`, which serves nothing — so the gate can go green while no human can see anything. Goal: one stable, bookmarkable URL — `preview.faun-snapper.ts.net` — that always serves **the preview a cluster_admin has selected**, plus a small OIDC-gated console at `oz.faun-snapper.ts.net` to select, rebuild, and delete previews. ## Acceptance criteria 1. `https://preview.faun-snapper.ts.net/` serves the selected preview at the **root path** (no path prefix), so absolute asset paths, cookies, and client-side routers behave as they would in prod. 2. Switching the active preview requires no admin-console action, changes nothing on the Tailscale side, and re-approves nothing. 3. `oz` refuses anyone not in Pocket ID's `cluster_admins` group. 4. An unauthenticated tailnet member can view the active preview but cannot change, rebuild, or tear down anything. 5. A preview can be deleted from `oz` (collected on the next reaper pass, shown as pending until then), and is deleted automatically when its PR is closed — merged or abandoned — without an admin-console action. 6. **A deleted or expired preview can be rebuilt in one action** — the console's Build button, or a manual pipeline run on the PR's branch — without pushing a commit. 7. Previews expire after 30 days by default; when one goes away the selection clears itself, and `preview.*` serves a "no preview selected" page rather than pointing at a namespace that no longer exists. 8. `python -m pytest tests/ -v` passes offline, including new guards that the preview namespace's render contains **no** Ingress and that the console/router RBAC is bounded (no secrets, no wildcards, no Ingress create). ## Design ### Two hostnames, both permanent | hostname | what | |---|---| | `oz.faun-snapper.ts.net` | the console: lists previews, selects one, rebuilds, deletes | | `preview.faun-snapper.ts.net` | the router: proxies everything to the active preview | Both Ingresses use the shared `ingress-proxies` ProxyGroup, so no new tailnet device is created. Each is a Tailscale Service, so each needs **one** approval at setup (or one `autoApprovers.services` entry each, then zero). After that neither is ever touched again — which is the entire point of this design. ### Data path ``` client → ingress-proxies-0 (TLS) → preview-router:8080 → orchestrator.preview-<n>.svc:80 → pod ``` The router resolves its target from the API, not from a mounted file: a projected ConfigMap can lag by the kubelet sync period plus cache TTL (a minute or more), which would make switching *slower* than doing nothing. The router watches namespaces and Services and swaps the upstream in memory. In-flight requests finish against the old target; there is no dark window. ### State: a label on the selected preview, not a ConfigMap `milner.dev/preview-active: "true"` as a **label** on the selected `preview-<n>` namespace. Rejected alternative: a `preview-active` ConfigMap in `preview-system`. It survives its target's teardown, so a reaped preview leaves the router pointing at nothing until someone re-selects. A label on the namespace cannot outlive the namespace, which is why criterion 7 falls out for free. ### Preview namespaces keep having no Ingress `apps/preview/kustomization.yaml` deletes the Ingress and must keep doing so. The reason currently written in that file — *"a preview must not mint a tailnet node"* — is **wrong**: with a shared ProxyGroup an Ingress creates a Tailscale *Service*, not a device. The real reasons are: - the operator allows only one Ingress per hostname per cluster (`ingress-for-pg.go`: *"multiple Ingresses for the same hostname in the same cluster are not allowed"*), so `preview.*` can only ever point at one namespace at a time; and - previews are not individually reachable — exposure is an explicit admin action, not a side effect of opening a PR. Fix that comment and the docstring of `tests/test_static.py::test_preview_overlay_strips_the_tailscale_ingress` to say so. ## Prerequisites (critical path first) **P0.1 — publish the images to zot.** `oz` is hand-written, so it needs an image — and this repo has never built one (no `Dockerfile`, no build step anywhere). That is the critical path for this issue and is probably worth its own ticket. **Blocked by** the zot work landing on `main` (`feat/48-zot-oci-service`, unmerged at the time of writing). *How the kubelet pulls it — verify before building, because it decides whether a Terraform change is needed.* The repo already proves a **node** can reach a `*.ts.net` name: `control-plane/main.tf` sets `cluster.apiServer.extraArgs."oidc-issuer-url" = "https://id.faun-snapper.ts.net"`, and dashboard login works — so the kube-apiserver fetches discovery and JWKS over the tailnet from the control plane. If the same holds for the worker, the kubelet can pull `zot.faun-snapper.ts.net/<repo>` over HTTPS with a real cert and **no Talos change at all**. Test it (debug pod on the worker: `curl -sv https://zot.faun-snapper.ts.net/v2/`). If it does *not* hold, the fallback is `zot.zot.svc:5000` over plain HTTP, which needs `machine.registries` in the worker's Talos config — a change in `worker/` plus a `terraform apply` you run — and it makes the worker trust an unauthenticated registry. *Where it is built and how it is pinned.* Build in Woodpecker (`woodpecker-runtime` is already `privileged`, so `buildah`/`kaniko` is viable) and push to zot. Tag with the commit SHA (`sha-<sha>`) and reference that tag in the manifest: no digest write-back loop, so CI still never commits to the repo, and the tag names its own commit. Watch the interaction with zot's catch-all retention tier (20 most recent, or pushed within 90 days) — an old `sha-*` tag ages out, so bump the manifest rather than letting a stale pin turn into `ImagePullBackOff`. The alternative is `:main` + `imagePullPolicy: Always` + a pod annotation carrying the SHA to force a rollout. That is mutable by design, which is exactly what `policy/k8s.rego` rejects `:latest` for — so it is a deliberate exception, not a free win. Either way the first push has to precede the manifest that references it, or Flux reports not-ready against an image that does not exist yet. **P0.2 — verify pod-reachable OIDC.** `curl https://id.faun-snapper.ts.net/.well-known/openid-configuration` **from a pod**. The node-side path is proven (P0.1), but pods resolve through CoreDNS, not necessarily the node's resolver, and the same class of problem already cost time with Pocket ID's `internalAppUrl`. If it fails, `oz` needs an internal issuer URL while its redirect URI stays external. **P0.3 — Pocket ID client.** Create an OIDC client for `oz` with redirect URI `https://oz.faun-snapper.ts.net/callback`, and confirm the ID token carries a `groups` claim containing `cluster_admins`. **P0.4 — teardown: three triggers, one recipe.** | trigger | who | when | |---|---|---| | delete button | `oz` (console) | immediately, on the reviewer's say-so | | PR closed | Woodpecker on `pull_request_closed` | after merge — and after an abandoned PR, which strands a preview today | | expiry | `preview-reaper` CronJob | the only mechanism for a preview no event will ever collect | The expiry row is not redundant with the other two: the close-triggered pipeline is driven by the *same* Forgejo delivery as everything else, so both automatic paths fail together — and neither covers a PR that is simply left open. Expiry is the only path that works with the forge unreachable, Woodpecker down, and no event ever arriving. The recipe, in this order (the Kustomization first, so its pruner collects the workloads): 1. delete Kustomization `milner-pr-<n>` in `preview-system` 2. delete ConfigMap `preview-<n>-values` in `preview-system` 3. delete namespace `preview-<n>` 4. delete GitRepository `milner-pr-<n>` in `preview-system` So that one recipe is addressable from three callers, label **all four** objects with `milner.dev/preview-pr: "<n>"`. Every teardown then reduces to `kubectl delete <kind> -l milner.dev/preview-pr=<n>`, and the namespace label is also what `oz` lists previews from. That lets the reaper stop judging four objects independently: it should find expired previews from the **namespace** annotation alone (`milner.dev/preview-expires`), then delete all four by label. Today each object is judged on its own annotation, so extending or deleting one of them leaves a window where the Kustomization is reaped and the namespace is not — which prunes the workloads out from under whoever is looking at the preview. This changes shipped behaviour, so `tests/test_static.py::test_preview_reaper_deletes_expired_resources_by_annotation` (and its ≥4-occurrence assertion, which now reads `.woodpecker/gates.yaml`) must change with it. **TTL: 30 days, up from 4 hours.** A preview is an artifact you come back to, not a four-hour window: at 4h a preview activated today for a demo tomorrow is already gone, and rebuildable only by spending a pipeline run (P0.5). A month is long enough that expiry is pure hygiene and never interrupts a review — which is why the issue does **not** need an "exempt the active preview" rule. Rejected for the record: exempting the selected namespace from expiry; unnecessary at a 30-day TTL, and it would let one namespace hold a PVC indefinitely. Budget note, since this is the change with a resource consequence: the preview overlay's 1 Gi PVC uses the `ephemeral` class, which lives at `/var/mnt/data/ephemeral` on the **same 100 GiB shared data disk** as `persistent/`. The per-namespace ResourceQuota does not bound the total, so the live-preview count is bounded only by "PRs opened in the last 30 days" — roughly 10 GiB if you open ten a month. Acceptable, but it is the number to watch, and it is an argument for using the delete button rather than waiting for expiry. Keep the TTL in **one** place — a Flux-owned `preview-config` ConfigMap in `preview-system` with a `ttl_seconds` key, read by L3 (`kubectl get -o jsonpath`) and by `oz`. Two literals in two languages is the same drift bug as the stand-in image duplicated across `l2-dry-run` and `l3-preview`. *Identity for each path — decided.* - **CI already has it.** `preview-deployer` holds namespace create **and delete**, plus CRUD on kustomizations/gitrepositories/configmaps in `preview-system` (`preview-system/rbac.yaml`). No new grant. - **`oz` gets patch, not delete.** Its Delete button patches the namespace's `milner.dev/preview-expires` to now — a verb it already needs for activation — and the reaper collects the preview on its next pass. Three things to build around: - the console must show a **pending** state ("deleting — collected by HH:MM") instead of removing the row, or the button looks broken and gets clicked twice; - the lag *is* the reaper's schedule, so shorten it. `*/15` was chosen when the only consumer was a 4-hour expiry; with a user-facing delete in front of it, `*/5` is the right cadence, and it costs a 10m-CPU/32Mi pod five times an hour instead of three; - a push during the window re-runs L3 and re-creates the preview. That is correct — a new commit legitimately rebuilds it — but a run overlapping the reaper's delete pass can leave the Kustomization not-ready until the next L3 run re-applies it. The race exists today, just narrower. Why this over the alternatives: the console can then **expire but never destroy**, which is a stronger statement than any grant list. `rbac.yaml` gives the reaper its own identity precisely because *"deleting namespaces cluster-wide is strictly more dangerous than creating them, and the CronJob runs unattended"* — (b) leaves that boundary intact and gives teardown one implementation, shared with expiry. Rejected: **(a)** `namespaces delete` on `oz` — a compromised console could delete `vault` or `flux-system`; **(c)** a one-shot Job under the reaper's SA — immediate and identity-preserving, but it adds a Job-spawning path and a second way to say "delete this" for a latency nobody is waiting on. *Woodpecker mechanics — decided: split into a directory.* `pull_request_closed` is currently filtered out by the root `when` in `.woodpecker.yaml`, and Woodpecker loads either `.woodpecker.yaml` **or** the `.woodpecker/` directory, not both. So: - `.woodpecker.yaml` → `.woodpecker/gates.yaml`: the four validation gates, unchanged apart from the items below; - `.woodpecker/cleanup.yaml`: the close-triggered teardown, with `when: [{event: pull_request_closed}]` on the step. The required status check therefore changes from `ci/woodpecker/pr/woodpecker` to `ci/woodpecker/pr/gates`. **Confirm the branch-protection setting before merging this.** It is documented as requiring the pattern `ci/woodpecker/pr/*`, which still matches — but if Forgejo is holding a literal check name rather than the pattern, the rename blocks every merge until it is updated, and that is the one step here whose mistake is not self-announcing. The cleanup pipeline reports under the `pull_request_closed` event, so it never contributes a check to an open PR. Rejected alternative: `- event: pull_request_closed` on the root `when` plus per-step `when` blocks on all four gate steps. It keeps one file and the check name, at the cost of every gate step carrying an event list for a concern that is not theirs. **P0.5 — rebuild a deleted preview in one action.** With a 30-day TTL, expiry is rare; but a preview deleted from `oz` (or expired, or collected after a merge) has to come back cheaply. Today the only route is pushing a commit to the PR, which re-runs L3 with `CI_COMMIT_PULL_REQUEST` set. Requiring an empty commit for a preview you deleted by mistake is the wrong ergonomics. Three routes, cheapest first: 1. **Push any commit to the PR** — works today, no changes, but pollutes history and needs a dirty-tree dance. Keep as the fallback. 2. **Run the gates pipeline on the PR's branch** from the Woodpecker UI (or restart a previous PR pipeline). Needs one change: L3's `when` currently allows only `event: pull_request`, so a manual or branch run never reaches it. 3. **Build button in `oz`** — `POST /api/repos/{repo_id}/pipelines` with a Bearer PAT and body `{branch, variables}`, which Woodpecker documents as *"Trigger a manual pipeline"* and injects into the step environment. Needs the same `when` change plus a token. The change that makes 2 and 3 both work is PR-number resolution in L3, since a manual run has no `CI_COMMIT_PULL_REQUEST`: ``` PR="${CI_COMMIT_PULL_REQUEST:-${PREVIEW_PR:-}}" # last resort: ask the forge which open PR has this branch as its head [ -n "$PR" ] || PR=$(…/api/v1/repos/tepichord/milner/pulls?state=open | match head.ref == $CI_COMMIT_BRANCH) case "$PR" in ''|*[!0-9]*) echo "no PR for this run — nothing to preview"; exit 0 ;; esac ``` The `exit 0` guard matters: a manual run of the gates on `main` must not try to create a preview. The forge lookup is what makes route 2 work with no variables at all, and it is the only tier that needs the network — the repo is public, so it can be anonymous. For route 3, `oz` needs: - a Woodpecker PAT (Vault → VSO → env), and the **in-cluster** endpoint `http://woodpecker-server.woodpecker.svc.cluster.local` rather than the tailnet hostname, which sidesteps P0.2's resolution question for this call; - the branch to trigger on. For a preview whose namespace still exists this is a label, but for a *deleted* one it is gone — so `oz` should list open PRs from the forge (`GET /api/v1/repos/tepichord/milner/pulls?state=open`) and correlate them with live preview namespaces. That also fixes the console's discoverability problem: without it, a deleted preview vanishes from a namespace-derived list and the Build button has nothing to attach to. Rejected: `oz` recreating the preview objects itself. That duplicates L3's recipe (constructing a GitRepository pinned to the branch, a Kustomization with the right target namespace and substitutions) inside the console, and widens its grants to kustomizations, gitrepositories, and configmaps. Triggering the pipeline that already knows how keeps one implementation. ## Platform work 1. `preview-system/router.yaml` — router Deployment + Service + Ingress (`tls.hosts: [preview]`, `tailscale.com/proxy-group: ingress-proxies`), listening on `:8080`, image pinned by tag, resources + limits, `runAsNonRoot`, `readOnlyRootFilesystem`, all capabilities dropped. Target resolution: the namespace labelled `milner.dev/preview-active`, then the Service in it labelled `milner.dev/preview-service: "true"` (label-selected rather than hard-coding `orchestrator`, so a second app works later). Refuse any target that is not `*.preview-<n>.svc.cluster.local` for a namespace that exists and carries the preview labels. Pass `Upgrade`/`Connection` through for WebSockets and set `X-Forwarded-*`. Serve a "no preview selected" page when the label is absent, and a distinct page when the namespace exists but has no ready endpoints. 2. `preview-system/oz.yaml` — console Deployment + Service + Ingress (`tls.hosts: [oz]`), plus its SA and RBAC **in this layer**, so the app, its SA, and its grants land atomically under `wait: true` and the preview identities stay in one place. 3. `preview-system/config.yaml` — the `preview-config` ConfigMap holding `ttl_seconds`. 4. `preview-system/rbac.yaml` — add a `preview-console` SA with `namespaces` get/list/watch, `namespaces` **patch**, and `services` get/list/watch. No delete verb of any kind, and no Ingress create/delete: the console asks the reaper to collect, it never destroys anything itself (P0.4). Extend `tests/test_static.py` so the existing "no secrets, no wildcard" ClusterRole check covers this identity too, and assert it holds no `delete` verb. 5. `preview-system/reaper.yaml` — expiry from the namespace, deletion by label (P0.4), and the schedule shortened to `*/5 * * * *`. 6. Register all of it in `preview-system/kustomization.yaml`. Where it lives: recommended `preview-system/`, keeping all preview machinery in one layer. `apps/oz/` also works but splits the RBAC from the other preview identities and applies in an earlier layer. ## Console (`oz`) behaviour - OIDC authorization-code flow against Pocket ID; require `cluster_admins` in the `groups` claim; signed, `httpOnly`, `Secure`, `SameSite` session cookie; session key from Vault via VSO; CSRF token on every state-changing POST. - Lists open PRs (forge API) alongside live preview namespaces, so each row shows one of: **live** (activate / delete), **expired or never built** (build), plus the active marker, the source branch, and the expiry countdown. - **Activate**: clears `milner.dev/preview-active` from any other preview, sets it on the chosen one, and extends that namespace's expiry to the full TTL. - **Build**: triggers the gates pipeline on that PR's branch (P0.5) and reports the pipeline URL so the operator can watch it. - **Delete**: behind a confirmation, patches the namespace's expiry to now (P0.4) and moves the row to a pending state naming when collection will happen — the console never deletes anything itself. - No free-form input reaches an object spec or the pipeline trigger — the only accepted values are a validated PR number and a validated preview identifier. Renders no user-controlled HTML. ## CI work (`.woodpecker/`) 1. `l3-preview`: replace the `registry.k8s.io/pause:3.10` stand-in with an HTTP server, so an activated preview shows something. Constraints from `apps/base/deployment.yaml`: non-root (uid 65535), `readOnlyRootFilesystem`, and the Service's `targetPort: http` is **8080**. `hashicorp/http-echo` fits (static binary, `-listen=:8080`); nginx-family images do not run under that context. Add matching `args` in the preview overlay so the page names the PR. 2. `l3-preview`: add `milner.dev/preview-pr: "<n>"` (and the source branch) as a **label** at namespace creation, and put the same PR label on the values ConfigMap, the GitRepository, and the Kustomization (P0.4's recipe depends on all four). 3. `l3-preview`: read the TTL from `preview-config` instead of the literal `4 * 3600`, and rewrite the comment above it — the current one ("short enough that a forgotten PR does not hold a namespace overnight") argues for the behaviour that is being replaced. 4. `l3-preview`: PR-number resolution per P0.5 (`CI_COMMIT_PULL_REQUEST` → `PREVIEW_PR` → forge lookup by branch), with the non-numeric/absent guard, and add `{event: manual}` to L3's own `when`. 5. `l2-dry-run`: keep the stand-in literal in sync with L3 and add a test asserting the two match — they are two copies of one fact today. 6. `l3-preview`: drop the hard-coded `healthChecks: Deployment/orchestrator` and rely on `wait: true`. The current check names a Deployment regardless of what was applied, so renaming or adding a workload is never health-checked. 7. `.woodpecker/cleanup.yaml` (P0.4): a `pull_request_closed` step implementing the four deletes by label. It must be idempotent — the preview may already be gone (expired, or deleted from `oz`) — and must not fail the run when there is nothing to delete. Move the existing file to `.woodpecker/gates.yaml` in the same commit, and update its header comment: it currently spells out the context derivation as *"`.woodpecker.yaml` gives `woodpecker`"*, which is now wrong — and that paragraph is the documentation of the exact string branch protection depends on. 8. Leave `path: ./apps/preview` alone. Previewing "what the PR changed" is a separate issue: with one app in the repo the mapping is near-degenerate, and the convention that scales (`apps/<name>/{base,prod,preview}`, changed paths → that overlay) is a restructure, not a CI tweak. Note the preview overlay **already** needs no `${domain}`: it deletes the Ingress and patches `storage_class` to `ephemeral`, so `${image}` is the only placeholder left and L2's substitution needs no change. Add a guard test anyway — every `${...}` in `rendered-apps-preview.yaml` must have a value in the L3 values ConfigMap, so the next placeholder added to the base cannot silently break the dry run. ## Docs - `OPERATIONS.md` §2: the two Ingresses and their one-time approval, plus optional `autoApprovers.services` entries for `svc:oz` and `svc:preview`; §11d: a preview drill (activate → verify → delete → **rebuild** → verify the same namespace returned → expire → verify the selection cleared); the teardown paths and which identity performs each; the TTL and where it is configured. - `AGENTS.md` repo map: the `preview-system/` row, and the `apps/preview/` row (which currently repeats the "must not mint a tailnet node" claim). - `ARCHITECTURE.md` and `PLAN.md`: the console, the router, and the image build. - `README.md` if it describes the preview pipeline. ## Verification Offline: `python -m pytest tests/ -v`; `kustomize build apps/preview` (must contain no Ingress); `flux build kustomization preview-system --path=... --kustomization-file=...`; `kubeconform -strict -ignore-missing-schemas` + `conftest test --policy policy/` over the render. Live, in order: 1. Two approvals in the admin console (or the two ACL entries), once. 2. Activate PR A → `preview.*` serves A at `/`, and the page identifies A. 3. Activate PR B → B within a second or two, with **no** admin-console prompt and **no** change to the ProxyGroup's serve config or the tailnet's service state (capture before/after). 4. **Delete from `oz`** → the row moves to pending, then within one reaper tick all four objects are gone, the console no longer lists it as live, and if it was active `preview.*` serves the "no preview selected" page. 5. **Rebuild it** from `oz` → the same `preview-<n>` namespace returns and reconciles to Ready, with the expiry reset to the full TTL. Repeat via a manual pipeline run on the branch, then once on `main` to confirm the run logs "nothing to preview" and exits 0. 6. **Close a PR** → the cleanup pipeline deletes all four objects; re-running it on an already-deleted preview succeeds without error. 7. A non-admin Pocket ID user gets 403 from `oz`; an unauthenticated tailnet member can view `preview.*` but can neither select, rebuild, nor delete. ## Non-goals Per-PR hostnames or slot pools; path-prefixed previews (`preview.*/45`); previewing what a PR changed rather than a fixed overlay; building the previewed app's own image (only `oz`/router images are in scope); Postgres — the selection lives in the cluster deliberately, so there is one source of truth. ## Evidence for the claims above - `cmd/k8s-operator/ingress-for-pg.go` — one Ingress per hostname per cluster, cluster-wide and path-blind: `// It is invalid to have multiple Ingress resources for the same Tailscale Service in one cluster.` → `found duplicate Ingress %q for hostname %q - multiple Ingresses for the same hostname in the same cluster are not allowed`. - `cmd/k8s-operator/ingress.go::handlersForIngress` — the proxy backend is the Service ClusterIP, and the Ingress path is appended: `Proxy: proto + net.JoinHostPort(svc.Spec.ClusterIP, fmt.Sprint(port)) + path`. - Woodpecker API — `POST /repos/{repo_id}/pipelines`, *"Trigger a manual pipeline"*, Bearer PAT, `PipelineOptions { branch: string; variables: Record<string, string> }`; the CLI documents the same custom parameters as "custom parameters to inject into the step environment". This is what makes P0.5's routes 2 and 3 possible. - `control-plane/main.tf` — `cluster.apiServer.extraArgs."oidc-issuer-url" = "https://id.faun-snapper.ts.net"` with working OIDC login: a node reaching a `*.ts.net` name is already load-bearing in this cluster, which is why P0.1 tests the tailnet path before falling back to `machine.registries`. - `data/main.tf` — the shared data disk is 100 GiB; `infrastructure/local-path/values.yaml` puts the `ephemeral` class (Delete) at `/var/mnt/data/ephemeral` on that same disk, which is the budget the 30-day TTL spends. - `preview-system/rbac.yaml` — `preview-deployer` already holds namespace delete and the three preview-system kinds, so the cleanup pipeline needs no new grant; and the reaper is a separate identity *because* cluster-wide namespace deletion is the dangerous verb, which is what makes P0.4(a) a real decision. - Tailscale docs, ProxyGroup ingress — *"Creates a Tailscale Service for the Ingress ProxyGroup"*, and each unique hostname costs a Let's Encrypt certificate (50 unique hostnames per week) — why per-PR hostnames were rejected. - Tailscale docs, Tags — only *device* tags are configurable (`operatorConfig.defaultTags`, `proxyConfig.defaultTags`); there is no supported way to tag an operator-created Service, which is why per-PR auto-approval is unavailable and a constant hostname is the only zero-touch option.