tepichord/milner

Fork 0

Create Pocket ID service on Milner #34

New issue

Closed

opened 2026-09-09 12:54:51 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-09 12:54:51 -07:00

Owner

Create Pocket ID service on Milner so that OIDC can be utilized across applications (namely vault)
Update README and ARCHITECTURE files to accommodate

- Create Pocket ID service on Milner so that OIDC can be utilized across applications (namely vault) - Update README and ARCHITECTURE files to accommodate

tepichord added this to the MVP project

2026-09-09 12:54:51 -07:00

tepichord added the

labels

2026-09-09 12:55:00 -07:00

tepichord commented

2026-09-09 14:26:37 -07:00

Author

Owner

Naina:

Overview

Deploy Pocket ID as the cluster OIDC provider on Milner — an arm64-capable StatefulSet with SQLite on a new worker data volume, exposed to the tailnet at pocket-id.faun-snapper.ts.net through the existing ingress-proxies ProxyGroup (same pattern as Vault). Then wire Vault as the first OIDC consumer (JWT/OIDC auth method, manual client registration + role/policy in OPERATIONS.md) and update the docs to match.

Technical Approach

Add a Pocket ID data volume to the worker (worker/main.tf)

# local: pocket_id_disk_name = "${var.cluster_name}-worker-1-pocket-id.qcow2"
resource "libvirt_volume" "pocket_id_data" {
  name     = local.pocket_id_disk_name
  pool     = var.pool_name
  capacity = 10737418240 # 10 GiB
  target = { format = { type = "qcow2" } }
}

Attach as /dev/vdc (third entry in libvirt_domain.worker disks) and add a second Talos UserVolumeConfig in the machine-config patch.¹ Today the worker has exactly two user disks (vda worker, vdb vault — the ISO is a scsi sda cdrom), so a third disk entry does land on vdc. Make both disk selectors unambiguous — vault's current bare !system_disk would match two user disks after this lands:² The vault UserVolumeConfig at worker/main.tf is diskSelector { match = "!system_disk" } with only minSize = 10GB — nothing that would exclude a second user disk once the pocket-id volume is added.

# vault volume → pin to the 20 GiB disk
diskSelector = { match = "!system_disk && disk.size > 15u * GiB" }
# pocket-id volume → 10 GiB disk → /var/mnt/pocket-id (auto)
name = "pocket-id"
diskSelector = { match = "!system_disk && disk.size < 15u * GiB" }

The size split matches reality: vault's qcow2 is 20 GiB (capacity = 21474836480), so > 15u * GiB vs < 15u * GiB keeps the two selectors mutually exclusive.³

Mount is activated only at boot → terraform -chdir=worker apply -replace='libvirt_domain.worker' (vault precedent, OPERATIONS.md §4e).⁴ OPERATIONS.md §4e documents exactly this: the mount is activated "only at boot" and the recovery is terraform -chdir=worker apply -replace='libvirt_domain.worker'.

Create the pocket-id/ Terraform root mirroring vault/ — no new ProxyGroup (cluster-scoped ingress-proxies is owned by the vault root's state):⁵ The ingress-proxies ProxyGroup manifest (no namespace → cluster-scoped) is declared in vault/main.tf, and OPERATIONS.md §1/§3 confirm it is Terraform-owned by the vault root.

# pocket-id/main.tf (namespace + helm + ingress, cp remote-state providers identical to vault/)
resource "helm_release" "pocket_id" {
  name       = "pocket-id"
  repository = "oci://ghcr.io/hobbit44/pocket-id-helm"
  chart      = "pocket-id"
  version    = "2.4.0"
  namespace  = kubernetes_namespace_v1.pocket_id.metadata[0].name
  values     = [file("${path.module}/values.yaml")]
}

resource "kubernetes_ingress_v1" "pocket_id" {
  metadata {
    name      = "pocket-id"
    namespace = kubernetes_namespace_v1.pocket_id.metadata[0].name
    annotations = { "tailscale.com/proxy-group" = "ingress-proxies" }
  }
  spec {
    ingress_class_name = "tailscale"
    tls { hosts = ["pocket-id"] }
    default_backend {
      service {
        name = "pocket-id"
        port { number = 1411 }
      }
    }
  }
}

pocket-id/values.yaml — hobbit44 chart defaults need four overrides (probes default to /health, which Pocket ID doesn't serve; PVC default needs no provisioner; config-as-env; worker pinning):⁶ The chart's livenessProbe/readinessProbe default to httpGet.path: /health; Pocket ID's canonical health endpoint is /healthz.⁷ (Nuance: v2.14.0 also serves /health as an alias of /healthz, so the chart's /health probe is not strictly dead — but /healthz is the documented, backend-native endpoint.)

image:
  tag: "v2.14.0"            # chart AppVersion v2.4.0 is stale; track latest app release
config:
  appUrl: "https://pocket-id.faun-snapper.ts.net"
  internalAppUrl: "http://pocket-id.pocket-id.svc.cluster.local:1411"
  trustProxy: "true"
persistence:
  enabled: false            # no StorageClass on Milner → hostPath, vault-style
volumes:
  - name: pocket-id-data
    hostPath: { path: /var/mnt/pocket-id }
volumeMounts:
  - name: pocket-id-data
    mountPath: /app/data    # SQLite data/pocket-id.db + data/uploads live here
livenessProbe:
  httpGet: { path: /healthz, port: http }
readinessProbe:
  httpGet: { path: /healthz, port: http }
nodeSelector:
  node-role: worker

Annotations on the values above: v2.14.0 is the current app release (2026-08-18) while the published hobbit44 chart 2.4.0 pins appVersion: 2.4.0 and its statefulset defaults image.tag to .Chart.AppVersion, so the explicit tag override is required⁸. "No StorageClass on Milner" is consistent with the repo: nothing installs a provisioner today and vault uses a hostPath (dataStorage.enabled: false + volume to /var/mnt/vault) — note only woodpecker/values.yaml (a "coming soon" workload) references local-path⁹. The /app/data target matches the official v2.14.0 compose mount ./data:/app/data and the app's SQLite/upload defaults (data/pocket-id.db, data/uploads)¹⁰¹¹. ⚠ Caveat: the published chart 2.4.0 unconditionally mounts the data volume at /app/backend/data (not /app/data) and does not gate that volumeMount on persistence.enabled, so the plan's hostPath volume + /app/data volumeMount must be reconciled with the chart's fixed /app/backend/data mount at install time. node-role: worker matches the worker machine-config nodeLabel¹².

Namespace gets the privileged PSA label (hostPath), exactly like the vault namespace.¹³ The vault namespace carries pod-security.kubernetes.io/enforce=privileged.

Imperative setup runbook → OPERATIONS.md (§5 Pocket ID; renumber old §5 Client devices → §6, §6 Bring-up order → §7, fixing internal refs):¹⁴ Current headings: §5 "Client devices" (L201), §6 "Bring-up order (from a wiped cluster)" (L222); there is no §7 today, so the renumber is clean.
- First-launch admin: browse https://pocket-id.faun-snapper.ts.net/setup from a tailnet device (passkey only, no default creds; needs HTTPS for WebAuthn).¹⁵ Pocket ID docs: HTTPS is required because the app uses the WebAuthn API; the first admin is created through the UI (/setup route exists in the app) — there is no env-var/default-password bootstrap.
- Register the Vault OIDC client in Settings → OIDC Clients: name Vault, confidential, callback URLs https://vault.faun-snapper.ts.net/ui/vault/auth/oidc/oidc/callback + http://localhost:8250/oidc/callback (CLI flow).
- Approve the new tailnet service in the admin console (Vault README §Prereq step 4 pattern).¹⁶ The approval pattern is real — OPERATIONS.md §2 item 4: "When a workload service first advertises, approve it — until then the MagicDNS name resolves but the service does not route." (Minor pointer fix: in vault/README.md the approve step is Prereq 3, not 4 — item 4 there is client versions.)
Vault OIDC auth method (imperative, OPERATIONS.md) — provider-agnostic HashiCorp tutorial flow:¹⁷ The auth enable oidc / auth/oidc/config / role shape and the CLI callback http://localhost:8250/oidc/callback match Vault's documented JWT/OIDC auth method (a localhost redirect URI is required for vault login -method=oidc; 8250 is the documented default listener).

vault auth enable oidc
vault write auth/oidc/config \
    oidc_discovery_url="https://pocket-id.faun-snapper.ts.net" \
    oidc_client_id="<uuid from Pocket ID>" \
    oidc_client_secret="<secret from Pocket ID>" \
    default_role="default"
vault write auth/oidc/role/default \
    user_claim="sub" \
    allowed_redirect_uris="https://vault.faun-snapper.ts.net/ui/vault/auth/oidc/oidc/callback" \
    allowed_redirect_uris="http://localhost:8250/oidc/callback" \
    token_policies="default"
# admin policy: path "sys/auth/oidc" { capabilities = ["create","read","update","delete","sudo"] } + auth/oidc/* + sys/policies/acl/*

Docs: add Pocket ID to README.md workload list, ARCHITECTURE.md pod topology + network diagram + service table, AGENTS.md repo map; tests/test_static.py (pocket-id/ to terraform fmt dirs, pocket-id/values.yaml to yamllint list) and tests/test_terraform.py (pocket-id root).

Implementation Details

Files created: pocket-id/main.tf, pocket-id/values.yaml, pocket-id/README.md, pocket-id/tests/plan.tftest.hcl + tests/fixtures/cp.tfstate (copy of vault's fixture).
Files modified: worker/main.tf, tests/test_static.py, tests/test_terraform.py, README.md, ARCHITECTURE.md, AGENTS.md, OPERATIONS.md, vault/README.md (OIDC section pointer).
Dependencies: hobbit44 chart 2.4.0 from oci://ghcr.io/hobbit44/pocket-id-helm;¹⁸ official image ghcr.io/pocket-id/pocket-id:v2.14.0 is multi-arch (arm64 ✓);¹⁹ vault-helm 0.34.1 supports server.hostAliases if ever needed.²⁰ (2.4.0 verified present on GHCR: the OCI repo hobbit44/pocket-id-helm/pocket-id lists a 2.4.0 tag; vault-helm v0.34.1 documents server.hostAliases and its statefulset template renders it.)
Constraints: arm64-only images; no storage class (hostPath via worker qcow2, privileged PSA); TLS always at the operator edge (LE cert), plaintext HTTP in-cluster; secrets/client id/secret never in repo or TF state (hand-run only).
Decision point for reviewer — Vault server-side reachability. Pocket ID's issuer is always APP_URL (https://pocket-id.faun-snapper.ts.net), and Vault is a strict go-oidc client: its oidc_discovery_url must equal the issuer and be reachable+trusted from the Vault pod.²¹ INTERNAL_APP_URL alone does not fix this (it rewrites only token/userinfo/jwks endpoints; issuer/auth endpoints stay on APP_URL — upstream discussion pocket-id/pocket-id#1169 is unanswered).²²²³ (Verified in well_known_controller.go: issuer, authorization_endpoint, end_session_endpoint, device_authorization_endpoint all stay on APP_URL; only token_endpoint/userinfo_endpoint/introspection_endpoint/jwks_uri/PAR move to INTERNAL_APP_URL. Discussion #1169 is open with zero replies as of 2026-09-09, so it is indeed unanswered.) Workable topologies, in order of preference:
1. In-cluster TLS + private CA: CoreDNS hosts override or vault pod hostAliases mapping pocket-id.faun-snapper.ts.net → pocket-id ClusterIP, Pocket ID serving native TLS (TLS_CERT_FILE/TLS_KEY_FILE, PR #1429) with a small private CA, Vault pinning it via oidc_discovery_ca_pem.²⁴ (PR #1429 "feat: add TLS support for HTTP/2 server" introduced TLS_CERT_FILE/TLS_KEY_FILE; both env vars exist in the app's env_config.go.) Browser path unaffected (operator edge keeps its LE cert).
2. Tailscale egress ProxyGroup + DNSConfig so the Vault pod reaches the ts.net VIP over the tailnet (adds operator-level egress infra — more moving parts).
- Until the reviewer picks one, steps 4–5 document the registration/config; the chosen topology lands as a follow-on commit within this issue.
Out-of-scope notes: no Terraform declarative for OIDC client registration (Pocket ID has no stable API — the community pocket-id-operator CRD is explicitly out of scope)²⁵ (a community aclerici38/pocket-id-operator CRD project exists, but there is no official, stable client-management API to drive from Terraform — hand-run registration is the only supported path); no Postgres (SQLite on local qcow2 is the right homelab scale); no group restriction on the Vault client (leave default open, revisit later); Woodpecker CI OIDC is future work and not blocked by this design.

Acceptance Criteria

worker/main.tf shows a second 10 GiB pocket-id qcow2 + UserVolumeConfig (mount /var/mnt/pocket-id) with unambiguous disk selectors; terraform -chdir=worker plan/apply + worker -replace activates it.
pocket-id/ root passes terraform fmt, terraform -chdir=pocket-id test (mock-provider plan assertions: namespace privileged-PSA label, chart/version, ingress class tailscale + proxy-group: ingress-proxies annotation + tls.hosts=["pocket-id"]), and updated pytest lint lists are green.
Pocket ID pod is Ready on the worker (/healthz probe), data persists under /app/data across pod restarts, and https://pocket-id.faun-snapper.ts.net serves the UI + /.well-known/openid-configuration from a tailnet device.
Admin passkey created at /setup; a Vault OIDC client exists in Pocket ID with both callback URLs.
Vault JWT/OIDC auth method is enabled and configured per OPERATIONS.md (auth enable/config/role/policy commands committed there; login flow verified end-to-end once the reviewer picks the reachability topology).
README, ARCHITECTURE, AGENTS, OPERATIONS, vault/README updated and cross-references consistent.

Testing Plan

Static: python -m pytest tests/ (terraform fmt incl. pocket-id/, yamllint on pocket-id/values.yaml, ruff).
Plan assertions: terraform -chdir=pocket-id test — namespace PSA label, helm chart/repo/version, image tag v2.14.0, persistence disabled + hostPath mount at /app/data, /healthz probes, ingress pinned name + proxy-group annotation.
Worker root: extend worker/tests/plan.tftest.hcl — assert both UserVolumeConfigs exist and selectors are mutually exclusive.
Live smoke (manual, after apply): kubectl -n pocket-id exec → curl :1411/healthz; curl https://pocket-id.faun-snapper.ts.net/.well-known/openid-configuration from a tailnet device; kill the pod and confirm SQLite survives; browser vault login -method=oidc completes.

Repo: Pod - Hashicorp Vault (#33) — the pattern this mirrors (namespace, Helm, ProxyGroup ingress, hostPath volume).
Upstream: pocket-id #857 / PR #858 — INTERNAL_APP_URL, discussion #1169 — issuer mismatch for in-cluster clients, hashicorp/vault #25024 — strict OIDC issuer check.

Research Findings

Best-maintained chart: hobbit44/pocket-id-helm (2.4.0, OCI ghcr.io/hobbit44/pocket-id-helm) — the one Pocket ID docs link to; actively renovate-updated; StatefulSet + PVC default, but its default /health probes are wrong (must be /healthz) and its 8 GiB PVC default needs no provisioner (disable + hostPath).
Expose via the existing cluster-scoped ingress-proxies ProxyGroup (reused across namespaces) with tailscale.com/proxy-group annotation + tls.hosts=["pocket-id"]; don't declare a second ProxyGroup. Set TRUST_PROXY=true since it sits behind the operator.
Storage: SQLite at /app/data (chart's mountPath); official image is multi-arch arm64; data dir needs local block storage (docs advise against NFS), so a dedicated worker qcow2 + hostPath matches Milner's no-provisioner reality.
Vault wiring uses the JWT/OIDC auth method (redirect URIs must match exactly between the Pocket ID client and Vault role — CLI http://localhost:8250/oidc/callback + UI https://vault.faun-snapper.ts.net/ui/vault/auth/oidc/oidc/callback); Pocket ID issues aud=client id, so no extra bound_audiences needed.
No default admin credentials — /setup passkey on first launch; admin console service approval is required before the MagicDNS name routes.

Pre-Mortem

The Vault↔Pocket ID issuer-reachability decision is left open and never converges — Vault is a strict go-oidc client that requires oidc_discovery_url to equal Pocket ID's issuer (always the external APP_URL) and be reachable and trusted from the Vault pod, yet the plan defers choosing a topology (in-cluster TLS + private CA vs Tailscale egress) to a follow-on commit. Any slip in the DNS override, private-CA pinning, or cert wiring — or the follow-on commit never landing — fails the end-to-end Vault OIDC login acceptance criterion and leaves the deployment half-done.
Worker reconfiguration disrupts the cluster or strands the Pocket ID data volume — activating the boot-only mount requires terraform apply -replace of libvirt_domain.worker, which reboots the same node already hosting Vault, and the new size-based disk selectors must stay mutually exclusive with the existing vault selector. A selector or qcow2 attach mistake leaves /var/mnt/pocket-id unmounted (Pocket ID CrashLoopBackOff or data loss) and risks Vault availability during the forced replace.
Helm service-port and probe wiring mismatches the chart's actual defaults — the plan's ingress backend and internalAppUrl reference port 1411 while the hobbit44 chart's ClusterIP service exposes port 80 (targeting the app's internal 1411), so the discovery URL advertised to Vault or the ingress route can point at a dead port.²⁶ ⚠ Correction: this premise does not hold for chart 2.4.0 or main — both set service.port: 1411 and the statefulset's containerPort: {{ .Values.service.port }} (the service targetPort is the named http port), so the ClusterIP service and the plan's :1411 backend agree. The probe-side risk stands: default probes are /health and the pod flaps unless overridden to /healthz. The chart's default probes also hit /health rather than Pocket ID's real /healthz, so any missed override makes the pod flap on failed probes.

Naina: ## Overview Deploy Pocket ID as the cluster OIDC provider on Milner — an arm64-capable StatefulSet with SQLite on a new worker data volume, exposed to the tailnet at `pocket-id.faun-snapper.ts.net` through the existing `ingress-proxies` ProxyGroup (same pattern as Vault). Then wire Vault as the first OIDC consumer (JWT/OIDC auth method, manual client registration + role/policy in OPERATIONS.md) and update the docs to match. ## Technical Approach 1. **Add a Pocket ID data volume to the worker** (`worker/main.tf`) ```hcl # local: pocket_id_disk_name = "${var.cluster_name}-worker-1-pocket-id.qcow2" resource "libvirt_volume" "pocket_id_data" { name = local.pocket_id_disk_name pool = var.pool_name capacity = 10737418240 # 10 GiB target = { format = { type = "qcow2" } } } ``` Attach as `/dev/vdc` (third entry in `libvirt_domain.worker` `disks`) and add a second Talos `UserVolumeConfig` in the machine-config patch.[1](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/worker/main.tf?display=source#L232) Today the worker has exactly two user disks (`vda` worker, `vdb` vault — the ISO is a scsi `sda` cdrom), so a third disk entry does land on `vdc`. **Make both disk selectors unambiguous** — vault's current bare `!system_disk` would match two user disks after this lands:[2](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/worker/main.tf?display=source#L337) The vault `UserVolumeConfig` at `worker/main.tf` is `diskSelector { match = "!system_disk" }` with only `minSize = 10GB` — nothing that would exclude a second user disk once the pocket-id volume is added. ```hcl # vault volume → pin to the 20 GiB disk diskSelector = { match = "!system_disk && disk.size > 15u * GiB" } # pocket-id volume → 10 GiB disk → /var/mnt/pocket-id (auto) name = "pocket-id" diskSelector = { match = "!system_disk && disk.size < 15u * GiB" } ``` The size split matches reality: vault's qcow2 is 20 GiB (`capacity = 21474836480`), so `> 15u * GiB` vs `< 15u * GiB` keeps the two selectors mutually exclusive.[3](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/worker/main.tf?display=source#L139) Mount is activated only at boot → `terraform -chdir=worker apply -replace='libvirt_domain.worker'` (vault precedent, OPERATIONS.md §4e).[4](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/OPERATIONS.md?display=source#L172) OPERATIONS.md §4e documents exactly this: the mount is activated "only at boot" and the recovery is `terraform -chdir=worker apply -replace='libvirt_domain.worker'`. 2. **Create the `pocket-id/` Terraform root** mirroring `vault/` — no new ProxyGroup (cluster-scoped `ingress-proxies` is owned by the vault root's state):[5](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/main.tf?display=source#L110) The `ingress-proxies` ProxyGroup manifest (no namespace → cluster-scoped) is declared in `vault/main.tf`, and OPERATIONS.md §1/§3 confirm it is Terraform-owned by the vault root. ```hcl # pocket-id/main.tf (namespace + helm + ingress, cp remote-state providers identical to vault/) resource "helm_release" "pocket_id" { name = "pocket-id" repository = "oci://ghcr.io/hobbit44/pocket-id-helm" chart = "pocket-id" version = "2.4.0" namespace = kubernetes_namespace_v1.pocket_id.metadata[0].name values = [file("${path.module}/values.yaml")] } resource "kubernetes_ingress_v1" "pocket_id" { metadata { name = "pocket-id" namespace = kubernetes_namespace_v1.pocket_id.metadata[0].name annotations = { "tailscale.com/proxy-group" = "ingress-proxies" } } spec { ingress_class_name = "tailscale" tls { hosts = ["pocket-id"] } default_backend { service { name = "pocket-id" port { number = 1411 } } } } } ``` 3. **`pocket-id/values.yaml`** — hobbit44 chart defaults need four overrides (probes default to `/health`, which Pocket ID doesn't serve; PVC default needs no provisioner; config-as-env; worker pinning):[6](https://github.com/hobbit44/pocket-id-helm/blob/main/values.yaml#L105) The chart's `livenessProbe`/`readinessProbe` default to `httpGet.path: /health`; Pocket ID's canonical health endpoint is `/healthz`.[7](https://github.com/pocket-id/pocket-id/blob/main/backend/internal/controller/healthz_controller.go#L27) (Nuance: v2.14.0 also serves `/health` as an alias of `/healthz`, so the chart's `/health` probe is not strictly dead — but `/healthz` is the documented, backend-native endpoint.) ```yaml image: tag: "v2.14.0" # chart AppVersion v2.4.0 is stale; track latest app release config: appUrl: "https://pocket-id.faun-snapper.ts.net" internalAppUrl: "http://pocket-id.pocket-id.svc.cluster.local:1411" trustProxy: "true" persistence: enabled: false # no StorageClass on Milner → hostPath, vault-style volumes: - name: pocket-id-data hostPath: { path: /var/mnt/pocket-id } volumeMounts: - name: pocket-id-data mountPath: /app/data # SQLite data/pocket-id.db + data/uploads live here livenessProbe: httpGet: { path: /healthz, port: http } readinessProbe: httpGet: { path: /healthz, port: http } nodeSelector: node-role: worker ``` Annotations on the values above: v2.14.0 is the current app release (2026-08-18) while the published hobbit44 chart 2.4.0 pins `appVersion: 2.4.0` and its statefulset defaults `image.tag` to `.Chart.AppVersion`, so the explicit tag override is required[8](https://github.com/pocket-id/pocket-id/releases/tag/v2.14.0). "No StorageClass on Milner" is consistent with the repo: nothing installs a provisioner today and vault uses a hostPath (`dataStorage.enabled: false` + volume to `/var/mnt/vault`) — note only `woodpecker/values.yaml` (a "coming soon" workload) references `local-path`[9](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/values.yaml?display=source#L39). The `/app/data` target matches the official v2.14.0 compose mount `./data:/app/data` and the app's SQLite/upload defaults (`data/pocket-id.db`, `data/uploads`)[10](https://raw.githubusercontent.com/pocket-id/pocket-id/v2.14.0/docker-compose.yml)[11](https://github.com/pocket-id/pocket-id/blob/main/backend/internal/common/env_config.go). ⚠ Caveat: the **published chart 2.4.0** unconditionally mounts the data volume at `/app/backend/data` (not `/app/data`) and does not gate that volumeMount on `persistence.enabled`, so the plan's hostPath volume + `/app/data` volumeMount must be reconciled with the chart's fixed `/app/backend/data` mount at install time. `node-role: worker` matches the worker machine-config nodeLabel[12](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/worker/main.tf?display=source#L326). Namespace gets the privileged PSA label (`hostPath`), exactly like the vault namespace.[13](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/main.tf?display=source#L75) The vault namespace carries `pod-security.kubernetes.io/enforce=privileged`. 4. **Imperative setup runbook → `OPERATIONS.md`** (§5 Pocket ID; renumber old §5 Client devices → §6, §6 Bring-up order → §7, fixing internal refs):[14](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/OPERATIONS.md?display=source#L201) Current headings: §5 "Client devices" (L201), §6 "Bring-up order (from a wiped cluster)" (L222); there is no §7 today, so the renumber is clean. - First-launch admin: browse `https://pocket-id.faun-snapper.ts.net/setup` from a tailnet device (passkey only, no default creds; needs HTTPS for WebAuthn).[15](https://pocket-id.org/docs/setup/installation) Pocket ID docs: HTTPS is required because the app uses the WebAuthn API; the first admin is created through the UI (`/setup` route exists in the app) — there is no env-var/default-password bootstrap. - Register the Vault OIDC client in Settings → OIDC Clients: name `Vault`, confidential, callback URLs `https://vault.faun-snapper.ts.net/ui/vault/auth/oidc/oidc/callback` + `http://localhost:8250/oidc/callback` (CLI flow). - Approve the new tailnet service in the admin console (Vault README §Prereq step 4 pattern).[16](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/OPERATIONS.md?display=source#L58) The approval pattern is real — OPERATIONS.md §2 item 4: "When a workload service first advertises, **approve it** — until then the MagicDNS name resolves but the service does not route." (Minor pointer fix: in `vault/README.md` the approve step is Prereq **3**, not 4 — item 4 there is client versions.) 5. **Vault OIDC auth method (imperative, OPERATIONS.md)** — provider-agnostic HashiCorp tutorial flow:[17](https://developer.hashicorp.com/vault/docs/auth/jwt) The `auth enable oidc` / `auth/oidc/config` / role shape and the CLI callback `http://localhost:8250/oidc/callback` match Vault's documented JWT/OIDC auth method (a localhost redirect URI is required for `vault login -method=oidc`; 8250 is the documented default listener). ```bash vault auth enable oidc vault write auth/oidc/config \ oidc_discovery_url="https://pocket-id.faun-snapper.ts.net" \ oidc_client_id="<uuid from Pocket ID>" \ oidc_client_secret="<secret from Pocket ID>" \ default_role="default" vault write auth/oidc/role/default \ user_claim="sub" \ allowed_redirect_uris="https://vault.faun-snapper.ts.net/ui/vault/auth/oidc/oidc/callback" \ allowed_redirect_uris="http://localhost:8250/oidc/callback" \ token_policies="default" # admin policy: path "sys/auth/oidc" { capabilities = ["create","read","update","delete","sudo"] } + auth/oidc/* + sys/policies/acl/* ``` 6. **Docs**: add Pocket ID to `README.md` workload list, `ARCHITECTURE.md` pod topology + network diagram + service table, `AGENTS.md` repo map; `tests/test_static.py` (`pocket-id/` to `terraform fmt` dirs, `pocket-id/values.yaml` to yamllint list) and `tests/test_terraform.py` (pocket-id root). ## Implementation Details - **Files created:** `pocket-id/main.tf`, `pocket-id/values.yaml`, `pocket-id/README.md`, `pocket-id/tests/plan.tftest.hcl` + `tests/fixtures/cp.tfstate` (copy of vault's fixture). - **Files modified:** `worker/main.tf`, `tests/test_static.py`, `tests/test_terraform.py`, `README.md`, `ARCHITECTURE.md`, `AGENTS.md`, `OPERATIONS.md`, `vault/README.md` (OIDC section pointer). - **Dependencies:** hobbit44 chart 2.4.0 from `oci://ghcr.io/hobbit44/pocket-id-helm`;[18](https://github.com/hobbit44/pocket-id-helm) official image `ghcr.io/pocket-id/pocket-id:v2.14.0` is multi-arch (arm64 ✓);[19](https://github.com/pocket-id/pocket-id/releases/tag/v2.14.0) vault-helm 0.34.1 supports `server.hostAliases` if ever needed.[20](https://github.com/hashicorp/vault-helm/blob/v0.34.1/values.yaml) (2.4.0 verified present on GHCR: the OCI repo `hobbit44/pocket-id-helm/pocket-id` lists a `2.4.0` tag; vault-helm v0.34.1 documents `server.hostAliases` and its statefulset template renders it.) - **Constraints:** arm64-only images; no storage class (hostPath via worker qcow2, privileged PSA); TLS always at the operator edge (LE cert), plaintext HTTP in-cluster; secrets/client id/secret never in repo or TF state (hand-run only). - **Decision point for reviewer — Vault server-side reachability.** Pocket ID's issuer is always `APP_URL` (`https://pocket-id.faun-snapper.ts.net`), and Vault is a strict go-oidc client: its `oidc_discovery_url` must equal the issuer *and* be reachable+trusted from the Vault pod.[21](https://github.com/hashicorp/vault/issues/25024) `INTERNAL_APP_URL` alone does **not** fix this (it rewrites only token/userinfo/jwks endpoints; issuer/auth endpoints stay on `APP_URL` — upstream discussion pocket-id/pocket-id#1169 is unanswered).[22](https://github.com/pocket-id/pocket-id/blob/main/backend/internal/controller/well_known_controller.go)[23](https://github.com/pocket-id/pocket-id/discussions/1169) (Verified in `well_known_controller.go`: `issuer`, `authorization_endpoint`, `end_session_endpoint`, `device_authorization_endpoint` all stay on `APP_URL`; only `token_endpoint`/`userinfo_endpoint`/`introspection_endpoint`/`jwks_uri`/PAR move to `INTERNAL_APP_URL`. Discussion #1169 is open with zero replies as of 2026-09-09, so it is indeed unanswered.) Workable topologies, in order of preference: 1. **In-cluster TLS + private CA**: CoreDNS hosts override or vault pod `hostAliases` mapping `pocket-id.faun-snapper.ts.net` → pocket-id ClusterIP, Pocket ID serving native TLS (`TLS_CERT_FILE`/`TLS_KEY_FILE`, PR #1429) with a small private CA, Vault pinning it via `oidc_discovery_ca_pem`.[24](https://github.com/pocket-id/pocket-id/pull/1429) (PR #1429 "feat: add TLS support for HTTP/2 server" introduced `TLS_CERT_FILE`/`TLS_KEY_FILE`; both env vars exist in the app's `env_config.go`.) Browser path unaffected (operator edge keeps its LE cert). 2. **Tailscale egress ProxyGroup + DNSConfig** so the Vault pod reaches the ts.net VIP over the tailnet (adds operator-level egress infra — more moving parts). - Until the reviewer picks one, steps 4–5 document the registration/config; the chosen topology lands as a follow-on commit within this issue. - **Out-of-scope notes:** no Terraform declarative for OIDC client registration (Pocket ID has no stable API — the community `pocket-id-operator` CRD is explicitly out of scope)[25](https://github.com/aclerici38/pocket-id-operator) (a community `aclerici38/pocket-id-operator` CRD project exists, but there is no official, stable client-management API to drive from Terraform — hand-run registration is the only supported path); no Postgres (SQLite on local qcow2 is the right homelab scale); no group restriction on the Vault client (leave default open, revisit later); Woodpecker CI OIDC is future work and not blocked by this design. ## Acceptance Criteria - [ ] `worker/main.tf` shows a second 10 GiB `pocket-id` qcow2 + `UserVolumeConfig` (mount `/var/mnt/pocket-id`) with unambiguous disk selectors; `terraform -chdir=worker plan/apply` + worker `-replace` activates it. - [ ] `pocket-id/` root passes `terraform fmt`, `terraform -chdir=pocket-id test` (mock-provider plan assertions: namespace privileged-PSA label, chart/version, ingress class `tailscale` + `proxy-group: ingress-proxies` annotation + `tls.hosts=["pocket-id"]`), and updated `pytest` lint lists are green. - [ ] Pocket ID pod is `Ready` on the worker (`/healthz` probe), data persists under `/app/data` across pod restarts, and `https://pocket-id.faun-snapper.ts.net` serves the UI + `/.well-known/openid-configuration` from a tailnet device. - [ ] Admin passkey created at `/setup`; a `Vault` OIDC client exists in Pocket ID with both callback URLs. - [ ] Vault JWT/OIDC auth method is enabled and configured per OPERATIONS.md (auth enable/config/role/policy commands committed there; login flow verified end-to-end once the reviewer picks the reachability topology). - [ ] README, ARCHITECTURE, AGENTS, OPERATIONS, vault/README updated and cross-references consistent. ## Testing Plan - Static: `python -m pytest tests/` (terraform fmt incl. `pocket-id/`, yamllint on `pocket-id/values.yaml`, ruff). - Plan assertions: `terraform -chdir=pocket-id test` — namespace PSA label, helm chart/repo/version, image tag `v2.14.0`, persistence disabled + hostPath mount at `/app/data`, `/healthz` probes, ingress pinned name + proxy-group annotation. - Worker root: extend `worker/tests/plan.tftest.hcl` — assert both `UserVolumeConfig`s exist and selectors are mutually exclusive. - Live smoke (manual, after apply): `kubectl -n pocket-id exec` → `curl :1411/healthz`; `curl https://pocket-id.faun-snapper.ts.net/.well-known/openid-configuration` from a tailnet device; kill the pod and confirm SQLite survives; browser `vault login -method=oidc` completes. ## Related Issues/PRs - Repo: [Pod - Hashicorp Vault (#33)](https://git.gloom.garden/tepichord/milner/pulls/33) — the pattern this mirrors (namespace, Helm, ProxyGroup ingress, hostPath volume). - Upstream: [pocket-id #857 / PR #858 — INTERNAL_APP_URL](https://github.com/pocket-id/pocket-id/pull/858), [discussion #1169 — issuer mismatch for in-cluster clients](https://github.com/pocket-id/pocket-id/discussions/1169), [hashicorp/vault #25024 — strict OIDC issuer check](https://github.com/hashicorp/vault/issues/25024). ## Research Findings - Best-maintained chart: **hobbit44/pocket-id-helm** (2.4.0, OCI `ghcr.io/hobbit44/pocket-id-helm`) — the one Pocket ID docs link to; actively renovate-updated; StatefulSet + PVC default, but its default `/health` probes are wrong (must be `/healthz`) and its 8 GiB PVC default needs no provisioner (disable + hostPath). - Expose via the existing cluster-scoped `ingress-proxies` ProxyGroup (reused across namespaces) with `tailscale.com/proxy-group` annotation + `tls.hosts=["pocket-id"]`; don't declare a second ProxyGroup. Set `TRUST_PROXY=true` since it sits behind the operator. - Storage: SQLite at `/app/data` (chart's `mountPath`); official image is multi-arch arm64; data dir needs local block storage (docs advise against NFS), so a dedicated worker qcow2 + hostPath matches Milner's no-provisioner reality. - Vault wiring uses the JWT/OIDC auth method (redirect URIs must match exactly between the Pocket ID client and Vault role — CLI `http://localhost:8250/oidc/callback` + UI `https://vault.faun-snapper.ts.net/ui/vault/auth/oidc/oidc/callback`); Pocket ID issues `aud=client id`, so no extra `bound_audiences` needed. - No default admin credentials — `/setup` passkey on first launch; admin console service approval is required before the MagicDNS name routes. ## Pre-Mortem 1. **The Vault↔Pocket ID issuer-reachability decision is left open and never converges** — Vault is a strict go-oidc client that requires `oidc_discovery_url` to equal Pocket ID's issuer (always the external `APP_URL`) and be reachable and trusted from the Vault pod, yet the plan defers choosing a topology (in-cluster TLS + private CA vs Tailscale egress) to a follow-on commit. Any slip in the DNS override, private-CA pinning, or cert wiring — or the follow-on commit never landing — fails the end-to-end Vault OIDC login acceptance criterion and leaves the deployment half-done. 2. **Worker reconfiguration disrupts the cluster or strands the Pocket ID data volume** — activating the boot-only mount requires `terraform apply -replace` of `libvirt_domain.worker`, which reboots the same node already hosting Vault, and the new size-based disk selectors must stay mutually exclusive with the existing vault selector. A selector or qcow2 attach mistake leaves `/var/mnt/pocket-id` unmounted (Pocket ID CrashLoopBackOff or data loss) and risks Vault availability during the forced replace. 3. **Helm service-port and probe wiring mismatches the chart's actual defaults** — the plan's ingress backend and `internalAppUrl` reference port 1411 while the hobbit44 chart's ClusterIP service exposes port 80 (targeting the app's internal 1411), so the discovery URL advertised to Vault or the ingress route can point at a dead port.[26](https://github.com/hobbit44/pocket-id-helm/blob/main/values.yaml#L71) ⚠ Correction: this premise does **not** hold for chart 2.4.0 or main — both set `service.port: 1411` and the statefulset's `containerPort: {{ .Values.service.port }}` (the service `targetPort` is the named `http` port), so the ClusterIP service and the plan's `:1411` backend agree. The probe-side risk stands: default probes are `/health` and the pod flaps unless overridden to `/healthz`. The chart's default probes also hit `/health` rather than Pocket ID's real `/healthz`, so any missed override makes the pod flap on failed probes.

tepichord added

and removed

labels

2026-09-09 14:41:08 -07:00

tepichord referenced this issue

2026-09-09 16:17:00 -07:00

Add Postgres Database to Milner #35

tepichord referenced this issue from a commit