tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Pod - Woodpecker CI #37

Merged
tepichord merged 34 commits from feat/19-pod-woodpecker-ci into main 2026-09-10 15:39:06 -07:00
Conversation 0 Commits 34 Files changed 47 +6041 -128
tepichord commented 2026-09-09 15:28:46 -07:00
Owner
Copy link

Closes #19

Automated PR created by forgejo-done adapter.

Closes #19 Automated PR created by forgejo-done adapter.
- Two namespaces: woodpecker (baseline PSA) and woodpecker-runtime
  (privileged PSA for root clone step)
- Pinned helm_release: OCI chart woodpecker v3.7.3 in woodpecker namespace
- Tailscale ingress reusing vault-owned ingress-proxies ProxyGroup,
  backend woodpecker-server:80, tls.hosts [woodpecker]

Refs: #19
- .gitignore matches vault pattern (.terraform, *.tfstate, etc.)
- .terraform.lock.hcl generated by terraform init (kubernetes ~>2.35,
  helm ~>2.17)

Refs: #19
- Server: WOODPECKER_HOST, WOODPECKER_ADMIN, Forgejo on git.gloom.garden,
  woodpecker-forgejo-oauth envFrom, local-path PVC 10Gi
- Agent: kubernetes backend -> woodpecker-runtime, local-path, RWX false,
  volume size 2G, 1 replica, worker nodeSelector
- Fixes skeleton bugs: runtime namespace was woodpecker (now woodpecker-runtime),
  RWX was true (now false for local-path RWO)

Refs: #19
- plan.tftest.hcl: mock kubernetes/helm providers, assert PSA labels
  (baseline/privileged), helm_release chart/version/namespace, ingress
  proxy-group annotation and backend service
- fixtures/cp.tfstate: copied from vault tests fixture

Refs: #19
- test_static.py: add woodpecker/ to terraform fmt -check -recursive dirs
- test_terraform.py: add test_terraform_test_woodpecker() function

Refs: #19
- Prerequisites, Forgejo OAuth registration with exact callback URL,
  Vault kv put, one-time kubectl secret sync
- Terraform deploy and post-apply verification steps
- Out-of-scope notes for VSO, local-path, and agent secret

Refs: #19
- Forgejo OAuth app registration with exact callback URL
- Vault kv put for OAuth credentials
- One-time kubectl secret sync to cluster
- Terraform deploy and Tailscale service approval
- Post-deploy verification steps

Refs: #19
Refs: #19
The woodpecker agent chart (v3.7.3) renders replicas from
.Values.replicaCount; the previous agent.replicas key was silently
ignored and the chart default of 2 applied.
tepichord added this to the MVP project 2026-09-09 17:40:07 -07:00 
Worker-attached storage tier for dynamic PVCs, serving Woodpecker today and
future workloads (Forgejo, RabbitMQ...).

- worker/main.tf: attach milner-worker-1-data.qcow2 (vdc, 100 GiB) and add
  UserVolumeConfig 'data' -> /var/mnt/data. Vault selector narrowed from
  '!system_disk' to a CEL size band ('< 40u * GiB') so the two dedicated
  disks stay mutually exclusive regardless of attach order; data uses the
  complementary '>= 40u * GiB' band.
- storage/ (new root): local-path-provisioner v0.0.37 (official OCI chart)
  in a privileged local-path-storage namespace, serving two contract-named
  StorageClasses rooted on the data disk: 'persistent' (default, durable,
  backed up) and 'ephemeral' (scratch, never backed up). Values + README
  document the contracts.
- worker tests: +data volume/vdc/UserVolumeConfig runs (19 total, cdrom and
  eject-index updates). storage tests: namespace PSA, pinned OCI chart, and
  per-class contract assertions (4 runs).
- harness/docs: storage/ in fmt + terraform test runners; OPERATIONS.md §8
  (contracts, first-time reboot, deploy, smoke test, capacity/backup);
  AGENTS/ARCHITECTURE/worker-README updated.

Ephemerality is a retention contract, not a second disk: both classes share
the data volume, differentiated by backing dir and backup coverage.
- values.yaml: server + agent PVCs -> persistent (durable, backed up);
  WOODPECKER_BACKEND_K8S_STORAGE_CLASS -> ephemeral (per-pipeline scratch,
  never backed up). Both classes come from the storage/ root.
- README: prerequisite is now the storage root; storage tiers documented.
values.yaml declared volumes/volumeMounts at the top level, which the
hashicorp/vault chart ignores. Vault's file backend was therefore writing
to the container overlay (ephemeral), not the worker's dedicated vault
qcow2 (/var/mnt/vault) as documented. The worker reboot recreated the pod,
the overlay vanished, and Vault reported 'not initialized' with no data on
the disk.

Moved the wiring under server: (chart keys server.volumes /
server.volumeMounts); helm template now renders the hostPath volume +
/vault/data mount into the StatefulSet. Requires a fresh vault operator
init — the old barrier state is unrecoverable.
hostPath volumes do not receive fsGroup ownership, so a freshly provisioned
vault disk is root:root while the chart runs Vault as uid 100 — 'vault
operator init' failed with 'mkdir /vault/data/core: permission denied'.

- values.yaml: root init container chowns /vault/data (100:1000) before
  Vault starts; server.statefulSet.securityContext.pod relaxed to
  runAsNonRoot: false only to permit that container — Vault itself keeps
  running as uid 100. Durable across disk re-provisioning and reinstall.
- tests/test_static.py: regression test parsing vault/values.yaml to assert
  volumes/volumeMounts nest under server: (top-level was silently ignored and
  sent Vault data to the container overlay), dataStorage stays disabled, and
  the chown/securityContext wiring is intact.

Verified live: vault pod uid=100(vault), /vault/data owned vault:vault,
write test OK.
- .env.example: every credential needed to bring the cluster up — Tailscale
  operator OAuth client id/secret, Vault root token (unseal keys optional and
  commented, preferring the read -s prompt), Woodpecker/Forgejo OAuth app
  credentials — each annotated with where it is created and used. Copy to
  .env, chmod 600, source before working.
- .gitignore: ignore .env / .env.* while keeping .env.example.
- README: Credentials subsection under Getting Started.
- OPERATIONS.md §4c-bis: enable the generic KV v2 engine after first init
  (fresh Vault has no secret/ mount, so kv put fails 403); §7b now passes
  VAULT_TOKEN into the pod and references the mount.
The Woodpecker Helm release waits for pod readiness, and the server pod
resolves extraSecretNamesForEnvFrom at container start — so the
woodpecker-forgejo-oauth Secret must exist before the full apply, or the
install fails with 'context deadline exceeded'. The sync snippet also did not
pass VAULT_TOKEN into the pod (kv get 403s without it).

- OPERATIONS §7c: targeted namespace apply -> idempotent secret sync via
  --dry-run=client | kubectl apply (re-run to rotate); §7d notes the ordering.
- woodpecker/README: same corrected snippet and ordering note.
The server logged 'WOODPECKER_GRPC_SECRET is not set; generated a temporary
random secret', which invalidates issued agent tokens on every restart — the
same class of durability bug as the Vault overlay issue.

- Vault gains secret/woodpecker/grpc (generated once, kept as source of
  truth) synced to the woodpecker-grpc Secret; values.yaml injects it into
  both server and agent via extraSecretNamesForEnvFrom. The Forgejo OAuth
  Secret stays server-only.
- OPERATIONS §7c documents generation + sync and that both Secrets must exist
  before the full apply; woodpecker/README notes the managed secret.

Verified live: server no longer warns, server+agent 1/1 Running.
Follow-up to the incident where a whole-table dump of Woodpecker's
server_configs exposed jwt-secret and signature-private-key:

- AGENTS.md: new hard rule against dumping credential-bearing state (no
  select *, no whole-table dumps of server_configs/secrets/agents/forge token
  columns/Vault paths); schema via names or explicit column allowlists, print
  verdicts (counts, hashes, MATCH/DIFFERENT), read comparison values via stdin
  so they never reach argv or a log line.
- OPERATIONS.md: §1 row recording that both keys are server-generated into
  server_configs, and §7g runbook for rotating them (deletion is rotation -
  the server regenerates missing keys on start), including the verified
  scale-to-0 / rw-probe delete / scale-up flow, hash-based verification, and
  the two side effects (sessions invalidated, forge webhook token stale ->
  re-sync the repo).
- woodpecker/README.md: out-of-scope note pointing at §7g and stating why
  jwt-secret exposure is account-impersonation.
Woodpecker exposes no Settings -> repair/re-sync path; each repository has a
Repair action, which re-creates the forge webhook with a fresh hook token once
jwt-secret (and with it every hook token) has been rotated. Confirmed against
the UI during the first rotation drill.
The forge (git.gloom.garden) is off-tailnet: it cannot resolve MagicDNS names
and the ingress proxies sit on 100.64.0.0/10, so every delivery to the
tailnet-only Ingress failed with 'lookup woodpecker.faun-snapper.ts.net ...
no such host' - no push event ever reached Woodpecker.

- woodpecker/main.tf: new 'woodpecker-hook' Ingress with
  tailscale.com/funnel=true on a dedicated hostname, routing only /api/hook
  (Prefix is the operator's only supported path type). The rule deliberately
  sets no 'host': the operator derives the hostname from the Ingress name and
  rejects rules that set one (InvalidIngressBackend, observed live). The UI and
  API stay tailnet-only on the existing Ingress.
- tests: plan assertions for the funnel annotation, dedicated hostname,
  hook-only path, Prefix path type, and backend.
- docs: OPERATIONS §7h (rationale, the two admin-console prerequisites - service
  approval and the funnel nodeAttr for tag:k8s - the Forgejo webhook host edit,
  and the Repair-rewrites-the-URL caveat), §1 row, woodpecker/README note, and
  the ARCHITECTURE network topology note plus diagram edge.
The Vault (20 GiB) and shared-data (100 GiB) qcow2s hold the only data in
the cluster that cannot be rebuilt from code, yet they lived in the worker
root — whose cleanup.sh destroys them.

Move them to a new data/ root that no teardown script runs:

- prevent_destroy plus a preservation-marker precondition, because the
  libvirt provider reads a directory-pool volume by its host-path key and
  silently drops it from state when the file has moved, which would let a
  later apply create an empty disk where the data used to be;
- cleanup.sh preserves by moving (never copying or deleting) to a sibling
  preserved/ directory with a sha256 manifest; restore.sh verifies the
  digests, moves them back, re-imports if state lost track, and refuses to
  finish unless terraform plan reports no changes;
- worker/main.tf keeps permanent removed { lifecycle { destroy = false } }
  blocks recording the handover and reads the volume names from data/ via
  terraform_remote_state, so the attached files cannot drift from the files
  that root protects;
- control-plane/ and worker/ cleanup.sh sweep orphaned libvirt domains left
  by a failed create (drill 4: "Failed to Wait for IP Address" defined a
  domain in libvirt but never persisted it to state, so no destroy could
  reach it), scoped to their own VM names.

The persistent StorageClass switches to reclaimPolicy Retain and
pathPattern {{ .PVC.Namespace }}/{{ .PVC.Name }}/ so a reinstall adopts the
existing directory instead of orphaning a UID-named one; ephemeral keeps
Delete and UID-derived names deliberately.

Verified by the drill-4 teardown and reinstall (OPERATIONS.md §9).
The forge (git.gloom.garden) is off-tailnet and its resolver answers
NXDOMAIN for the whole ts.net zone, so no Tailscale Funnel hostname can
ever reach it. Publish only /api/hook on ci.vando.dev instead:

- tunnel with a single ingress route to woodpecker-server:80 and a proxied
  CNAME at the tunnel hostname;
- zone rules on the Free plan: rate limiting (429), POST-only (403), and an
  optional source-IP allowlist, all keyed on the path and reusable for
  future webhook apps;
- two cloudflared connectors under restricted Pod Security Admission,
  non-root, read-only root filesystem, token via env from a Secret.

Retire the Funnel Ingress from woodpecker/ (it also carried no
network-layer control at all) and point WOODPECKER_EXPERT_WEBHOOK_HOST at
https://ci.vando.dev so Activate/Repair register a URL the forge can
actually resolve. The UI stays tailnet-only on the woodpecker Ingress.
- OPERATIONS.md §9: durable-volume protection, the teardown + reinstall
  procedure with preservation, the drill ladder, and the traps that cost
  real time (§9e) — provider forgetting moved volumes, the pool cache lie
  after an out-of-band move, and preconditions running on every plan;
- §7h: the Cloudflare Tunnel endpoint (why Funnel could not work), the edge
  rules, and the off-tailnet verification recipe;
- §7i: rotating the per-entity token secrets (users.hash, repos.hash);
- README/ARCHITECTURE/AGENTS: the new data/ and cloudflare/ roots, the
  apply order, and the credential-handling hard rule.
Static guards for the things that must not silently regress:

- the durable volumes stay out of the worker root, remain behind data/'s
  prevent_destroy and marker precondition, the marker stays gitignored, and
  restore.sh keeps its precondition override;
- WOODPECKER_EXPERT_WEBHOOK_HOST matches a published cloudflare/ route and
  is not a ts.net name;
- Funnel stays retired from woodpecker/ while the UI keeps the shared
  proxy group;
- both teardown scripts sweep orphaned domains scoped to their own VMs.

Plus terraform test harnesses for the new data/ and cloudflare/ roots.
nixpkgs' plain `helm` attr is an unrelated polyphonic synthesizer, so the
Kubernetes chart manager is the `kubernetes-helm` attribute. OPERATIONS §3
installs the Tailscale operator with the helm CLI.
Two drill-4 findings.

Teardown order. The two places people actually read disagreed — OPERATIONS.md
§9c's code block ran control-plane/cleanup.sh first (the prose two paragraphs
down already said worker-first), and data/cleanup.sh's epilogue plus
worker/cleanup.sh's banner printed the reverse. Control-plane cleanup rm -rf's
the pool directory, so the worker must already be down (its VM holds its
OS-disk file inside that pool; the sweep deletes it out from under a running
domain and terraform destroy hits libvirt pool-in-use errors) and preserve must
come before the sweep or it takes the durable qcow2s with it. Order is now
worker -> preserve -> control-plane, checked by hand.

Manifest lifecycle. data/restore.sh deletes preserved/manifest.json on success
so `preserved/` non-empty means exactly "a preservation is outstanding". Left
behind, the manifest describes volumes that have been written to since, so a
later restore would fail its own digest check. A second restore run with
neither marker nor manifest now exits 0; a marker without a manifest stays a
hard error.

Verified by the drill-4 rerun: teardown + reinstall with Vault and Woodpecker
data preserved (same sqlite inode 201326721, same marker digest, same Vault
secret versions), services re-approved, all 7 roots plan clean.
The drill-4 rerun showed the corrected worker -> preserve -> control-plane
order only lived in OPERATIONS.md §9c/§9e; the READMEs and AGENTS.md still
printed the old (unsafe) order, so the next person to follow them would hit
the same pool-in-use mess. Sweep:

- README.md (control-plane and worker runbooks), AGENTS.md, worker/README.md
  and data/README.md all state worker -> preserve -> control-plane now.
- data/README.md's host layout notes that preserved/manifest.json exists
  only while a preservation is outstanding (restore.sh deletes it).
- OPERATIONS.md §2.4 and §7e say the pending service approval is listed under
  the *advertising proxy device* (`milner-ingress-proxies`), not the
  `tailscale-operator` device — the confusion the rerun surfaced.
- §9d: the drill-4 row shows the corrected order and the clean rerun; a new
  paragraph records the rerun anchors; node-name examples are labelled as
  per-run rather than fixed.
OPERATIONS.md §9d used to record the drill ladder's evidence as literals: the
sqlite inode, node names and UIDs, the etcd member ID, the cluster CA, PV UIDs,
secret digests. Half of those regenerate on every reinstall, so the section had
to be hand-edited after each drill — it was edited again during the 2026-09-10
rerun — and was simply wrong in between. A runbook that is wrong between drills
cannot be trusted to say what "passing" means, which is its only job.

Those values are per-run observations, so they now live here instead, in a
gitignored state file:

  drills/check.sh snapshot                  # before
  drills/check.sh verify --mode restart     # after; non-zero exit on regression

The modes encode what each rung is allowed to change: `clean` (drill 1) wants
everything identical, `restart` (2-3) additionally requires the boot ids to
differ or the node never actually restarted, and `reinstall` (4) compares only
the data, reporting regenerated identity as information. That distinction is
the point — drill 4 is a new cluster, so comparing identity by hand is what
makes a passing drill read as a failure.

Nothing prints a value. Vault is read through `vault status` and
`vault kv metadata get`; the marker is hashed inside vault-0 and only the
16-char digest crosses the boundary; the token arrives on stdin, never argv.
The sqlite facts come from a read-only privileged pod on the worker, which
reads a *copy* of the database, because an in-place read of a live sqlite can
fail on a hot WAL.

Verified against the live cluster: snapshot collects 20 predicates (sqlite
inode 201326721, Vault cluster id, both PVC hostPaths), `verify --mode clean`
passes on all 20, `--mode restart` fails exactly the two boot ids because
nothing has restarted, and `--mode reinstall` passes with identity reported as
information. shellcheck clean; state file gitignored.

AGENTS.md gains the repo-map row and a note that this is not part of the
offline suite, since it needs a live cluster.
Follows the harness added in the previous commit. §9d now carries the ladder
table with its `--mode`, the two classes of drill and what each may change, the
predicates that must hold in both, and an explicit list of what regenerates on
every install and must never be recorded or compared. The per-run values are
gone; `drills/check.sh` remembers them.

The rewrite also surfaced three stale or missing things:

- §7's secret-rotation probe still globbed the pre-§9f path layout
  (`pvc-<uid>_woodpecker_<claim>`). That layout no longer exists, so the
  snippet would have died with IndexError. `pathPattern` makes the path
  deterministic, so it is now a literal.
- §9c's reinstall steps never re-pointed the client configs. `apply`
  regenerates the cluster CA while `~/.talos/config` keeps the old one, so
  `talosctl` reports "certificate signed by unknown authority" and `etcd
  members` returns nothing, on a cluster that is otherwise perfectly healthy.
  This was not hypothetical: it was the state of this workstation when this
  work started, and it is now a step in §9c plus a trap in §9e.
- §9e gains the traps that were buried in §9d's narrative: the newline that
  `vault kv get -field` appends (which gave one secret two digests and nearly
  read as a failed drill), the `sys/token/` exclusion, and the stale-talosconfig
  symptom.

tests/test_docs.py guards the rot from returning, and that §9d keeps pointing
at the harness rather than drifting back into holding values itself.
Two things the first live run of the harness turned up.

`vault kv metadata get -format=json` carries `created_time` twice: once at
`data.created_time` and again inside `data.versions.<n>`. `updated_time` appears
only once. The extraction collected every match, so `c` was a two-line string
and `u` a one-line one — `created == updated` was therefore false for all three
Woodpecker paths, reporting secrets as rewritten when none had been touched
since they were created. It takes the first match per field now, and
tests/test_static.py runs the script's own expressions against a fixture with
the nested copy, so dropping the `head -n1` fails the suite rather than the next
drill.

Rerun against the live cluster with the token loaded: all three paths read
`v1/created==updated:yes`, and `marker_digest` comes out `bbb2250a5b3a3774` —
byte-identical to the value recorded for drill 4. That is the first independent
confirmation that the digest convention in §9e (`printf %s` around
`vault kv get -field`) is the one the drill actually used, and the first time
the KV predicates have been exercised at all.

The token now comes from .env when it is not already exported. Only that one
variable is read rather than sourcing the file: .env.example documents
`set -a && . ./.env && set +a`, which would export the Tailscale, Forgejo and
Cloudflare credentials into every kubectl/talosctl child process this script
spawns — more exposure than reading a single secret needs. §9e records the
repeated-field trap alongside the newline one.

Verified: 25 predicates collected, `verify --mode clean` passes 24/24 with the
KV predicates included (0 skipped, 0 failed).
A dotenv file is the operator's interface to their own shell; no script should
read one. Loading it inside the harness meant parsing it — a second dotenv
implementation, with its own quoting and comment rules to get wrong — and pulled
every other credential in the file (Tailscale, Forgejo, Cloudflare) into the
environment of each kubectl and talosctl child the script spawns.

The harness now reads $VAULT_TOKEN like any other environment variable, and the
operator loads it once with the flow already documented in README.md:

  set -a && . ./.env && set +a

Because a smaller predicate set silently narrows what a drill proves, `verify`
now says so outright — "VAULT_TOKEN is not set, so the Vault KV predicates were
not checked at all" — rather than quietly reporting a shorter list. `snapshot`
already warned.

AGENTS.md records the rule under Code Conventions, and tests/test_static.py
enforces it: no executable line in the harness may name a dotenv file. Comments
still may, so the reasoning can be written down next to the code.

Verified both paths against the live cluster. Without the token: 21 predicates
collected, `verify --mode clean` 20 checked / 0 failed, with the NOTICE. With
.env loaded by the shell: 25 predicates, 24 checked / 0 failed / 0 skipped,
every KV path `v1/created==updated:yes` and `marker_digest=bbb2250a5b3a3774`.
`VAULT_TOKEN` was exported, so every `kubectl` and `talosctl` call the
harness makes inherited it. Drop the export attribute before the first
fork; the shell keeps the value, the children do not.

Also pin `-c vault` on both execs into vault-0, so the calls do not
depend on kubectl's container-defaulting heuristic (which currently
prints a note on every invocation). The init container sitting next to
vault in that pod is not a safe default to leave unpinned.

Add a static guard for both: the unexport must be present, and every
line reaching into the pod must name the container.

Also note why `vault_sealed` is recorded but never compared, which is
the reason a snapshot holds one more line than a verify reports.
§4f gives the drill ladder its own credential: the `drill-check` policy
reads KV metadata plus the one drill marker, and deliberately cannot
read `secret/data/woodpecker/*` — metadata without bytes. Mints a 2 h
token from the root token and revokes it after.

§9d gains a table of which predicates need a credential at all. Most do
not: sqlite, PVC, and mount facts come from the debug pod, and
`vault_cluster` is unauthenticated but load-bearing, since a wiped or
re-initialised Vault mints a new cluster_id. The token only adds the
layer above that, which is worth having for a deliberate drill and
overkill for a routine reboot.

§4g records `vault operator generate-root`, including that the root
token appears in local argv because the unseal key has to arrive on
stdin, and that rotating a credential without updating its masking rule
fails silently — masking looks healthy while it has stopped masking.

.env.example's VAULT_TOKEN comment no longer claims the token comes
only from `vault operator init`.
tepichord deleted branch feat/19-pod-woodpecker-ci 2026-09-10 15:39:06 -07:00
tepichord referenced this pull request from a commit 2026-09-10 15:39:07 -07:00
Pod - Woodpecker CI (#37)
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner!37
No description provided.