tepichord/milner

Fork 0

Pod - Woodpecker CI #19

New issue

Closed

opened 2026-08-19 23:34:29 -07:00 by tepichord · 1 comment

tepichord commented

2026-08-19 23:34:29 -07:00

Owner

Define IaC code to stand up Woodpecker service for forgejo CI/CD.
Use the helm chart to set up woodpecker:

agent:
  environment:
    WOODPECKER_BACKEND: kubernetes
    WOODPECKER_BACKEND_K8S_NAMESPACE: woodpecker-runtime
    # Optional: if you need to pull private images
    # WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES: <name_of_secret>

Get a single woodpecker server, and woodpecker agent/runner pod running
Secrets needed by Woodpecker to be managed by the HC Vault instance on milner.

- Define IaC code to stand up Woodpecker service for forgejo CI/CD. - Use the helm chart to set up woodpecker: ```yaml agent: environment: WOODPECKER_BACKEND: kubernetes WOODPECKER_BACKEND_K8S_NAMESPACE: woodpecker-runtime # Optional: if you need to pull private images # WOODPECKER_BACKEND_K8S_PULL_SECRET_NAMES: <name_of_secret> ``` - Get a single woodpecker server, and woodpecker agent/runner pod running - Secrets needed by Woodpecker to be managed by the HC Vault instance on milner.

tepichord added the

labels

2026-08-19 23:34:29 -07:00

tepichord added this to the MVP project

2026-08-19 23:34:29 -07:00

tepichord referenced this issue

2026-09-08 16:03:17 -07:00

Pod - Hashicorp Vault #18

tepichord added

and removed

labels

2026-09-09 12:34:39 -07:00

tepichord commented

2026-09-09 13:28:54 -07:00

Author

Owner

Naina:

Overview

Deploy Woodpecker CI (single server + one Kubernetes-backend agent) with a new woodpecker/ Terraform root that mirrors the vault/ pattern: two namespaces, one pinned helm_release from the official Woodpecker chart, and a Tailscale operator Ingress reusing the existing ingress-proxies ProxyGroup. Forgejo OAuth credentials are sourced from the in-cluster Vault via a one-time documented sync; runtime build pods land in the woodpecker-runtime namespace on local-path storage.

Technical Approach

Add woodpecker/main.tf — copy the providers/remote-state preamble verbatim from vault/main.tf (control_plane_state_path, cluster_name variables; terraform_remote_state.cp; kubernetes + helm providers from local.kc), then create the namespaces, release, and ingress:¹

resource "kubernetes_namespace_v1" "woodpecker" {
  metadata {
    name = "woodpecker"
    labels = { "pod-security.kubernetes.io/enforce" = "baseline" }   # chart pods: non-privileged, PVC only
  }
}

# Must exist before helm install: the agent chart renders its Role/RoleBinding
# INTO this namespace. Privileged: default clone step (plugin-git) runs as root.
resource "kubernetes_namespace_v1" "woodpecker_runtime" {
  metadata {
    name = "woodpecker-runtime"
    labels = { "pod-security.kubernetes.io/enforce" = "privileged" }
  }
}

resource "helm_release" "woodpecker" {
  name       = "woodpecker"                    # keep: defaults like woodpecker-server:9000 assume it
  repository = "oci://ghcr.io/woodpecker-ci/helm"
  chart      = "woodpecker"
  version    = "3.7.3"                         # app 3.18.x; verify latest at implement time
  namespace  = kubernetes_namespace_v1.woodpecker.metadata[0].name
  values     = [file("${path.module}/values.yaml")]
  depends_on = [kubernetes_namespace_v1.woodpecker, kubernetes_namespace_v1.woodpecker_runtime]
}

Assumption check — the agent chart renders its Role/RoleBinding INTO WOODPECKER_BACKEND_K8S_NAMESPACE, so that namespace must pre-exist² ³; the default clone step (plugin-git) runs as root, hence the privileged runtime namespace⁴.

Add woodpecker/.gitignore — copy vault/.gitignore (.terraform, *.tfstate, !tests/fixtures/*.tfstate, terraform.tfvars, *.auto.tfvars)⁵. Commit the .terraform.lock.hcl generated by terraform -chdir=woodpecker init.
Expand woodpecker/values.yaml (fixes the two skeleton bugs: runtime namespace was woodpecker, RWX left at chart-default true)⁶:

---
server:
  env:
    WOODPECKER_HOST: https://woodpecker.faun-snapper.ts.net   # OAuth redirect + webhook base URL
    WOODPECKER_ADMIN: <forgejo-admin-usernames>               # not woodpecker,admin
    WOODPECKER_FORGEJO: "true"
    WOODPECKER_FORGEJO_URL: https://git.gloom.garden
  extraSecretNamesForEnvFrom:
    - woodpecker-forgejo-oauth    # keys WOODPECKER_FORGEJO_CLIENT/_SECRET (Vault-backed, step 6)
  nodeSelector:
    node-role: worker
  persistentVolume:
    storageClass: local-path       # server SQLite DB lives at /var/lib/woodpecker
    size: 10Gi

agent:
  replicas: 1
  nodeSelector:
    node-role: worker
  env:
    WOODPECKER_BACKEND: kubernetes
    WOODPECKER_BACKEND_K8S_NAMESPACE: woodpecker-runtime
    WOODPECKER_BACKEND_K8S_STORAGE_CLASS: local-path
    WOODPECKER_BACKEND_K8S_STORAGE_RWX: "false"   # local-path is RWO-only; chart default true won't bind
    WOODPECKER_BACKEND_K8S_VOLUME_SIZE: 2G        # chart default 10G
  persistence:
    storageClass: local-path
    size: 1Gi

Assumption check — chart defaults: agent env ships WOODPECKER_BACKEND_K8S_STORAGE_RWX: true and WOODPECKER_BACKEND_K8S_VOLUME_SIZE: 10G⁷; local-path provisioned volumes are single-node (RWO) unless a shared filesystem is configured⁸.

Keep server.createAgentSecret at the default (true): the chart-generated woodpecker-default-agent-secret is an internal trust secret that never needs operator knowledge.⁹ ¹⁰

Add the Tailscale ingress (reuse ingress-proxies — ProxyGroup is cluster-scoped, owned by vault/main.tf; no new ProxyGroup resource):¹¹

resource "kubernetes_ingress_v1" "woodpecker" {
  metadata {
    name      = "woodpecker"
    namespace = kubernetes_namespace_v1.woodpecker.metadata[0].name
    annotations = {
      "tailscale.com/proxy-group" = "ingress-proxies"
    }
  }
  spec {
    ingress_class_name = "tailscale"
    tls {
      hosts = ["woodpecker"]          # -> woodpecker.faun-snapper.ts.net (LE cert)
    }
    default_backend {
      service {
        name = "woodpecker-server"    # chart Service http port 80 -> container :8000
        port { number = 80 }
      }
    }
  }
  depends_on = [helm_release.woodpecker]
}

No external gRPC (9000) ingress: agents are in-cluster and reach woodpecker-server:9000 over the Service.¹² ¹³

Add woodpecker/tests/ — copy vault/tests/fixtures/cp.tfstate, then plan.tftest.hcl mirroring vault/tests/plan.tftest.hcl with mock kubernetes/helm providers:¹⁴ ¹⁵

run "namespaces_have_psa_labels" {
  command = plan

  assert {
    condition     = kubernetes_namespace_v1.woodpecker.metadata[0].labels["pod-security.kubernetes.io/enforce"] == "baseline"
    error_message = "woodpecker namespace must be baseline PSA"
  }
  assert {
    condition     = kubernetes_namespace_v1.woodpecker_runtime.metadata[0].labels["pod-security.kubernetes.io/enforce"] == "privileged"
    error_message = "woodpecker-runtime namespace must be privileged PSA (root clone step)"
  }
}

run "helm_release_uses_official_chart_pinned" {
  command = plan

  assert {
    condition     = helm_release.woodpecker.chart == "woodpecker"
    error_message = "release must use the umbrella woodpecker chart"
  }
  assert {
    condition     = helm_release.woodpecker.version == "3.7.3"
    error_message = "chart version must be pinned"
  }
  assert {
    condition     = helm_release.woodpecker.namespace == "woodpecker"
    error_message = "release must deploy to the woodpecker namespace"
  }
}

run "ingress_targets_woodpecker_server_via_existing_proxy_group" {
  command = plan

  assert {
    condition     = kubernetes_ingress_v1.woodpecker.metadata[0].annotations["tailscale.com/proxy-group"] == "ingress-proxies"
    error_message = "ingress must reuse the vault-owned ingress-proxies ProxyGroup"
  }
  assert {
    condition     = kubernetes_ingress_v1.woodpecker.spec[0].tls[0].hosts[0] == "woodpecker"
    error_message = "tls.hosts must pin the MagicDNS name to woodpecker"
  }
  assert {
    condition     = kubernetes_ingress_v1.woodpecker.spec[0].default_backend[0].service[0].name == "woodpecker-server"
    error_message = "backend must be the woodpecker-server Service"
  }
  assert {
    condition     = kubernetes_ingress_v1.woodpecker.spec[0].default_backend[0].service[0].port[0].number == 80
    error_message = "backend must target the chart's http Service port 80"
  }
}

Wire the root test suites — tests/test_static.py: add woodpecker/ to the terraform fmt -check -recursive dirs¹⁶; tests/test_terraform.py: add a test_terraform_test_woodpecker() mirroring the vault one¹⁷.
Document the out-of-band flow — rewrite woodpecker/README.md (Terraform deploy + secrets) and add a Woodpecker section to OPERATIONS.md:

# 1. Forgejo (admin console / user settings): register OAuth2 app
#    callback URL MUST be exactly https://woodpecker.faun-snapper.ts.net/authorize
# 2. Vault is the source of truth:
kubectl -n vault exec vault-0 -- vault kv put secret/woodpecker/forgejo-oauth \
  WOODPECKER_FORGEJO_CLIENT=<client-id> WOODPECKER_FORGEJO_SECRET=<client-secret>
# 3. One-time sync into the cluster (before/soon after `terraform apply`):
kubectl -n woodpecker create secret generic woodpecker-forgejo-oauth \
  --from-literal=WOODPECKER_FORGEJO_CLIENT="$(kubectl -n vault exec vault-0 -- vault kv get -field=WOODPECKER_FORGEJO_CLIENT secret/woodpecker/forgejo-oauth)" \
  --from-literal=WOODPECKER_FORGEJO_SECRET="$(kubectl -n vault exec vault-0 -- vault kv get -field=WOODPECKER_FORGEJO_SECRET secret/woodpecker/forgejo-oauth)"
# 4. Tailscale admin console: approve the new woodpecker.faun-snapper.ts.net service

Assumption check — Forgejo OAuth: the callback URL must be exactly https://<host>/authorize¹⁸; the kubectl -n vault exec vault-0 -- vault kv pattern matches existing ops usage (which passes -address=http://127.0.0.1:8200)¹⁹.

Apply order on a fresh cluster: vault root first (owns the ProxyGroup the ingress annotation references), then woodpecker.²⁰ Nothing secret is ever written to the repo.

Implementation Details

Files to modify/create

woodpecker/main.tf (new) — providers/remote-state preamble copied from vault/main.tf; namespaces, helm_release, ingress above.
woodpecker/values.yaml (modify) — full file in step 3.
woodpecker/values.yaml lint note: yamllint already checks this file (tests/test_static.py).
woodpecker/.gitignore (new), woodpecker/.terraform.lock.hcl (new, committed), woodpecker/tests/plan.tftest.hcl + woodpecker/tests/fixtures/cp.tfstate (new).
woodpecker/README.md (rewrite), OPERATIONS.md (new §Woodpecker), tests/test_static.py, tests/test_terraform.py (modify).
No changes to control-plane/, worker/, or vault/.

Dependencies / constraints

Chart woodpecker 3.7.3 (agent 2.0.1 / server 3.0.1 subcharts, app 3.18.0)²¹ ²²; alternative repository = "https://woodpecker-ci.org/" (deprecated-but-functional, used in the chart README's own snippet) if OCI pulls fail²³.
Terraform providers match vault/main.tf: kubernetes ~> 2.35, helm ~> 2.17.²⁴
Release name stays woodpecker in namespace woodpecker so chart defaults (woodpecker-server:9000, agent secret name) need no overrides.²⁵
Runtime pods/clones need egress to git.gloom.garden (webhooks + clone) — verify reachability out-of-band ^unverified.
arm64: official server/agent/plugin-git images ship linux/arm64 — no image changes needed.²⁶ ²⁷ ²⁸

Out-of-scope notes (discovered, not requested by the issue)

local-path StorageClass is referenced but not provisioned anywhere in this repo²⁹ (Talos ships none; Vault uses hostPath, not PVCs³⁰). Verify kubectl get storageclass local-path exists before apply ^unverified (out-of-band cluster check), otherwise deploy local-path-provisioner first (separate change).
Vault Secrets Operator (VSO) is planned in PLAN.md/ARCHITECTURE.md but not deployed — this plan uses a one-time documented sync instead of VSO-synced secrets.³¹ ³²
Build-time pipeline secrets from Vault would need a secret extension (WOODPECKER_SECRET_EXTENSION_ENDPOINT)³³; no native Vault store exists (woodpecker-ci/woodpecker#929)³⁴. Optional future work.
WOODPECKER_AGENT_SECRET stays chart-generated (auto-rotated internal trust secret), only external Forgejo credentials are Vault-managed.³⁵

Acceptance Criteria

woodpecker/ root plans offline: terraform -chdir=woodpecker test passes with mock providers and a copied cp.tfstate fixture.
main.tf creates woodpecker (baseline PSA) and woodpecker-runtime (privileged PSA) namespaces, a pinned helm_release (chart woodpecker, version 3.7.3, namespace woodpecker), and a Tailscale ingress (ingress-proxies annotation, tls.hosts: ["woodpecker"], backend woodpecker-server:80) — no duplicate ProxyGroup.
values.yaml sets server env (WOODPECKER_HOST, real WOODPECKER_ADMIN, Forgejo on git.gloom.garden, woodpecker-forgejo-oauth envFrom) and agent env (Kubernetes backend → woodpecker-runtime, local-path, RWX false, replicas: 1, worker nodeSelector); yamllint and terraform fmt -check are green.
Root suites updated: tests/test_static.py formats woodpecker/, tests/test_terraform.py runs terraform test on the new root.
No secrets or credentials appear anywhere in the repo (git grep for placeholder/real OAuth values finds none); Forgejo OAuth creds live only in Vault, synced to the woodpecker-forgejo-oauth Secret, with steps + exact callback URL in woodpecker/README.md and OPERATIONS.md.

Testing Plan

Offline (all run under nix develop, no cluster):

python -m pytest tests/ -v                       # static + terraform tests incl. new woodpecker root
terraform -chdir=woodpecker test -no-color       # mock-provider plan assertions

Live, after human approval and apply (terraform -chdir=woodpecker apply):
- kubectl -n woodpecker get statefulset,pvc — woodpecker-server-0 and one agent Ready, both PVCs Bound on local-path.
- kubectl -n woodpecker logs <agent-pod> — agent connected to woodpecker-server:9000 (no auth errors).
- https://woodpecker.faun-snapper.ts.net loads (LE cert) after admin-console service approval; log in with a WOODPECKER_ADMIN Forgejo account.
- Enable a test repo with a .woodpecker.yml; verify the pipeline spawns step pods in woodpecker-runtime (kubectl -n woodpecker-runtime get pods) and a clone succeeds (proves Forgejo reachability + OAuth).
Edge/negative: stop the agent StatefulSet and confirm pipelines queue rather than error; delete the woodpecker-forgejo-oauth Secret and confirm the server pod fails env resolution (proves the wiring is live).

#19 Pod - Woodpecker CI (this issue) — Phase 3 of PLAN.md.
Precedent work: vault root + vault README — the ProxyGroup/ingress pattern this plan mirrors.
Future follow-ups (not part of this issue): VSO deployment (ARCHITECTURE.md §6), Woodpecker secret extension for build-time Vault secrets, local-path provisioner if absent.

Research Findings

Chart 3.7.3 (2026-08-30, app 3.18.x) is current; the server subchart is a StatefulSet on HTTP :8000 / gRPC :9000, and the chart Service exposes http on port 80 → 8000. Install is oci://ghcr.io/woodpecker-ci/helm/woodpecker; the https://woodpecker-ci.org/ repo is deprecated-but-functional.
The agent's default env already targets the Kubernetes backend with a runtime namespace and storage class knobs, but WOODPECKER_BACKEND_K8S_STORAGE_RWX defaults to true — unchangeable on local-path (RWO hostPath) — so it must be forced false.
Forgejo integrates via server env (WOODPECKER_FORGEJO=true/_URL/_CLIENT/_SECRET); the OAuth callback URL must be exactly https://<host>/authorize. No dedicated Helm values exist — they go in server.env/secrets.
Woodpecker has no native Vault secret store (woodpecker-ci/woodpecker#929); operational env secrets can come from Vault via VSO-synced Kubernetes Secrets consumed with extraSecretNamesForEnvFrom. VSO isn't deployed yet, so this plan syncs once manually.
Official images (server/agent/plugin-git) are published for linux/arm64; the chart ships resources: {} by design, and step pods (not the agent) consume CPU/memory in the runtime namespace.
The default clone step runs as root, so the workload namespace needs the privileged PSA label on Talos (baseline is fine for the chart's own woodpecker namespace pods).
ProxyGroups are cluster-scoped, so the vault-owned ingress-proxies is reusable from the woodpecker namespace; in-cluster agents need no external gRPC exposure.

Pre-Mortem

local-path StorageClass is missing or the PVCs never bind — The plan flags that no storage provider is provisioned anywhere in the repo and Talos ships none, yet the entire deploy (server SQLite PVC, agent PVC, pipeline workspace volumes) depends on local-path existing in-cluster. If that out-of-band check is skipped and the class is absent, the StatefulSets stay Pending and Woodpecker never comes up, which no offline terraform test can catch.
Forgejo is unreachable from the cluster or the OAuth host/callback is misconfigured — Connectivity from the cluster/tailnet to git.gloom.garden was explicitly flagged as unverified, and Woodpecker needs it for webhooks, repo clones, and OAuth login. If WOODPECKER_HOST or the registered callback URL doesn't exactly match https://woodpecker.faun-snapper.ts.net/authorize, login and pipeline triggers fail even though the cluster-side rollout looks healthy.
A human out-of-band step is missed or drifts from the Terraform state — The Forgejo OAuth secret sync from Vault, the admin-console approval of the new Tailscale service, and the pre-existing ingress-proxies ProxyGroup are all manual prerequisites not owned by the woodpecker/ root. Missing any of them yields a release that appears applied but cannot authenticate, route traffic, or run builds, and the divergence is invisible to the automated test suite.

Naina: ## Overview Deploy Woodpecker CI (single server + one Kubernetes-backend agent) with a new `woodpecker/` Terraform root that mirrors the `vault/` pattern: two namespaces, one pinned `helm_release` from the official Woodpecker chart, and a Tailscale operator Ingress reusing the existing `ingress-proxies` ProxyGroup. Forgejo OAuth credentials are sourced from the in-cluster Vault via a one-time documented sync; runtime build pods land in the `woodpecker-runtime` namespace on `local-path` storage. ## Technical Approach 1. **Add `woodpecker/main.tf`** — copy the providers/remote-state preamble verbatim from `vault/main.tf` (`control_plane_state_path`, `cluster_name` variables; `terraform_remote_state.cp`; kubernetes + helm providers from `local.kc`), then create the namespaces, release, and ingress:[1](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/main.tf?display=source#L19) ```hcl resource "kubernetes_namespace_v1" "woodpecker" { metadata { name = "woodpecker" labels = { "pod-security.kubernetes.io/enforce" = "baseline" } # chart pods: non-privileged, PVC only } } # Must exist before helm install: the agent chart renders its Role/RoleBinding # INTO this namespace. Privileged: default clone step (plugin-git) runs as root. resource "kubernetes_namespace_v1" "woodpecker_runtime" { metadata { name = "woodpecker-runtime" labels = { "pod-security.kubernetes.io/enforce" = "privileged" } } } resource "helm_release" "woodpecker" { name = "woodpecker" # keep: defaults like woodpecker-server:9000 assume it repository = "oci://ghcr.io/woodpecker-ci/helm" chart = "woodpecker" version = "3.7.3" # app 3.18.x; verify latest at implement time namespace = kubernetes_namespace_v1.woodpecker.metadata[0].name values = [file("${path.module}/values.yaml")] depends_on = [kubernetes_namespace_v1.woodpecker, kubernetes_namespace_v1.woodpecker_runtime] } ``` _Assumption check — the agent chart renders its Role/RoleBinding INTO `WOODPECKER_BACKEND_K8S_NAMESPACE`, so that namespace must pre-exist[2](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/agent/templates/role.yaml) [3](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/agent/templates/rolebinding.yaml); the default clone step (plugin-git) runs as root, hence the `privileged` runtime namespace[4](https://github.com/woodpecker-ci/woodpecker/issues/5346)._ 2. **Add `woodpecker/.gitignore`** — copy `vault/.gitignore` (`.terraform`, `*.tfstate`, `!tests/fixtures/*.tfstate`, `terraform.tfvars`, `*.auto.tfvars`)[5](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/.gitignore?display=source#L1). Commit the `.terraform.lock.hcl` generated by `terraform -chdir=woodpecker init`. 3. **Expand `woodpecker/values.yaml`** (fixes the two skeleton bugs: runtime namespace was `woodpecker`, RWX left at chart-default `true`)[6](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/woodpecker/values.yaml?display=source#L9): ```yaml --- server: env: WOODPECKER_HOST: https://woodpecker.faun-snapper.ts.net # OAuth redirect + webhook base URL WOODPECKER_ADMIN: <forgejo-admin-usernames> # not woodpecker,admin WOODPECKER_FORGEJO: "true" WOODPECKER_FORGEJO_URL: https://git.gloom.garden extraSecretNamesForEnvFrom: - woodpecker-forgejo-oauth # keys WOODPECKER_FORGEJO_CLIENT/_SECRET (Vault-backed, step 6) nodeSelector: node-role: worker persistentVolume: storageClass: local-path # server SQLite DB lives at /var/lib/woodpecker size: 10Gi agent: replicas: 1 nodeSelector: node-role: worker env: WOODPECKER_BACKEND: kubernetes WOODPECKER_BACKEND_K8S_NAMESPACE: woodpecker-runtime WOODPECKER_BACKEND_K8S_STORAGE_CLASS: local-path WOODPECKER_BACKEND_K8S_STORAGE_RWX: "false" # local-path is RWO-only; chart default true won't bind WOODPECKER_BACKEND_K8S_VOLUME_SIZE: 2G # chart default 10G persistence: storageClass: local-path size: 1Gi ``` _Assumption check — chart defaults: agent env ships `WOODPECKER_BACKEND_K8S_STORAGE_RWX: true` and `WOODPECKER_BACKEND_K8S_VOLUME_SIZE: 10G`[7](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/agent/values.yaml); local-path provisioned volumes are single-node (RWO) unless a shared filesystem is configured[8](https://github.com/rancher/local-path-provisioner)._ Keep `server.createAgentSecret` at the default (`true`): the chart-generated `woodpecker-default-agent-secret` is an internal trust secret that never needs operator knowledge.[9](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/server/values.yaml) [10](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/server/templates/secret.yaml) 4. **Add the Tailscale ingress** (reuse `ingress-proxies` — ProxyGroup is cluster-scoped, owned by `vault/main.tf`; no new ProxyGroup resource):[11](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/main.tf?display=source#L107) ```hcl resource "kubernetes_ingress_v1" "woodpecker" { metadata { name = "woodpecker" namespace = kubernetes_namespace_v1.woodpecker.metadata[0].name annotations = { "tailscale.com/proxy-group" = "ingress-proxies" } } spec { ingress_class_name = "tailscale" tls { hosts = ["woodpecker"] # -> woodpecker.faun-snapper.ts.net (LE cert) } default_backend { service { name = "woodpecker-server" # chart Service http port 80 -> container :8000 port { number = 80 } } } } depends_on = [helm_release.woodpecker] } ``` No external gRPC (9000) ingress: agents are in-cluster and reach `woodpecker-server:9000` over the Service.[12](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/agent/values.yaml) [13](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/server/templates/service.yaml) 5. **Add `woodpecker/tests/`** — copy `vault/tests/fixtures/cp.tfstate`, then `plan.tftest.hcl` mirroring `vault/tests/plan.tftest.hcl` with mock kubernetes/helm providers:[14](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/tests/plan.tftest.hcl?display=source#L1) [15](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/tests/fixtures/cp.tfstate?display=source#L1) ```hcl run "namespaces_have_psa_labels" { command = plan assert { condition = kubernetes_namespace_v1.woodpecker.metadata[0].labels["pod-security.kubernetes.io/enforce"] == "baseline" error_message = "woodpecker namespace must be baseline PSA" } assert { condition = kubernetes_namespace_v1.woodpecker_runtime.metadata[0].labels["pod-security.kubernetes.io/enforce"] == "privileged" error_message = "woodpecker-runtime namespace must be privileged PSA (root clone step)" } } run "helm_release_uses_official_chart_pinned" { command = plan assert { condition = helm_release.woodpecker.chart == "woodpecker" error_message = "release must use the umbrella woodpecker chart" } assert { condition = helm_release.woodpecker.version == "3.7.3" error_message = "chart version must be pinned" } assert { condition = helm_release.woodpecker.namespace == "woodpecker" error_message = "release must deploy to the woodpecker namespace" } } run "ingress_targets_woodpecker_server_via_existing_proxy_group" { command = plan assert { condition = kubernetes_ingress_v1.woodpecker.metadata[0].annotations["tailscale.com/proxy-group"] == "ingress-proxies" error_message = "ingress must reuse the vault-owned ingress-proxies ProxyGroup" } assert { condition = kubernetes_ingress_v1.woodpecker.spec[0].tls[0].hosts[0] == "woodpecker" error_message = "tls.hosts must pin the MagicDNS name to woodpecker" } assert { condition = kubernetes_ingress_v1.woodpecker.spec[0].default_backend[0].service[0].name == "woodpecker-server" error_message = "backend must be the woodpecker-server Service" } assert { condition = kubernetes_ingress_v1.woodpecker.spec[0].default_backend[0].service[0].port[0].number == 80 error_message = "backend must target the chart's http Service port 80" } } ``` 6. **Wire the root test suites** — `tests/test_static.py`: add `woodpecker/` to the `terraform fmt -check -recursive` dirs[16](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/tests/test_static.py?display=source#L14); `tests/test_terraform.py`: add a `test_terraform_test_woodpecker()` mirroring the vault one[17](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/tests/test_terraform.py?display=source#L34). 7. **Document the out-of-band flow** — rewrite `woodpecker/README.md` (Terraform deploy + secrets) and add a Woodpecker section to `OPERATIONS.md`: ```bash # 1. Forgejo (admin console / user settings): register OAuth2 app # callback URL MUST be exactly https://woodpecker.faun-snapper.ts.net/authorize # 2. Vault is the source of truth: kubectl -n vault exec vault-0 -- vault kv put secret/woodpecker/forgejo-oauth \ WOODPECKER_FORGEJO_CLIENT=<client-id> WOODPECKER_FORGEJO_SECRET=<client-secret> # 3. One-time sync into the cluster (before/soon after `terraform apply`): kubectl -n woodpecker create secret generic woodpecker-forgejo-oauth \ --from-literal=WOODPECKER_FORGEJO_CLIENT="$(kubectl -n vault exec vault-0 -- vault kv get -field=WOODPECKER_FORGEJO_CLIENT secret/woodpecker/forgejo-oauth)" \ --from-literal=WOODPECKER_FORGEJO_SECRET="$(kubectl -n vault exec vault-0 -- vault kv get -field=WOODPECKER_FORGEJO_SECRET secret/woodpecker/forgejo-oauth)" # 4. Tailscale admin console: approve the new woodpecker.faun-snapper.ts.net service ``` _Assumption check — Forgejo OAuth: the callback URL must be exactly `https://<host>/authorize`[18](https://woodpecker-ci.org/docs/next/administration/configuration/forges/forgejo); the `kubectl -n vault exec vault-0 -- vault kv` pattern matches existing ops usage (which passes `-address=http://127.0.0.1:8200`)[19](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/OPERATIONS.md?display=source#L137)._ Apply order on a fresh cluster: `vault` root first (owns the ProxyGroup the ingress annotation references), then `woodpecker`.[20](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/main.tf?display=source#L148) Nothing secret is ever written to the repo. ## Implementation Details **Files to modify/create** - `woodpecker/main.tf` (new) — providers/remote-state preamble copied from `vault/main.tf`; namespaces, `helm_release`, ingress above. - `woodpecker/values.yaml` (modify) — full file in step 3. - `woodpecker/values.yaml` lint note: `yamllint` already checks this file (`tests/test_static.py`). - `woodpecker/.gitignore` (new), `woodpecker/.terraform.lock.hcl` (new, committed), `woodpecker/tests/plan.tftest.hcl` + `woodpecker/tests/fixtures/cp.tfstate` (new). - `woodpecker/README.md` (rewrite), `OPERATIONS.md` (new §Woodpecker), `tests/test_static.py`, `tests/test_terraform.py` (modify). - No changes to `control-plane/`, `worker/`, or `vault/`. **Dependencies / constraints** - Chart `woodpecker` **3.7.3** (agent 2.0.1 / server 3.0.1 subcharts, app 3.18.0)[21](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/Chart.yaml) [22](https://github.com/woodpecker-ci/helm/blob/3.7.3/CHANGELOG.md); alternative `repository = "https://woodpecker-ci.org/"` (deprecated-but-functional, used in the chart README's own snippet) if OCI pulls fail[23](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/README.md). - Terraform providers match `vault/main.tf`: kubernetes `~> 2.35`, helm `~> 2.17`.[24](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/main.tf?display=source#L4) - Release name stays `woodpecker` in namespace `woodpecker` so chart defaults (`woodpecker-server:9000`, agent secret name) need no overrides.[25](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/agent/values.yaml) - Runtime pods/clones need egress to `git.gloom.garden` (webhooks + clone) — verify reachability out-of-band _unverified_. - arm64: official server/agent/plugin-git images ship `linux/arm64` — no image changes needed.[26](https://hub.docker.com/r/woodpeckerci/woodpecker-server/tags) [27](https://hub.docker.com/r/woodpeckerci/woodpecker-agent/tags) [28](https://hub.docker.com/r/woodpeckerci/plugin-git/tags) **Out-of-scope notes (discovered, not requested by the issue)** - `local-path` StorageClass is referenced but **not provisioned anywhere in this repo**[29](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/woodpecker/values.yaml?display=source#L4) (Talos ships none; Vault uses hostPath, not PVCs[30](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/values.yaml?display=source#L45)). Verify `kubectl get storageclass local-path` exists before apply _unverified_ (out-of-band cluster check), otherwise deploy local-path-provisioner first (separate change). - Vault Secrets Operator (VSO) is planned in PLAN.md/ARCHITECTURE.md but not deployed — this plan uses a one-time documented sync instead of VSO-synced secrets.[31](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/PLAN.md?display=source#L30) [32](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/ARCHITECTURE.md?display=source#L194) - Build-time pipeline secrets from Vault would need a secret extension (`WOODPECKER_SECRET_EXTENSION_ENDPOINT`)[33](https://woodpecker-ci.org/docs/next/usage/extensions/secret-extension); no native Vault store exists (woodpecker-ci/woodpecker#929)[34](https://github.com/woodpecker-ci/woodpecker/issues/929). Optional future work. - `WOODPECKER_AGENT_SECRET` stays chart-generated (auto-rotated internal trust secret), only external Forgejo credentials are Vault-managed.[35](https://github.com/woodpecker-ci/helm/blob/3.7.3/charts/woodpecker/charts/agent/values.yaml) ## Acceptance Criteria - [ ] `woodpecker/` root plans offline: `terraform -chdir=woodpecker test` passes with mock providers and a copied `cp.tfstate` fixture. - [ ] `main.tf` creates `woodpecker` (baseline PSA) and `woodpecker-runtime` (privileged PSA) namespaces, a pinned `helm_release` (chart `woodpecker`, version `3.7.3`, namespace `woodpecker`), and a Tailscale ingress (`ingress-proxies` annotation, `tls.hosts: ["woodpecker"]`, backend `woodpecker-server:80`) — no duplicate ProxyGroup. - [ ] `values.yaml` sets server env (`WOODPECKER_HOST`, real `WOODPECKER_ADMIN`, Forgejo on `git.gloom.garden`, `woodpecker-forgejo-oauth` envFrom) and agent env (Kubernetes backend → `woodpecker-runtime`, `local-path`, RWX `false`, `replicas: 1`, worker `nodeSelector`); `yamllint` and `terraform fmt -check` are green. - [ ] Root suites updated: `tests/test_static.py` formats `woodpecker/`, `tests/test_terraform.py` runs `terraform test` on the new root. - [ ] No secrets or credentials appear anywhere in the repo (`git grep` for placeholder/real OAuth values finds none); Forgejo OAuth creds live only in Vault, synced to the `woodpecker-forgejo-oauth` Secret, with steps + exact callback URL in `woodpecker/README.md` and `OPERATIONS.md`. ## Testing Plan - Offline (all run under `nix develop`, no cluster): ```bash python -m pytest tests/ -v # static + terraform tests incl. new woodpecker root terraform -chdir=woodpecker test -no-color # mock-provider plan assertions ``` - Live, after human approval and apply (`terraform -chdir=woodpecker apply`): - `kubectl -n woodpecker get statefulset,pvc` — `woodpecker-server-0` and one agent `Ready`, both PVCs `Bound` on `local-path`. - `kubectl -n woodpecker logs <agent-pod>` — agent connected to `woodpecker-server:9000` (no auth errors). - `https://woodpecker.faun-snapper.ts.net` loads (LE cert) after admin-console service approval; log in with a `WOODPECKER_ADMIN` Forgejo account. - Enable a test repo with a `.woodpecker.yml`; verify the pipeline spawns step pods in `woodpecker-runtime` (`kubectl -n woodpecker-runtime get pods`) and a clone succeeds (proves Forgejo reachability + OAuth). - Edge/negative: stop the agent StatefulSet and confirm pipelines queue rather than error; delete the `woodpecker-forgejo-oauth` Secret and confirm the server pod fails env resolution (proves the wiring is live). ## Related Issues/PRs - [#19 Pod - Woodpecker CI](https://git.gloom.garden/tepichord/milner/issues/19) (this issue) — Phase 3 of [PLAN.md](PLAN.md). - Precedent work: [vault root](vault/main.tf) + [vault README](vault/README.md) — the ProxyGroup/ingress pattern this plan mirrors. - Future follow-ups (not part of this issue): VSO deployment (ARCHITECTURE.md §6), Woodpecker secret extension for build-time Vault secrets, `local-path` provisioner if absent. ## Research Findings - Chart **3.7.3** (2026-08-30, app 3.18.x) is current; the server subchart is a StatefulSet on HTTP `:8000` / gRPC `:9000`, and the chart Service exposes http on port 80 → 8000. Install is `oci://ghcr.io/woodpecker-ci/helm/woodpecker`; the `https://woodpecker-ci.org/` repo is deprecated-but-functional. - The agent's default env already targets the Kubernetes backend with a runtime namespace and storage class knobs, but `WOODPECKER_BACKEND_K8S_STORAGE_RWX` defaults to `true` — unchangeable on `local-path` (RWO hostPath) — so it must be forced `false`. - Forgejo integrates via server env (`WOODPECKER_FORGEJO=true/_URL/_CLIENT/_SECRET`); the OAuth callback URL must be exactly `https://<host>/authorize`. No dedicated Helm values exist — they go in `server.env`/secrets. - Woodpecker has no native Vault secret store (woodpecker-ci/woodpecker#929); operational env secrets can come from Vault via VSO-synced Kubernetes Secrets consumed with `extraSecretNamesForEnvFrom`. VSO isn't deployed yet, so this plan syncs once manually. - Official images (server/agent/plugin-git) are published for `linux/arm64`; the chart ships `resources: {}` by design, and step pods (not the agent) consume CPU/memory in the runtime namespace. - The default clone step runs as root, so the workload namespace needs the `privileged` PSA label on Talos (baseline is fine for the chart's own `woodpecker` namespace pods). - ProxyGroups are cluster-scoped, so the vault-owned `ingress-proxies` is reusable from the woodpecker namespace; in-cluster agents need no external gRPC exposure. ## Pre-Mortem 1. **`local-path` StorageClass is missing or the PVCs never bind** — The plan flags that no storage provider is provisioned anywhere in the repo and Talos ships none, yet the entire deploy (server SQLite PVC, agent PVC, pipeline workspace volumes) depends on `local-path` existing in-cluster. If that out-of-band check is skipped and the class is absent, the StatefulSets stay `Pending` and Woodpecker never comes up, which no offline `terraform test` can catch. 2. **Forgejo is unreachable from the cluster or the OAuth host/callback is misconfigured** — Connectivity from the cluster/tailnet to git.gloom.garden was explicitly flagged as unverified, and Woodpecker needs it for webhooks, repo clones, and OAuth login. If `WOODPECKER_HOST` or the registered callback URL doesn't exactly match `https://woodpecker.faun-snapper.ts.net/authorize`, login and pipeline triggers fail even though the cluster-side rollout looks healthy. 3. **A human out-of-band step is missed or drifts from the Terraform state** — The Forgejo OAuth secret sync from Vault, the admin-console approval of the new Tailscale service, and the pre-existing `ingress-proxies` ProxyGroup are all manual prerequisites not owned by the `woodpecker/` root. Missing any of them yields a release that appears applied but cannot authenticate, route traffic, or run builds, and the divergence is invisible to the automated test suite.

tepichord added

and removed

labels

2026-09-09 13:38:58 -07:00

tepichord referenced this issue from a commit

2026-09-09 15:28:38 -07:00

feat(woodpecker): add .gitignore and .terraform.lock.hcl

tepichord referenced this issue from a commit