Serve Woodpecker pipeline secrets from Vault

tepichord commented

2026-09-17 10:25:16 -07:00

Owner

Why

Woodpecker has no way to give a pipeline a secret that VSO manages. VSO writes
Kubernetes Secrets, and those reach a step only as environment variables of the
server or the agent, while a step's from_secret resolves through Woodpecker's
own secret store or a secret-extension HTTP endpoint. Both are empty today —
which is how annia-infra's merged-PR pipeline came to run with an unset
$FORGEJO_TOKEN and quietly do nothing.

This adds the endpoint: lazysteff's woodpecker-vault-secret-extension, pinned to
upstream commit a568dce (tag v0.2.4), plus its Vault credentials and the
VaultStaticSecret that carries them. A pipeline secret then has one home —
Vault — at secret/forgejo/token, the same PAT oz will read.

Pieces

woodpecker-secret-extension/Dockerfile — built by hand and pushed to zot
(§9k step 3), because the image tag has to exist before the manifest that pins
it can merge. oz already has this shape, so the header documents the build
rather than adding a CI job.
gitops/.../apps/woodpecker/secret-extension.yaml — the extension's config
(one rule: repository tepichord/annia-infra, event pull_request_closed,
secret FORGEJO_TOKEN read from forgejo/token), its Deployment
(distroless non-root, read-only root, probes, requests and limits) and a
ClusterIP Service. No Ingress: only the server calls it.
gitops/.../secrets/woodpecker-secret-extension.yaml — the AppRole
role_id/secret_id and Woodpecker's Ed25519 public key, synced from
secret/woodpecker/secret-extension with a rolloutRestartTarget.
apps/woodpecker/values.yaml — WOODPECKER_SECRET_EXTENSION_ENDPOINT and
WOODPECKER_EXTENSIONS_ALLOWED_HOSTS: private (the allowed-host matcher
matches the resolved ClusterIP, and external — the default — refuses it).
OPERATIONS.md §9k is the procedure that reproduces all of this, including
what a signing-key rotation does to it; §9g and §9c are updated for the new
consumer. tests/test_gitops.py pins what kustomize and the policy tests
cannot see: the manifest tag against the Dockerfile, the server's endpoint
host and port against the rendered Service, and every *_file in the config
against the volume that mounts it.

Two things worth an explicit sign-off

The rule sets allow_forks: true, and that is forced rather than chosen.
Woodpecker cannot express "same-repo PR": pipeline.from_fork is
json:"from_fork,omitempty", so a closed pull request in this repository
omits the field and the extension reads the silence as forked. The gate that
actually holds is Woodpecker's own and sits upstream of the extension — a
repository's default approval mode is forks, so a fork's pipeline is parked
in blocked before any secret is fetched. tepichord/annia-infra must stay
at forks or stricter: at none, this rule would hand the forge token to a
fork's pipeline, which runs the fork's own workflow file.
The extension authenticates with an AppRole, which is the documented
exception to the Kubernetes-auth rule in AGENTS.md. The program supports
only AppRole and static tokens, so it holds a role_id/secret_id pair in a
Kubernetes Secret instead of projecting a service-account token. The policy
it gets is one KV path plus token self-lookup and renew.

Deployment order

The Vault half (§9k steps 1–2, root token: enable AppRole, write the policy and
role, mint the credentials, then write the PAT and the extension's three values)
has to happen before this merges. The extension authenticates to Vault before it serves and exits on a
failed login, so with the manifests merged first the pods crash-loop and the
apps Kustomization never reaches Ready.

The image is already in zot: milner/woodpecker-secret-extension:0.2.4.

Checks

kustomize build and kubeconform -strict on both roots, conftest 48/48
(apps) and 32/32 (secrets), hadolint on the Dockerfile, yamllint on the
values, pytest 192 passed / 8 skipped.

### Why Woodpecker has no way to give a pipeline a secret that VSO manages. VSO writes Kubernetes Secrets, and those reach a step only as environment variables of the server or the agent, while a step's `from_secret` resolves through Woodpecker's own secret store or a secret-extension HTTP endpoint. Both are empty today — which is how `annia-infra`'s merged-PR pipeline came to run with an unset `$FORGEJO_TOKEN` and quietly do nothing. This adds the endpoint: lazysteff's `woodpecker-vault-secret-extension`, pinned to upstream commit `a568dce` (tag v0.2.4), plus its Vault credentials and the `VaultStaticSecret` that carries them. A pipeline secret then has one home — Vault — at `secret/forgejo/token`, the same PAT `oz` will read. ### Pieces - `woodpecker-secret-extension/Dockerfile` — built by hand and pushed to zot (§9k step 3), because the image tag has to exist before the manifest that pins it can merge. `oz` already has this shape, so the header documents the build rather than adding a CI job. - `gitops/.../apps/woodpecker/secret-extension.yaml` — the extension's config (one rule: repository `tepichord/annia-infra`, event `pull_request_closed`, secret `FORGEJO_TOKEN` read from `forgejo/token`), its Deployment (distroless non-root, read-only root, probes, requests and limits) and a ClusterIP Service. No Ingress: only the server calls it. - `gitops/.../secrets/woodpecker-secret-extension.yaml` — the AppRole `role_id`/`secret_id` and Woodpecker's Ed25519 public key, synced from `secret/woodpecker/secret-extension` with a `rolloutRestartTarget`. - `apps/woodpecker/values.yaml` — `WOODPECKER_SECRET_EXTENSION_ENDPOINT` and `WOODPECKER_EXTENSIONS_ALLOWED_HOSTS: private` (the allowed-host matcher matches the resolved ClusterIP, and `external` — the default — refuses it). - `OPERATIONS.md` §9k is the procedure that reproduces all of this, including what a signing-key rotation does to it; §9g and §9c are updated for the new consumer. `tests/test_gitops.py` pins what `kustomize` and the policy tests cannot see: the manifest tag against the Dockerfile, the server's endpoint host and port against the rendered Service, and every `*_file` in the config against the volume that mounts it. ### Two things worth an explicit sign-off 1. **The rule sets `allow_forks: true`, and that is forced rather than chosen.** Woodpecker cannot express "same-repo PR": `pipeline.from_fork` is `json:"from_fork,omitempty"`, so a closed pull request in this repository omits the field and the extension reads the silence as *forked*. The gate that actually holds is Woodpecker's own and sits upstream of the extension — a repository's default approval mode is `forks`, so a fork's pipeline is parked in `blocked` before any secret is fetched. `tepichord/annia-infra` must stay at `forks` or stricter: at `none`, this rule would hand the forge token to a fork's pipeline, which runs the fork's own workflow file. 2. **The extension authenticates with an AppRole, which is the documented exception to the Kubernetes-auth rule in `AGENTS.md`.** The program supports only AppRole and static tokens, so it holds a `role_id`/`secret_id` pair in a Kubernetes Secret instead of projecting a service-account token. The policy it gets is one KV path plus token self-lookup and renew. ### Deployment order The Vault half (§9k steps 1–2, root token: enable AppRole, write the policy and role, mint the credentials, then write the PAT and the extension's three values) has to happen before this merges. The extension authenticates to Vault *before* it serves and exits on a failed login, so with the manifests merged first the pods crash-loop and the `apps` Kustomization never reaches Ready. The image is already in zot: `milner/woodpecker-secret-extension:0.2.4`. ### Checks `kustomize build` and `kubeconform -strict` on both roots, `conftest` 48/48 (apps) and 32/32 (secrets), `hadolint` on the Dockerfile, `yamllint` on the values, `pytest` 192 passed / 8 skipped.

tepichord added 4 commits

2026-09-17 10:25:16 -07:00

Grow storage to 2 TiB and upgrade Talos to v1.13.9

ci/woodpecker/pr/gates Pipeline failed

Details

ci/woodpecker/pull_request_closed/cleanup Pipeline was successful

Details

d163ce473f

The shared data disk goes 100 GiB -> 2 TiB, the worker system disk 40 GiB ->
1 TiB and the control-plane disk 40 GiB -> 128 GiB; Vault's 20 GiB disk is
unchanged. Memory moves to control-plane 3 GB / worker 15 GB (the host's 18 GiB
split), and both nodes move to Talos v1.13.9.

Two things the resize exposed, both fixed here:

* A Talos user volume does not follow its disk on its own. A `partition` volume
  is created at its provisioning size and stays there unless the
  UserVolumeConfig sets `provisioning.grow: true`, so bumping the qcow2 alone
  left /var/mnt/data at 100 GiB. Both worker volumes now declare grow.

* OPERATIONS 12d's reconcile step was wrong: a `terraform apply` does not adopt
  the host's new capacity (the provider's Read keeps the state value), so the
  plan reports an in-place capacity update that the provider then refuses, on
  every retry. The working procedure is `state rm` + `import` by pool path --
  what data/restore.sh already does -- and an import reads back more than
  Create ever wrote, so every volume now ignores the provider-reported
  metadata (the ignore_changes lists were incomplete). They are
  version-sensitive: control-plane still pins libvirt 0.9.8, which decides
  `allocation` itself.

Refreshes the docs and plan-time assertions to the new sizes, corrects the
stale "20 GB worker" preview-quota comment, and ignores the state backups a
12d reconcile leaves behind.

Serve Woodpecker pipeline secrets from Vault 8a228477db

Woodpecker's own secret store is empty and its secret-extension endpoint is
unset, so a step that declares `from_secret: FORGEJO_TOKEN` receives nothing.
annia-infra's merged-PR pipeline needed that token and failed on its absence:
the call to the forge went out unauthenticated, Forgejo answered 404, and the
`grep` after it exited 1 under the `/bin/sh -e` Woodpecker runs a step with.

VSO cannot close this. It writes Kubernetes Secrets, which reach a step only as
environment variables of the server or the agent, while `from_secret` resolves
through Woodpecker's own store or a secret-extension HTTP endpoint. This adds
the endpoint: lazysteff's Vault extension, pinned to upstream commit a568dce
(tag v0.2.4) and built by hand into zot, because the tag has to exist in the
registry before the manifest that pins it can merge -- the same deal as `oz`,
and the Dockerfile header carries the build.

The extension authenticates to Vault with an AppRole (policy, role and the
`secret-id` handling are §9k) and reads `secret/forgejo/token`, which is where
the Forgejo PAT lives for the whole cluster -- `oz` reads the same path.

* `allow_forks: true` is forced, not chosen. `pipeline.from_fork` is
  `json:"from_fork,omitempty"`, so a closed PR in this repository omits the
  field and the extension reads the silence as "forked". The gate that actually
  holds is Woodpecker's own `require_approval: forks`, which parks a fork's
  pipeline in `blocked` before any secret is fetched; keep annia-infra at
  `forks` or stricter.
* `WOODPECKER_EXTENSIONS_ALLOWED_HOSTS: private`, because the allowed-host
  matcher matches the resolved address -- a ClusterIP -- and the default
  (`external`) refuses it.

The public key is a Vault value too, not a committed one: the server generates
it, and a copy baked into this file would turn a key rotation into a silent loss
of every secret (Woodpecker logs a failing extension and runs the pipeline
anyway).

Pin the secret extension's wiring in tests 03b5fb3a36

Three ways the deployment can be wrong without `kustomize` or the policy tests
noticing, all of them silent until a pipeline asks for a secret:

* a manifest tag that drifts from the `EXTENSION_VERSION` the Dockerfile builds
  (the image is hand-built, so nothing else ties the two together),
* a server pointed at a Service name or port that the rendered manifests do not
  define, or at a host the extension-allowed-hosts policy refuses,
* a config file asking for files the volume does not mount under that name --
  `role_id_file`, `secret_id_file` and `public_key_file` are paths, and the
  mount is what decides whether they exist.

The Vault CR count moves to 5 with the extension's own `VaultStaticSecret`.

Document the pipeline secret extension (OPERATIONS 9k)

ci/woodpecker/pr/gates Pipeline failed

Details

72361c0db6

VSO and Woodpecker's `from_secret` never meet, so the gap needs a procedure:
enable the AppRole, write the policy and the three Vault values, build the image,
and let Flux do the rest -- in that order, because the AppRole credential, the
image tag and the manifests each have something that must exist first.

9g's signing-key rotation gains the extension as a second user-visible effect,
and 9c lists the new `VaultStaticSecret`.

tepichord added 1 commit

2026-09-17 10:37:18 -07:00

Write the extension's Vault values without putting them in argv

ci/woodpecker/pr/gates Pipeline failed

Details

16102713e7

9k step 2 wrote the role_id as a literal in the same `kv put` as the public key,
because only one value per invocation can come from the stdin that vault-0's
wrapper shares with the token. That exception was documented but not needed: the
three values write one per call, each from stdin, and the caveat goes away.

The key fetch also gains `-f`. Without it a failed fetch or a proxy's error page
is stored as the public key, and the extension then rejects every signature
("invalid signature") for a reason nothing in the KV entry explains.

tepichord force-pushed feat/woodpecker-vault-secret from 16102713e7

ci/woodpecker/pr/gates Pipeline failed

Details

to 5eaf24d766

ci/woodpecker/pr/gates Pipeline was successful

Details

ci/woodpecker/pull_request_closed/cleanup Pipeline was successful

Details

2026-09-17 11:04:10 -07:00

Compare