tepichord/milner

Fork 0

Give zot a push credential, and keep every read open #70

Merged

tepichord merged 1 commit from feat/zot-htpasswd-auth into main

2026-09-17 15:49:31 -07:00

tepichord commented

2026-09-17 15:24:22 -07:00

Owner

Description

The registry had no http.auth at all, so every push was anonymous — including
a pull request's own pipeline, which runs privileged. That matters here
because the cluster pulls milner/woodpecker-secret-extension:0.2.4 unpinned,
with the extension's AppRole behind it: an anonymous push endpoint is a path to
replacing an image a pod then runs.

zot reads htpasswd from a file and cannot be handed a password, while a
pipeline cannot log in with a hash — so there is one password in two
representations, and they have to be written together or they drift into a 401
nobody can explain later:

Where	What	Read by
`secret/ci/zot` → `ZOT_TOKEN`	the cleartext	the pipeline that pushes, via the secret extension
`secret/zot/htpasswd` → `HTPASSWD`	the bcrypt hash	zot, via the `VaultStaticSecret` `zot/zot-htpasswd`

set-credential.sh is the one operation that writes both: it derives the hash
from the cleartext it read or minted, so there is no second rotation path. The
hash reaches the pod as a Secret the chart mounts whole at /secret, which is
the only way to get it there without putting it in git (mountSecret: true
would render it from secretFiles.htpasswd).

What the access policy says

Reads stay anonymous. That is not a detail of the config but the shape of it:
zot consults defaultPolicy only for a named caller and anonymousPolicy
only for an anonymous one (isPermitted, pkg/api/authz.go), and every puller
of this registry is anonymous — the secret extension, oz, the preview router,
and the kubelet. So ** is read-only for anonymous callers, and every
defaultPolicy mirrors its anonymousPolicy, because a logged-in client is not
an anonymous one: without the mirror, podman login followed by a pull of
milner/… would turn a working pull into a 401.

Two repositories keep anonymous write, and they are different cases:

preview/** is permanent. A per-PR push cannot be given a credential at all —
the pipeline runs the PR's own workflow file and the extension refuses every
pull event — and those images are what the l3 preview runs.
milner/** is meant to be deleted. The main halves of build-oz and
build-router push milner/oz and milner/router unattended, and cannot be
given a credential yet: the extension's AppRole is granted
secret/forgejo/token and no other path. Once the ci/ grant and a rule for
this repository exist, those steps take --creds and this entry loses its two
policies — in that order, because the credential has to be in place before the
exception can go.

annia-ci is the only user in the password file, so the credential is scoped by
who holds it rather than by a prefix allowlist that every new publisher would
have to remember to extend.

Order

zot reads the file at start-up and exits 1 without it (Error: open /secret/HTPASSWD: no such file or directory). The chart rolls the pod when the
config changes and the secrets layer is applied after infrastructure, so a pod
that restarts into this config retries until VSO has synced the Secret — a
window with an end because the Vault side (both entries, plus the zot-read
policy and zot role from §4c-ter) was done first, and a CrashLoopBackOff on
the registry every pull goes through when it was not.

Changes

infrastructure/zot/set-credential.sh — provision (default), --rotate,
verify. No value in argv, nothing echoed, the root-only half printed for an
operator session unless ROOT_VAULT_TOKEN is set.
infrastructure/zot/values.yaml — http.auth.htpasswd + the access-control
tree, with the two traps (auth alone authorizes nothing; the two policies are
not alternatives) written down where they bite.
secrets/vaultauth-zot.yaml, secrets/zot-htpasswd.yaml — the namespace's
VaultAuth and the Secret the chart mounts. No rolloutRestartTargets: zot
watches the file and its parent directory, so a rotation is picked up in
place.
OPERATIONS.md — the zot-read policy and zot role in §4c-ter, and a
podman login in §9k step 3's hand-run push.
infrastructure/zot/README.md — the policy table, the credential, the
ordering, and the evidence table.
tests/test_gitops.py — anonymous reads stay, ** stays read-only for
anonymous callers, and the file the config names is the one VSO writes.

Evidence

zot verify accepts this exact config.json; on a throwaway v2.1.21 with it and
the hash Vault holds: anonymous GET /v2/ 200, GET /v2/_catalog 200,
POST /v2/milner/… 202, POST /v2/preview/… 202, POST /v2/annia/… 401;
annia-ci GET /v2/_catalog 200 and POST 202 on annia, milner,
library; a wrong password 401. The log says loaded htpasswd file users: 1
and access control is being enabled.

pytest tests/ -q: 206 passed.

## Description The registry had no `http.auth` at all, so every push was anonymous — including a pull request's own pipeline, which runs `privileged`. That matters here because the cluster pulls `milner/woodpecker-secret-extension:0.2.4` unpinned, with the extension's AppRole behind it: an anonymous push endpoint is a path to replacing an image a pod then runs. zot reads htpasswd from a *file* and cannot be handed a password, while a pipeline cannot log in with a hash — so there is one password in two representations, and they have to be written together or they drift into a 401 nobody can explain later: | Where | What | Read by | |---|---|---| | `secret/ci/zot` → `ZOT_TOKEN` | the cleartext | the pipeline that pushes, via the secret extension | | `secret/zot/htpasswd` → `HTPASSWD` | the bcrypt hash | zot, via the `VaultStaticSecret` `zot/zot-htpasswd` | `set-credential.sh` is the one operation that writes both: it derives the hash from the cleartext it read or minted, so there is no second rotation path. The hash reaches the pod as a Secret the chart mounts whole at `/secret`, which is the only way to get it there without putting it in git (`mountSecret: true` would render it from `secretFiles.htpasswd`). ## What the access policy says Reads stay anonymous. That is not a detail of the config but the shape of it: zot consults `defaultPolicy` only for a *named* caller and `anonymousPolicy` only for an anonymous one (`isPermitted`, `pkg/api/authz.go`), and every puller of this registry is anonymous — the secret extension, `oz`, the preview router, and the kubelet. So `**` is read-only for anonymous callers, and every `defaultPolicy` mirrors its `anonymousPolicy`, because a logged-in client is not an anonymous one: without the mirror, `podman login` followed by a pull of `milner/…` would turn a working pull into a 401. Two repositories keep anonymous *write*, and they are different cases: - `preview/**` is permanent. A per-PR push cannot be given a credential at all — the pipeline runs the PR's own workflow file and the extension refuses every pull event — and those images are what the l3 preview runs. - `milner/**` is meant to be deleted. The main halves of `build-oz` and `build-router` push `milner/oz` and `milner/router` unattended, and cannot be given a credential yet: the extension's AppRole is granted `secret/forgejo/token` and no other path. Once the `ci/` grant and a rule for this repository exist, those steps take `--creds` and this entry loses its two policies — in that order, because the credential has to be in place before the exception can go. `annia-ci` is the only user in the password file, so the credential is scoped by who holds it rather than by a prefix allowlist that every new publisher would have to remember to extend. ## Order zot reads the file at start-up and exits 1 without it (`Error: open /secret/HTPASSWD: no such file or directory`). The chart rolls the pod when the config changes and the secrets layer is applied after infrastructure, so a pod that restarts into this config retries until VSO has synced the Secret — a window with an end because the Vault side (both entries, plus the `zot-read` policy and `zot` role from §4c-ter) was done first, and a CrashLoopBackOff on the registry every pull goes through when it was not. ## Changes - `infrastructure/zot/set-credential.sh` — `provision` (default), `--rotate`, `verify`. No value in argv, nothing echoed, the root-only half printed for an operator session unless `ROOT_VAULT_TOKEN` is set. - `infrastructure/zot/values.yaml` — `http.auth.htpasswd` + the access-control tree, with the two traps (auth alone authorizes nothing; the two policies are not alternatives) written down where they bite. - `secrets/vaultauth-zot.yaml`, `secrets/zot-htpasswd.yaml` — the namespace's VaultAuth and the Secret the chart mounts. No `rolloutRestartTargets`: zot watches the file and its parent directory, so a rotation is picked up in place. - `OPERATIONS.md` — the `zot-read` policy and `zot` role in §4c-ter, and a `podman login` in §9k step 3's hand-run push. - `infrastructure/zot/README.md` — the policy table, the credential, the ordering, and the evidence table. - `tests/test_gitops.py` — anonymous reads stay, `**` stays read-only for anonymous callers, and the file the config names is the one VSO writes. ## Evidence `zot verify` accepts this exact `config.json`; on a throwaway v2.1.21 with it and the hash Vault holds: anonymous `GET /v2/` 200, `GET /v2/_catalog` 200, `POST /v2/milner/…` 202, `POST /v2/preview/…` 202, `POST /v2/annia/…` 401; `annia-ci` `GET /v2/_catalog` 200 and `POST` 202 on `annia`, `milner`, `library`; a wrong password 401. The log says `loaded htpasswd file users: 1` and `access control is being enabled`. `pytest tests/ -q`: 206 passed.

tepichord added 1 commit

2026-09-17 15:24:23 -07:00

Give zot a push credential, and keep every read open

ci/woodpecker/pr/gates Pipeline failed

Details

47edfd48ab

## Description

The registry had no `http.auth` at all, so every push was anonymous — including
a pull request's own pipeline, which runs `privileged`. That matters here
because the cluster pulls `milner/woodpecker-secret-extension:0.2.4` unpinned,
with the extension's AppRole behind it: an anonymous push endpoint is a path to
replacing an image a pod then runs.

zot reads htpasswd from a *file* and cannot be handed a password, while a
pipeline cannot log in with a hash — so there is one password in two
representations, and they have to be written together or they drift into a 401
nobody can explain later:

| Where | What | Read by |
|---|---|---|
| `secret/ci/zot` → `ZOT_TOKEN` | the cleartext | the pipeline that pushes, via the secret extension |
| `secret/zot/htpasswd` → `HTPASSWD` | the bcrypt hash | zot, via the `VaultStaticSecret` `zot/zot-htpasswd` |

`set-credential.sh` is the one operation that writes both: it derives the hash
from the cleartext it read or minted, so there is no second rotation path. The
hash reaches the pod as a Secret the chart mounts whole at `/secret`, which is
the only way to get it there without putting it in git (`mountSecret: true`
would render it from `secretFiles.htpasswd`).

## What the access policy says

Reads stay anonymous. That is not a detail of the config but the shape of it:
zot consults `defaultPolicy` only for a *named* caller and `anonymousPolicy`
only for an anonymous one (`isPermitted`, `pkg/api/authz.go`), and every puller
of this registry is anonymous — the secret extension, `oz`, the preview router,
and the kubelet. So `**` is read-only for anonymous callers, and every
`defaultPolicy` mirrors its `anonymousPolicy`, because a logged-in client is not
an anonymous one: without the mirror, `podman login` followed by a pull of
`milner/…` would turn a working pull into a 401.

Two repositories keep anonymous *write*, and they are different cases:

- `preview/**` is permanent. A per-PR push cannot be given a credential at all —
the pipeline runs the PR's own workflow file and the extension refuses every
pull event — and those images are what the l3 preview runs.
- `milner/**` is meant to be deleted. The main halves of `build-oz` and
`build-router` push `milner/oz` and `milner/router` unattended, and cannot be
given a credential yet: the extension's AppRole is granted
`secret/forgejo/token` and no other path. Once the `ci/` grant and a rule for
this repository exist, those steps take `--creds` and this entry loses its two
policies — in that order, because the credential has to be in place before the
exception can go.

`annia-ci` is the only user in the password file, so the credential is scoped by
who holds it rather than by a prefix allowlist that every new publisher would
have to remember to extend.

## Order

zot reads the file at start-up and exits 1 without it (`Error: open
/secret/HTPASSWD: no such file or directory`). The chart rolls the pod when the
config changes and the secrets layer is applied after infrastructure, so a pod
that restarts into this config retries until VSO has synced the Secret — a
window with an end because the Vault side (both entries, plus the `zot-read`
policy and `zot` role from §4c-ter) was done first, and a CrashLoopBackOff on
the registry every pull goes through when it was not.

## Changes

- `infrastructure/zot/set-credential.sh` — `provision` (default), `--rotate`,
`verify`. No value in argv, nothing echoed, the root-only half printed for an
operator session unless `ROOT_VAULT_TOKEN` is set.
- `infrastructure/zot/values.yaml` — `http.auth.htpasswd` + the access-control
tree, with the two traps (auth alone authorizes nothing; the two policies are
not alternatives) written down where they bite.
- `secrets/vaultauth-zot.yaml`, `secrets/zot-htpasswd.yaml` — the namespace's
VaultAuth and the Secret the chart mounts. No `rolloutRestartTargets`: zot
watches the file and its parent directory, so a rotation is picked up in
place.
- `OPERATIONS.md` — the `zot-read` policy and `zot` role in §4c-ter, and a
`podman login` in §9k step 3's hand-run push.
- `infrastructure/zot/README.md` — the policy table, the credential, the
ordering, and the evidence table.
- `tests/test_gitops.py` — anonymous reads stay, `**` stays read-only for
anonymous callers, and the file the config names is the one VSO writes.

## Evidence

`zot verify` accepts this exact `config.json`; on a throwaway v2.1.21 with it and
the hash Vault holds: anonymous `GET /v2/` 200, `GET /v2/_catalog` 200,
`POST /v2/milner/…` 202, `POST /v2/preview/…` 202, `POST /v2/annia/…` 401;
`annia-ci` `GET /v2/_catalog` 200 and `POST` 202 on `annia`, `milner`,
`library`; a wrong password 401. The log says `loaded htpasswd file users: 1`
and `access control is being enabled`.

`pytest tests/ -q`: 206 passed.

tepichord force-pushed feat/zot-htpasswd-auth from 47edfd48ab

ci/woodpecker/pr/gates Pipeline failed

Details

to 881f121c95

ci/woodpecker/pr/gates Pipeline failed

Details

2026-09-17 15:31:26 -07:00

Compare

tepichord force-pushed feat/zot-htpasswd-auth from 881f121c95

ci/woodpecker/pr/gates Pipeline failed

Details