tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Add Postgres Database to Milner #40

Merged
tepichord merged 39 commits from feat/35-add-postgres-database into main 2026-09-19 11:26:26 -07:00
Conversation 6 Commits 39 Files changed 34 +1472 -176
tepichord commented 2026-09-09 18:47:40 -07:00
Owner
Copy link

Closes #35

Automated PR created by forgejo-done adapter.

Closes #35 Automated PR created by forgejo-done adapter.
Deploys CloudNativePG operator (chart 0.29.0) + Rancher local-path
provisioner (chart 0.0.37) + single-instance Cluster CR on the worker.
Three namespaces: local-path-storage (privileged), cnpg-system (default),
postgres (restricted).

Refs: #35
Tests PSA labels, helm chart/version/repo, Cluster CR fields
(instances, storageClass, initdb, nodeSelector). Fixture copied
from vault.

Refs: #35
- Add libvirt_volume.postgres_data (50 GiB qcow2) at vdc
- Add postgres UserVolumeConfig with size-band exclusive selector (>40 GiB)
- Tighten vault selector from bare !system_disk to 15-40 GiB range
- All data-disk selectors are now mutually exclusive

Refs: #35
- Fix disk index shifts: cdrom at disks[3], eject disk count 2→3
- Update vault selector assertion to size-band range
- Add postgres volume, vdc attach, and selector exclusivity tests
- Patches count 2→3 for vault + postgres UserVolumeConfigs

Refs: #35
- Include postgres/ in terraform fmt -check dirs
- Include postgres/values.yaml in yamllint files
- Add test_terraform_test_postgres() runner

Refs: #35
- README: add CloudNativePG Postgres to workload list, add runbook section
- ARCHITECTURE: add Postgres to topology, service table, storage distribution
- AGENTS: add postgres/ to repo map
- OPERATIONS: add Postgres section (deploy, connect, teardown durability, pg_dump)

Refs: #35
Refs: #35
The postgres Cluster manifest hard-coded namespace = "postgres" without
referencing kubernetes_namespace_v1.postgres, so Terraform had no edge
forcing the namespace to be created first. On a fresh apply the manifest
could be submitted before the namespace existed and the apply would fail.
Reference the namespace resource (the pattern vault/main.tf already uses)
so the dependency is explicit.
The acceptance criteria and testing plan call for the provisioner's
root path to be covered; the suite only checked chart/version/repo/
namespace. Assert the rendered values point nodePathMap at
/var/mnt/postgres.
postgres/README.md and OPERATIONS.md §4d claimed the 50 GiB qcow2 was
un-managed via a removed{lifecycle{destroy=false}} block in
worker/main.tf. No such block exists (the volume is a normally managed
libvirt_volume, matching the Vault and pocket-id volumes), so a full
terraform destroy / cleanup.sh does remove it. Describe the actual
behaviour and point at the pg_dump runbook instead.

Renumbering Vault from §4 to §5 (to make room for Postgres) also left
stale references: OPERATIONS.md's state table, bring-up order, and the
vault/worker READMEs still pointed at the old section numbers.
tepichord commented 2026-09-10 15:40:35 -07:00
Author
Owner
Copy link

Merge changes in from main

Merge changes in from main
tepichord added this to the MVP project 2026-09-10 15:40:38 -07:00 
The CNPG operator is installed in cnpg-system while the Cluster CR lives
in postgres. The chart's config.clusterWide defaults to false, which
restricts the operator to its own namespace, so the Cluster would silently
never reconcile while all other plan assertions still pass. Assert the
value, matching how storage/tests/plan.tftest.hcl guards its load-bearing
Helm values.
tepichord commented 2026-09-11 19:20:33 -07:00
Author
Owner
Copy link

Naina:
Not a change summary — one heads-up while resolving the merge with main.

Two storage/ references remain on main and are therefore out of scope for this PR, so I left them untouched:

  • data/README.md:102 — the re-provision list still reads storage/ vault/ woodpecker/, but the storage/ root was removed by the Flux migration.
  • worker/main.tf:275 — a comment says /var/mnt/data is the "backing store for the storage/ StorageClasses"; the provisioner is now Flux-managed under gitops/clusters/milner/infrastructure/local-path/.

They are stale on main independently of this branch. Happy to clean them up here if you'd prefer, otherwise they belong in a follow-up.

Naina: Not a change summary — one heads-up while resolving the merge with `main`. Two `storage/` references remain on `main` and are therefore **out of scope** for this PR, so I left them untouched: - `data/README.md:102` — the re-provision list still reads `storage/ vault/ woodpecker/`, but the `storage/` root was removed by the Flux migration. - `worker/main.tf:275` — a comment says `/var/mnt/data` is the "backing store for the `storage/` StorageClasses"; the provisioner is now Flux-managed under `gitops/clusters/milner/infrastructure/local-path/`. They are stale on `main` independently of this branch. Happy to clean them up here if you'd prefer, otherwise they belong in a follow-up.
The comment claimed config.clusterWide defaults to false; the chart
defaults to true. Shorten the rationale to a one-line assertion purpose.

Refs: #40
§11a told the operator to run `terraform -chdir=postgres plan` before the
`-target=helm_release.cnpg` apply, but that is exactly the plan that fails
on a fresh cluster: `kubernetes_manifest` resolves the Cluster CR's GVK at
plan time and the CRD only exists after the operator applies. The surrounding
text and the README runbook already order init -> targeted apply -> plan, so
this line contradicted both.

Refs: #40
style(tests): match skipif decorator formatting in test_terraform
All checks were successful
ci/woodpecker/pr/woodpecker Pipeline was successful
Details
d0ece136ec 
test_terraform_test_postgres carried the only multi-line
`@pytest.mark.skipif(...)` in the file; the four sibling tests (and ruff
format's output) keep it on one line.

Refs: #40
tepichord commented 2026-09-15 15:23:15 -07:00
Author
Owner
Copy link

Significant architectural changes were made to milner, please merge the changes into this branch and integrate them with the Postgres changes.

Significant architectural changes were made to milner, please merge the changes into this branch and integrate them with the Postgres changes.
chore(postgres): drop unused cluster_name variable (#40)
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
0307f041f8 
The postgres root never references it: cluster identity comes from the
control-plane state's kubeconfig output, as in kubernetes-dashboard/.
Every other root that declares cluster_name uses it to name libvirt
resources; here it was dead input.
Keep main's new VictoriaLogs section as §12 and renumber the Postgres
runbook to §13, dropping main's stale duplicate Bring-up order section
(§8 remains the canonical list).
docs(postgres): point backup cross-reference at §13e (#40)
Some checks failed
ci/woodpecker/pr/gates Pipeline failed
Details
ce248f0c05 
Refs: #40
docs(postgres): correct provisioner kind and worker mount step (#40)
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
37598698ef
Merge branch 'main' into feat/35-add-postgres-database
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
b0f4999e0c
tepichord commented 2026-09-17 14:03:56 -07:00
Author
Owner
Copy link

Make postgres an app managed by flux + kustomization.

Make postgres an app managed by flux + kustomization.
Move the CloudNativePG operator and the single-instance Cluster out of the
Terraform root and into the GitOps tree: the operator chart is a HelmRelease
in the infrastructure layer (a layer earlier than the Cluster CR it defines)
and cluster.yaml is wired into the apps layer, with both namespaces declared
under infrastructure/namespaces/. The apps Kustomization gets a
healthCheckExprs entry so wait: true gates on the Cluster's Ready condition.
test(postgres): port the deleted plan guards into test_gitops (#40)
Some checks failed
ci/woodpecker/pr/gates Pipeline failed
Details
47d92eea63
tepichord commented 2026-09-18 08:48:04 -07:00
Author
Owner
Copy link

The woodpecker build failed in l1-validate for the following reason:

kubeconform -strict -ignore-missing-schemas -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' rendered-*.yaml
+ kubeconform -strict -ignore-missing-schemas -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' rendered-*.yaml
rendered-apps.yaml - Cluster postgres is invalid: problem validating schema. Check JSON formatting: jsonschema: '/spec' does not validate with https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/postgresql.cnpg.io/cluster_v1.json#/properties/spec/additionalProperties: additionalProperties 'nodeSelector' not allowed
The woodpecker build failed in l1-validate for the following reason: ``` kubeconform -strict -ignore-missing-schemas -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' rendered-*.yaml + kubeconform -strict -ignore-missing-schemas -schema-location default -schema-location 'https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/{{.Group}}/{{.ResourceKind}}_{{.ResourceAPIVersion}}.json' rendered-*.yaml rendered-apps.yaml - Cluster postgres is invalid: problem validating schema. Check JSON formatting: jsonschema: '/spec' does not validate with https://raw.githubusercontent.com/datreeio/CRDs-catalog/main/postgresql.cnpg.io/cluster_v1.json#/properties/spec/additionalProperties: additionalProperties 'nodeSelector' not allowed ```
The Cluster CR carried a top-level `spec.nodeSelector`, which the CNPG
Cluster CRD does not declare: `-strict` kubeconform rejected it in the L1
gate with "additionalProperties 'nodeSelector' not allowed". CNPG keeps
node placement in its own AffinityConfiguration, so the selector moves to
`spec.affinity.nodeAffinity` as required (not preferred) node affinity —
the `persistent` class is node-local, so anywhere but the worker the
claim never binds and the Cluster never starts.

The ported plan guard asserts the new structure, plus the absence of
`spec.nodeSelector` so the schema-valid form is what stays.
refactor(postgres): pin the worker via affinity.nodeSelector (#40)
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
42cd58f0b1 
CNPG's AffinityConfiguration exposes its own `nodeSelector` as the 1:1
equivalent of the top-level field the CRD rejects, and it is inherently
required. The required-node-affinity term was a more verbose way to say
the same thing and matches no other node placement in the repo, so use
the operator's own field and assert it in the ported plan guard.
Merge branch 'main' into feat/35-add-postgres-database
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
9ae0c6f98b
tepichord commented 2026-09-18 23:05:23 -07:00
Author
Owner
Copy link

Are there any services/pods on milner that can swap from their existing file based data storage to database driven with postgres?

Are there any services/pods on milner that can swap from their existing file based data storage to database driven with postgres?
refactor(postgres): move the Cluster into its own data layer (#40)
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
6059dbf06f 
A database-backed app names a `Database` and a `DatabaseRole` in its own
manifests, and a layer is applied atomically — so while the Cluster and the app
share the `apps` layer, the app starts before its database exists and converges
only by restart backoff. A HelmRelease cannot `dependsOn` a custom resource, so
the ordering has to be structural rather than declared per app.

So the Cluster moves to a `data` layer between `secrets` and `apps`: the VSO
Secrets the future DatabaseRoles name come from `secrets`, and the apps that
name them wait on `data`. The `healthCheckExprs` entry moves with the Cluster —
in a layer that does not hold it the expression matches nothing, and `wait: true`
would report Ready before Postgres is up without saying so.

`test_data_layer_sits_between_secrets_and_apps` pins the chain, which side the
health check is on, and which layer owns the Cluster. No consumer is switched
here: this change is only the layer that makes switching one possible.

Note for the first reconcile on a live cluster: if the `apps` Kustomization
already has `Cluster/postgres` in its inventory it will prune the object while
`data` adopts it. Check with `flux` and clear
`/var/mnt/data/persistent/postgres/` first — `initdb` against a PVC that already
holds a data directory is not a recoverable state.
feat(woodpecker): move the CI server's state into Postgres (#40)
All checks were successful
ci/woodpecker/pr/gates Pipeline was successful
Details
888de46706 
The server's whole state was `woodpecker.sqlite` on a 10 GiB `persistent`
claim: repo metadata, build history, build logs (the log store defaults to
`database`) and the two signing keys. All of it becomes rows in the new
`postgres` cluster, so the server keeps no volume at all and the durable
tier holds one data directory instead of one file per app.

Shape, one decision per object:

- `data/postgres/roles.yaml` + `databases.yaml`: one `DatabaseRole` and one
  `Database` per app, in the Cluster's own namespace. The role is the owner,
  which is what lets the app's migrations write `public` on PG 15+
  (`pg_database_owner`) without this layer declaring a schema, and both
  reclaim policies are stated `retain` — a pruned manifest must not drop a
  role or a database. `ensure: absent` is not available here: the reconciler
  refuses it for `DatabaseRole`, pointing at the reclaim policy.
- `data.yaml` gains `healthCheckExprs` for `DatabaseRole`/`Database`
  (`current` only — the operator writes the outcome to `status.applied`, so a
  refusal and a not-yet-reconciled object look alike and a `failed`
  expression would flap). This is the same gate the Cluster got, one level
  down: `apps` must not start a server against a database that is not there.
- Credential: `secret/woodpecker/database` in Vault is the one source, synced
  into two Secrets because a `passwordSecret` is a LocalObjectReference and
  the operator reads it in its own namespace. The `postgres` copy is
  basic-auth and labelled `k8s.enterprisedb.io/reload` (the rotation
  mechanism); the `woodpecker` copy is Opaque and holds only a DSN templated
  from the same password, so the two cannot drift. The password is hex, not
  base64: the DSN is a URL and VSO templates it as a plain string.
- `apps/woodpecker`: driver `postgres`, DSN through `valueFrom.secretKeyRef`,
  and `persistentVolume.enabled: false` stated *explicitly* — the chart's
  default is `true`, so omitting the key inherits a 10 GiB claim holding
  nothing but the file nobody opens any more. The agent's `agent.conf` cache
  moves to `ephemeral`: it is a derived agent id, and a stale one is a
  permanent crashloop, not a cold start.

The cut-over is a hand-cut, recorded in docs/woodpecker-ci.md §9l (nothing
ported; `volumeClaimTemplates` and PVC `storageClassName` are immutable, so
both StatefulSets and the agent's claim are deleted first). It is not only the
database that moves: the extension's copy of the server's *public* signing key
(§9k) is derived from a row that is now regenerated, so §9l step 5 re-fetches
it and §9g's rotation points at the same step.

Drills follow the data: `drills/check.sh` now compares the Postgres data
directory's inode and live row counts (psql over the pod's Unix socket, peer
auth, no credential), and selects durable claims **by class** instead of by
name so preview churn cannot produce false diffs. A chart-render guard in
tests/test_chart_sources.py is what keeps `persistentVolume: false` honest.
tepichord deleted branch feat/35-add-postgres-database 2026-09-19 11:26:26 -07:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner!40
No description provided.