Determine incremental deployment methodology #27
Labels
No labels
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
tepichord/milner#27
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Naina:
Naina:
Updated on 2026-08-22 based on maintainer feedback.
Overview
Define Milner's incremental deployment methodology and commit it to documentation in absolute terms so agents adhere closely. Per maintainer comment #7594, this is strictly a methodology + documentation issue — NO code changes. The strategy: option (a) Terraform delta is the default everywhere past the VM/base-infra boundary, option (b)
terraform apply -replaceis used only at the VM-level base infrastructure for disposable nodes (keeping the persistent volume/network/pool separate and protected), and option (c) per-change migration scripts are rejected outright. The deliverables are the methodology text and anARCHITECTURE.mdcomponent mapping; all current-environment friction (partial control-plane deployment) is explicitly out of scope.Technical Approach
Record the layer boundary — which layers have a real in-place delta and what to do per layer.
staged_if_needing_reboot), never reset the nodeDocument what (a) looks like in practice — one change, apply, only that reconciles.
Document what (b) looks like in practice — swap only the disposable VM, keep the disk/network/pool.
Keep the domain and persistent volume as separate resources so the volume is never a replacement target:
Define cleanup.sh's exact purpose in absolute terms: full cluster teardown only.
-replaceswaps the disposable node but does not remove the environment (network, pool, kubeconfig/talos contexts);cleanup.shis the confirmation-gated full removal tool, never a partial-recovery tool (recovery is-replace), and it must never globallypkillQEMU or silently delete state/persistent data. (Documenting the contract; editing the script itself is out of scope.)Frame adoption paths, not divergent instruction sets. Record the single-graph framing: clean install = apply from empty; incremental adopt/recover = import existing UUID → review plan → apply (same
main.tf, different entry points).Write the methodology in absolute terms in
README.mdandARCHITECTURE.md: what (a)/(b) are in practice, best practices for each ((a): plan before apply, confirmation-gated, never commit state/secrets, prefer-replaceover destroy+recreate; (b): volume separate withprevent_destroy, review every-replace, never hand-runvirsh/qemu-img), the layer boundary table, and the clean-install vs incremental-adopt framing. All commands run insidenix develop.Map every
ARCHITECTURE.mdcomponent to (a)/(b)../vault_dataImplementation Details
README.md— add/adjust the "How Changes Ship" section to state the methodology, the layer-boundary table, what (a)/(b) are in practice, cleanup's exact purpose, and best practices in absolute terms.ARCHITECTURE.md— add the per-component (a)/(b) mapping table and the layer-boundary (VM → Talos config → pods/Helm) framing.control-plane/README.md— note the clean-install vs incremental-adopt pathways and cleanup purpose.control-plane/main.tfedits, noimport {}blocks, no.terraform.lock.hcl, nocleanup.shedits, noplan.tftest.hcl/test changes. Documentation may referenceimport/-replaceadoption but must not assume a provider version the repo is not yet on (repo pinslibvirt ~>0.7; domain-UUID import is confirmed on 0.9.6+ — flag the provider-version dependency).terraform apply/talosctlwithout explicit confirmation; no manualvirsh/qemu-imglifecycle); no secrets/certs committed; docs are example-forward and absolute in tone.Acceptance Criteria
README.mddocuments the methodology in absolute terms: (a) default past the VM boundary, (b)-replaceonly for disposable VM-level base infra, (c) rejected — plus best practices for each.README.mddefines cleanup.sh's exact purpose as full cluster teardown only (never partial recovery, never broadpkill/silent data or state deletion), and notes recovery is-replace.README.mdrecords the layer-boundary table (VM/disk/network/pool, Talos config, pods/Helm/Compose) and the clean-install vs incremental-adopt single-graph framing.ARCHITECTURE.mdmaps every component (control plane, worker-large, worker-sim, HC Vault, Woodpecker) to its normal (a) and recovery (b) method.cleanup.share modified — this is documentation-only, per comment #7594.Testing Plan
bash -n/ markdown checks that any doc-embedded commands are syntactically valid and example-forward.python -m pytest tests/ -vinsidenix developto confirm no doc change breaks existing lints.README.mdandARCHITECTURE.mdstate the methodology in absolute terms and the(a)/(b)/(c)decision and per-component table match this comment..tf,.sh,.hcl, or test files).Related Issues
Planning comment aside, respond to this comment with your thoughts on potential options to take for this methodology. Lay them out and compare each of the options, including where they might apply.
Naina:
I recommend option (a) as the default: Terraform’s reviewed plan/apply already converges partial infrastructure and applies only the declared delta. Use option (b) for disposable boundaries such as a control-plane or worker VM: replace the domain with
terraform apply -replace=...while keeping the network, pool, and data volume protected; a full disk reset remains a separate, explicit destructive operation. Use option (c) only for thin, repeatable adoption/recovery sequencing—not as per-change migrations—because scripts otherwise become a second source of truth. That maps to Terraform deltas plus targeted replacement for the control plane and workers, idempotent compose/Helm reconciliation for HC Vault and Woodpecker, and helper scripts only around those workflows.Okay ignoring the planning comment and looking at your recommendation:
Naina:
Yes, I agree with all of that — you've basically got the right shape. Let me make it concrete per option rather than abstract.
(a) —
terraform applyas a deltaThis is just how Terraform behaves against stateful resources. Change one thing, apply, and only that changes:
One value changed → only that release is reconciled. Toward pods/releases, this is the whole story.
(b) — replace only the disposable VM
This is where Terraform cannot diff in place. A VM created by
virt-installfrom an ISO isn't a stateful resource, so the only honest "delta" for the VM itself is recreate:Only the domain is replaced; network, pool, and the persistent data volume stay untouched (separate
libvirt_volumewithprevent_destroy). Past the VM boundary, config is a delta again, not a reset:(c) — per-change migration scripts
Agreed, drop it as a methodology. It becomes a second source of truth you have to keep in sync with reality. An adoption/recovery helper (import an existing VM once) is fine, but not a migration script per change.
Layers:
So exactly your read: (b) only at the VM-level base infrastructure (and even then only the disposable node, keeping the disk separate), and past that, (a) is used pretty much wholesale.
On
cleanup.sh— yes, it should reset to a known-good state, but scoped to what Terraform owns: destroy managed resources + remove the talosctl client context, never globallypkill qemuor delete persistent storage. I'll fold this into the plan revision.Naina:
Yes, a cleanup script still earns its keep —
terraform apply -replaceonly swaps the disposable VM; it doesn't tear down the whole control-plane environment (network, pool, kubeconfig/talos contexts). When you want to fully remove a cluster you'd otherwise hand-runterraform destroyplustalosctl config removeplus kubeconfig cleanup, so cleanup.sh stays — but rebased to Terraform ownership: validate target → show exact disposable targets → confirmation-gatedterraform destroy→talosctl config remove. No globalpkill qemu, no silent state or persistent-data deletion.For how (b) gets defined and run: it's a README documentation section with the concrete command —
terraform plan -replace='libvirt_domain.control_plane'to recover the disposable node while the volume stays protected. And yes, there are effectively two pathway docs (clean-install bootstrap vs incremental adopt/recover), but they share ONE Terraform graph, not two divergent instruction sets: the clean path is an apply from empty, the incremental path is import existing UUID → review plan → apply. Samemain.tf, different entry points.This is purely out of scope for this issue, this issue is simply a methodology and document issue. NO code changes will be done. I would like to define a strategy to best ensure success in the future, not concerned with the current issues at play that conflict with the strategy. Update the planning comment to reflect that.