Determine testing methodology #11
Labels
No labels
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
wontfix
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
tepichord/milner#11
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Naina:
Updated on 2026-08-19 based on maintainer feedback.
Overview
A pytest-driven static testing strategy with a single entry point:
python -m pytest tests/ -v. No cluster orchestration — just offline validation. The deliverable covers static linters,terraform testwith mock providers, and BATS tests for shell helpers, all unified under pytest.Technical Approach
Add test dependencies to
flake.nixCreate pytest config
Add static validation tests
Add
terraform testwith mock providersNote:
override_during = planrequires Terraform ≥1.11.0. The pinned nixpkgs ships 1.15.x. Provisioners (null_resource) cannot be mocked —command = planskips them automatically.Add BATS tests for shell helpers
Fix
cleanup.shtypoAdd source guard to
create-worker-vm.shImplementation Details
Files to create:
pyproject.tomladdopts = "-v")tests/test_static.pytests/test_terraform.pyterraform testwrappertests/test_shell.pycontrol-plane/tests/plan.tftest.hclworker/tests/test_create_worker_vm.batsFiles to modify:
flake.nixpython3,pytest,shellcheck,hadolint,yamllint,bats.withLibraries,ruff,terraformcontrol-plane/cleanup.sh|| truec→|| true(line 22)worker/vm-setup/create-worker-vm.sh[[ "${BASH_SOURCE[0]}" == "${0}" ]]guard beforemain "$@"Constraints:
terraform testwithoverride_during = planrequires Terraform ≥1.11.0null_resourceprovisioners cannot be mocked (hashicorp/terraform#36469) —command = planskips themterraform apply— tests work fully offlineOut-of-scope notes:
talosctl cluster create docker, podman socket mounting) removed per maintainer directive in comment #6943Acceptance Criteria
python -m pytest tests/ -vruns all tests with clear pass/fail per testterraform testpasses incontrol-plane/without a running libvirt daemonbats worker/tests/passes all helper function testscleanup.shline 22 reads|| true(not|| truec)Testing Plan
Related Issues/PRs
override_provisioner(open, blocks provisioner mocking)are you sure there are enough permissions in the container for this to work? Especially since I won’t allow docker and it must use podman.
Naina:
Good questions — I dug into both.
On podman vs docker:
talosctl cluster create dockerworks with podman via socket compatibility. You setDOCKER_HOST=unix:///run/podman/podman.sock(or the rootful socket path) and talosctl talks to podman's Docker-compatible API. This has been confirmed working since Podman v4.2.0 / Talos v1.2.3. On macOS, the podman machine needs to be rootful (podman machine set --rootful=true) because the containers run in--privilegedmode and need access to/dev/kmsgand kernel parameter modifications.On permissions: The containers run
--privileged, so they need a real container runtime with full access. This means the integration test cannot run inside a Nix sandbox or a restricted CI container — it has to be invoked from thenix developdev shell where podman has direct access. That's why in the updated plan it's an opt-in test (explicit marker/flag), not somethingnix flake checkwould attempt.On the unified runner: Agreed on pytest. It gives us a single
pytest tests/ -ventry point with structured output (test name, pass/fail, errors) and lets us organize the integration tests, static checks, and terraform tests in one place. The integration tests get a pytest marker so they're skipped by default unless you opt in.I'll update the plan to focus on the comprehensive integration test as the primary deliverable, with pytest as the orchestrator.
So what would have to change in the annia-infra project (you can look at the code under /workspace/annia-infra) to make it work with the
talosctl cluster create dockercommand? You can find the script used to create the environment you're in atstart-workspace.shNaina:
The main change is mounting the host's podman socket into the workspace container. Right now
start-workspace.shdoesn't give the container access to any container runtime, sotalosctl cluster create dockerhas nothing to talk to.Two lines in
start-workspace.shwould do it:The socket-mount approach is better than podman-in-podman because the talos containers run
--privilegedwith--security-opt seccomp=unconfinedand need access to/dev/kmsgand/proc/sys/*. With socket mounting, those containers are created directly on the host where they get full privileges naturally — the workspace container itself doesn't need--privileged.On macOS, the podman machine must be rootful (
podman machine set --rootful=true) because rootless podman doesn't propagate privileged capabilities to nested containers, which causes talos nodes to fail with permission denied errors. The Dockerfile and flake.nix in annia-infra don't need changes —talosctlis already in the milner flake vianix develop.And what would the considerations be when annia-infra eventually becomes a Kubernetes pod instead of a podman container? Is there an alternative there or would podman in-podman be the only option in that case?
Naina:
Three options, ordered by complexity:
Socket mounting from the node (simplest). Mount the node's container runtime socket into the annia-infra pod via a
hostPathvolume pointing to/run/podman/podman.sock(or/run/containerd/containerd.sock). The talosctl containers then run directly on the node, same as today. The catch: this requires aprivilegedPod Security Standard namespace because the kubelet'shostPathtypeSockethas known issues — it may create a directory instead of waiting for a socket (kubernetes/kubernetes#128814). Security-wise, you're exposing the node's container runtime API to the pod, which is a container escape path. For a single-node dev cluster this is acceptable; for a multi-tenant production node it's a hard no.Docker-in-Docker / Podman-in-Podman. Run a container runtime daemon inside the workspace pod itself. Requires
privileged: trueon the pod and a volume for the daemon's storage. Same security tradeoffs as socket mounting but self-contained — no host socket exposure. The talos containers run inside the nested daemon. This is how KinD-in-pod works (kubernetes-sigs/kind#303 has working pod specs).Sysbox runtime (most secure). Use Nestybox Sysbox as the container runtime on the node. Sysbox allows running Docker/KinD/talosctl inside unprivileged pods — no
privilegedflag, no host socket mount, proper isolation. The tradeoff is that Sysbox has to be installed on every cluster node and isn't part of standard Kubernetes distributions. Long-term this is the right answer, but it's a node-level change that needs ops buy-in.None of these are milner's problem to solve — they all live in the annia-infra orchestration layer. The milner tests just need
DOCKER_HOSTpointed at whatever socket the workspace provides.I’ve soured on the full orchestration integration test suite. Let’s rework the planning comment to just incorporate the static testing that we’ve been talking about with the pytest suites.