Determine testing methodology #11

Closed
opened 2026-08-18 12:20:06 -07:00 by tepichord · 8 comments
Owner
  • I would like a proper and thorough method to test that the infrastructural commands could work, but need to evaluate if it’s possible beyond a simple terraform plan command
  • the first thing that comes to mind is the use of minikube and podman, perhaps a testing script can be created to simulate a small form of the k8 cluster? Although this misses testing whether the actual cluster can be created from the terraform file.
  • explore some other options for testing and report them in the planning comment.
- I would like a proper and thorough method to test that the infrastructural commands could work, but need to evaluate if it’s possible beyond a simple terraform plan command - the first thing that comes to mind is the use of minikube and podman, perhaps a testing script can be created to simulate a small form of the k8 cluster? Although this misses testing whether the actual cluster can be created from the terraform file. - explore some other options for testing and report them in the planning comment.
tepichord added this to the MVP project 2026-08-18 12:20:06 -07:00
Author
Owner

Naina:
Updated on 2026-08-19 based on maintainer feedback.

Overview

A pytest-driven static testing strategy with a single entry point: python -m pytest tests/ -v. No cluster orchestration — just offline validation. The deliverable covers static linters, terraform test with mock providers, and BATS tests for shell helpers, all unified under pytest.

Technical Approach

  1. Add test dependencies to flake.nix

    buildInputs = with pkgs; [
      # ... existing ...
      python3
      python3Packages.pytest
      shellcheck
      hadolint
      yamllint
      (bats.withLibraries (p: [ p.bats-support p.bats-assert ]))
      ruff
      terraform
    ];
    
  2. Create pytest config

    # pyproject.toml
    [tool.pytest.ini_options]
    addopts = "-v"
    
  3. Add static validation tests

    # tests/test_static.py
    import subprocess, pathlib
    
    def test_terraform_fmt():
        r = subprocess.run(["terraform", "fmt", "-check", "-recursive", "control-plane/"],
                           capture_output=True)
        assert r.returncode == 0, r.stderr.decode()
    
    def test_shellcheck():
        scripts = ["control-plane/cleanup.sh", "worker/vm-setup/create-worker-vm.sh"]
        r = subprocess.run(["shellcheck"] + scripts, capture_output=True)
        assert r.returncode == 0, r.stderr.decode()
    
    def test_hadolint():
        r = subprocess.run(["hadolint", "hc-vault/hc_vault/Dockerfile"], capture_output=True)
        assert r.returncode == 0, r.stderr.decode()
    
    def test_yamllint():
        files = ["hc-vault/docker-compose.yml", "woodpecker/values.yaml"]
        r = subprocess.run(["yamllint"] + files, capture_output=True)
        assert r.returncode == 0, r.stderr.decode()
    
    def test_python_lint():
        r = subprocess.run(["ruff", "check", "tests/"], capture_output=True)
        assert r.returncode == 0, r.stderr.decode()
    
  4. Add terraform test with mock providers

    # control-plane/tests/plan.tftest.hcl
    mock_provider "libvirt" {
      override_during = plan
    }
    mock_provider "talos" {
      override_during = plan
    }
    
    run "network_has_correct_subnet" {
      command = plan
      assert {
        condition     = libvirt_network.talos_bridge.addresses[0] == "10.5.0.1"
        error_message = "Expected 10.5.0.1 subnet"
      }
    }
    
    run "disk_is_40gb" {
      command = plan
      assert {
        condition     = libvirt_volume.control_plane_disk.size == 42949672960
        error_message = "Expected 40GB disk"
      }
    }
    
    # tests/test_terraform.py
    import subprocess
    def test_terraform_test():
        r = subprocess.run(["terraform", "-chdir=control-plane", "test", "-no-color"],
                           capture_output=True)
        assert r.returncode == 0, r.stderr.decode()
    

    Note: override_during = plan requires Terraform ≥1.11.0. The pinned nixpkgs ships 1.15.x. Provisioners (null_resource) cannot be mocked — command = plan skips them automatically.

  5. Add BATS tests for shell helpers

    # worker/tests/test_create_worker_vm.bats
    setup() {
      bats_load_library bats-support
      bats_load_library bats-assert
      source "${BATS_TEST_DIRNAME}/../vm-setup/create-worker-vm.sh"
    }
    
    @test "generate_mac produces valid MAC" {
      run generate_mac "test-vm"
      assert_success
      assert_output --regexp '^52:54:00:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}$'
    }
    
    @test "create_vm_dir creates directory" {
      local result
      result=$(create_vm_dir "test-vm")
      [[ -d "$result" ]]
    }
    
    # tests/test_shell.py
    import subprocess
    def test_bats():
        r = subprocess.run(["bats", "worker/tests/"], capture_output=True)
        assert r.returncode == 0, r.stdout.decode()
    
  6. Fix cleanup.sh typo

    -sudo rm -rf "${HOME}/.talos/clusters/${cluster_name}" || truec
    +sudo rm -rf "${HOME}/.talos/clusters/${cluster_name}" || true
    
  7. Add source guard to create-worker-vm.sh

    -main "$@"
    +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then
    +  main "$@"
    +fi
    

Implementation Details

Files to create:

File Purpose
pyproject.toml Pytest config (addopts = "-v")
tests/test_static.py terraform fmt, shellcheck, hadolint, yamllint, ruff
tests/test_terraform.py terraform test wrapper
tests/test_shell.py BATS wrapper
control-plane/tests/plan.tftest.hcl Mock provider test definitions
worker/tests/test_create_worker_vm.bats BATS tests for helper functions

Files to modify:

File Change
flake.nix Add python3, pytest, shellcheck, hadolint, yamllint, bats.withLibraries, ruff, terraform
control-plane/cleanup.sh Fix || truec|| true (line 22)
worker/vm-setup/create-worker-vm.sh Add [[ "${BASH_SOURCE[0]}" == "${0}" ]] guard before main "$@"

Constraints:

  • terraform test with override_during = plan requires Terraform ≥1.11.0
  • null_resource provisioners cannot be mocked (hashicorp/terraform#36469) — command = plan skips them
  • No terraform apply — tests work fully offline

Out-of-scope notes:

  • Integration tests (talosctl cluster create docker, podman socket mounting) removed per maintainer directive in comment #6943
  • annia-infra workspace changes (socket mounting, K8s pod migration) tracked separately — see earlier discussion

Acceptance Criteria

  • python -m pytest tests/ -v runs all tests with clear pass/fail per test
  • terraform test passes in control-plane/ without a running libvirt daemon
  • bats worker/tests/ passes all helper function tests
  • cleanup.sh line 22 reads || true (not || truec)

Testing Plan

# Static checks + terraform + BATS, all offline
nix develop --command python -m pytest tests/ -v

# Terraform mock validation alone
nix develop --command python -m pytest tests/test_terraform.py -v

# BATS alone
nix develop --command bats worker/tests/
Naina: *Updated on 2026-08-19 based on maintainer feedback.* #### Overview A pytest-driven static testing strategy with a single entry point: `python -m pytest tests/ -v`. No cluster orchestration — just offline validation. The deliverable covers static linters, `terraform test` with mock providers, and BATS tests for shell helpers, all unified under pytest. #### Technical Approach 1. **Add test dependencies to `flake.nix`** ```nix buildInputs = with pkgs; [ # ... existing ... python3 python3Packages.pytest shellcheck hadolint yamllint (bats.withLibraries (p: [ p.bats-support p.bats-assert ])) ruff terraform ]; ``` 2. **Create pytest config** ```toml # pyproject.toml [tool.pytest.ini_options] addopts = "-v" ``` 3. **Add static validation tests** ```python # tests/test_static.py import subprocess, pathlib def test_terraform_fmt(): r = subprocess.run(["terraform", "fmt", "-check", "-recursive", "control-plane/"], capture_output=True) assert r.returncode == 0, r.stderr.decode() def test_shellcheck(): scripts = ["control-plane/cleanup.sh", "worker/vm-setup/create-worker-vm.sh"] r = subprocess.run(["shellcheck"] + scripts, capture_output=True) assert r.returncode == 0, r.stderr.decode() def test_hadolint(): r = subprocess.run(["hadolint", "hc-vault/hc_vault/Dockerfile"], capture_output=True) assert r.returncode == 0, r.stderr.decode() def test_yamllint(): files = ["hc-vault/docker-compose.yml", "woodpecker/values.yaml"] r = subprocess.run(["yamllint"] + files, capture_output=True) assert r.returncode == 0, r.stderr.decode() def test_python_lint(): r = subprocess.run(["ruff", "check", "tests/"], capture_output=True) assert r.returncode == 0, r.stderr.decode() ``` 4. **Add `terraform test` with mock providers** ```hcl # control-plane/tests/plan.tftest.hcl mock_provider "libvirt" { override_during = plan } mock_provider "talos" { override_during = plan } run "network_has_correct_subnet" { command = plan assert { condition = libvirt_network.talos_bridge.addresses[0] == "10.5.0.1" error_message = "Expected 10.5.0.1 subnet" } } run "disk_is_40gb" { command = plan assert { condition = libvirt_volume.control_plane_disk.size == 42949672960 error_message = "Expected 40GB disk" } } ``` ```python # tests/test_terraform.py import subprocess def test_terraform_test(): r = subprocess.run(["terraform", "-chdir=control-plane", "test", "-no-color"], capture_output=True) assert r.returncode == 0, r.stderr.decode() ``` **Note:** `override_during = plan` requires Terraform ≥1.11.0. The pinned nixpkgs ships 1.15.x. Provisioners (`null_resource`) cannot be mocked — `command = plan` skips them automatically. 5. **Add BATS tests for shell helpers** ```bash # worker/tests/test_create_worker_vm.bats setup() { bats_load_library bats-support bats_load_library bats-assert source "${BATS_TEST_DIRNAME}/../vm-setup/create-worker-vm.sh" } @test "generate_mac produces valid MAC" { run generate_mac "test-vm" assert_success assert_output --regexp '^52:54:00:[0-9a-f]{2}:[0-9a-f]{2}:[0-9a-f]{2}$' } @test "create_vm_dir creates directory" { local result result=$(create_vm_dir "test-vm") [[ -d "$result" ]] } ``` ```python # tests/test_shell.py import subprocess def test_bats(): r = subprocess.run(["bats", "worker/tests/"], capture_output=True) assert r.returncode == 0, r.stdout.decode() ``` 6. **Fix `cleanup.sh` typo** ```diff -sudo rm -rf "${HOME}/.talos/clusters/${cluster_name}" || truec +sudo rm -rf "${HOME}/.talos/clusters/${cluster_name}" || true ``` 7. **Add source guard to `create-worker-vm.sh`** ```diff -main "$@" +if [[ "${BASH_SOURCE[0]}" == "${0}" ]]; then + main "$@" +fi ``` #### Implementation Details **Files to create:** | File | Purpose | |------|---------| | `pyproject.toml` | Pytest config (`addopts = "-v"`) | | `tests/test_static.py` | terraform fmt, shellcheck, hadolint, yamllint, ruff | | `tests/test_terraform.py` | `terraform test` wrapper | | `tests/test_shell.py` | BATS wrapper | | `control-plane/tests/plan.tftest.hcl` | Mock provider test definitions | | `worker/tests/test_create_worker_vm.bats` | BATS tests for helper functions | **Files to modify:** | File | Change | |------|--------| | `flake.nix` | Add `python3`, `pytest`, `shellcheck`, `hadolint`, `yamllint`, `bats.withLibraries`, `ruff`, `terraform` | | `control-plane/cleanup.sh` | Fix `\|\| truec` → `\|\| true` (line 22) | | `worker/vm-setup/create-worker-vm.sh` | Add `[[ "${BASH_SOURCE[0]}" == "${0}" ]]` guard before `main "$@"` | **Constraints:** - `terraform test` with `override_during = plan` requires Terraform ≥1.11.0 - `null_resource` provisioners cannot be mocked (hashicorp/terraform#36469) — `command = plan` skips them - No `terraform apply` — tests work fully offline **Out-of-scope notes:** - Integration tests (`talosctl cluster create docker`, podman socket mounting) removed per maintainer directive in comment #6943 - annia-infra workspace changes (socket mounting, K8s pod migration) tracked separately — see [earlier discussion](https://git.gloom.garden/tepichord/milner/issues/11#issuecomment-6908) #### Acceptance Criteria - [ ] `python -m pytest tests/ -v` runs all tests with clear pass/fail per test - [ ] `terraform test` passes in `control-plane/` without a running libvirt daemon - [ ] `bats worker/tests/` passes all helper function tests - [ ] `cleanup.sh` line 22 reads `|| true` (not `|| truec`) #### Testing Plan ```bash # Static checks + terraform + BATS, all offline nix develop --command python -m pytest tests/ -v # Terraform mock validation alone nix develop --command python -m pytest tests/test_terraform.py -v # BATS alone nix develop --command bats worker/tests/ ``` #### Related Issues/PRs - [hashicorp/terraform#36469](https://github.com/hashicorp/terraform/pull/36469) — `override_provisioner` (open, blocks provisioner mocking) - [hashicorp/terraform#35459](https://github.com/hashicorp/terraform/issues/35459) — Original provisioner mocking issue
Author
Owner
  • I really am just interested in 6 but a very comprehensive version of it that stands up the whole cluster with an associated test suite to validate aspects about the cluster.
  • 1 and 2 are good to have as well

talosctl cluster create docker

are you sure there are enough permissions in the container for this to work? Especially since I won’t allow docker and it must use podman.

  • Other than that there needs to be more structure to how the tests are organized, they are scattered across the repo and I can’t rely on an agent to remember to call them all correctly. Maybe just utilizing python we can have a single test script that also is very clear about what it ran, its results, and the errors it encountered.
- I really am just interested in 6 but a very comprehensive version of it that stands up the whole cluster with an associated test suite to validate aspects about the cluster. - 1 and 2 are good to have as well > talosctl cluster create docker are you sure there are enough permissions in the container for this to work? Especially since I won’t allow docker and it must use podman. - Other than that there needs to be more structure to how the tests are organized, they are scattered across the repo and I can’t rely on an agent to remember to call them all correctly. Maybe just utilizing python we can have a single test script that also is very clear about what it ran, its results, and the errors it encountered.
Author
Owner

Naina:
Good questions — I dug into both.

On podman vs docker: talosctl cluster create docker works with podman via socket compatibility. You set DOCKER_HOST=unix:///run/podman/podman.sock (or the rootful socket path) and talosctl talks to podman's Docker-compatible API. This has been confirmed working since Podman v4.2.0 / Talos v1.2.3. On macOS, the podman machine needs to be rootful (podman machine set --rootful=true) because the containers run in --privileged mode and need access to /dev/kmsg and kernel parameter modifications.

On permissions: The containers run --privileged, so they need a real container runtime with full access. This means the integration test cannot run inside a Nix sandbox or a restricted CI container — it has to be invoked from the nix develop dev shell where podman has direct access. That's why in the updated plan it's an opt-in test (explicit marker/flag), not something nix flake check would attempt.

On the unified runner: Agreed on pytest. It gives us a single pytest tests/ -v entry point with structured output (test name, pass/fail, errors) and lets us organize the integration tests, static checks, and terraform tests in one place. The integration tests get a pytest marker so they're skipped by default unless you opt in.

I'll update the plan to focus on the comprehensive integration test as the primary deliverable, with pytest as the orchestrator.

Naina: Good questions — I dug into both. **On podman vs docker:** `talosctl cluster create docker` works with podman via socket compatibility. You set `DOCKER_HOST=unix:///run/podman/podman.sock` (or the rootful socket path) and talosctl talks to podman's Docker-compatible API. This has been confirmed working since Podman v4.2.0 / Talos v1.2.3. On macOS, the podman machine needs to be rootful (`podman machine set --rootful=true`) because the containers run in `--privileged` mode and need access to `/dev/kmsg` and kernel parameter modifications. **On permissions:** The containers run `--privileged`, so they need a real container runtime with full access. This means the integration test **cannot** run inside a Nix sandbox or a restricted CI container — it has to be invoked from the `nix develop` dev shell where podman has direct access. That's why in the updated plan it's an opt-in test (explicit marker/flag), not something `nix flake check` would attempt. **On the unified runner:** Agreed on pytest. It gives us a single `pytest tests/ -v` entry point with structured output (test name, pass/fail, errors) and lets us organize the integration tests, static checks, and terraform tests in one place. The integration tests get a pytest marker so they're skipped by default unless you opt in. I'll update the plan to focus on the comprehensive integration test as the primary deliverable, with pytest as the orchestrator.
Author
Owner

So what would have to change in the annia-infra project (you can look at the code under /workspace/annia-infra) to make it work with the talosctl cluster create docker command? You can find the script used to create the environment you're in at start-workspace.sh

So what would have to change in the annia-infra project (you can look at the code under /workspace/annia-infra) to make it work with the `talosctl cluster create docker` command? You can find the script used to create the environment you're in at `start-workspace.sh`
Author
Owner

Naina:
The main change is mounting the host's podman socket into the workspace container. Right now start-workspace.sh doesn't give the container access to any container runtime, so talosctl cluster create docker has nothing to talk to.

Two lines in start-workspace.sh would do it:

# Detect platform and mount podman socket
if [[ "$(uname)" == "Darwin" ]]; then
    SOCKET_PATH="/var/run/docker.sock"  # podman machine forwards here
else
    SOCKET_PATH="/run/podman/podman.sock"
fi
CREATE_ARGS+=( -v "${SOCKET_PATH}:/run/podman/podman.sock:Z" )
CREATE_ARGS+=( -e "DOCKER_HOST=unix:///run/podman/podman.sock" )

The socket-mount approach is better than podman-in-podman because the talos containers run --privileged with --security-opt seccomp=unconfined and need access to /dev/kmsg and /proc/sys/*. With socket mounting, those containers are created directly on the host where they get full privileges naturally — the workspace container itself doesn't need --privileged.

On macOS, the podman machine must be rootful (podman machine set --rootful=true) because rootless podman doesn't propagate privileged capabilities to nested containers, which causes talos nodes to fail with permission denied errors. The Dockerfile and flake.nix in annia-infra don't need changes — talosctl is already in the milner flake via nix develop.

Naina: The main change is mounting the host's podman socket into the workspace container. Right now `start-workspace.sh` doesn't give the container access to any container runtime, so `talosctl cluster create docker` has nothing to talk to. Two lines in `start-workspace.sh` would do it: ```bash # Detect platform and mount podman socket if [[ "$(uname)" == "Darwin" ]]; then SOCKET_PATH="/var/run/docker.sock" # podman machine forwards here else SOCKET_PATH="/run/podman/podman.sock" fi CREATE_ARGS+=( -v "${SOCKET_PATH}:/run/podman/podman.sock:Z" ) CREATE_ARGS+=( -e "DOCKER_HOST=unix:///run/podman/podman.sock" ) ``` The socket-mount approach is better than podman-in-podman because the talos containers run `--privileged` with `--security-opt seccomp=unconfined` and need access to `/dev/kmsg` and `/proc/sys/*`. With socket mounting, those containers are created directly on the host where they get full privileges naturally — the workspace container itself doesn't need `--privileged`. On macOS, the podman machine must be rootful (`podman machine set --rootful=true`) because rootless podman doesn't propagate privileged capabilities to nested containers, which causes talos nodes to fail with permission denied errors. The Dockerfile and flake.nix in annia-infra don't need changes — `talosctl` is already in the milner flake via `nix develop`.
Author
Owner

And what would the considerations be when annia-infra eventually becomes a Kubernetes pod instead of a podman container? Is there an alternative there or would podman in-podman be the only option in that case?

And what would the considerations be when annia-infra eventually becomes a Kubernetes pod instead of a podman container? Is there an alternative there or would podman in-podman be the only option in that case?
Author
Owner

Naina:
Three options, ordered by complexity:

Socket mounting from the node (simplest). Mount the node's container runtime socket into the annia-infra pod via a hostPath volume pointing to /run/podman/podman.sock (or /run/containerd/containerd.sock). The talosctl containers then run directly on the node, same as today. The catch: this requires a privileged Pod Security Standard namespace because the kubelet's hostPath type Socket has known issues — it may create a directory instead of waiting for a socket (kubernetes/kubernetes#128814). Security-wise, you're exposing the node's container runtime API to the pod, which is a container escape path. For a single-node dev cluster this is acceptable; for a multi-tenant production node it's a hard no.

Docker-in-Docker / Podman-in-Podman. Run a container runtime daemon inside the workspace pod itself. Requires privileged: true on the pod and a volume for the daemon's storage. Same security tradeoffs as socket mounting but self-contained — no host socket exposure. The talos containers run inside the nested daemon. This is how KinD-in-pod works (kubernetes-sigs/kind#303 has working pod specs).

Sysbox runtime (most secure). Use Nestybox Sysbox as the container runtime on the node. Sysbox allows running Docker/KinD/talosctl inside unprivileged pods — no privileged flag, no host socket mount, proper isolation. The tradeoff is that Sysbox has to be installed on every cluster node and isn't part of standard Kubernetes distributions. Long-term this is the right answer, but it's a node-level change that needs ops buy-in.

None of these are milner's problem to solve — they all live in the annia-infra orchestration layer. The milner tests just need DOCKER_HOST pointed at whatever socket the workspace provides.

Naina: Three options, ordered by complexity: **Socket mounting from the node (simplest).** Mount the node's container runtime socket into the annia-infra pod via a `hostPath` volume pointing to `/run/podman/podman.sock` (or `/run/containerd/containerd.sock`). The talosctl containers then run directly on the node, same as today. The catch: this requires a `privileged` Pod Security Standard namespace because the kubelet's `hostPath` type `Socket` has known issues — it may create a directory instead of waiting for a socket (kubernetes/kubernetes#128814). Security-wise, you're exposing the node's container runtime API to the pod, which is a container escape path. For a single-node dev cluster this is acceptable; for a multi-tenant production node it's a hard no. **Docker-in-Docker / Podman-in-Podman.** Run a container runtime daemon inside the workspace pod itself. Requires `privileged: true` on the pod and a volume for the daemon's storage. Same security tradeoffs as socket mounting but self-contained — no host socket exposure. The talos containers run inside the nested daemon. This is how KinD-in-pod works (kubernetes-sigs/kind#303 has working pod specs). **Sysbox runtime (most secure).** Use Nestybox Sysbox as the container runtime on the node. Sysbox allows running Docker/KinD/talosctl inside unprivileged pods — no `privileged` flag, no host socket mount, proper isolation. The tradeoff is that Sysbox has to be installed on every cluster node and isn't part of standard Kubernetes distributions. Long-term this is the right answer, but it's a node-level change that needs ops buy-in. None of these are milner's problem to solve — they all live in the annia-infra orchestration layer. The milner tests just need `DOCKER_HOST` pointed at whatever socket the workspace provides.
Author
Owner

I’ve soured on the full orchestration integration test suite. Let’s rework the planning comment to just incorporate the static testing that we’ve been talking about with the pytest suites.

I’ve soured on the full orchestration integration test suite. Let’s rework the planning comment to just incorporate the static testing that we’ve been talking about with the pytest suites.
Sign in to join this conversation.
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner#11
No description provided.