tepichord/milner

Fork 0

Create Kubernetes Dashboard #39

New issue

Closed

opened 2026-09-09 18:08:16 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-09 18:08:16 -07:00

Owner

Research and integrate a web-accessible tool for kubernetes that will allow the admin to view and manage the health of the kubernetes cluster, including insight on resource usage and nodes.

- Research and integrate a web-accessible tool for kubernetes that will allow the admin to view and manage the health of the kubernetes cluster, including insight on resource usage and nodes.

tepichord added the

labels

2026-09-09 18:08:16 -07:00

tepichord added this to the MVP project

2026-09-09 18:08:16 -07:00

tepichord commented

2026-09-09 18:26:18 -07:00

Author

Owner

Naina:

Overview

Deploy Headlamp — the official Kubernetes SIG UI successor to the archived Dashboard — as a Terraform-managed Helm component in a dedicated namespace, exposed at dashboard.faun-snapper.ts.net via the existing Tailscale operator ProxyGroup. The deployment follows the established vault/ pattern: new Terraform root, helm_release, kubernetes_ingress_v1¹.

Open scope question: The issue requests "insight on resource usage." Headlamp's CPU/memory charts require metrics-server, which is not in the cluster today. Two options: (a) include metrics-server as a companion Terraform root, or (b) ship Headlamp's node/pod inventory views first and add metrics-server in a follow-up. Recommend (b) — keeping this issue focused.

Technical Approach

Create kubernetes-dashboard/main.tf — new Terraform root mirroring vault/main.tf structure: remote state from CP, provider config, namespace, Helm release, Ingress.

Create kubernetes-dashboard/values.yaml — Headlamp Helm chart config:

replicaCount: 1

image:
  tag: "v0.45.0"

nodeSelector:
  node-role: worker

resources:
  requests:
    cpu: 50m
    memory: 128Mi
  limits:
    cpu: 100m
    memory: 256Mi

clusterRoleBinding:
  create: false

config:
  # MVP auth: user pastes a ServiceAccount token.
  # unsafeUseServiceAccountToken: false (default) — users must supply
  # their own token; the pod SA is not an implicit cluster-admin path.
  unsafeUseServiceAccountToken: false

Create kubernetes-dashboard/cluster-role.tf — read-only ClusterRole + binding for the Headlamp pod SA (replacing the chart's cluster-admin default²):

resource "kubernetes_cluster_role_v1" "headlamp" {
  metadata {
    name = "headlamp-view"
  }

  rule {
    api_groups = ["*"]
    resources  = ["*"]
    verbs      = ["get", "list", "watch"]
  }
}

resource "kubernetes_cluster_role_binding_v1" "headlamp" {
  metadata {
    name = "headlamp-view"
  }

  role_ref {
    api_group = "rbac.authorization.k8s.io"
    kind      = "ClusterRole"
    name      = kubernetes_cluster_role_v1.headlamp.metadata[0].name
  }

  subject {
    kind      = "ServiceAccount"
    name      = "headlamp"
    namespace = kubernetes_namespace_v1.dashboard.metadata[0].name
  }
}

Create kubernetes-dashboard/namespace.tf — dedicated namespace with baseline PSA (Headlamp runs runAsNonRoot: true³, no privileged containers):

resource "kubernetes_namespace_v1" "dashboard" {
  metadata {
    name = "dashboard"
    labels = {
      "pod-security.kubernetes.io/enforce" = "baseline"
    }
  }
}

Add Tailscale Ingress to main.tf — reuse existing ingress-proxies ProxyGroup (confirmed: one ProxyGroup handles multiple Ingress resources⁴):

resource "kubernetes_ingress_v1" "dashboard" {
  metadata {
    name      = "headlamp"
    namespace = kubernetes_namespace_v1.dashboard.metadata[0].name
    annotations = {
      "tailscale.com/proxy-group" = "ingress-proxies"
    }
  }

  spec {
    ingress_class_name = "tailscale"

    tls {
      hosts = ["dashboard"]
    }

    default_backend {
      service {
        name = "headlamp"
        port {
          number = 80
        }
      }
    }

    depends_on = [
      helm_release.headlamp,
    ]
  }
}

Reference ProxyGroup from vault/main.tf — the existing kubernetes_manifest.vault_proxy_group resource is already Terraform-owned⁵. The new root's Ingress only needs depends_on on the Helm release; the ProxyGroup exists cluster-wide. No second ProxyGroup resource needed.

Create kubernetes-dashboard/remote-state.tf — standard remote state block (copy from vault/):

terraform {
  required_version = ">= 1.7"
  required_providers {
    kubernetes = {
      source  = "hashicorp/kubernetes"
      version = "~> 2.35"
    }
    helm = {
      source  = "hashicorp/helm"
      version = "~> 2.17"
    }
  }
}

variable "control_plane_state_path" {
  type    = string
  default = "../control-plane/terraform.tfstate"
}

data "terraform_remote_state" "cp" {
  backend = "local"
  config = {
    path = var.control_plane_state_path
  }
}

locals {
  cp = data.terraform_remote_state.cp.outputs
  kc = yamldecode(local.cp.kubeconfig)
}

provider "kubernetes" {
  host                   = local.kc.clusters[0].cluster.server
  cluster_ca_certificate = base64decode(local.kc.clusters[0].cluster["certificate-authority-data"])
  client_certificate     = base64decode(local.kc.users[0].user["client-certificate-data"])
  client_key             = base64decode(local.kc.users[0].user["client-key-data"])
}

provider "helm" {
  kubernetes {
    host                   = local.kc.clusters[0].cluster.server
    cluster_ca_certificate = base64decode(local.kc.clusters[0].cluster["certificate-authority-data"])
    client_certificate     = base64decode(local.kc.users[0].user["client-certificate-data"])
    client_key             = base64decode(local.kc.users[0].user["client-key-data"])
  }
}

Create kubernetes-dashboard/.gitignore — exclude state files:
```
*.tfstate
.terraform/
```

Update ARCHITECTURE.md — add Dashboard to the service table and network topology diagram:

| Dashboard UI | Tailscale operator ingress | 80 | HTTPS | dashboard.faun-snapper.ts.net | deployed |

Update OPERATIONS.md — add §7 documenting Headlamp bring-up: approve dashboard service in Tailscale admin console, create a ServiceAccount + token for login, note that Talos has no node-shell binary (expected, not a bug).

Implementation Details

Files to create

kubernetes-dashboard/main.tf — namespace, Helm release, Ingress, remote state, providers
kubernetes-dashboard/values.yaml — Headlamp chart values (image tag, resources, nodeSelector, RBAC override)
kubernetes-dashboard/cluster-role.tf — read-only ClusterRole + binding
kubernetes-dashboard/.gitignore — state exclusions

Files to modify

ARCHITECTURE.md — add Dashboard row to service table and network diagram
OPERATIONS.md — add §7 (Dashboard bring-up/access)

Dependencies

Headlamp Helm chart headlamp/headlamp 0.45.0 from https://kubernetes-sigs.github.io/headlamp/⁶
Existing ingress-proxies ProxyGroup (created in vault/main.tf)
Tailscale operator (§3 in OPERATIONS.md)

Constraints

arm64-only cluster: Headlamp image ghcr.io/headlamp-k8s/headlamp:v0.45.0 is multi-arch (linux/arm64 verified)⁷
Headlamp runs as non-root (runAsUser: 100), no hostPath — baseline PSA sufficient⁸
No metrics-server in cluster today: resource usage charts will show "Install metrics server" placeholder
Headlamp's node shell feature does not work on Talos (no shell binary in node OS) — expected⁹
L7 Ingress only supports Prefix path type (Tailscale limitation — fine for this use case)¹⁰

Out-of-scope notes

metrics-server deployment — needed for live CPU/memory charts; recommended as a follow-up issue
OIDC/SSO via tsidp — Tailscale's OIDC identity provider is experimental¹¹, pre-1.0, no Helm chart; needs apiserver config changes. Defer to a follow-up
Prometheus plugin — Headlamp supports Prometheus for richer metrics; requires separate Prometheus stack

Acceptance Criteria

terraform -chdir=kubernetes-dashboard plan succeeds with no errors
Headlamp pod runs in dashboard namespace on worker-large node (verified via kubectl get pods -n dashboard -o wide)
Headlamp UI is accessible at https://dashboard.faun-snapper.ts.net from a tailnet device
The Headlamp pod SA has read-only ClusterRole (not cluster-admin) — verified via kubectl describe clusterrolebinding headlamp-view
terraform destroy -chdir=kubernetes-dashboard removes all resources cleanly

Testing Plan

terraform fmt -check -recursive kubernetes-dashboard/ — formatting lint
shellcheck kubernetes-dashboard/*.sh (if any scripts added)
yamllint kubernetes-dashboard/values.yaml — Helm values lint
terraform validate -chdir=kubernetes-dashboard — syntax/provider validation
Manual smoke test: kubectl -n dashboard get pods, confirm Running; curl https://dashboard.faun-snapper.ts.net returns 200
Verify Tailscale service appears as "Advertising" in admin console after apply

Issue #39 (this issue) — parent
No related PRs yet

Research Findings

The upstream kubernetes/dashboard project is archived and unmaintained (Jan 2026); the Kubernetes project formally recommends Headlamp as its successor under SIG UI.
Headlamp is the only option with native arm64 multi-arch images (ghcr.io/headlamp-k8s/headlamp), single-container footprint (~60 MiB RSS), and in-cluster Helm deployment matching the repo's patterns.
Headlamp's Helm chart defaults to cluster-admin for its pod SA (PR #4437 to change this was closed unmerged); the plan explicitly overrides this with a read-only ClusterRole.
One existing ProxyGroup (ingress-proxies) can serve multiple Ingress resources, so no second ProxyGroup is needed for the dashboard.
Headlamp requires metrics-server for CPU/memory resource usage charts; the cluster currently has no metrics-server. Sidero documents two Talos-compatible installation paths (insecure kubelet TLS or cert-approver).
Tailscale's tsidp OIDC provider is experimental and "rather tricky" to deploy in Kubernetes; recommend deferring SSO to a follow-up.
Headlamp has a known upstream memory leak issue (#5285) with progressive backend cache growth when many short-lived tokens log in; low risk for a single-admin homelab but worth noting in OPERATIONS.md.

Naina: ## Overview Deploy [Headlamp](https://github.com/kubernetes-sigs/headlamp) — the official Kubernetes SIG UI successor to the archived Dashboard — as a Terraform-managed Helm component in a dedicated namespace, exposed at `dashboard.faun-snapper.ts.net` via the existing Tailscale operator ProxyGroup. The deployment follows the established `vault/` pattern: new Terraform root, `helm_release`, `kubernetes_ingress_v1`[1](https://git.gloom.garden/tepichord/annia-infra/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/main.tf?display=source#L87). **Open scope question:** The issue requests "insight on resource usage." Headlamp's CPU/memory charts require [metrics-server](https://docs.siderolabs.com/kubernetes-guides/monitoring-and-observability/deploy-metrics-server), which is not in the cluster today. Two options: (a) include metrics-server as a companion Terraform root, or (b) ship Headlamp's node/pod inventory views first and add metrics-server in a follow-up. Recommend (b) — keeping this issue focused. ## Technical Approach 1. **Create `kubernetes-dashboard/main.tf`** — new Terraform root mirroring `vault/main.tf` structure: remote state from CP, provider config, namespace, Helm release, Ingress. 2. **Create `kubernetes-dashboard/values.yaml`** — Headlamp Helm chart config: ```yaml replicaCount: 1 image: tag: "v0.45.0" nodeSelector: node-role: worker resources: requests: cpu: 50m memory: 128Mi limits: cpu: 100m memory: 256Mi clusterRoleBinding: create: false config: # MVP auth: user pastes a ServiceAccount token. # unsafeUseServiceAccountToken: false (default) — users must supply # their own token; the pod SA is not an implicit cluster-admin path. unsafeUseServiceAccountToken: false ``` 3. **Create `kubernetes-dashboard/cluster-role.tf`** — read-only ClusterRole + binding for the Headlamp pod SA (replacing the chart's `cluster-admin` default[2](https://github.com/kubernetes-sigs/headlamp/issues/4435)): ```hcl resource "kubernetes_cluster_role_v1" "headlamp" { metadata { name = "headlamp-view" } rule { api_groups = ["*"] resources = ["*"] verbs = ["get", "list", "watch"] } } resource "kubernetes_cluster_role_binding_v1" "headlamp" { metadata { name = "headlamp-view" } role_ref { api_group = "rbac.authorization.k8s.io" kind = "ClusterRole" name = kubernetes_cluster_role_v1.headlamp.metadata[0].name } subject { kind = "ServiceAccount" name = "headlamp" namespace = kubernetes_namespace_v1.dashboard.metadata[0].name } } ``` 4. **Create `kubernetes-dashboard/namespace.tf`** — dedicated namespace with `baseline` PSA (Headlamp runs `runAsNonRoot: true`[3](https://github.com/kubernetes-sigs/headlamp/blob/b3b39c52/charts/headlamp/values.yaml), no privileged containers): ```hcl resource "kubernetes_namespace_v1" "dashboard" { metadata { name = "dashboard" labels = { "pod-security.kubernetes.io/enforce" = "baseline" } } } ``` 5. **Add Tailscale Ingress to `main.tf`** — reuse existing `ingress-proxies` ProxyGroup (confirmed: one ProxyGroup handles multiple Ingress resources[4](https://tailscale.com/docs/kubernetes-operator/concepts/proxygroup)): ```hcl resource "kubernetes_ingress_v1" "dashboard" { metadata { name = "headlamp" namespace = kubernetes_namespace_v1.dashboard.metadata[0].name annotations = { "tailscale.com/proxy-group" = "ingress-proxies" } } spec { ingress_class_name = "tailscale" tls { hosts = ["dashboard"] } default_backend { service { name = "headlamp" port { number = 80 } } } depends_on = [ helm_release.headlamp, ] } } ``` 6. **Reference ProxyGroup from `vault/main.tf`** — the existing `kubernetes_manifest.vault_proxy_group` resource is already Terraform-owned[5](https://git.gloom.garden/tepichord/annia-infra/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/vault/main.tf?display=source#L107). The new root's Ingress only needs `depends_on` on the Helm release; the ProxyGroup exists cluster-wide. No second ProxyGroup resource needed. 7. **Create `kubernetes-dashboard/remote-state.tf`** — standard remote state block (copy from `vault/`): ```hcl terraform { required_version = ">= 1.7" required_providers { kubernetes = { source = "hashicorp/kubernetes" version = "~> 2.35" } helm = { source = "hashicorp/helm" version = "~> 2.17" } } } variable "control_plane_state_path" { type = string default = "../control-plane/terraform.tfstate" } data "terraform_remote_state" "cp" { backend = "local" config = { path = var.control_plane_state_path } } locals { cp = data.terraform_remote_state.cp.outputs kc = yamldecode(local.cp.kubeconfig) } provider "kubernetes" { host = local.kc.clusters[0].cluster.server cluster_ca_certificate = base64decode(local.kc.clusters[0].cluster["certificate-authority-data"]) client_certificate = base64decode(local.kc.users[0].user["client-certificate-data"]) client_key = base64decode(local.kc.users[0].user["client-key-data"]) } provider "helm" { kubernetes { host = local.kc.clusters[0].cluster.server cluster_ca_certificate = base64decode(local.kc.clusters[0].cluster["certificate-authority-data"]) client_certificate = base64decode(local.kc.users[0].user["client-certificate-data"]) client_key = base64decode(local.kc.users[0].user["client-key-data"]) } } ``` 8. **Create `kubernetes-dashboard/.gitignore`** — exclude state files: ``` *.tfstate .terraform/ ``` 9. **Update `ARCHITECTURE.md`** — add Dashboard to the service table and network topology diagram: ``` | Dashboard UI | Tailscale operator ingress | 80 | HTTPS | dashboard.faun-snapper.ts.net | deployed | ``` 10. **Update `OPERATIONS.md`** — add §7 documenting Headlamp bring-up: approve `dashboard` service in Tailscale admin console, create a ServiceAccount + token for login, note that Talos has no node-shell binary (expected, not a bug). ## Implementation Details ### Files to create - `kubernetes-dashboard/main.tf` — namespace, Helm release, Ingress, remote state, providers - `kubernetes-dashboard/values.yaml` — Headlamp chart values (image tag, resources, nodeSelector, RBAC override) - `kubernetes-dashboard/cluster-role.tf` — read-only ClusterRole + binding - `kubernetes-dashboard/.gitignore` — state exclusions ### Files to modify - `ARCHITECTURE.md` — add Dashboard row to service table and network diagram - `OPERATIONS.md` — add §7 (Dashboard bring-up/access) ### Dependencies - Headlamp Helm chart `headlamp/headlamp` **0.45.0** from `https://kubernetes-sigs.github.io/headlamp/`[6](https://artifacthub.io/packages/helm/headlamp/headlamp) - Existing `ingress-proxies` ProxyGroup (created in `vault/main.tf`) - Tailscale operator (§3 in OPERATIONS.md) ### Constraints - arm64-only cluster: Headlamp image `ghcr.io/headlamp-k8s/headlamp:v0.45.0` is multi-arch (linux/arm64 verified)[7](https://github.com/kubernetes-sigs/headlamp/releases/tag/v0.45.0) - Headlamp runs as non-root (`runAsUser: 100`), no hostPath — `baseline` PSA sufficient[8](https://github.com/kubernetes-sigs/headlamp/blob/b3b39c52/charts/headlamp/values.yaml) - No metrics-server in cluster today: resource usage charts will show "Install metrics server" placeholder - Headlamp's `node shell` feature does not work on Talos (no shell binary in node OS) — expected[9](https://github.com/kubernetes-sigs/headlamp/issues/3648) - L7 Ingress only supports `Prefix` path type (Tailscale limitation — fine for this use case)[10](https://tailscale.com/docs/kubernetes-operator/reference/limitations) ### Out-of-scope notes - **metrics-server deployment** — needed for live CPU/memory charts; recommended as a follow-up issue - **OIDC/SSO via tsidp** — Tailscale's OIDC identity provider is experimental[11](https://github.com/tailscale/tsidp), pre-1.0, no Helm chart; needs apiserver config changes. Defer to a follow-up - **Prometheus plugin** — Headlamp supports Prometheus for richer metrics; requires separate Prometheus stack ## Acceptance Criteria - [ ] `terraform -chdir=kubernetes-dashboard plan` succeeds with no errors - [ ] Headlamp pod runs in `dashboard` namespace on `worker-large` node (verified via `kubectl get pods -n dashboard -o wide`) - [ ] Headlamp UI is accessible at `https://dashboard.faun-snapper.ts.net` from a tailnet device - [ ] The Headlamp pod SA has read-only ClusterRole (not cluster-admin) — verified via `kubectl describe clusterrolebinding headlamp-view` - [ ] `terraform destroy -chdir=kubernetes-dashboard` removes all resources cleanly ## Testing Plan - `terraform fmt -check -recursive kubernetes-dashboard/` — formatting lint - `shellcheck kubernetes-dashboard/*.sh` (if any scripts added) - `yamllint kubernetes-dashboard/values.yaml` — Helm values lint - `terraform validate -chdir=kubernetes-dashboard` — syntax/provider validation - Manual smoke test: `kubectl -n dashboard get pods`, confirm Running; curl `https://dashboard.faun-snapper.ts.net` returns 200 - Verify Tailscale service appears as "Advertising" in admin console after apply ## Related Issues/PRs - Issue #39 (this issue) — parent - No related PRs yet ## Research Findings - The upstream `kubernetes/dashboard` project is **archived and unmaintained** (Jan 2026); the Kubernetes project formally recommends Headlamp as its successor under SIG UI. - Headlamp is the only option with native arm64 multi-arch images (`ghcr.io/headlamp-k8s/headlamp`), single-container footprint (~60 MiB RSS), and in-cluster Helm deployment matching the repo's patterns. - Headlamp's Helm chart defaults to `cluster-admin` for its pod SA (PR #4437 to change this was closed unmerged); the plan explicitly overrides this with a read-only ClusterRole. - One existing ProxyGroup (`ingress-proxies`) can serve multiple Ingress resources, so no second ProxyGroup is needed for the dashboard. - Headlamp requires metrics-server for CPU/memory resource usage charts; the cluster currently has no metrics-server. Sidero documents two Talos-compatible installation paths (insecure kubelet TLS or cert-approver). - Tailscale's `tsidp` OIDC provider is experimental and "rather tricky" to deploy in Kubernetes; recommend deferring SSO to a follow-up. - Headlamp has a known upstream memory leak issue (#5285) with progressive backend cache growth when many short-lived tokens log in; low risk for a single-admin homelab but worth noting in OPERATIONS.md.

tepichord added

and removed

labels

2026-09-09 18:32:00 -07:00

tepichord referenced this issue from a commit

2026-09-10 12:00:44 -07:00

feat(dashboard): add Headlamp Helm deployment with Tailscale ingress

tepichord referenced this issue from a commit