Integrate VictoriaLogs in milner #60

New issue

Closed

opened 2026-09-15 10:27:42 -07:00 by tepichord · 1 comment

tepichord commented 2026-09-15 10:27:42 -07:00

Owner

Copy link

1. Namespace

• Add gitops/clusters/milner/infrastructure/namespaces/logging.yaml with pod-security.kubernetes.io/enforce: privileged — the collector needs hostPath: /var/log/pods +
  hostNetwork, both rejected by baseline.

2. Store (VictoriaLogs server)

• New gitops/clusters/milner/infrastructure/victoria-logs/: ocirepository.yaml (oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-single, pinned tag, layerSelector),
  helmrelease.yaml (chartRef + valuesFrom ConfigMap victoria-logs-values), values.yaml → configMapGenerator, kustomization.yaml with namespace: logging (copy
  zot/kustomization.yaml shape).
• values.yaml: server.retentionPeriod: 7d and retentionDiskSpaceUsage as a hard cap, persistentVolume.enabled: true / size: 20Gi / storageClassName: persistent, explicit
  CPU+memory resources (required by policy/k8s.rego), nodeSelector: node-role: worker, ClusterIP service. Pin the image tag — no :latest.

3. Collector

• Pod logs: the official victoria-logs-collector chart as a second OCIRepository/HelmRelease in the same dir, remoteWrite[0].url=http://-server:9428.
• Talos host logs: the chart's vector config needs a host-network UDP/TCP json_lines receiver added (Talos destination format), or a small standalone Vector DaemonSet per the
  Talos docs example. Then point machine.logging.destinations at 127.0.0.1: on each node.

4. Talos machine config (Terraform — needs explicit confirmation)

• Add the logging destination to worker/main.tf and control-plane/main.tf; kernel logs via KmsgLogConfig (note extraKernelArgs only applies on a Talos upgrade).

5. Wiring + gates

• Add the namespace, victoria-logs/, victoria-logs-collector/ to infrastructure/kustomization.yaml (infrastructure layer only — no CRDs cross-layer here).
• Add the new values.yaml to the yamllint list in tests/test_static.py; tests/test_gitops.py LAYERS already covers schema/conftest for infrastructure.

6. Agent access

• In-cluster: plain HTTP — /select/logsql/query, /select/logsql/return, /select/logsql/tail on :9428. That alone is enough for an agent running with kubectl.
• Off-cluster: add a Tailscale operator Ingress (logs.faun-snapper.ts.net) like zot//vault/; remember one service approval in the admin console.
• Caveat: VictoriaLogs OSS has no built-in auth. Don't publish it beyond the tailnet; rely on tailnet ACLs, or front it with a proxy/authenticated UI if you want a token. If
  you do mint a token, it goes Vault-first via a VaultStaticSecret.
• Optional: VictoriaLogs MCP server Deployment for tool-based querying; Grafana only if you want the dashboards.

7. Storage budget

• Increase size of 100GiB shared data disk to 1TB
• The 20 GiB PVC lands on the shared 1 TB data disk alongside Woodpecker (10 GiB), Zot (15 GiB), Pocket ID — set retention/disk cap so logs can't crowd them.

8. Mirror/migrate existing logs — per-app capture strategy

Goal of this step: walk every log source milner runs today, decide how its logs reach VictoriaLogs, and record the answer. Most apps need nothing per-app — the DaemonSet
collector already mirrors their stdout once it's running; the real decisions are exceptions, retention, and what stays out.

┌───────────────────────────────────────────────────┬─────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐
│ App (namespace) │ Log surface │ Strategy in VictoriaLogs │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ Talos host: machined, apid, trustd, etcd, │ Not stdout; │ machine.logging.destinations (json_lines TCP/UDP) + KmsgLogConfig → collector host-network │
│ kubelet, containerd, syslogd + kernel (all 3 VMs) │ talosctl logs/dmesg │ receiver. Terraform change, user confirmation required. │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ kube-system: Control-plane static pods │ Pod stdout │ Automatic — collector mirrors. Cross-check against the Talos service stream; they duplicate │
│ (apiserver, controller-manager, scheduler, etcd), │ │ deliberately. │
│ CoreDNS, metrics-server, CNI/kube-proxy │ │ │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ flux-system: source/kustomize/helm/notification │ Pod stdout │ Automatic. Flux errors are the highest-value stream for debugging drift. │
│ controllers │ │ │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ vault │ Pod stdout + audit │ Stdout automatic. Audit log is not stdout — needs a Vault audit device decision (file on the PVC │
│ │ log (separate │ + collector tail, or syslog device). Contains credentials: put it in a separate stream with │
│ │ device) │ tighter access, or leave it out of the agent-queryable store. Also note login failures stay │
│ │ │ invisible until verbose_oidc_logging=true (OPERATIONS.md §5f) — a config change, not a pipeline │
│ │ │ change. │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ vault-secrets-operator │ Pod stdout │ Automatic. Sync/rotation failures surface here, not in the app. │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ pocket-id │ Pod stdout │ Automatic. │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ woodpecker (server + agent) │ Pod stdout │ Automatic, but lifecycle only — build output is not here (see below). │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ woodpecker-runtime (step pods) │ Step stdout, │ Best-effort mirror of {namespace="woodpecker-runtime"}. Not authoritative. │
│ │ deleted right after │ │
│ │ the step │ │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ zot │ Pod stdout (+ HTTP │ Automatic; enable access logging in Zot config if pull/push audit is wanted. │
│ │ access log if │ │
│ │ enabled in its │ │
│ │ config) │ │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ local-path-storage │ Pod stdout │ Automatic. │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ tailscale: operator + ingress-proxies ProxyGroup │ Pod stdout │ Automatic. Ingress/proxy errors explain certificate and approval problems. │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ cloudflare: cloudflared connectors │ Pod stdout │ Automatic. The only public-surface component — keep its stream. │
│ (cloudflare/cloudflared.tf) │ │ │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ dashboard: Headlamp │ Pod stdout │ Automatic. │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ preview-system: router, oz console, reaper │ Pod stdout; reaper │ Automatic; CronJob pods vanish quickly — same tail-read race as Woodpecker steps, low volume. │
│ CronJob │ pods are ephemeral │ │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ preview- (ephemeral) │ Pod stdout │ Automatic via namespace label; useful while reviewing a PR. │
├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤
│ evolved-npcs (planned): orchestrator + │ Pod stdout │ Automatic, but the high-volume outlier. Decide retention/sampling and keep it off the worker-sim │
│ KEDA-scaled simulation pods │ │ disk. │
└───────────────────────────────────────────────────┴─────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘

Woodpecker findings (the one source that doesn't migrate by collection)

• WOODPECKER_LOG_STORE defaults to database — here sqlite on the persistent PVC. The agent streams step output over gRPC to the server, which persists it and serves it at
  /api/repos//logs//; the server/agent pod logs contain no step output.
• Step stdout does exist as pod logs for the seconds the step pod lives in woodpecker-runtime, so the collector catches it best-effort — but it's one pod per step (not per
  pipeline), ordering is lost, and a step that never scheduled leaves nothing.
• A pipeline in error state never started: its reason is the pipeline object's errors[], not any log. A collector cannot see this at all.
• Therefore: keep the API as the authoritative reader and wrap it in a woodpecker-logs skill (the id-vs-pid trap and base64/sort_by(.line) decode are already written up in
  OPERATIONS.md §9j) — that resolves pipeline → step id → log → text, and reads errors[] too.
• Mirror the woodpecker-runtime stream into VictoriaLogs purely for cross-source correlation; label it as best-effort so nobody trusts it as the record.
• Only if build logs must be first-class in the store: WOODPECKER_LOG_STORE=addon with a go-plugin binary writing labeled lines to VL, or =file as a cheaper intermediate that
  at least makes them plain files on the existing PVC.

Verification + record

• For each row: run {namespace=""} (and {job=...} for the Talos stream) in VL, confirm fresh lines arrive, and note the exact stream/label selector.
• Record the per-app mapping and the non-stdout exceptions in OPERATIONS.md as the reason a future reader knows why Woodpecker build logs live behind an API and not in the
  store.

1. Namespace - Add gitops/clusters/milner/infrastructure/namespaces/logging.yaml with pod-security.kubernetes.io/enforce: privileged — the collector needs hostPath: /var/log/pods + hostNetwork, both rejected by baseline. 2. Store (VictoriaLogs server) - New gitops/clusters/milner/infrastructure/victoria-logs/: ocirepository.yaml (oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-single, pinned tag, layerSelector), helmrelease.yaml (chartRef + valuesFrom ConfigMap victoria-logs-values), values.yaml → configMapGenerator, kustomization.yaml with namespace: logging (copy zot/kustomization.yaml shape). - values.yaml: server.retentionPeriod: 7d and retentionDiskSpaceUsage as a hard cap, persistentVolume.enabled: true / size: 20Gi / storageClassName: persistent, explicit CPU+memory resources (required by policy/k8s.rego), nodeSelector: node-role: worker, ClusterIP service. Pin the image tag — no :latest. 3. Collector - Pod logs: the official victoria-logs-collector chart as a second OCIRepository/HelmRelease in the same dir, remoteWrite[0].url=http://<release>-server:9428. - Talos host logs: the chart's vector config needs a host-network UDP/TCP json_lines receiver added (Talos destination format), or a small standalone Vector DaemonSet per the Talos docs example. Then point machine.logging.destinations at 127.0.0.1:<port> on each node. 4. Talos machine config (Terraform — needs explicit confirmation) - Add the logging destination to worker/main.tf and control-plane/main.tf; kernel logs via KmsgLogConfig (note extraKernelArgs only applies on a Talos upgrade). 5. Wiring + gates - Add the namespace, victoria-logs/, victoria-logs-collector/ to infrastructure/kustomization.yaml (infrastructure layer only — no CRDs cross-layer here). - Add the new values.yaml to the yamllint list in tests/test_static.py; tests/test_gitops.py LAYERS already covers schema/conftest for infrastructure. 6. Agent access - In-cluster: plain HTTP — /select/logsql/query, /select/logsql/return, /select/logsql/tail on :9428. That alone is enough for an agent running with kubectl. - Off-cluster: add a Tailscale operator Ingress (logs.faun-snapper.ts.net) like zot//vault/; remember one service approval in the admin console. - Caveat: VictoriaLogs OSS has no built-in auth. Don't publish it beyond the tailnet; rely on tailnet ACLs, or front it with a proxy/authenticated UI if you want a token. If you do mint a token, it goes Vault-first via a VaultStaticSecret. - Optional: VictoriaLogs MCP server Deployment for tool-based querying; Grafana only if you want the dashboards. 7. Storage budget - Increase size of 100GiB shared data disk to 1TB - The 20 GiB PVC lands on the shared 1 TB data disk alongside Woodpecker (10 GiB), Zot (15 GiB), Pocket ID — set retention/disk cap so logs can't crowd them. 8. Mirror/migrate existing logs — per-app capture strategy Goal of this step: walk every log source milner runs today, decide how its logs reach VictoriaLogs, and record the answer. Most apps need nothing per-app — the DaemonSet collector already mirrors their stdout once it's running; the real decisions are exceptions, retention, and what stays out. ┌───────────────────────────────────────────────────┬─────────────────────┬───────────────────────────────────────────────────────────────────────────────────────────────────┐ │ App (namespace) │ Log surface │ Strategy in VictoriaLogs │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ Talos host: machined, apid, trustd, etcd, │ Not stdout; │ machine.logging.destinations (json_lines TCP/UDP) + KmsgLogConfig → collector host-network │ │ kubelet, containerd, syslogd + kernel (all 3 VMs) │ talosctl logs/dmesg │ receiver. Terraform change, user confirmation required. │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ kube-system: Control-plane static pods │ Pod stdout │ Automatic — collector mirrors. Cross-check against the Talos service stream; they duplicate │ │ (apiserver, controller-manager, scheduler, etcd), │ │ deliberately. │ │ CoreDNS, metrics-server, CNI/kube-proxy │ │ │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ flux-system: source/kustomize/helm/notification │ Pod stdout │ Automatic. Flux errors are the highest-value stream for debugging drift. │ │ controllers │ │ │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ vault │ Pod stdout + audit │ Stdout automatic. Audit log is not stdout — needs a Vault audit device decision (file on the PVC │ │ │ log (separate │ + collector tail, or syslog device). Contains credentials: put it in a separate stream with │ │ │ device) │ tighter access, or leave it out of the agent-queryable store. Also note login failures stay │ │ │ │ invisible until verbose_oidc_logging=true (OPERATIONS.md §5f) — a config change, not a pipeline │ │ │ │ change. │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ vault-secrets-operator │ Pod stdout │ Automatic. Sync/rotation failures surface here, not in the app. │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ pocket-id │ Pod stdout │ Automatic. │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ woodpecker (server + agent) │ Pod stdout │ Automatic, but lifecycle only — build output is not here (see below). │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ woodpecker-runtime (step pods) │ Step stdout, │ Best-effort mirror of {namespace="woodpecker-runtime"}. Not authoritative. │ │ │ deleted right after │ │ │ │ the step │ │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ zot │ Pod stdout (+ HTTP │ Automatic; enable access logging in Zot config if pull/push audit is wanted. │ │ │ access log if │ │ │ │ enabled in its │ │ │ │ config) │ │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ local-path-storage │ Pod stdout │ Automatic. │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ tailscale: operator + ingress-proxies ProxyGroup │ Pod stdout │ Automatic. Ingress/proxy errors explain certificate and approval problems. │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ cloudflare: cloudflared connectors │ Pod stdout │ Automatic. The only public-surface component — keep its stream. │ │ (cloudflare/cloudflared.tf) │ │ │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ dashboard: Headlamp │ Pod stdout │ Automatic. │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ preview-system: router, oz console, reaper │ Pod stdout; reaper │ Automatic; CronJob pods vanish quickly — same tail-read race as Woodpecker steps, low volume. │ │ CronJob │ pods are ephemeral │ │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ preview-<n> (ephemeral) │ Pod stdout │ Automatic via namespace label; useful while reviewing a PR. │ ├───────────────────────────────────────────────────┼─────────────────────┼───────────────────────────────────────────────────────────────────────────────────────────────────┤ │ evolved-npcs (planned): orchestrator + │ Pod stdout │ Automatic, but the high-volume outlier. Decide retention/sampling and keep it off the worker-sim │ │ KEDA-scaled simulation pods │ │ disk. │ └───────────────────────────────────────────────────┴─────────────────────┴───────────────────────────────────────────────────────────────────────────────────────────────────┘ Woodpecker findings (the one source that doesn't migrate by collection) - WOODPECKER_LOG_STORE defaults to database — here sqlite on the persistent PVC. The agent streams step output over gRPC to the server, which persists it and serves it at /api/repos/<repo>/logs/<pipeline>/<step>; the server/agent pod logs contain no step output. - Step stdout does exist as pod logs for the seconds the step pod lives in woodpecker-runtime, so the collector catches it best-effort — but it's one pod per step (not per pipeline), ordering is lost, and a step that never scheduled leaves nothing. - A pipeline in error state never started: its reason is the pipeline object's errors[], not any log. A collector cannot see this at all. - Therefore: keep the API as the authoritative reader and wrap it in a woodpecker-logs skill (the id-vs-pid trap and base64/sort_by(.line) decode are already written up in OPERATIONS.md §9j) — that resolves pipeline → step id → log → text, and reads errors[] too. - Mirror the woodpecker-runtime stream into VictoriaLogs purely for cross-source correlation; label it as best-effort so nobody trusts it as the record. - Only if build logs must be first-class in the store: WOODPECKER_LOG_STORE=addon with a go-plugin binary writing labeled lines to VL, or =file as a cheaper intermediate that at least makes them plain files on the existing PVC. Verification + record - For each row: run {namespace="<ns>"} (and {job=...} for the Talos stream) in VL, confirm fresh lines arrive, and note the exact stream/label selector. - Record the per-app mapping and the non-stdout exceptions in OPERATIONS.md as the reason a future reader knows why Woodpecker build logs live behind an API and not in the store.

tepichord added the

status

planning

enhancement

effort

medium

labels 2026-09-15 10:27:42 -07:00

tepichord added this to the MVP project 2026-09-15 10:27:42 -07:00

tepichord commented 2026-09-15 11:25:21 -07:00

Author

Owner

Copy link

Naina:

Overview

Add a centralised log store to milner: VictoriaLogs (server), vlagent (pod-stdout DaemonSet), a logging namespace and a Tailscale ingress — all Flux-managed, copying the Zot layout. Two of the issue's steps are not part of this plan's definition of done: Talos host-log forwarding (step 4) and the 100 GiB → 1 TiB resize (step 7). Both are gated on your explicit go-ahead, and the resize turned out to be blocked twice (details below). Phase 1 needs no resize — 31 GiB of persistent claims today + 20 GiB for logs = 51 GiB of 100 GiB.

Technical Approach

1. Namespace — infrastructure/namespaces/logging.yaml, privileged PSA (vlagent mounts hostPath: /var/log/pods, which baseline forbids^88b):

   labels:
     pod-security.kubernetes.io/enforce: privileged
   

2. Server chart source — infrastructure/victoria-logs/ocirepository.yaml (tag verified against the registry¹):

   url: oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-single
   layerSelector: {mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"}
   ref: {tag: "0.13.9"}   # appVersion v1.52.0
   

3. Server values — the chart uses server.persistentVolume.*^22b, not persistence.*/pvc.* as zot does; retentionDiskSpaceUsage is a real chart key³ and service.clusterIP defaults to None⁴:

   server:
     fullnameOverride: victoria-logs      # Service + StatefulSet are then `victoria-logs`
     image: {repository: victoriametrics/victoria-logs, tag: v1.52.0}
     retentionPeriod: 7d
     retentionDiskSpaceUsage: 18GiB       # hard cap under the 20Gi claim
     persistentVolume: {enabled: true, existingClaim: victoria-logs-data}
     service: {type: ClusterIP, clusterIP: ""}   # chart default is `None` (headless)
     nodeSelector: {node-role: worker}
     resources:
       requests: {cpu: 100m, memory: 256Mi}
       limits: {cpu: "1", memory: 1Gi}
   ingress: {enabled: false}   # this dir owns the Ingress; see step 6
   vector: {enabled: false}    # comes back with the Talos phase
   

4. Flux-owned PVC — infrastructure/victoria-logs/persistentvolumeclaim.yaml, same shape as zot-data:

   metadata: {name: victoria-logs-data, namespace: logging}
   spec:
     accessModes: [ReadWriteOnce]
     storageClassName: persistent
     resources: {requests: {storage: 20Gi}}
   

5. Collector — infrastructure/victoria-logs-collector/, its own OCIRepository + HelmRelease + values.yaml (tag verified; the chart ships victoriametrics/vlagent, mounts hostPath volumes and declares no hostNetwork⁵; an un-pathed remoteWrite.url targets /insert/native⁶):

   # ocirepository.yaml
   url: oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-collector
   ref: {tag: "0.3.7"}
   

   # values.yaml — the chart is vlagent, not Vector: hostPath only, no hostNetwork
   fullnameOverride: victoria-logs-collector
   image: {repository: victoriametrics/vlagent, tag: v1.52.0}
   remoteWrite:
     - url: http://victoria-logs:9428    # no path -> /insert/native
   tolerations:                          # no-op if the CP node is untainted
     - {key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule}
   resources:
     requests: {cpu: 100m, memory: 128Mi}
     limits: {cpu: "1", memory: 512Mi}
   

6. Ingress — infrastructure/victoria-logs/ingress.yaml, copied from zot⁷; port 9428, tls.hosts: [logs] → logs.faun-snapper.ts.net:

   metadata: {annotations: {tailscale.com/proxy-group: ingress-proxies}}
   spec:
     ingressClassName: tailscale
     tls: [{hosts: [logs]}]
     defaultBackend: {service: {name: victoria-logs, port: {number: 9428}}}
   

7. Wiring:

   # infrastructure/kustomization.yaml
   - namespaces/logging.yaml
   - victoria-logs
   - victoria-logs-collector
   

   # tests/test_static.py::test_yamllint — append
   "gitops/clusters/milner/infrastructure/victoria-logs/values.yaml",
   "gitops/clusters/milner/infrastructure/victoria-logs-collector/values.yaml",
   

8. Docs — three edits:

   • OPERATIONS.md §10: the resize procedure and its blockers; the logs service approval; the query recipes.

     kubectl -n logging port-forward svc/victoria-logs 9428:9428 &
     curl -s http://127.0.0.1:9428/select/logsql/query --data-urlencode 'query=*' | head
     

   • ARCHITECTURE.md: storage budget row (+20 GiB) and the logs.faun-snapper.ts.net endpoint.
   • Per-app capture table: transcribe the issue's table, with Woodpecker build logs marked not collected (they stream over gRPC into SQLite, never to stdout — the collector catches only best-effort stdout from ephemeral step pods).

Implementation Details

Create

Path	Contents
infrastructure/namespaces/logging.yaml	Namespace, enforce: privileged
infrastructure/victoria-logs/{kustomization,ocirepository,helmrelease,values,persistentvolumeclaim,ingress}.yaml	server
infrastructure/victoria-logs-collector/{kustomization,ocirepository,helmrelease,values}.yaml	vlagent DaemonSet

Modify

Path	Change
infrastructure/kustomization.yaml	+3 resources
tests/test_static.py	+2 values.yaml paths in test_yamllint
OPERATIONS.md §10	logging runbook, resize procedure, query access
ARCHITECTURE.md	storage budget, network endpoint

Constraints

• policy/k8s.rego rules 1–2 still bite: pinned tags and CPU+memory limits on every container of both workloads. Rule 3 only inspects container.securityContext.privileged, so no policy exemption is needed for the hostPath pod — the privileged PSA label is what admits it⁹.
• Pin server.image.tag and the vlagent tag explicitly; both charts default them to appVersion, which the L1 gate would accept but which makes the running version invisible in review.
• persistent is WaitForFirstConsumer¹⁰, so the PVC stays Pending until the server pod schedules. Flux reporting the Kustomization not-ready in that window is the class working as designed.
• Do not set vector.enabled: true yet — a receiver with no sender is dead weight until step 4 is approved.

Out-of-scope notes (deferred, need your go-ahead)

• Talos host logs (step 4). machine.logging.destinations emits raw newline-delimited JSON over TCP/UDP; vlagent only speaks HTTP, so a Vector receiver is required. The victoria-logs-single OCI artifact bundles the Vector subchart, so no extra Flux source is needed:

  vector:
    enabled: true
    podHostNetwork: true
    dnsPolicy: ClusterFirstWithHostNet
    image: {repository: timberio/vector, tag: 0.57.0-distroless-libc}   # arm64 published
    customConfig:
      sources:
        talos: {type: socket, address: 0.0.0.0:6051, mode: tcp, decoding: {codec: json}, host_key: __host}
      sinks:
        vlogs: {type: elasticsearch, inputs: [talos]}   # endpoints auto-filled as /insert/elasticsearch
  

  # worker/main.tf and control-plane/main.tf config_patches
  yamlencode({
    machine = {
      logging = {
        destinations = [{
          endpoint  = "tcp://127.0.0.1:6051/"
          format    = "json_lines"
          extraTags = { cluster = "milner", node = "worker-1" }
        }]
      }
    }
  })
  

  Two caveats to settle before applying: destinations are appended under strategic-merge, so re-applying duplicates them (talosctl edit mc cleans up), and KmsgLogConfig is probably unnecessary because kernel messages already arrive in the service stream as talos-service=kernel.

• Disk resize (step 7). Leaving data/main.tf's data_volume_size at 100 GiB. Changing it makes terraform plan dirty and unapplyable:

  • dmacvicar/libvirt 0.9.9 VolumeResource.Update() errors with "Storage volumes cannot be updated. All changes require replacement.", and prevent_destroy = true forbids the replacement — so the plan shows an in-place update that apply refuses.
  • Talos v1.11.1 does not grow disk-type user volumes when the backing disk expands (siderolabs/talos#12991; fixed by #13057, present only in ≥ v1.13.0), so even a hand-run qemu-img resize may leave /var/mnt/data at 100 GiB.

  When it is wanted: qemu-img resize … +900G, then reconcile state (terraform apply -refresh-only or a state rm/import pair), then verify talosctl get volumestatus -o yaml and df -h /var/mnt/data.

• VictoriaLogs OSS has no auth. The ingress stays tailnet-only and nothing else publishes it; if a token is ever wanted it goes Vault-first via a VaultStaticSecret.

• Adjacent, not requested: victoria-logs-mcp (same registry) and the chart's dashboards.enabled — both only if you want them.

Acceptance Criteria

• kustomize build gitops/clusters/milner/infrastructure renders Namespace/logging with pod-security.kubernetes.io/enforce: privileged plus both new HelmReleases and the victoria-logs-data PVC.
• victoria-logs is Ready/Healthy; kubectl -n logging get pvc victoria-logs-data is Bound on the persistent class at 20Gi and was created by Flux, not the chart (server.persistentVolume.existingClaim set).
• kubectl -n logging get ds victoria-logs-collector reports DESIRED == READY, and /select/logsql/query returns entries with kubernetes.pod_namespace.
• pytest tests/test_static.py -k "yamllint or conftest" and pytest tests/test_gitops.py pass, including test_all_namespaces_have_psa_enforce and the L1 Rego gate over the rendered logging workloads.
• pytest tests/test_chart_sources.py resolves both new OCI sources (network-gated).
• OPERATIONS.md §10 documents the logs service approval, the query commands, the per-app capture table and the resize blockers; ARCHITECTURE.md reflects the new claim and endpoint.

Testing Plan

Case	Command
Render	kustomize build gitops/clusters/milner/infrastructure
Schema + namespace guard	pytest tests/test_gitops.py -k "kubeconform or namespaces or helmreleases"
L1 policy	pytest tests/test_static.py -k conftest
Static	pytest tests/test_static.py -k "yamllint or terraform_fmt"
Chart reachability	pytest tests/test_chart_sources.py (needs network)
Live ingest	kubectl -n logging port-forward svc/victoria-logs 9428:9428 then curl --data-urlencode 'query=kubernetes.pod_namespace:logging' http://127.0.0.1:9428/select/logsql/query
Live tail	curl -N --data-urlencode 'query=*' http://127.0.0.1:9428/select/logsql/tail
Flag parsing	kubectl -n logging logs sts/victoria-logs shows no arg-parse error for -retentionPeriod=7d / -retention.maxDiskSpaceUsageBytes=18GiB
Host logs (deferred)	after the Talos phase: talosctl logs machined and look for talos-service: machined in VictoriaLogs

Related Issues/PRs

• #50 — Zot OCI registry: the layout this copies (oci:// URL convention, Flux-owned PVC, Tailscale Ingress).
• #43 — Kubernetes Dashboard: infrastructure-layer + ingress wiring precedent.
• siderolabs/talos#12991 and siderolabs/talos#13057 — user-volume growth; tracks the resize blocker.
• Issue steps 4 and 7 stay unimplemented until you confirm in this thread.

Research Findings

• victoria-logs-single 0.13.9 and victoria-logs-collector 0.3.7 both exist as OCI artifacts under ghcr.io/victoriametrics/helm-charts, and both server/collector images publish linux/arm64.
• The collector chart is vlagent, not Vector: it ingests HTTP only and needs hostPath but not hostNetwork, so the issue's "hostPath + hostNetwork" applies to the Talos-receiver half alone.
• The victoria-logs-single OCI artifact bundles the Vector subchart (0.57.0), so a host-network json_lines receiver needs no additional Flux source; timberio/vector:0.57.0-distroless-libc publishes arm64.
• Talos v1.11 machine.logging.destinations accepts tcp:///udp:// endpoints with format: json_lines and applies without a reboot; kernel logs already ride the service stream, making KmsgLogConfig usually redundant.
• /select/logsql/return (issue step 6) is not a real endpoint; the read paths are /select/logsql/query and /select/logsql/tail on :9428.
• The disk resize is blocked twice independently: the libvirt provider refuses volume updates while prevent_destroy forbids replacement, and Talos v1.11.1 will not grow the user volume even after the qcow2 grows.

Pre-Mortem

1. The chart values keys are wrong and nothing renders the chart to catch it. The plan nests fullnameOverride under server: (only the collector chart puts it at top level) and depends on server.persistentVolume.existingClaim, service.clusterIP: "" and retentionDiskSpaceUsage: 18GiB without ever running helm template against the pinned 0.13.9 chart. Helm silently ignores unknown keys, so the release installs with defaults: the Service/StatefulSet keep a release-derived name, the Ingress backend and vlagent's http://victoria-logs:9428 resolve to nothing, and every static test still passes while /select/logsql/query returns no entries.
2. The two deferred steps are read as an incomplete delivery. Steps 4 (Talos host logs) and 7 (disk resize) are stated parts of issue #60's definition of done, so a reviewer diffing the PR against the issue can send everything back and discard the working Phase 1. The gate lives only as "need your go-ahead" in a comment thread, so it has no recorded owner and no expiry.
3. The gated follow-ups fail on their first real apply. Talos merges machine.logging.destinations by appending arrays, so a re-applied Terraform patch duplicates destinations, and tcp://127.0.0.1:6051/ under hostNetwork makes every node report as 127.0.0.1. When the resize is attempted, the libvirt provider's refusal to update volumes plus prevent_destroy = true leaves terraform apply failing and data/ state dirty, blocking unrelated changes to that root.

Naina: ## Overview Add a centralised log store to milner: VictoriaLogs (server), vlagent (pod-stdout DaemonSet), a `logging` namespace and a Tailscale ingress — all Flux-managed, copying the Zot layout. Two of the issue's steps are **not** part of this plan's definition of done: Talos host-log forwarding (step 4) and the 100 GiB → 1 TiB resize (step 7). Both are gated on your explicit go-ahead, and the resize turned out to be blocked twice (details below). Phase 1 needs no resize — 31 GiB of `persistent` claims today + 20 GiB for logs = 51 GiB of 100 GiB. ## Technical Approach 1. **Namespace** — `infrastructure/namespaces/logging.yaml`, privileged PSA (vlagent mounts `hostPath: /var/log/pods`, which baseline forbids^{[8](https://kubernetes.io/docs/concepts/security/pod-security-standards/)}^{[8b](https://git.gloom.garden/tepichord/milner/src/commit/e5e4a25dc58a64a5fcb77499111c7df685bad4a2/gitops/clusters/milner/infrastructure/namespaces/zot.yaml?display=source#L6)}): ```yaml labels: pod-security.kubernetes.io/enforce: privileged ``` 2. **Server chart source** — `infrastructure/victoria-logs/ocirepository.yaml` (tag verified against the registry^{[1](https://github.com/VictoriaMetrics/helm-charts/blob/victoria-logs-single-0.13.9/charts/victoria-logs-single/Chart.yaml)}): ```yaml url: oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-single layerSelector: {mediaType: "application/vnd.cncf.helm.chart.content.v1.tar+gzip"} ref: {tag: "0.13.9"} # appVersion v1.52.0 ``` 3. **Server values** — the chart uses `server.persistentVolume.*`^{[2](https://github.com/VictoriaMetrics/helm-charts/blob/victoria-logs-single-0.13.9/charts/victoria-logs-single/values.yaml)}^{[2b](https://git.gloom.garden/tepichord/milner/src/commit/e5e4a25dc58a64a5fcb77499111c7df685bad4a2/gitops/clusters/milner/infrastructure/zot/values.yaml?display=source#L131)}, **not** `persistence.*`/`pvc.*` as zot does; `retentionDiskSpaceUsage` is a real chart key^{[3](https://docs.victoriametrics.com/helm/victoria-logs-single/)} and `service.clusterIP` defaults to `None`^{[4](https://github.com/VictoriaMetrics/helm-charts/blob/victoria-logs-single-0.13.9/charts/victoria-logs-single/values.yaml)}: ```yaml server: fullnameOverride: victoria-logs # Service + StatefulSet are then `victoria-logs` image: {repository: victoriametrics/victoria-logs, tag: v1.52.0} retentionPeriod: 7d retentionDiskSpaceUsage: 18GiB # hard cap under the 20Gi claim persistentVolume: {enabled: true, existingClaim: victoria-logs-data} service: {type: ClusterIP, clusterIP: ""} # chart default is `None` (headless) nodeSelector: {node-role: worker} resources: requests: {cpu: 100m, memory: 256Mi} limits: {cpu: "1", memory: 1Gi} ingress: {enabled: false} # this dir owns the Ingress; see step 6 vector: {enabled: false} # comes back with the Talos phase ``` 4. **Flux-owned PVC** — `infrastructure/victoria-logs/persistentvolumeclaim.yaml`, same shape as `zot-data`: ```yaml metadata: {name: victoria-logs-data, namespace: logging} spec: accessModes: [ReadWriteOnce] storageClassName: persistent resources: {requests: {storage: 20Gi}} ``` 5. **Collector** — `infrastructure/victoria-logs-collector/`, its own `OCIRepository` + `HelmRelease` + `values.yaml` (tag verified; the chart ships `victoriametrics/vlagent`, mounts `hostPath` volumes and declares no `hostNetwork`^{[5](https://github.com/VictoriaMetrics/helm-charts/blob/victoria-logs-collector-0.3.7/charts/victoria-logs-collector/values.yaml)}; an un-pathed `remoteWrite.url` targets `/insert/native`^{[6](https://github.com/VictoriaMetrics/helm-charts/blob/victoria-logs-collector-0.3.7/charts/victoria-logs-collector/values.yaml)}): ```yaml # ocirepository.yaml url: oci://ghcr.io/victoriametrics/helm-charts/victoria-logs-collector ref: {tag: "0.3.7"} ``` ```yaml # values.yaml — the chart is vlagent, not Vector: hostPath only, no hostNetwork fullnameOverride: victoria-logs-collector image: {repository: victoriametrics/vlagent, tag: v1.52.0} remoteWrite: - url: http://victoria-logs:9428 # no path -> /insert/native tolerations: # no-op if the CP node is untainted - {key: node-role.kubernetes.io/control-plane, operator: Exists, effect: NoSchedule} resources: requests: {cpu: 100m, memory: 128Mi} limits: {cpu: "1", memory: 512Mi} ``` 6. **Ingress** — `infrastructure/victoria-logs/ingress.yaml`, copied from zot^{[7](https://git.gloom.garden/tepichord/milner/src/commit/e5e4a25dc58a64a5fcb77499111c7df685bad4a2/gitops/clusters/milner/infrastructure/zot/ingress.yaml?display=source#L7)}; port `9428`, `tls.hosts: [logs]` → `logs.faun-snapper.ts.net`: ```yaml metadata: {annotations: {tailscale.com/proxy-group: ingress-proxies}} spec: ingressClassName: tailscale tls: [{hosts: [logs]}] defaultBackend: {service: {name: victoria-logs, port: {number: 9428}}} ``` 7. **Wiring**: ```yaml # infrastructure/kustomization.yaml - namespaces/logging.yaml - victoria-logs - victoria-logs-collector ``` ```python # tests/test_static.py::test_yamllint — append "gitops/clusters/milner/infrastructure/victoria-logs/values.yaml", "gitops/clusters/milner/infrastructure/victoria-logs-collector/values.yaml", ``` 8. **Docs** — three edits: - `OPERATIONS.md` §10: the resize procedure *and* its blockers; the `logs` service approval; the query recipes. ```bash kubectl -n logging port-forward svc/victoria-logs 9428:9428 & curl -s http://127.0.0.1:9428/select/logsql/query --data-urlencode 'query=*' | head ``` - `ARCHITECTURE.md`: storage budget row (+20 GiB) and the `logs.faun-snapper.ts.net` endpoint. - Per-app capture table: transcribe the issue's table, with Woodpecker build logs marked **not collected** (they stream over gRPC into SQLite, never to stdout — the collector catches only best-effort stdout from ephemeral step pods). ## Implementation Details **Create** | Path | Contents | |---|---| | `infrastructure/namespaces/logging.yaml` | Namespace, `enforce: privileged` | | `infrastructure/victoria-logs/{kustomization,ocirepository,helmrelease,values,persistentvolumeclaim,ingress}.yaml` | server | | `infrastructure/victoria-logs-collector/{kustomization,ocirepository,helmrelease,values}.yaml` | vlagent DaemonSet | **Modify** | Path | Change | |---|---| | `infrastructure/kustomization.yaml` | +3 resources | | `tests/test_static.py` | +2 `values.yaml` paths in `test_yamllint` | | `OPERATIONS.md` §10 | logging runbook, resize procedure, query access | | `ARCHITECTURE.md` | storage budget, network endpoint | **Constraints** - `policy/k8s.rego` rules 1–2 still bite: pinned tags and CPU+memory limits on every container of both workloads. Rule 3 only inspects `container.securityContext.privileged`, so **no policy exemption is needed** for the hostPath pod — the `privileged` PSA label is what admits it^{[9](https://git.gloom.garden/tepichord/milner/src/commit/e5e4a25dc58a64a5fcb77499111c7df685bad4a2/policy/k8s.rego?display=source#L99)}. - Pin `server.image.tag` and the vlagent tag explicitly; both charts default them to `appVersion`, which the L1 gate would accept but which makes the running version invisible in review. - `persistent` is `WaitForFirstConsumer`^{[10](https://git.gloom.garden/tepichord/milner/src/commit/e5e4a25dc58a64a5fcb77499111c7df685bad4a2/gitops/clusters/milner/infrastructure/local-path/values.yaml?display=source#L69)}, so the PVC stays `Pending` until the server pod schedules. Flux reporting the Kustomization not-ready in that window is the class working as designed. - Do not set `vector.enabled: true` yet — a receiver with no sender is dead weight until step 4 is approved. **Out-of-scope notes (deferred, need your go-ahead)** - **Talos host logs (step 4).** `machine.logging.destinations` emits raw newline-delimited JSON over TCP/UDP; vlagent only speaks HTTP, so a Vector receiver is required. The `victoria-logs-single` OCI artifact *bundles* the Vector subchart, so no extra Flux source is needed: ```yaml vector: enabled: true podHostNetwork: true dnsPolicy: ClusterFirstWithHostNet image: {repository: timberio/vector, tag: 0.57.0-distroless-libc} # arm64 published customConfig: sources: talos: {type: socket, address: 0.0.0.0:6051, mode: tcp, decoding: {codec: json}, host_key: __host} sinks: vlogs: {type: elasticsearch, inputs: [talos]} # endpoints auto-filled as /insert/elasticsearch ``` ```hcl # worker/main.tf and control-plane/main.tf config_patches yamlencode({ machine = { logging = { destinations = [{ endpoint = "tcp://127.0.0.1:6051/" format = "json_lines" extraTags = { cluster = "milner", node = "worker-1" } }] } } }) ``` Two caveats to settle before applying: destinations are **appended** under strategic-merge, so re-applying duplicates them (`talosctl edit mc` cleans up), and `KmsgLogConfig` is probably unnecessary because kernel messages already arrive in the service stream as `talos-service=kernel`. - **Disk resize (step 7).** Leaving `data/main.tf`'s `data_volume_size` at 100 GiB. Changing it makes `terraform plan` dirty and unapplyable: - `dmacvicar/libvirt` 0.9.9 `VolumeResource.Update()` errors with *"Storage volumes cannot be updated. All changes require replacement."*, and `prevent_destroy = true` forbids the replacement — so the plan shows an in-place update that apply refuses. - Talos v1.11.1 does not grow `disk`-type user volumes when the backing disk expands (siderolabs/talos#12991; fixed by #13057, present only in ≥ v1.13.0), so even a hand-run `qemu-img resize` may leave `/var/mnt/data` at 100 GiB. When it is wanted: `qemu-img resize … +900G`, then reconcile state (`terraform apply -refresh-only` or a `state rm`/`import` pair), then verify `talosctl get volumestatus -o yaml` and `df -h /var/mnt/data`. - **VictoriaLogs OSS has no auth.** The ingress stays tailnet-only and nothing else publishes it; if a token is ever wanted it goes Vault-first via a `VaultStaticSecret`. - **Adjacent, not requested:** `victoria-logs-mcp` (same registry) and the chart's `dashboards.enabled` — both only if you want them. ## Acceptance Criteria - [ ] `kustomize build gitops/clusters/milner/infrastructure` renders `Namespace/logging` with `pod-security.kubernetes.io/enforce: privileged` plus both new HelmReleases and the `victoria-logs-data` PVC. - [ ] `victoria-logs` is `Ready`/`Healthy`; `kubectl -n logging get pvc victoria-logs-data` is `Bound` on the `persistent` class at 20Gi and was created by Flux, not the chart (`server.persistentVolume.existingClaim` set). - [ ] `kubectl -n logging get ds victoria-logs-collector` reports `DESIRED == READY`, and `/select/logsql/query` returns entries with `kubernetes.pod_namespace`. - [ ] `pytest tests/test_static.py -k "yamllint or conftest"` and `pytest tests/test_gitops.py` pass, including `test_all_namespaces_have_psa_enforce` and the L1 Rego gate over the rendered logging workloads. - [ ] `pytest tests/test_chart_sources.py` resolves both new OCI sources (network-gated). - [ ] `OPERATIONS.md` §10 documents the `logs` service approval, the query commands, the per-app capture table and the resize blockers; `ARCHITECTURE.md` reflects the new claim and endpoint. ## Testing Plan | Case | Command | |---|---| | Render | `kustomize build gitops/clusters/milner/infrastructure` | | Schema + namespace guard | `pytest tests/test_gitops.py -k "kubeconform or namespaces or helmreleases"` | | L1 policy | `pytest tests/test_static.py -k conftest` | | Static | `pytest tests/test_static.py -k "yamllint or terraform_fmt"` | | Chart reachability | `pytest tests/test_chart_sources.py` (needs network) | | Live ingest | `kubectl -n logging port-forward svc/victoria-logs 9428:9428` then `curl --data-urlencode 'query=kubernetes.pod_namespace:logging' http://127.0.0.1:9428/select/logsql/query` | | Live tail | `curl -N --data-urlencode 'query=*' http://127.0.0.1:9428/select/logsql/tail` | | Flag parsing | `kubectl -n logging logs sts/victoria-logs` shows no arg-parse error for `-retentionPeriod=7d` / `-retention.maxDiskSpaceUsageBytes=18GiB` | | Host logs (deferred) | after the Talos phase: `talosctl logs machined` and look for `talos-service: machined` in VictoriaLogs | ## Related Issues/PRs - #50 — Zot OCI registry: the layout this copies (`oci://` URL convention, Flux-owned PVC, Tailscale Ingress). - #43 — Kubernetes Dashboard: infrastructure-layer + ingress wiring precedent. - siderolabs/talos#12991 and siderolabs/talos#13057 — user-volume growth; tracks the resize blocker. - Issue steps 4 and 7 stay unimplemented until you confirm in this thread. ## Research Findings - `victoria-logs-single` **0.13.9** and `victoria-logs-collector` **0.3.7** both exist as OCI artifacts under `ghcr.io/victoriametrics/helm-charts`, and both server/collector images publish `linux/arm64`. - The collector chart is **vlagent**, not Vector: it ingests HTTP only and needs hostPath but not hostNetwork, so the issue's "hostPath + hostNetwork" applies to the Talos-receiver half alone. - The `victoria-logs-single` OCI artifact bundles the Vector subchart (`0.57.0`), so a host-network `json_lines` receiver needs no additional Flux source; `timberio/vector:0.57.0-distroless-libc` publishes arm64. - Talos v1.11 `machine.logging.destinations` accepts `tcp://`/`udp://` endpoints with `format: json_lines` and applies without a reboot; kernel logs already ride the service stream, making `KmsgLogConfig` usually redundant. - `/select/logsql/return` (issue step 6) is not a real endpoint; the read paths are `/select/logsql/query` and `/select/logsql/tail` on `:9428`. - The disk resize is blocked twice independently: the libvirt provider refuses volume updates while `prevent_destroy` forbids replacement, and Talos v1.11.1 will not grow the user volume even after the qcow2 grows. ## Pre-Mortem 1. **The chart values keys are wrong and nothing renders the chart to catch it.** The plan nests `fullnameOverride` under `server:` (only the collector chart puts it at top level) and depends on `server.persistentVolume.existingClaim`, `service.clusterIP: ""` and `retentionDiskSpaceUsage: 18GiB` without ever running `helm template` against the pinned `0.13.9` chart. Helm silently ignores unknown keys, so the release installs with defaults: the Service/StatefulSet keep a release-derived name, the Ingress backend and vlagent's `http://victoria-logs:9428` resolve to nothing, and every static test still passes while `/select/logsql/query` returns no entries. 2. **The two deferred steps are read as an incomplete delivery.** Steps 4 (Talos host logs) and 7 (disk resize) are stated parts of issue #60's definition of done, so a reviewer diffing the PR against the issue can send everything back and discard the working Phase 1. The gate lives only as "need your go-ahead" in a comment thread, so it has no recorded owner and no expiry. 3. **The gated follow-ups fail on their first real apply.** Talos merges `machine.logging.destinations` by appending arrays, so a re-applied Terraform patch duplicates destinations, and `tcp://127.0.0.1:6051/` under `hostNetwork` makes every node report as `127.0.0.1`. When the resize is attempted, the libvirt provider's refusal to update volumes plus `prevent_destroy = true` leaves `terraform apply` failing and `data/` state dirty, blocking unrelated changes to that root.

tepichord added

status

implement

and removed

status

planning

labels 2026-09-15 11:39:34 -07:00

tepichord referenced this issue from a commit 2026-09-15 14:52:58 -07:00

feat(infra): wire victoria-logs and victoria-logs-collector into infrastructure layer

tepichord referenced this issue from a commit 2026-09-15 14:52:58 -07:00

test(static): add victoria-logs values.yaml paths to yamllint gate

tepichord referenced this issue from a commit 2026-09-15 14:52:58 -07:00

docs(operations): add §12 VictoriaLogs logging runbook

tepichord referenced this issue from a commit 2026-09-15 14:52:58 -07:00

docs(architecture): add VictoriaLogs endpoint and update storage budget

tepichord referenced this issue from a commit 2026-09-15 14:52:59 -07:00

docs(operations): fix section numbering and cross-references after VictoriaLogs addition

tepichord referenced this issue from a commit 2026-09-15 14:52:59 -07:00

fix(victoria-logs): read the ingress key at the path the chart reads

tepichord referenced this issue from a commit 2026-09-15 14:52:59 -07:00

docs(operations): fix VictoriaLogs query syntax and storage budget

tepichord referenced this issue from a commit 2026-09-15 14:52:59 -07:00

docs(architecture): keep the status column honest for the new endpoint

tepichord referenced this issue from a commit 2026-09-15 14:52:59 -07:00

test(gitops): guard victoria-logs values against the chart key shape

tepichord referenced this issue from a pull request that will close it, 2026-09-15 14:53:08 -07:00

Integrate VictoriaLogs in milner #62

tepichord added

status

reviewing

and removed

status

implement

labels 2026-09-15 14:53:09 -07:00

tepichord referenced this issue from a commit 2026-09-15 16:07:08 -07:00

feat(preview): preview the victoria-logs store and point the router at it (#62)

tepichord closed this issue 2026-09-16 13:33:40 -07:00

tepichord referenced this issue from a commit 2026-09-16 13:33:40 -07:00

Integrate VictoriaLogs in milner (#62)

tepichord added

status

document

and removed

status

reviewing

labels 2026-09-16 13:33:57 -07:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

main

feat/109-nix-base

feat/109-layered-nix-store

feat/110-zot-push-credential

feat/90-annia-pause-lever-oz-console

feat/87-tailscale-egress-from-cluster

chore/ci-step-resources

docs/vault-secret-prompt

feat/42-support-backups-persistent-data

feat/41-integrate-restic-tfstate-backup

feat/3-setting-up-agent-development

kvm-kubernetes-setup

No results found.

Labels

Clear labels   

blocked

  

bug

Something is not working

  

duplicate

This issue or pull request already exists

  

effort

high

  

effort

low

  

effort

medium

  

effort

minimal

  

enhancement

New feature

  

help wanted

Need some help

  

invalid

Something is wrong

  

pr

feedback

Used for PRs, indicates that feedback has been given and agent needs to make changes

  

pr

reviewing

Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  

question

More information is needed

  

status

backlog

Ignore, no work needed yet, human will assign this work by changing to status/planning

  

status

document

Record findings, create flashcards, and document knowledge in knowledgebase for the future

  

status

done

Completed when PR is merged, no more work needed.

  

status

implement

Make the code changes. Should be done by a faster and cheaper model

  

status

pending

Give human a chance to review planning details

  

status

planning

Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  

status

reviewing

PR is created and feedback loop begins between human and agent

  

waiting

  

wontfix

This won't be fixed

No labels

blocked

bug

duplicate

effort

high

effort

low

effort

medium

effort

minimal

enhancement

help wanted

invalid

pr

feedback

pr

reviewing

question

status

backlog

status

document

status

done

status

implement

status

pending

status

planning

status

reviewing

waiting

wontfix

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

MVP

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

tepichord/milner#60

No description provided.