tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Create Tailscale egress from the cluster #87

New issue
Closed
opened 2026-09-22 14:01:28 -07:00 by tepichord · 1 comment
tepichord commented 2026-09-22 14:01:28 -07:00
Owner
Copy link
   This adds the cluster's first egress path: a Tailscale egress ProxyGroup, and an
   `ExternalName` Service the consumer dials by cluster DNS. It ends at the relay N1
   installs on `naina-server`; this issue is the cluster half only.

   The consumer dials `http://llm-gateway.annia.svc.cluster.local:4000`. Nothing in the
   cluster ever names the gateway's own hostname: `lain-llm.tailc23d6.ts.net` **does not
   resolve here at all** (different tailnet, different MagicDNS suffix — the resolver
   answers NXDOMAIN for the whole zone), so the Service names the host instead.

   ## Prerequisite: annia's namespace

   The Service is a namespaced object, and kustomize-controller will not place one it
   cannot find a namespace for. `infrastructure/namespaces/annia.yaml` must exist, listed
   in `infrastructure/kustomization.yaml`, following `namespaces/rabbitmq.yaml`:

   ```yaml
   apiVersion: v1
   kind: Namespace
   metadata:
     name: annia
     labels:
       pod-security.kubernetes.io/enforce: baseline
   ```

   Add it here **if it does not already exist** — #25 and the enqueuer issue both need the
   same namespace, so whichever lands first creates it and the others cite it. Never a
   second copy.

   ## Layer 1 — `infrastructure/tailscale.yaml` gains a second ProxyGroup

   That file currently holds exactly one object, the `ingress-proxies` ProxyGroup.

   ```yaml
   apiVersion: tailscale.com/v1alpha1
   kind: ProxyGroup
   metadata:
     name: egress-proxies
   spec:
     type: egress
     replicas: 1
   ```

   - **Why a second ProxyGroup instead of reusing `ingress-proxies`:** `spec.type` is the
     role, and an egress Service must name a ProxyGroup of type `egress`. This is the
     tree's second ProxyGroup and its **first egress one**.
   - **No admin-console approval step.** Egress advertises nothing, so unlike an ingress
     Service there is no approval to grant; the proxy's device joins tagged `tag:k8s`,
     which `docs/tailscale.md` §2.5 already auto-approves. Verify in the admin console
     that a new device appears as `milner-egress-proxies-0`.
   - The ProxyGroup is cluster-scoped, so it carries no namespace — the layer's guard
     (`test_every_namespaced_object_declares_a_namespace`) knows this kind and will not
     flag it.

   ## Layer 2 — the egress Service in annia's namespace

   `apps/annia/llm-gateway.yaml`, with `apps/annia/kustomization.yaml` carrying
   `namespace: annia` and listing it (mirroring `apps/rabbitmq/kustomization.yaml`), and
   `annia` added to `apps/kustomization.yaml`'s resources. That directory will grow to hold
   annia's own workloads; this is its first object.

   ```yaml
   apiVersion: v1
   kind: Service
   metadata:
     name: llm-gateway
     annotations:
       tailscale.com/proxy-group: egress-proxies
       tailscale.com/tailnet-fqdn: naina-server.faun-snapper.ts.net
   spec:
     type: ExternalName
     externalName: naina-server.faun-snapper.ts.net   # the operator overwrites this
     ports:
       - name: http
         port: 4000
         protocol: TCP
   ```

   Four rules the operator's own documentation pins down, each of which has a silent
   failure mode:

   1. **`tailscale.com/tailnet-fqdn` must be the full MagicDNS name.** A bare
      `naina-server` is not resolvable. `tailscale.com/tailnet-ip` is the mutually
      exclusive alternative; the FQDN is the one to use, so a re-addressed host needs no
      edit here.
   2. **Every port to be reached must be listed in `spec.ports`**, and the operator uses
      only `port`, `protocol` and `name` from each entry. Omit `4000` and nothing is
      reachable, with nothing reporting why.
   3. **The operator overwrites `spec.externalName`.** The literal above is the human
      record of the intent; do not expect it to survive reconciliation.
   4. **One ProxyGroup serves any number of egress Services.** A second consumer — Letta
      is the known one — adds its *own* Service in its *own* namespace, naming this same
      ProxyGroup. Not a shared Service.

   ## Placement is the point

   The Service goes in the **consumer's namespace**, not in `infrastructure`. A shared
   `llm-gateway` in an earlier layer would be inherited by every preview namespace, which
   hands a preview the live GPU box — and story 14 says a preview holds no real credential
   and no real power. In annia's namespace, a preview namespace simply does not have it; an
   app that wants a preview path gets a stub, the way `oz` uses its `PREVIEW-BUILD` marker.

   **If annia's Service ever declares `milner.dev/preview-paths`** — which is what would make
   annia previewable — this placement has to be revisited in that same change, not after.

   ## Naming

   `llm-gateway`, port `4000`. The consumer's `entrypoint.sh` already embeds `:4000` in its
   `mitmdump` reverse-proxy target, so the pod's only change is the hostname it dials: from
   the off-cluster name to `llm-gateway.annia.svc.cluster.local`. If the port ever moves to
   `80`, that target moves with it.

   ## Done when

   ```bash
   # The proxy device exists (admin console, or from a workstation):
   tailscale status | grep milner-egress-proxies

   # From a throwaway pod in annia's namespace — 401 means the gateway was reached
   # and asked for the API key, which is the whole point of this path:
   kubectl -n annia run probe --rm -it --image=curlimages/curl --restart=Never -- \
     curl -s -o /dev/null -w '%{http_code}\n' http://llm-gateway:4000/v1/models

   # And the DNS name the consumer actually uses:
   kubectl -n annia run probe --rm -it --image=busybox --restart=Never -- \
     nslookup llm-gateway.annia.svc.cluster.local

   # Offline, before anything is applied:
   kustomize build gitops/clusters/milner/infrastructure | grep -A6 'kind: ProxyGroup'
   kustomize build gitops/clusters/milner/apps | grep -A12 llm-gateway
   ```

   A `401` is the success signal. A connection refused or a timeout means N1's relay or its
   firewall rule is not in place; that path fails as "no route to host" from in-cluster,
   which looks like an ACL denial and is not one.

   ## Documentation

   - **`docs/tailscale.md`** — the new section N1 adds gets the cluster-side half: the
     ProxyGroup, the Service, the full-MagicDNS-name rule, the consumer-namespace placement
     and story 14's reason for it. Take the next free section number in the `docs/`
     sequence.
   - **`OPERATIONS.md` gets no row.** This is Flux-owned cluster state, not an out-of-band
     action, and §1 is the inventory of what Terraform does not own.
   - **`ARCHITECTURE.md` is not touched by this issue.**
````markdown This adds the cluster's first egress path: a Tailscale egress ProxyGroup, and an `ExternalName` Service the consumer dials by cluster DNS. It ends at the relay N1 installs on `naina-server`; this issue is the cluster half only. The consumer dials `http://llm-gateway.annia.svc.cluster.local:4000`. Nothing in the cluster ever names the gateway's own hostname: `lain-llm.tailc23d6.ts.net` **does not resolve here at all** (different tailnet, different MagicDNS suffix — the resolver answers NXDOMAIN for the whole zone), so the Service names the host instead. ## Prerequisite: annia's namespace The Service is a namespaced object, and kustomize-controller will not place one it cannot find a namespace for. `infrastructure/namespaces/annia.yaml` must exist, listed in `infrastructure/kustomization.yaml`, following `namespaces/rabbitmq.yaml`: ```yaml apiVersion: v1 kind: Namespace metadata: name: annia labels: pod-security.kubernetes.io/enforce: baseline ``` Add it here **if it does not already exist** — #25 and the enqueuer issue both need the same namespace, so whichever lands first creates it and the others cite it. Never a second copy. ## Layer 1 — `infrastructure/tailscale.yaml` gains a second ProxyGroup That file currently holds exactly one object, the `ingress-proxies` ProxyGroup. ```yaml apiVersion: tailscale.com/v1alpha1 kind: ProxyGroup metadata: name: egress-proxies spec: type: egress replicas: 1 ``` - **Why a second ProxyGroup instead of reusing `ingress-proxies`:** `spec.type` is the role, and an egress Service must name a ProxyGroup of type `egress`. This is the tree's second ProxyGroup and its **first egress one**. - **No admin-console approval step.** Egress advertises nothing, so unlike an ingress Service there is no approval to grant; the proxy's device joins tagged `tag:k8s`, which `docs/tailscale.md` §2.5 already auto-approves. Verify in the admin console that a new device appears as `milner-egress-proxies-0`. - The ProxyGroup is cluster-scoped, so it carries no namespace — the layer's guard (`test_every_namespaced_object_declares_a_namespace`) knows this kind and will not flag it. ## Layer 2 — the egress Service in annia's namespace `apps/annia/llm-gateway.yaml`, with `apps/annia/kustomization.yaml` carrying `namespace: annia` and listing it (mirroring `apps/rabbitmq/kustomization.yaml`), and `annia` added to `apps/kustomization.yaml`'s resources. That directory will grow to hold annia's own workloads; this is its first object. ```yaml apiVersion: v1 kind: Service metadata: name: llm-gateway annotations: tailscale.com/proxy-group: egress-proxies tailscale.com/tailnet-fqdn: naina-server.faun-snapper.ts.net spec: type: ExternalName externalName: naina-server.faun-snapper.ts.net # the operator overwrites this ports: - name: http port: 4000 protocol: TCP ``` Four rules the operator's own documentation pins down, each of which has a silent failure mode: 1. **`tailscale.com/tailnet-fqdn` must be the full MagicDNS name.** A bare `naina-server` is not resolvable. `tailscale.com/tailnet-ip` is the mutually exclusive alternative; the FQDN is the one to use, so a re-addressed host needs no edit here. 2. **Every port to be reached must be listed in `spec.ports`**, and the operator uses only `port`, `protocol` and `name` from each entry. Omit `4000` and nothing is reachable, with nothing reporting why. 3. **The operator overwrites `spec.externalName`.** The literal above is the human record of the intent; do not expect it to survive reconciliation. 4. **One ProxyGroup serves any number of egress Services.** A second consumer — Letta is the known one — adds its *own* Service in its *own* namespace, naming this same ProxyGroup. Not a shared Service. ## Placement is the point The Service goes in the **consumer's namespace**, not in `infrastructure`. A shared `llm-gateway` in an earlier layer would be inherited by every preview namespace, which hands a preview the live GPU box — and story 14 says a preview holds no real credential and no real power. In annia's namespace, a preview namespace simply does not have it; an app that wants a preview path gets a stub, the way `oz` uses its `PREVIEW-BUILD` marker. **If annia's Service ever declares `milner.dev/preview-paths`** — which is what would make annia previewable — this placement has to be revisited in that same change, not after. ## Naming `llm-gateway`, port `4000`. The consumer's `entrypoint.sh` already embeds `:4000` in its `mitmdump` reverse-proxy target, so the pod's only change is the hostname it dials: from the off-cluster name to `llm-gateway.annia.svc.cluster.local`. If the port ever moves to `80`, that target moves with it. ## Done when ```bash # The proxy device exists (admin console, or from a workstation): tailscale status | grep milner-egress-proxies # From a throwaway pod in annia's namespace — 401 means the gateway was reached # and asked for the API key, which is the whole point of this path: kubectl -n annia run probe --rm -it --image=curlimages/curl --restart=Never -- \ curl -s -o /dev/null -w '%{http_code}\n' http://llm-gateway:4000/v1/models # And the DNS name the consumer actually uses: kubectl -n annia run probe --rm -it --image=busybox --restart=Never -- \ nslookup llm-gateway.annia.svc.cluster.local # Offline, before anything is applied: kustomize build gitops/clusters/milner/infrastructure | grep -A6 'kind: ProxyGroup' kustomize build gitops/clusters/milner/apps | grep -A12 llm-gateway ``` A `401` is the success signal. A connection refused or a timeout means N1's relay or its firewall rule is not in place; that path fails as "no route to host" from in-cluster, which looks like an ACL denial and is not one. ## Documentation - **`docs/tailscale.md`** — the new section N1 adds gets the cluster-side half: the ProxyGroup, the Service, the full-MagicDNS-name rule, the consumer-namespace placement and story 14's reason for it. Take the next free section number in the `docs/` sequence. - **`OPERATIONS.md` gets no row.** This is Flux-owned cluster state, not an out-of-band action, and §1 is the inventory of what Terraform does not own. - **`ARCHITECTURE.md` is not touched by this issue.** ````
tepichord added this to the MVP project 2026-09-22 14:01:32 -07:00
tepichord commented 2026-09-22 14:27:09 -07:00
Author
Owner
Copy link

Naina:

Overview

Add the cluster's first egress path: an egress ProxyGroup plus an ExternalName Service in a new annia namespace, so in-cluster workloads reach naina-server.faun-snapper.ts.net:4000 at http://llm-gateway.annia.svc.cluster.local:4000. This is the cluster half only — the relay on naina-server is #86, and the Service deliberately lives in the consumer namespace so preview namespaces never inherit a live GPU path (story 14).

Technical Approach

  1. Create gitops/clusters/milner/infrastructure/namespaces/annia.yaml — only if it does not already exist (#25 and the enqueuer issue both need it; first to land creates it, the rest cite it).

    apiVersion: v1
kind: Namespace
metadata:
  name: annia
  labels:
    pod-security.kubernetes.io/enforce: baseline

    List namespaces/annia.yaml in gitops/clusters/milner/infrastructure/kustomization.yaml (after namespaces/postgres.yaml).

  2. Append a second ProxyGroup to gitops/clusters/milner/infrastructure/tailscale.yaml.

    apiVersion: tailscale.com/v1alpha1
kind: ProxyGroup
metadata:
  name: egress-proxies
spec:
  type: egress
  replicas: 1

    spec.type is immutable and the only required field; replicas defaults to 2, so 1 must be explicit to match ingress-proxies.1234

  3. Create gitops/clusters/milner/apps/annia/llm-gateway.yaml.

    apiVersion: v1
kind: Service
metadata:
  name: llm-gateway
  annotations:
    tailscale.com/proxy-group: egress-proxies
    tailscale.com/tailnet-fqdn: naina-server.faun-snapper.ts.net
spec:
  type: ExternalName
  externalName: naina-server.faun-snapper.ts.net   # the operator overwrites this
  ports:
    - name: http
      port: 4000
      protocol: TCP

    externalName cannot be omitted (the API server rejects an empty one); the literal is the human record of intent, not a value that survives reconciliation.56

  4. Create gitops/clusters/milner/apps/annia/kustomization.yaml (mirrors apps/rabbitmq/kustomization.yaml).

    apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
namespace: annia
resources:
  - llm-gateway.yaml

  5. Add annia to gitops/clusters/milner/apps/kustomization.yaml's resources. Without this the Service never renders; the existing apps.yaml Flux Kustomization picks the directory up unchanged.18

  6. Add docs/tailscale.md §17, "Cluster egress" — ProxyGroup, Service, the full-MagicDNS-name rule, tailnet-ip as the mutually exclusive alternative, the consumer-namespace placement and story 14's reason, and why egress needs no admin approval. Do not claim failover for replicas: 1.

Implementation Details

Files

Path Action
gitops/clusters/milner/infrastructure/namespaces/annia.yaml create (if absent)
gitops/clusters/milner/infrastructure/kustomization.yaml add namespace entry
gitops/clusters/milner/infrastructure/tailscale.yaml append egress-proxies
gitops/clusters/milner/apps/annia/llm-gateway.yaml create
gitops/clusters/milner/apps/annia/kustomization.yaml create
gitops/clusters/milner/apps/kustomization.yaml add annia
docs/tailscale.md new §17
tests/test_gitops.py one new shape guard

Dependencies — none beyond what is already installed; operator v1.102.3 supports egress ProxyGroups and the tailnet-fqdn path.78 No RBAC change (the operator's ClusterRole already covers proxygroups, services, endpointslices)91011, no ACL change (tag:k8s-operator already owns tag:k8s).12

Constraints

  • tailscale.com/tailnet-fqdn must be the full MagicDNS name, and exactly one of tailnet-fqdn/tailnet-ip may be set.13
  • Every port to be reached must be listed in spec.ports; a missing 4000 fails silently.1415
  • ProxyGroup is cluster-scoped — no namespace, already in CLUSTER_SCOPED_KINDS.1617
  • The operator places egress proxy pods in the privileged tailscale namespace, not in annia; annia's baseline PSA does not admit them.1920
  • One ProxyGroup serves many Services: a second consumer (Letta) adds its own Service in its own namespace, naming egress-proxies.

Out-of-scope notes

  • No OPERATIONS.md row (Flux-owned state) and no ARCHITECTURE.md change, per the issue.
  • The VSO tripwire (7 VaultAuths / 11 VaultStaticSecrets) reads only the secrets and preview-system layers, so an annia VaultStaticSecret later would not move it.
  • Device hostname caveat: the operator derives <proxygroup>-<ordinal> = egress-proxies-0, not milner-egress-proxies-0 — the docs' milner- prefix implies an out-of-band hostnamePrefix/ProxyClass that grep finds nowhere in-tree. Verify via ProxyGroup status, not a hard-coded name.
  • docs/tailscale.md §3 pins v1.102.3; 1.102.4 exists but is orthogonal to this issue.

Acceptance Criteria

  • infrastructure layer renders a second ProxyGroup named egress-proxies, type: egress, replicas: 1.
  • apps layer renders Service/llm-gateway in namespace annia with type: ExternalName, both annotations, and port 4000.
  • annia namespace exists with pod-security.kubernetes.io/enforce: baseline, and is created only if not already present.
  • pytest tests/test_gitops.py passes, including the new egress-shape guard.
  • Live: the egress proxy device has joined with no admin-console approval (kubectl get proxygroup egress-proxies -o jsonpath='{.status.devices[*].hostname}{\"\n\"}').
  • Live: Service/llm-gateway reports TailscaleEgressSvcReady=true, and once #86's relay is in place the probe returns 401.
  • docs/tailscale.md §17 documents the four rules and the placement rationale.

Testing Plan

# Offline: both layers build and show the new objects.
kustomize build gitops/clusters/milner/infrastructure | grep -A6 'kind: ProxyGroup'
kustomize build gitops/clusters/milner/apps | grep -A12 llm-gateway

# Offline gate.
pytest tests/test_gitops.py -k "kustomize_build or kubeconform or namespaced_object or psa"

# New guard: apps/annia Service binds the egress ProxyGroup and names the full
# MagicDNS FQDN in tailscale.com/tailnet-fqdn.
pytest tests/test_gitops.py -k egress

# Live readiness, then the probe (401 = reached the gateway and it asked for a key).
kubectl -n annia wait svc/llm-gateway --for=condition=TailscaleEgressSvcReady=true
kubectl -n annia run probe --rm -it --image=curlimages/curl --restart=Never -- \
  curl -s -o /dev/null -w '%{http_code}\n' http://llm-gateway:4000/v1/models
kubectl -n annia run probe --rm -it --image=busybox --restart=Never -- \
  nslookup llm-gateway.annia.svc.cluster.local

A connection-refused or timeout means #86's relay/firewall is not in place; that is not an ACL denial.

Related Issues/PRs

  • #86 — Reach the tailnet-shared LLM gateway (the relay on naina-server; this is the cluster half).
  • #25 — Create Annia pod (also needs the annia namespace; cite, do not duplicate).
  • docs/preview-environment.md §11h — milner.dev/preview-paths; if annia's Service ever declares it, revisit this placement in the same change.

Research Findings

  • spec.type is the only required ProxyGroup field; egress is a valid enum and replicas defaults to 2, so the issue's manifest is complete and valid.
  • No new Kubernetes RBAC is needed: the operator's ClusterRole already covers proxygroups, core services, and endpointslices for HA egress, and the proxy pod runs in the privileged tailscale namespace.
  • spec.externalName must be set — the API server rejects an empty one — and the operator overwrites it with its generated ClusterIP FQDN, as the issue states.
  • Egress uses no annotation beyond tailscale.com/proxy-group and tailscale.com/tailnet-fqdn (the mutually exclusive alternative is tailscale.com/tailnet-ip); the operator adds a finalizer and cleans up only on Service deletion.
  • Device approval is automatic: the operator requests preauthorized auth keys, and docs/tailscale.md §2.1 already makes tag:k8s-operator own tag:k8s; autoApprovers.services is irrelevant because egress advertises nothing.
  • The default ProxyGroup device hostname is <proxygroup>-<ordinal> (egress-proxies-0), not the milner--prefixed form the docs imply, so verification should read ProxyGroup status rather than assume the prefix.

Pre-Mortem

  1. Live acceptance is coupled to #86's relay, so the issue cannot be closed independently — The probe and TailscaleEgressSvcReady checks only pass once the out-of-scope relay on naina-server exists, so the issue stalls open or gets closed on trust. A reviewer then misreads the timeout as an ACL/egress defect and "fixes" working manifests.
  2. Two issues race to own the annia namespace and the Kustomization.resources entry — "create only if absent" cannot be expressed in a committed git manifest, so #25 or the enqueuer adds the same file or a second namespaces/annia.yaml line, and the merge leaves a duplicate resource that breaks kustomize build in CI.
  3. TailscaleEgressSvcReady=true never becomes true despite correct YAML — Readiness depends on naina-server.faun-snapper.ts.net:4000 being resolvable and reachable from the egress proxy pods, which is outside the plan's control. The plan offers no fallback assertion that isolates cluster-side correctness, so a non-ready condition is indistinguishable from a manifest mistake.
Naina: ## Overview Add the cluster's first egress path: an `egress` ProxyGroup plus an `ExternalName` Service in a new `annia` namespace, so in-cluster workloads reach `naina-server.faun-snapper.ts.net:4000` at `http://llm-gateway.annia.svc.cluster.local:4000`. This is the cluster half only — the relay on `naina-server` is #86, and the Service deliberately lives in the consumer namespace so preview namespaces never inherit a live GPU path (story 14). ## Technical Approach 1. **Create `gitops/clusters/milner/infrastructure/namespaces/annia.yaml`** — only if it does not already exist (#25 and the enqueuer issue both need it; first to land creates it, the rest cite it). ```yaml apiVersion: v1 kind: Namespace metadata: name: annia labels: pod-security.kubernetes.io/enforce: baseline ``` List `namespaces/annia.yaml` in `gitops/clusters/milner/infrastructure/kustomization.yaml` (after `namespaces/postgres.yaml`). 2. **Append a second ProxyGroup to `gitops/clusters/milner/infrastructure/tailscale.yaml`.** ```yaml apiVersion: tailscale.com/v1alpha1 kind: ProxyGroup metadata: name: egress-proxies spec: type: egress replicas: 1 ``` `spec.type` is immutable and the only required field; `replicas` defaults to 2, so `1` must be explicit to match `ingress-proxies`.<sup>[1](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/deploy/crds/tailscale.com_proxygroups.yaml#L79)</sup><sup>[2](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/deploy/crds/tailscale.com_proxygroups.yaml#L161)</sup><sup>[3](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/deploy/crds/tailscale.com_proxygroups.yaml#L126)</sup><sup>[4](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/infrastructure/tailscale.yaml?display=source#L7)</sup> 3. **Create `gitops/clusters/milner/apps/annia/llm-gateway.yaml`.** ```yaml apiVersion: v1 kind: Service metadata: name: llm-gateway annotations: tailscale.com/proxy-group: egress-proxies tailscale.com/tailnet-fqdn: naina-server.faun-snapper.ts.net spec: type: ExternalName externalName: naina-server.faun-snapper.ts.net # the operator overwrites this ports: - name: http port: 4000 protocol: TCP ``` `externalName` cannot be omitted (the API server rejects an empty one); the literal is the human record of intent, not a value that survives reconciliation.<sup>[5](https://github.com/kubernetes/kubernetes/blob/v1.34.1/pkg/apis/core/validation/validation.go#L6351)</sup><sup>[6](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/egress-services.go#L215)</sup> 4. **Create `gitops/clusters/milner/apps/annia/kustomization.yaml`** (mirrors `apps/rabbitmq/kustomization.yaml`). ```yaml apiVersion: kustomize.config.k8s.io/v1beta1 kind: Kustomization namespace: annia resources: - llm-gateway.yaml ``` 5. **Add `annia` to `gitops/clusters/milner/apps/kustomization.yaml`'s `resources`.** Without this the Service never renders; the existing `apps.yaml` Flux Kustomization picks the directory up unchanged.<sup>[18](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/gitops/clusters/milner/apps.yaml?display=source#L8)</sup> 6. **Add `docs/tailscale.md` §17, "Cluster egress"** — ProxyGroup, Service, the full-MagicDNS-name rule, `tailnet-ip` as the mutually exclusive alternative, the consumer-namespace placement and story 14's reason, and why egress needs no admin approval. Do not claim failover for `replicas: 1`. ## Implementation Details **Files** | Path | Action | | --- | --- | | `gitops/clusters/milner/infrastructure/namespaces/annia.yaml` | create (if absent) | | `gitops/clusters/milner/infrastructure/kustomization.yaml` | add namespace entry | | `gitops/clusters/milner/infrastructure/tailscale.yaml` | append `egress-proxies` | | `gitops/clusters/milner/apps/annia/llm-gateway.yaml` | create | | `gitops/clusters/milner/apps/annia/kustomization.yaml` | create | | `gitops/clusters/milner/apps/kustomization.yaml` | add `annia` | | `docs/tailscale.md` | new §17 | | `tests/test_gitops.py` | one new shape guard | **Dependencies** — none beyond what is already installed; operator `v1.102.3` supports `egress` ProxyGroups and the `tailnet-fqdn` path.<sup>[7](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/deploy/crds/tailscale.com_proxygroups.yaml#L156)</sup><sup>[8](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/egress-services.go#L215)</sup> No RBAC change (the operator's ClusterRole already covers `proxygroups`, `services`, `endpointslices`)<sup>[9](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/deploy/chart/templates/operator-rbac.yaml#L23)</sup><sup>[10](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/deploy/chart/templates/operator-rbac.yaml#L32)</sup><sup>[11](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/deploy/chart/templates/operator-rbac.yaml#L35)</sup>, no ACL change (`tag:k8s-operator` already owns `tag:k8s`).<sup>[12](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/docs/tailscale.md?display=source#L7)</sup> **Constraints** - `tailscale.com/tailnet-fqdn` must be the full MagicDNS name, and exactly one of `tailnet-fqdn`/`tailnet-ip` may be set.<sup>[13](https://tailscale.com/docs/kubernetes-operator/egress/access-tailnet-service)</sup> - Every port to be reached must be listed in `spec.ports`; a missing `4000` fails silently.<sup>[14](https://tailscale.com/docs/kubernetes-operator/egress/access-tailnet-service)</sup><sup>[15](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/egress-services.go#L608)</sup> - ProxyGroup is cluster-scoped — no namespace, already in `CLUSTER_SCOPED_KINDS`.<sup>[16](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/tests/test_gitops.py?display=source#L195)</sup><sup>[17](https://github.com/tailscale/tailscale/blob/v1.102.3/cmd/k8s-operator/deploy/crds/tailscale.com_proxygroups.yaml#L16)</sup> - The operator places egress proxy pods in the privileged `tailscale` namespace, not in `annia`; `annia`'s `baseline` PSA does not admit them.<sup>[19](https://git.gloom.garden/tepichord/milner/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/docs/tailscale.md?display=source#L69)</sup><sup>[20](https://tailscale.com/docs/kubernetes-operator/reference/rbac)</sup> - One ProxyGroup serves many Services: a second consumer (Letta) adds its own Service in its own namespace, naming `egress-proxies`. **Out-of-scope notes** - No `OPERATIONS.md` row (Flux-owned state) and no `ARCHITECTURE.md` change, per the issue. - The VSO tripwire (`7 VaultAuths / 11 VaultStaticSecrets`) reads only the `secrets` and `preview-system` layers, so an `annia` VaultStaticSecret later would not move it. - Device hostname caveat: the operator derives `<proxygroup>-<ordinal>` = `egress-proxies-0`, not `milner-egress-proxies-0` — the docs' `milner-` prefix implies an out-of-band `hostnamePrefix`/`ProxyClass` that grep finds nowhere in-tree. Verify via ProxyGroup status, not a hard-coded name. - `docs/tailscale.md` §3 pins v1.102.3; 1.102.4 exists but is orthogonal to this issue. ## Acceptance Criteria - [ ] `infrastructure` layer renders a second ProxyGroup named `egress-proxies`, `type: egress`, `replicas: 1`. - [ ] `apps` layer renders `Service/llm-gateway` in namespace `annia` with `type: ExternalName`, both annotations, and port `4000`. - [ ] `annia` namespace exists with `pod-security.kubernetes.io/enforce: baseline`, and is created only if not already present. - [ ] `pytest tests/test_gitops.py` passes, including the new egress-shape guard. - [ ] Live: the egress proxy device has joined with no admin-console approval (`kubectl get proxygroup egress-proxies -o jsonpath='{.status.devices[*].hostname}{\"\n\"}'`). - [ ] Live: `Service/llm-gateway` reports `TailscaleEgressSvcReady=true`, and once #86's relay is in place the probe returns `401`. - [ ] `docs/tailscale.md` §17 documents the four rules and the placement rationale. ## Testing Plan ```bash # Offline: both layers build and show the new objects. kustomize build gitops/clusters/milner/infrastructure | grep -A6 'kind: ProxyGroup' kustomize build gitops/clusters/milner/apps | grep -A12 llm-gateway # Offline gate. pytest tests/test_gitops.py -k "kustomize_build or kubeconform or namespaced_object or psa" # New guard: apps/annia Service binds the egress ProxyGroup and names the full # MagicDNS FQDN in tailscale.com/tailnet-fqdn. pytest tests/test_gitops.py -k egress # Live readiness, then the probe (401 = reached the gateway and it asked for a key). kubectl -n annia wait svc/llm-gateway --for=condition=TailscaleEgressSvcReady=true kubectl -n annia run probe --rm -it --image=curlimages/curl --restart=Never -- \ curl -s -o /dev/null -w '%{http_code}\n' http://llm-gateway:4000/v1/models kubectl -n annia run probe --rm -it --image=busybox --restart=Never -- \ nslookup llm-gateway.annia.svc.cluster.local ``` A connection-refused or timeout means #86's relay/firewall is not in place; that is not an ACL denial. ## Related Issues/PRs - #86 — Reach the tailnet-shared LLM gateway (the relay on `naina-server`; this is the cluster half). - #25 — Create Annia pod (also needs the `annia` namespace; cite, do not duplicate). - `docs/preview-environment.md` §11h — `milner.dev/preview-paths`; if annia's Service ever declares it, revisit this placement in the same change. ## Research Findings - `spec.type` is the only required ProxyGroup field; `egress` is a valid enum and `replicas` defaults to `2`, so the issue's manifest is complete and valid. - No new Kubernetes RBAC is needed: the operator's ClusterRole already covers `proxygroups`, core `services`, and `endpointslices` for HA egress, and the proxy pod runs in the privileged `tailscale` namespace. - `spec.externalName` must be set — the API server rejects an empty one — and the operator overwrites it with its generated ClusterIP FQDN, as the issue states. - Egress uses no annotation beyond `tailscale.com/proxy-group` and `tailscale.com/tailnet-fqdn` (the mutually exclusive alternative is `tailscale.com/tailnet-ip`); the operator adds a finalizer and cleans up only on Service deletion. - Device approval is automatic: the operator requests preauthorized auth keys, and `docs/tailscale.md` §2.1 already makes `tag:k8s-operator` own `tag:k8s`; `autoApprovers.services` is irrelevant because egress advertises nothing. - The default ProxyGroup device hostname is `<proxygroup>-<ordinal>` (`egress-proxies-0`), not the `milner-`-prefixed form the docs imply, so verification should read ProxyGroup status rather than assume the prefix. ## Pre-Mortem 1. **Live acceptance is coupled to #86's relay, so the issue cannot be closed independently** — The probe and `TailscaleEgressSvcReady` checks only pass once the out-of-scope relay on `naina-server` exists, so the issue stalls open or gets closed on trust. A reviewer then misreads the timeout as an ACL/egress defect and "fixes" working manifests. 2. **Two issues race to own the `annia` namespace and the `Kustomization.resources` entry** — "create only if absent" cannot be expressed in a committed git manifest, so #25 or the enqueuer adds the same file or a second `namespaces/annia.yaml` line, and the merge leaves a duplicate resource that breaks `kustomize build` in CI. 3. **`TailscaleEgressSvcReady=true` never becomes true despite correct YAML** — Readiness depends on `naina-server.faun-snapper.ts.net:4000` being resolvable and reachable from the egress proxy pods, which is outside the plan's control. The plan offers no fallback assertion that isolates cluster-side correctness, so a non-ready condition is indistinguishable from a manifest mistake.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/109-nix-base
feat/109-layered-nix-store
feat/110-zot-push-credential
feat/90-annia-pause-lever-oz-console
feat/87-tailscale-egress-from-cluster
chore/ci-step-resources
docs/vault-secret-prompt
feat/42-support-backups-persistent-data
feat/41-integrate-restic-tfstate-backup
feat/3-setting-up-agent-development
kvm-kubernetes-setup
No results found.
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner#87
No description provided.