tepichord/milner
1
1
Fork 0
Code Issues 8 Pull requests 5 Projects 1 Releases Packages Wiki Activity Actions

Reach the tailnet-shared LLM gateway #86

New issue
Closed
opened 2026-09-22 13:59:37 -07:00 by tepichord · 1 comment
tepichord commented 2026-09-22 13:59:37 -07:00
Owner
Copy link
  A cluster workload needs to reach an OpenAI-compatible inference gateway that lives on
  a **different** tailnet (`lain-llm.tailc23d6.ts.net`, `100.122.246.96`, plain HTTP on
  `:4000`, key-authenticated). No pod, and no Tailscale egress proxy, can reach it
  directly. This issue adds the one hop that can: a small TCP relay on `naina-server`,
  the libvirt host that runs this cluster's VMs.

  ## Why the cluster cannot reach the gateway itself

  `lain-llm` is shared as a **machine share to a user** (`vandomej@`), not to a tag.
  Tailscale shares a machine with a person, and the recipient accepts it; a node
  authenticated with a tagged auth key is not a person's node, so the share never
  appears to it. Every device the Tailscale operator creates is tagged **by
  construction** — the operator authenticates with a `tag:k8s-operator` OAuth client,
  and a tagged credential can only mint tagged devices. So `tag:k8s` devices — an
  egress ProxyGroup, a standalone proxy, or any in-cluster pod — cannot see `lain-llm`,
  and no ACL arrangement changes that.

  The untagged device that already sees it is `naina-server` (`100.72.140.122`,
  untagged, owned by `vandomej@`), which is also where the cluster's VMs run. Hence the
  path: **the cluster reaches the host as an ordinary same-tailnet egress target, and
  the host relays to `lain-llm`.**

  ## What was measured, this session, on `naina-server`

  | Check | Command | Result |
  |---|---|---|
  | Host reaches the gateway | `curl -m6 http://100.122.246.96:4000/health` on the host | `{"status":"ok","engine":{"loaded_model":"qwen3.8-27b",...}}` |
  | The gateway is key-authenticated | `curl -o /dev/null -w '%{http_code}' http://100.122.246.96:4000/v1/models` | `401` |
  | The host is the right identity | `tailscale ping --c=1 lain-llm.tailc23d6.ts.net` | `pong from lain-llm (100.122.246.96)` |
  | The cluster's ACL is not the blocker | netmap `PacketFilter` read from the egress/ingress proxy | **one rule: all tailnet sources → all destinations, all ports,
proto 6/17/1/58** |
  | The cluster reaches the host | `kubectl -n tailscale exec ingress-proxies-0 -c tailscale -- tailscale nc naina-server 22` | `CONNECTED` |
  | The cluster cannot reach the port | same command, port `4000` | `no route to host` |
  | Why | `firewall-cmd --get-active-zones` (no sudo needed) | `libvirt: cp-bridge virbr0` / `public (default): wlp1s0f0` — **`tailscale0` is in no zone**, so it
falls to `public`, which allows ssh only |
  | The gateway resolves only off-cluster | `getent hosts lain-llm.tailc23d6.ts.net` **inside the cluster** | NXDOMAIN — different tailnet, different MagicDNS
suffix. The cluster-side name must be `naina-server.faun-snapper.ts.net` |

  Two consequences worth stating, because both were assumed otherwise:

  - **No ACL change is needed, and no admin-console edit.** The blocker is on the host,
    not in the tailnet policy.
  - **A cluster workload depends on a process on the host for the first time in this
    repo.** That is the honest framing for `OPERATIONS.md` §1, which is the inventory of
    everything Terraform does not own.

  ## The change: one privileged session, two halves

  Neither half works alone. The firewall half is the one that fails silently — the
  symptom from the cluster is `no route to host`, indistinguishable from an ACL denial.

  **Half 1 — the relay unit** (proposed; `/usr/bin/socat` comes from dnf in this same
  session — `socat` is **not installed** on the host today):

  ```ini
  # /etc/systemd/system/llm-relay.service
  [Unit]
  Description=Relay to the shared inference gateway on lain-llm
  After=network-online.target tailscaled.service
  Wants=network-online.target
  Requires=tailscaled.service

  [Service]
  User=naina
  ExecStart=/usr/bin/socat TCP-LISTEN:4000,bind=100.72.140.122,fork,reuseaddr TCP:lain-llm.tailc23d6.ts.net:4000
  Restart=always
  RestartSec=2
  NoNewPrivileges=true
  PrivateTmp=true
  CapabilityBoundingSet=
  RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

  [Install]
  WantedBy=multi-user.target
  • A TCP forwarder, not an HTTP reverse proxy. Traffic inside the tailnet is
    already encrypted and the gateway speaks plain HTTP, so an HTTP-level proxy would
    only add a place to get paths and headers wrong. The pod's own mitmdump reverse
    proxy keeps its job; only its target hostname changes.
  • Bound to the tailnet address, not 0.0.0.0. That is why
    After=/Requires=tailscaled.service is load-bearing, and why Restart=always
    matters: a bind that races tailscaled at boot retries until it succeeds.
  • Targeted by MagicDNS name, so a re-addressed device costs a restart rather than
    an edit.
  • Runs as naina with no capability and no root after installation. The relay is
    unauthenticated and the tailnet ACL is allow-all, so the gateway's API key stays
    the only authentication on this path     — a dumb forwarder is the right shape.

Half 2 — the port opening:

sudo firewall-cmd --permanent --add-port=4000/tcp && sudo firewall-cmd --reload
  • Prefer the single port over --add-interface=tailscale0 --zone=trusted, which opens
    every port on the host to the tailnet.
  • --permanent survives reboots and reloads; a host upgrade that resets zone
    assignments would not, so the drift check below belongs in the runbook.
  • Port 4000 on the host is free (measured: no listener in ss -lnt).

What Terraform can and cannot own

Recommendation: no Terraform root for this relay. The evaluation, in short:

  • dmacvicar/libvirt manages domains, volumes, networks and pools — no host-process
    surface. There is no null_resource, remote-exec, hashicorp/local or SSH use
    anywhere in this tree.
  • The maintained path would be dlarochette/systemd (0.10.x, MIT, single maintainer),
    whose own README assumes root SSH and writes system units only. There is no
    firewalld provider    , so half this change — the half that actually makes it work —
    stays a hand-run command regardless.
  • It needs a privileged identity naina-server does not offer today: sudo -n fails
    for naina (password required) and Fedora's root login is key-only with no root key
    authorized here. That means a root key on a tailnet device, or a passwordless-sudoers
    or polkit rule — a new privileged credential path, in a repo whose rules keep
    credentials out of state.
  • Terraform in this repo is never applied automatically, so the gain is a plan that
    notices drift, for the price of a young third-party provider on the inference
    critical path.

AGENTS.md's rule is "no manual virsh/qemu-img" — it does not require Terraform
for host OS packages, so a documented system unit does not violate it.

Out of scope

  • The cluster-side path (the egress ProxyGroup and the ExternalName Service) is a
    separate issue; this issue ends at a reachable port on the host.
  • Any change to lain-llm and how it is shared.
  • Subnet routing/NAT, an untagged in-cluster device, and tailscale serve are all
    rejected, with reasons, so they do not come back: tailscale serve's target must be
    a service on the local machine, and it would be gated by the same firewalld zone.

Done when

# 1. From another tailnet device (was: no route to host):
tailscale nc naina-server 4000

# 2. End to end, on the host:
curl -s http://100.122.246.96:4000/health | jq -r .status          # ok
journalctl -u llm-relay -n 20                                       # the connection appears here

# 3. Drift check, worth keeping in the runbook:
firewall-cmd --get-active-zones    # tailscale0 must land in a zone that allows 4000

And the rows this change writes down:

  • OPERATIONS.md §1 — a new row naming the unit, naina-server as the host, systemd
    as what restarts it, and the drift check above as how it is verified. This is the
    first row where a running cluster component depends on a host process; the row
    should say so.
  • docs/tailscale.md — a new section, as a sibling of §6, covering the relay, the
    firewall rule, the allowed-reachability decision, and the names the cluster may use.
  • OPERATIONS.md's domain index — the existing docs/tailscale.md row extended to
    mention the relay.

ARCHITECTURE.md is not touched by this issue.

```markdown A cluster workload needs to reach an OpenAI-compatible inference gateway that lives on a **different** tailnet (`lain-llm.tailc23d6.ts.net`, `100.122.246.96`, plain HTTP on `:4000`, key-authenticated). No pod, and no Tailscale egress proxy, can reach it directly. This issue adds the one hop that can: a small TCP relay on `naina-server`, the libvirt host that runs this cluster's VMs. ## Why the cluster cannot reach the gateway itself `lain-llm` is shared as a **machine share to a user** (`vandomej@`), not to a tag. Tailscale shares a machine with a person, and the recipient accepts it; a node authenticated with a tagged auth key is not a person's node, so the share never appears to it. Every device the Tailscale operator creates is tagged **by construction** — the operator authenticates with a `tag:k8s-operator` OAuth client, and a tagged credential can only mint tagged devices. So `tag:k8s` devices — an egress ProxyGroup, a standalone proxy, or any in-cluster pod — cannot see `lain-llm`, and no ACL arrangement changes that. The untagged device that already sees it is `naina-server` (`100.72.140.122`, untagged, owned by `vandomej@`), which is also where the cluster's VMs run. Hence the path: **the cluster reaches the host as an ordinary same-tailnet egress target, and the host relays to `lain-llm`.** ## What was measured, this session, on `naina-server` | Check | Command | Result | |---|---|---| | Host reaches the gateway | `curl -m6 http://100.122.246.96:4000/health` on the host | `{"status":"ok","engine":{"loaded_model":"qwen3.8-27b",...}}` | | The gateway is key-authenticated | `curl -o /dev/null -w '%{http_code}' http://100.122.246.96:4000/v1/models` | `401` | | The host is the right identity | `tailscale ping --c=1 lain-llm.tailc23d6.ts.net` | `pong from lain-llm (100.122.246.96)` | | The cluster's ACL is not the blocker | netmap `PacketFilter` read from the egress/ingress proxy | **one rule: all tailnet sources → all destinations, all ports, proto 6/17/1/58** | | The cluster reaches the host | `kubectl -n tailscale exec ingress-proxies-0 -c tailscale -- tailscale nc naina-server 22` | `CONNECTED` | | The cluster cannot reach the port | same command, port `4000` | `no route to host` | | Why | `firewall-cmd --get-active-zones` (no sudo needed) | `libvirt: cp-bridge virbr0` / `public (default): wlp1s0f0` — **`tailscale0` is in no zone**, so it falls to `public`, which allows ssh only | | The gateway resolves only off-cluster | `getent hosts lain-llm.tailc23d6.ts.net` **inside the cluster** | NXDOMAIN — different tailnet, different MagicDNS suffix. The cluster-side name must be `naina-server.faun-snapper.ts.net` | Two consequences worth stating, because both were assumed otherwise: - **No ACL change is needed, and no admin-console edit.** The blocker is on the host, not in the tailnet policy. - **A cluster workload depends on a process on the host for the first time in this repo.** That is the honest framing for `OPERATIONS.md` §1, which is the inventory of everything Terraform does not own. ## The change: one privileged session, two halves Neither half works alone. The firewall half is the one that fails silently — the symptom from the cluster is `no route to host`, indistinguishable from an ACL denial. **Half 1 — the relay unit** (proposed; `/usr/bin/socat` comes from dnf in this same session — `socat` is **not installed** on the host today): ```ini # /etc/systemd/system/llm-relay.service [Unit] Description=Relay to the shared inference gateway on lain-llm After=network-online.target tailscaled.service Wants=network-online.target Requires=tailscaled.service [Service] User=naina ExecStart=/usr/bin/socat TCP-LISTEN:4000,bind=100.72.140.122,fork,reuseaddr TCP:lain-llm.tailc23d6.ts.net:4000 Restart=always RestartSec=2 NoNewPrivileges=true PrivateTmp=true CapabilityBoundingSet= RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX [Install] WantedBy=multi-user.target ``` - A **TCP forwarder, not an HTTP reverse proxy.** Traffic inside the tailnet is already encrypted and the gateway speaks plain HTTP, so an HTTP-level proxy would only add a place to get paths and headers wrong. The pod's own `mitmdump` reverse proxy keeps its job; only its target hostname changes. - **Bound to the tailnet address**, not `0.0.0.0`. That is why `After=/Requires=tailscaled.service` is load-bearing, and why `Restart=always` matters: a bind that races `tailscaled` at boot retries until it succeeds. - **Targeted by MagicDNS name**, so a re-addressed device costs a restart rather than an edit. - Runs as `naina` with no capability and no root after installation. The relay is unauthenticated and the tailnet ACL is allow-all, so **the gateway's API key stays the only authentication on this path** — a dumb forwarder is the right shape. **Half 2 — the port opening:** ```bash sudo firewall-cmd --permanent --add-port=4000/tcp && sudo firewall-cmd --reload ``` - Prefer the single port over `--add-interface=tailscale0 --zone=trusted`, which opens every port on the host to the tailnet. - `--permanent` survives reboots and reloads; a host **upgrade** that resets zone assignments would not, so the drift check below belongs in the runbook. - Port `4000` on the host is free (measured: no listener in `ss -lnt`). ## What Terraform can and cannot own Recommendation: **no Terraform root for this relay.** The evaluation, in short: - `dmacvicar/libvirt` manages domains, volumes, networks and pools — no host-process surface. There is no `null_resource`, `remote-exec`, `hashicorp/local` or SSH use anywhere in this tree. - The maintained path would be `dlarochette/systemd` (0.10.x, MIT, single maintainer), whose own README assumes **root SSH** and writes system units only. There is **no firewalld provider**, so half this change — the half that actually makes it work — stays a hand-run command regardless. - It needs a privileged identity `naina-server` does not offer today: `sudo -n` fails for `naina` (password required) and Fedora's root login is key-only with no root key authorized here. That means a root key on a tailnet device, or a passwordless-sudoers or polkit rule — a new privileged credential path, in a repo whose rules keep credentials out of state. - Terraform in this repo is never applied automatically, so the gain is a `plan` that notices drift, for the price of a young third-party provider on the inference critical path. `AGENTS.md`'s rule is "no manual `virsh`/`qemu-img`" — it does not require Terraform for host OS packages, so a documented system unit does not violate it. ## Out of scope - The cluster-side path (the egress ProxyGroup and the ExternalName Service) is a separate issue; this issue ends at a reachable port on the host. - Any change to `lain-llm` and how it is shared. - Subnet routing/NAT, an untagged in-cluster device, and `tailscale serve` are all rejected, with reasons, so they do not come back: `tailscale serve`'s target must be a service on the local machine, and it would be gated by the same firewalld zone. ## Done when ```bash # 1. From another tailnet device (was: no route to host): tailscale nc naina-server 4000 # 2. End to end, on the host: curl -s http://100.122.246.96:4000/health | jq -r .status # ok journalctl -u llm-relay -n 20 # the connection appears here # 3. Drift check, worth keeping in the runbook: firewall-cmd --get-active-zones # tailscale0 must land in a zone that allows 4000 ``` And the rows this change writes down: - `OPERATIONS.md` §1 — a new row naming the unit, `naina-server` as the host, systemd as what restarts it, and the drift check above as how it is verified. This is the first row where a *running* cluster component depends on a host process; the row should say so. - `docs/tailscale.md` — a new section, as a sibling of §6, covering the relay, the firewall rule, the allowed-reachability decision, and the names the cluster may use. - `OPERATIONS.md`'s domain index — the existing `docs/tailscale.md` row extended to mention the relay. `ARCHITECTURE.md` is not touched by this issue. ```
tepichord added this to the MVP project 2026-09-22 13:59:37 -07:00
tepichord commented 2026-09-22 14:38:45 -07:00
Author
Owner
Copy link

Naina:

Overview

A cluster workload must reach the OpenAI-compatible gateway lain-llm.tailc23d6.ts.net:4000, which is machine-shared to vandomej@ and therefore invisible to every tagged device the Tailscale operator can mint. This issue documents the hand-run fix: a socat TCP relay on naina-server — the untagged libvirt host that already sees the gateway — plus a firewalld opening for port 4000. unverified Documentation only; the relay and the firewall rule are applied by hand on the host, and ARCHITECTURE.md is untouched.

Technical Approach

  1. Add docs/tailscale.md §6b, a sibling of §6 — "Reach the shared gateway from the cluster". Lead with the reason, because it is why no ACL edit helps:

    lain-llm is machine-shared to vandomej@. Tailscale shares a machine with a user, and tagging a device replaces its user identity, so a tag:k8s device can never see it.1

    Record the symptoms and their bound:

    getent hosts lain-llm.tailc23d6.ts.net   # NXDOMAIN in-cluster: different tailnet, different MagicDNS suffix
kubectl -n tailscale exec ingress-proxies-0 -c tailscale -- tailscale nc naina-server 22    # CONNECTED
kubectl -n tailscale exec ingress-proxies-0 -c tailscale -- tailscale nc naina-server 4000  # no route to host

    State the names the cluster may use: naina-server.faun-snapper.ts.net:4000 (own tailnet). The cluster-side consumer is #87.

  2. Document the relay unit — /etc/systemd/system/llm-relay.service, applied by hand:

    [Unit]
Description=Relay to the shared inference gateway on lain-llm
After=network-online.target tailscaled.service
Wants=network-online.target
Requires=tailscaled.service
StartLimitIntervalSec=0

[Service]
User=naina
ExecStart=/usr/bin/socat -d TCP-LISTEN:4000,bind=100.72.140.122,fork,reuseaddr,max-children=64 TCP:lain-llm.tailc23d6.ts.net:4000
Restart=always
RestartSec=2
NoNewPrivileges=true
PrivateTmp=true
CapabilityBoundingSet=
RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK

[Install]
WantedBy=multi-user.target

    Four deltas from the unit in the issue body, each load-bearing:

    Change Why
    -d socat's default level logs no connection; the issue's own journalctl -u llm-relay check shows nothing without it.2
    StartLimitIntervalSec=0 the default is 5 starts / 10s3; with RestartSec=2 a boot bind race burns the budget and the relay stays dead until systemctl reset-failed
    +AF_NETLINK socat resolves the target via getaddrinfo, which opens a NETLINK_ROUTE socket4; without it the failure looks like a DNS problem
    max-children=64 unauthenticated on an allow-all tailnet; an unbounded fork is the cheapest way for any device to exhaust the host's process limit

    Add: Requires=tailscaled.service does not guarantee 100.72.140.122 is bindable (tailscaled is "ready" when LocalAPI is up, not when the address exists), so Restart=always is the mechanism that wins the race.

  3. Document the firewall half, and the reachability semantics

    sudo firewall-cmd --permanent --add-port=4000/tcp && sudo firewall-cmd --reload

    • tailscale0 is in no zone, so it inherits the default public — which is why --get-active-zones lists it under nothing.5

    • public has target: default → reject6, which is why the probe returned no route to host (EHOSTUNREACH)7 rather than hanging — indistinguishable from an ACL denial.

    • Zones are one-to-many: wlp1s0f0 is also in public, so the port opens on both interfaces. The bound bind=100.72.140.122 is the compensating control; say so. Scoped alternative if that is not enough:

      sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=100.64.0.0/10 port port=4000 protocol=tcp accept'
sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv6 source address=fd7a:115c:a1e0::/48 port port=4000 protocol=tcp accept'

    • Reject --add-interface=tailscale0 --zone=trusted: trusted accepts every port, NetworkManager can reassign the zone (tailscale#3416), and tailscale#15968 reports the interface going DOWN.

  4. Document the drift check — also the §1 row's verification:

    firewall-cmd --get-active-zones
sudo firewall-cmd --list-ports --zone="$(firewall-cmd --get-zone-of-interface=tailscale0)"

    It must assert both which zone tailscale0 lands in and that the zone allows 4000 — --list-ports --zone=public alone silently passes if DefaultZone ever changes.

  5. Add the OPERATIONS.md §1 row8 — the first row where a running cluster component depends on a host process:

    | LLM relay `llm-relay.service` on `naina-server` (`socat` TCP 4000 → `lain-llm.tailc23d6.ts.net:4000`; the only path to the shared inference gateway) | **hand-run on the libvirt host** — systemd unit + `firewall-cmd --add-port=4000/tcp` (docs/tailscale.md §6b). **The first running cluster component that depends on a host process:** the cluster reaches the gateway only while the host unit and its firewall opening are both up. Verified by `firewall-cmd --get-active-zones` plus that zone's `--list-ports` |

  6. Note in OPERATIONS.md §89 — no renumbering of its steps:

    > Host-side services are not part of this order: the relay on `naina-server`
> (docs/tailscale.md §6b) lives outside the cluster and survives a cluster
> wipe. Re-run its install only if the host was rebuilt.

  7. Extend the domain index row10

    | [docs/tailscale.md](docs/tailscale.md) | §2 Tailscale admin console, §3 operator bootstrap, §6 client devices, §6b relay to the shared gateway |

Implementation Details

Files

File Change
docs/tailscale.md new §6b after §6 (the last section)11
OPERATIONS.md one §1 row, one §8 note, one domain-index row

Nothing under gitops/, control-plane/, worker/, data/ or cloudflare/; ARCHITECTURE.md untouched.

Constraints

  • AGENTS.md's rule is "no manual virsh/qemu-img" — host OS packages and system units are not in that set, so a documented unit does not violate it.12
  • tests/test_docs.py::test_runbook_carries_no_volatile_identity_literals — the new prose must carry no 32+ hex literal, UUID, talos-* name or pvc-* name.13
  • No Terraform root: there is no firewalld provider14, and dlarochette/systemd needs root SSH naina-server does not offer unverified. Frame as documented, not automated.
  • No secrets in these docs — the gateway's API key never appears.

Out-of-scope notes

  • The cluster half (egress ProxyGroup + ExternalName Service) is #87.
  • tailscale serve is rejected: its target must be a service on the local machine, and it hits the same firewalld zone.
  • Alternative not chosen: systemd-socket-proxyd on a FreeBind=yes .socket removes the boot race entirely; socat stays because the issue chose it and it is one command.

Acceptance Criteria

  • docs/tailscale.md §6b exists as a sibling of §6 and covers the machine-share reason, the in-cluster symptoms, the unit, the firewall rule, the reachability/trusted decision, the names the cluster may use, and the drift check.
  • The documented unit carries -d, StartLimitIntervalSec=0, max-children and AF_NETLINK, and is otherwise behaviourally identical to the unit in the issue body.
  • OPERATIONS.md §1 has one new row naming llm-relay.service, naina-server, systemd, the drift check, and that it is the first running cluster component depending on a host process.
  • OPERATIONS.md §8 carries the host-side note, and the domain index row for docs/tailscale.md mentions the relay.
  • pytest tests/test_docs.py passes.

Testing Plan

  • pytest tests/test_docs.py — the runbook guards.
  • On naina-server, after the documented install: systemctl is-active llm-relay is active, and journalctl -u llm-relay -n 20 shows one line per connection (-d is what makes this check meaningful).
  • From another tailnet device: tailscale nc naina-server 4000 reaches CONNECTED (was no route to host).
  • The drift check returns the zone holding tailscale0 and that zone lists 4000/tcp.
  • Reboot the host once and confirm the relay is active afterwards — the Restart=always race.
  • sudo ausearch -m avc -ts recent shows no denials for llm-relay, and systemd-analyze security llm-relay.service is run once and recorded.

Related Issues/PRs

  • #86 — this issue: the host half (relay + firewall rule).
  • #87 — Create Tailscale egress from the cluster: the cluster half, which begins where this ends.
  • docs/tailscale.md §2/§3 — no admin-console or ACL change is part of this work.
  • AGENTS.md Hard Rules — the "no manual virsh/qemu-img" rule, read as not covering host OS packages.

Research Findings

  • socat is an official Fedora package with first-class aarch64 builds (socat-1.8.1.1-*), so plain dnf install socat is correct on Fedora Asahi. CVE-2024-54661 affects only versions below 1.8.0.3, so the runbook records rpm -q socat.
  • firewalld's default zone (public) is what an unassigned interface inherits, which is why tailscale0 never appears in --get-active-zones. Zones are one-to-many, so the opening is not scoped to the tailnet interface and the relay's bind= is the compensating control.
  • --add-interface=tailscale0 --zone=trusted is worse than one port: trusted accepts every host port, NetworkManager can reassign the zone (tailscale#3416), and tailscale#15968 reports the interface going DOWN.
  • systemd's start rate limiting (5 starts / 10s by default) can leave the relay dead after a boot bind race, defeating the Restart=always the unit relies on.
  • tailscaled signals "ready" before its tailnet address is bindable (tailscale#11504, closed as documented behaviour), so the 100.72.140.122 bind only ever succeeds by retry.
  • RestrictAddressFamilies must include AF_NETLINK: glibc's getaddrinfo/getifaddrs opens a NETLINK_ROUTE socket, and omitting it fails as something that looks like a DNS error.
  • Tailscale's sharing documentation states a machine cannot be shared with, or accessed by, tagged machines on another tailnet, and the feature request to lift that (#5321) is still open — so no ACL change can fix this.
  • systemd-socket-proxyd on a FreeBind=yes socket was the considered alternative; it removes the boot race, but socat remains the simpler, chosen design.

Pre-Mortem

  1. The docs-only change is unfalsifiable in the pipeline, so a broken relay ships as "done" — pytest tests/test_docs.py is the only automated check and it verifies nothing about the relay, while the plan's real done-when criteria (tailscale nc ... CONNECTED, the reboot survival test, ausearch) are hand-run on a host no agent can reach. A runbook whose unit was never installed passes review, and issue #87's cluster half is then built against a gateway that is still unreachable.

  2. The load-bearing unit deltas were inferred from research rather than a running relay — -d, StartLimitIntervalSec=0, max-children=64, AF_NETLINK and the belief that Requires=tailscaled.service plus Restart=always wins the bind race are all documented as fact but never validated on naina-server. If any inference is wrong, the relay dies silently after a reboot and the runbook confidently instructs the next operator to reproduce the failure.

  3. The hand-run host state drifts from the committed docs and nothing observes it — the relay is neither Terraform- nor Flux-managed, so bind=100.72.140.122, the public-zone default and the User=naina account are unversioned host facts that a rebuild, a tailnet address change, or a DefaultZone change can invalidate. The plan's entire value rests on those literals matching the host, and the mismatch only surfaces when the cluster-side consumer is tested.

Naina: ## Overview A cluster workload must reach the OpenAI-compatible gateway `lain-llm.tailc23d6.ts.net:4000`, which is machine-shared to `vandomej@` and therefore invisible to every tagged device the Tailscale operator can mint. This issue documents the hand-run fix: a `socat` TCP relay on `naina-server` — the untagged libvirt host that already sees the gateway — plus a firewalld opening for port 4000. _<sup>unverified</sup>_ Documentation only; the relay and the firewall rule are applied by hand on the host, and `ARCHITECTURE.md` is untouched. ## Technical Approach 1. **Add `docs/tailscale.md` §6b, a sibling of §6** — *"Reach the shared gateway from the cluster"*. Lead with the reason, because it is why no ACL edit helps: > `lain-llm` is machine-shared to `vandomej@`. Tailscale shares a machine with a *user*, and tagging a device replaces its user identity, so a `tag:k8s` device can never see it.<sup>[1](https://tailscale.com/docs/features/sharing)</sup> Record the symptoms and their bound: ```bash getent hosts lain-llm.tailc23d6.ts.net # NXDOMAIN in-cluster: different tailnet, different MagicDNS suffix kubectl -n tailscale exec ingress-proxies-0 -c tailscale -- tailscale nc naina-server 22 # CONNECTED kubectl -n tailscale exec ingress-proxies-0 -c tailscale -- tailscale nc naina-server 4000 # no route to host ``` State the names the cluster may use: `naina-server.faun-snapper.ts.net:4000` (own tailnet). The cluster-side consumer is #87. 2. **Document the relay unit** — `/etc/systemd/system/llm-relay.service`, applied by hand: ```ini [Unit] Description=Relay to the shared inference gateway on lain-llm After=network-online.target tailscaled.service Wants=network-online.target Requires=tailscaled.service StartLimitIntervalSec=0 [Service] User=naina ExecStart=/usr/bin/socat -d TCP-LISTEN:4000,bind=100.72.140.122,fork,reuseaddr,max-children=64 TCP:lain-llm.tailc23d6.ts.net:4000 Restart=always RestartSec=2 NoNewPrivileges=true PrivateTmp=true CapabilityBoundingSet= RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX AF_NETLINK [Install] WantedBy=multi-user.target ``` Four deltas from the unit in the issue body, each load-bearing: | Change | Why | |---|---| | `-d` | socat's default level logs no connection; the issue's own `journalctl -u llm-relay` check shows nothing without it.<sup>[2](http://www.dest-unreach.org/socat/doc/socat.html)</sup> | | `StartLimitIntervalSec=0` | the default is 5 starts / 10s<sup>[3](https://www.freedesktop.org/software/systemd/man/latest/systemd-system.conf.html)</sup>; with `RestartSec=2` a boot bind race burns the budget and the relay stays dead until `systemctl reset-failed` | | `+AF_NETLINK` | socat resolves the target via `getaddrinfo`, which opens a `NETLINK_ROUTE` socket<sup>[4](https://sourceware.org/git/?p=glibc.git;a=blob;f=sysdeps/unix/sysv/linux/ifaddrs.c;hb=HEAD)</sup>; without it the failure looks like a DNS problem | | `max-children=64` | unauthenticated on an allow-all tailnet; an unbounded `fork` is the cheapest way for any device to exhaust the host's process limit | Add: `Requires=tailscaled.service` does **not** guarantee `100.72.140.122` is bindable (`tailscaled` is "ready" when LocalAPI is up, not when the address exists), so `Restart=always` is the mechanism that wins the race. 3. **Document the firewall half, and the reachability semantics** ```bash sudo firewall-cmd --permanent --add-port=4000/tcp && sudo firewall-cmd --reload ``` - `tailscale0` is in no zone, so it inherits the default `public` — which is why `--get-active-zones` lists it under nothing.<sup>[5](https://firewalld.org/documentation/zone/default-zone)</sup> - `public` has `target: default` → **reject**<sup>[6](https://firewalld.org/documentation/zone/options)</sup>, which is why the probe returned `no route to host` (EHOSTUNREACH)<sup>[7](https://firewalld.org/documentation/zone/predefined-zones)</sup> rather than hanging — indistinguishable from an ACL denial. - Zones are one-to-many: `wlp1s0f0` is also in `public`, so the port opens on both interfaces. The bound `bind=100.72.140.122` is the compensating control; say so. Scoped alternative if that is not enough: ```bash sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv4 source address=100.64.0.0/10 port port=4000 protocol=tcp accept' sudo firewall-cmd --permanent --add-rich-rule='rule family=ipv6 source address=fd7a:115c:a1e0::/48 port port=4000 protocol=tcp accept' ``` - Reject `--add-interface=tailscale0 --zone=trusted`: `trusted` accepts every port, NetworkManager can reassign the zone (tailscale#3416), and tailscale#15968 reports the interface going DOWN. 4. **Document the drift check** — also the §1 row's verification: ```bash firewall-cmd --get-active-zones sudo firewall-cmd --list-ports --zone="$(firewall-cmd --get-zone-of-interface=tailscale0)" ``` It must assert *both* which zone `tailscale0` lands in and that the zone allows 4000 — `--list-ports --zone=public` alone silently passes if `DefaultZone` ever changes. 5. **Add the `OPERATIONS.md` §1 row**<sup>[8](https://git.gloom.garden/tepichord/annia-infra/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/OPERATIONS.md?display=source#L20)</sup> — the first row where a *running* cluster component depends on a host process: ```markdown | LLM relay `llm-relay.service` on `naina-server` (`socat` TCP 4000 → `lain-llm.tailc23d6.ts.net:4000`; the only path to the shared inference gateway) | **hand-run on the libvirt host** — systemd unit + `firewall-cmd --add-port=4000/tcp` (docs/tailscale.md §6b). **The first running cluster component that depends on a host process:** the cluster reaches the gateway only while the host unit and its firewall opening are both up. Verified by `firewall-cmd --get-active-zones` plus that zone's `--list-ports` | ``` 6. **Note in `OPERATIONS.md` §8**<sup>[9](https://git.gloom.garden/tepichord/annia-infra/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/OPERATIONS.md?display=source#L56)</sup> — no renumbering of its steps: ```markdown > Host-side services are not part of this order: the relay on `naina-server` > (docs/tailscale.md §6b) lives outside the cluster and survives a cluster > wipe. Re-run its install only if the host was rebuilt. ``` 7. **Extend the domain index row**<sup>[10](https://git.gloom.garden/tepichord/annia-infra/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/OPERATIONS.md?display=source#L125)</sup> ```markdown | [docs/tailscale.md](docs/tailscale.md) | §2 Tailscale admin console, §3 operator bootstrap, §6 client devices, §6b relay to the shared gateway | ``` ## Implementation Details **Files** | File | Change | |---|---| | `docs/tailscale.md` | new §6b after §6 (the last section)<sup>[11](https://git.gloom.garden/tepichord/annia-infra/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/docs/tailscale.md?display=source#L156)</sup> | | `OPERATIONS.md` | one §1 row, one §8 note, one domain-index row | Nothing under `gitops/`, `control-plane/`, `worker/`, `data/` or `cloudflare/`; `ARCHITECTURE.md` untouched. **Constraints** - `AGENTS.md`'s rule is "no manual `virsh`/`qemu-img`" — host OS packages and system units are not in that set, so a documented unit does not violate it.<sup>[12](https://git.gloom.garden/tepichord/annia-infra/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/AGENTS.md?display=source#L31)</sup> - `tests/test_docs.py::test_runbook_carries_no_volatile_identity_literals` — the new prose must carry no 32+ hex literal, UUID, `talos-*` name or `pvc-*` name.<sup>[13](https://git.gloom.garden/tepichord/annia-infra/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/tests/test_docs.py?display=source#L14)</sup> - No Terraform root: there is no firewalld provider<sup>[14](https://git.gloom.garden/tepichord/annia-infra/src/commit/354ba0bf8fb52de95c086e0e28d10e8a4e3a8882/control-plane/main.tf?display=source#L5)</sup>, and `dlarochette/systemd` needs root SSH `naina-server` does not offer _<sup>unverified</sup>_. Frame as **documented, not automated**. - No secrets in these docs — the gateway's API key never appears. **Out-of-scope notes** - The cluster half (egress ProxyGroup + `ExternalName` Service) is #87. - `tailscale serve` is rejected: its target must be a service on the local machine, and it hits the same firewalld zone. - Alternative not chosen: `systemd-socket-proxyd` on a `FreeBind=yes` `.socket` removes the boot race entirely; `socat` stays because the issue chose it and it is one command. ## Acceptance Criteria - [ ] `docs/tailscale.md` §6b exists as a sibling of §6 and covers the machine-share reason, the in-cluster symptoms, the unit, the firewall rule, the reachability/`trusted` decision, the names the cluster may use, and the drift check. - [ ] The documented unit carries `-d`, `StartLimitIntervalSec=0`, `max-children` and `AF_NETLINK`, and is otherwise behaviourally identical to the unit in the issue body. - [ ] `OPERATIONS.md` §1 has one new row naming `llm-relay.service`, `naina-server`, systemd, the drift check, and that it is the first running cluster component depending on a host process. - [ ] `OPERATIONS.md` §8 carries the host-side note, and the domain index row for `docs/tailscale.md` mentions the relay. - [ ] `pytest tests/test_docs.py` passes. ## Testing Plan - `pytest tests/test_docs.py` — the runbook guards. - On `naina-server`, after the documented install: `systemctl is-active llm-relay` is `active`, and `journalctl -u llm-relay -n 20` shows one line per connection (`-d` is what makes this check meaningful). - From another tailnet device: `tailscale nc naina-server 4000` reaches `CONNECTED` (was `no route to host`). - The drift check returns the zone holding `tailscale0` and that zone lists `4000/tcp`. - Reboot the host once and confirm the relay is `active` afterwards — the `Restart=always` race. - `sudo ausearch -m avc -ts recent` shows no denials for `llm-relay`, and `systemd-analyze security llm-relay.service` is run once and recorded. ## Related Issues/PRs - #86 — this issue: the host half (relay + firewall rule). - #87 — Create Tailscale egress from the cluster: the cluster half, which begins where this ends. - `docs/tailscale.md` §2/§3 — no admin-console or ACL change is part of this work. - `AGENTS.md` Hard Rules — the "no manual `virsh`/`qemu-img`" rule, read as not covering host OS packages. ## Research Findings - `socat` is an official Fedora package with first-class aarch64 builds (`socat-1.8.1.1-*`), so plain `dnf install socat` is correct on Fedora Asahi. CVE-2024-54661 affects only versions below `1.8.0.3`, so the runbook records `rpm -q socat`. - firewalld's default zone (`public`) is what an unassigned interface inherits, which is why `tailscale0` never appears in `--get-active-zones`. Zones are one-to-many, so the opening is not scoped to the tailnet interface and the relay's `bind=` is the compensating control. - `--add-interface=tailscale0 --zone=trusted` is worse than one port: `trusted` accepts every host port, NetworkManager can reassign the zone (tailscale#3416), and tailscale#15968 reports the interface going DOWN. - systemd's start rate limiting (5 starts / 10s by default) can leave the relay dead after a boot bind race, defeating the `Restart=always` the unit relies on. - `tailscaled` signals "ready" before its tailnet address is bindable (tailscale#11504, closed as documented behaviour), so the `100.72.140.122` bind only ever succeeds by retry. - `RestrictAddressFamilies` must include `AF_NETLINK`: glibc's `getaddrinfo`/`getifaddrs` opens a `NETLINK_ROUTE` socket, and omitting it fails as something that looks like a DNS error. - Tailscale's sharing documentation states a machine cannot be shared with, or accessed by, tagged machines on another tailnet, and the feature request to lift that (#5321) is still open — so no ACL change can fix this. - `systemd-socket-proxyd` on a `FreeBind=yes` socket was the considered alternative; it removes the boot race, but `socat` remains the simpler, chosen design. ## Pre-Mortem 1. **The docs-only change is unfalsifiable in the pipeline, so a broken relay ships as "done"** — `pytest tests/test_docs.py` is the only automated check and it verifies nothing about the relay, while the plan's real done-when criteria (`tailscale nc ... CONNECTED`, the reboot survival test, `ausearch`) are hand-run on a host no agent can reach. A runbook whose unit was never installed passes review, and issue #87's cluster half is then built against a gateway that is still unreachable. 2. **The load-bearing unit deltas were inferred from research rather than a running relay** — `-d`, `StartLimitIntervalSec=0`, `max-children=64`, `AF_NETLINK` and the belief that `Requires=tailscaled.service` plus `Restart=always` wins the bind race are all documented as fact but never validated on `naina-server`. If any inference is wrong, the relay dies silently after a reboot and the runbook confidently instructs the next operator to reproduce the failure. 3. **The hand-run host state drifts from the committed docs and nothing observes it** — the relay is neither Terraform- nor Flux-managed, so `bind=100.72.140.122`, the `public`-zone default and the `User=naina` account are unversioned host facts that a rebuild, a tailnet address change, or a `DefaultZone` change can invalidate. The plan's entire value rests on those literals matching the host, and the mismatch only surfaces when the cluster-side consumer is tested.
tepichord 2026-09-23 18:58:21 -07:00
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
main
feat/109-nix-base
feat/109-layered-nix-store
feat/110-zot-push-credential
feat/90-annia-pause-lever-oz-console
feat/87-tailscale-egress-from-cluster
chore/ci-step-resources
docs/vault-secret-prompt
feat/42-support-backups-persistent-data
feat/41-integrate-restic-tfstate-backup
feat/3-setting-up-agent-development
kvm-kubernetes-setup
No results found.
Labels
Clear labels   
blocked

   
bug

Something is not working

  
duplicate

This issue or pull request already exists

  
effort
high

  
effort
low

  
effort
medium

  
effort
minimal

  
enhancement

New feature

  
help wanted

Need some help

  
invalid

Something is wrong

  
pr
feedback
Used for PRs, indicates that feedback has been given and agent needs to make changes

  
pr
reviewing
Used for PRs, indicates that a human is waiting to review the pull request, no action needed by the agent

  
question

More information is needed

  
status
backlog
Ignore, no work needed yet, human will assign this work by changing to status/planning

  
status
document
Record findings, create flashcards, and document knowledge in knowledgebase for the future

  
status
done
Completed when PR is merged, no more work needed.

  
status
implement
Make the code changes. Should be done by a faster and cheaper model

  
status
pending
Give human a chance to review planning details

  
status
planning
Ready to begin. Fill out implementation details and things like acceptance criteria, etc. A smarter thoughtful llm should be delegated for this

  
status
reviewing
PR is created and feedback loop begins between human and agent

  
waiting

   
wontfix

This won't be fixed

No labels
blocked
bug
duplicate
effort
high
effort
low
effort
medium
effort
minimal
enhancement
help wanted
invalid
pr
feedback
pr
reviewing
question
status
backlog
status
document
status
done
status
implement
status
pending
status
planning
status
reviewing
waiting
wontfix
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
MVP
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
tepichord/milner#86
No description provided.