Reach the tailnet-shared LLM gateway

tepichord commented

2026-09-22 16:57:45 -07:00

Owner

Closes #86

Automated PR created by forgejo-done adapter.

Closes #86 Automated PR created by forgejo-done adapter.

tepichord added 2 commits

2026-09-22 16:57:45 -07:00

docs: add §6b relay runbook and OPERATIONS.md entries for the LLM gateway 47d626d7f8

- Add docs/tailscale.md §6b covering the socat TCP relay on naina-server,
  the firewalld port opening, why no ACL change helps, the drift check,
  and the names the cluster may use.
- Add §1 row in OPERATIONS.md naming llm-relay.service as the first
  running cluster component that depends on a host process.
- Add §8 blockquote note explaining host-side services survive cluster wipes.
- Extend the domain index row for docs/tailscale.md to include §6b.

Refs: #86

fix(docs): make the relay drift check observe the relay and its zone

ci/woodpecker/pr/gates Pipeline was successful

Details

66cb8be26f

The §6b drift check piped `firewall-cmd --get-active-zones` through
`grep -A1 tailscale`, which prints nothing: active zones are the zones with
a binding, and `tailscale0` has none — the section says so itself. A check
whose first line cannot answer the question it asks is a check that always
looks clean.

Name the zone that actually governs the interface (`--get-default-zone`,
which is what an unbound interface inherits), keep `--get-active-zones` so a
zone someone actually binds `tailscale0` to is visible, and check the unit
too: §1's row promises the check covers the host unit as well as the opening,
and the port can be open while the relay is dead.

tepichord added the

pr

reviewing

label

2026-09-22 16:57:45 -07:00

tepichord added this to the MVP project

2026-09-22 19:50:07 -07:00

tepichord added 13 commits

2026-09-23 18:52:11 -07:00

feat(infrastructure): add egress-proxies ProxyGroup to tailscale.yaml 36364642a0

Refs: #87

feat(infrastructure): add annia namespace with baseline PSA 2252b9fcba

Refs: #87

feat(apps): add llm-gateway ExternalName Service in annia namespace fe692f8082

Refs: #87

test(gitops): add egress ProxyGroup and llm-gateway shape guards caa71e0e2a

Refs: #87

docs(tailscale): add §17 cluster egress section 7a544627dc

Refs: #87

docs(tests): trim verbose docstrings on egress test functions c0c0db8d24

Refs: #87

docs(tailscale): correct the annia PSA rationale and cite story 14 5e1bb68740

The §17 placement rationale claimed `annia`'s `baseline` label is what
admits the egress proxy pods. It is not: the operator runs ProxyGroup pods
in its own `tailscale` namespace (§3a), the one whose `privileged` label
covers them, so `annia`'s label governs annia's own workloads only.

Also record why the Service sits in the consumer's namespace — story 14's
"no real credential and no real power", so a preview path to an LLM is a
stub — and the fourth rule from the issue: one egress ProxyGroup serves
every egress Service.

Refs: #87

test(gitops): assert the egress Service's externalName, fix a stale rationale

ci/woodpecker/pr/gates Pipeline was successful

Details

ci/woodpecker/pull_request_closed/cleanup Pipeline was successful

Details

b9d42aac7d

The shape guard's docstring names the `externalName` rule it did not
check: an ExternalName Service the API server accepts needs a non-empty
one, and the literal is the human record of intent the operator later
overwrites. Assert it, so dropping the field fails offline rather than at
admission.

The namespace guard's docstring also attributed the `baseline` label to
the egress proxy pods; those run in the operator's `privileged` `tailscale`
namespace (§3a). Say what the label actually governs.

Refs: #87

Merge #87 (cluster egress) into the #86 relay branch 4eb5cbafb7

#86 documents the host relay; #87 adds the cluster-side egress that uses it.
Both appended to docs/tailscale.md, so the whole conflict was the section
itself: §6b (host relay) and §17 (cluster egress) are two halves of one path,
and §17 was numbered against no scheme the file uses (it has §2/§3/§6/§6b).
Merging them as two sections would have kept the halves — and the reader —
apart.

Resolution:
- One §6b covering both halves: why the gateway is unreachable from the
  cluster (a machine shared to a user cannot be reached by a tag:k8s device,
  per Tailscale's sharing rules and the operator's egress Service contract),
  the host relay (§6b-1), the firewall (§6b-2), and the egress ProxyGroup +
  ExternalName Service (§6b-3).
- The relay unit is committed at host/llm-relay.service rather than inlined;
  §6b-1 installs that file (copy or symlink). tests/test_docs.py fails if the
  runbook inlines a second copy again, since the lines this unit was hardened
  for fail silently.
- §2.5 -> §2 step 5: the old reference pointed at a heading that does not
  exist.

Surfaces updated: OPERATIONS.md §1 (relay is a committed unit; the egress
ProxyGroup and llm-gateway Service are Flux-owned) and §8; ARCHITECTURE.md §7
network paragraph, diagram and ports table; AGENTS.md repo map.

host: take socat from home-manager, and make nix-first the stated rule 9b0a47fd33

The relay's socat was installed with `dnf` and named as /usr/bin/socat, so the
binary the unit actually ran was invisible to this repo: nothing here said
where it came from, and it could not drift *visibly* from the dev shell's
nixpkgs. That is the failure every runbook guard here exists to stop, one level
down.

- host/llm-relay.service runs /home/naina/.nix-profile/bin/socat, an absolute
  path into the home-manager profile. A systemd *system* unit does not read a
  user's profile, so the path has to be spelled out, and the profile is a GC
  root rather than a bare store path a later nixpkgs bump would orphan.
- The rule is stated where an operator and an agent read it (AGENTS.md Code
  Conventions, host/README.md), with the one exception it has: libvirtd and its
  firmware, which nix cannot place as the host's own system service.
- README's libvirt install now says why it is the exception, inline with the
  command rather than in a paragraph the command can be separated from.
- tests/test_docs.py asserts all three halves: the rule is stated, the unit
  runs a nix profile path, and every `dnf install` left in the repo names nix
  in the same block. A silent second rpm is what that fails on.

Companion change outside the repo: socat added to home.packages in
~/.config/home-manager/home.nix.

docs(tailscale): verify §6b-3 the way the path actually behaves f5b469b8c4

Three corrections from checking it against the live path, not from reading it
back:

- The Service check used `kubectl run --image=curlimages/curl:8.11.1`. A pod
  dialing its own Service ClusterIP is the hairpin case flannel does not do, and
  the runbook was about to tell a reader to run it from the egress proxy. The
  example now says why it has to be another pod, and uses one that is present
  (`pocket-id-0`) instead of an unpulled tag.
- `tailscale nc` takes no `-w`, and prints nothing on success rather than
  "CONNECTED"; it prints `no route to host` on failure. The comment claimed
  output that never appears.
- The drift check's curl hit `100.122.246.96` — the gateway directly, which the
  host reaches without the relay, so it proved the gateway and not the thing
  being drilled. It now probes the relay's own address, and the direct check is
  dropped rather than mislabelled.

Verified live: DNS resolves llm-gateway -> ts-llm-gateway-njqs4 -> 10.107.79.69
and a request from pocket-id-0 returns the gateway's /health.

fix(host): run llm-relay as a user unit, because SELinux forbids a system one 7964a00acc

Installed as a system unit the relay was silently dead. A system unit runs
in init_t, which may not read the user_home_t symlink the home-manager
profile path is reached through, so the exec was refused before socat
started:

  avc: denied { read } for comm="(socat)" name=".nix-profile"
    scontext=system_u:system_r:init_t:s0
    tcontext=unconfined_u:object_r:user_home_t:s0 tclass=lnk_file
  Failed at step EXEC ... status=203/EXEC

systemd reports that as `activating (auto-restart)` — a unit that looks
alive and serves nothing — so `is-active` alone reads the dead relay as
fine. A user manager runs unconfined and reads the profile normally; the
alternative, relabelling the nix store or the home symlinks so init_t may
execute it, is a host-wide policy change for one relay.

The unit is a user unit now, with `loginctl enable-linger` as part of the
install rather than a convenience: without linger a user service runs only
while the user is logged in, so the relay would vanish at logout. Ordering
after tailscaled.service goes with it — a user unit cannot order after a
system unit — which the runbook already argued Restart=always covers.

A guard holds both halves, because the failure is one that reports itself
as healthy.

fix(host): relay at -d -d, because -d logs no connection at all

ci/woodpecker/pr/gates Pipeline was successful

Details

ci/woodpecker/pull_request_closed/cleanup Pipeline was successful

Details

748c354782

The unit said `-d` was load-bearing so that `journalctl --user -u
llm-relay` would show the relay being used, and §6b-3 gave that as its
proof. Measured, it is not true: at socat's default level and at a single
`-d` a served connection adds nothing to the journal, so the check it was
written for — "the connection appears here" — can never pass, and reads a
busy relay as an idle one.

`-d -d` is the level that logs it, one `accepting connection from AF=2
<peer> on <listen>` and one `opening connection to <target>` per
connection. That is the check §6b-3 asks a reader to make, so it is the
level the unit runs at now.

tepichord merged commit 03e7e21def into main

2026-09-23 18:58:21 -07:00

tepichord deleted branch feat/86-reach-tailnet-shared-llm-gateway

2026-09-23 18:58:21 -07:00

tepichord referenced this pull request from a commit

2026-09-23 18:58:23 -07:00

Merge pull request 'Reach the tailnet-shared LLM gateway' (#93) from feat/86-reach-tailnet-shared-llm-gateway into main

tepichord removed the

pr

reviewing

label

2026-09-23 18:58:25 -07:00