This issue was "define the IaC code to create a pod running syncthing on Milner". That is still the core, but the pod acquires its purpose and a hard consumer, so the scope below is what the plan should cover. The body's one-line framing is superseded; the issue is not.

1. What this ships

An always-on Syncthing pod in annia, running the official upstream image at a pinned tag, on a single persistent-class ReadWriteOnce PVC that holds one folder: the operator's ~/.pi — Syncthing's folder id naccs-5znau, label pi. Send/receive, with the operator's dima (macOS) and this host (fedora) as the two registered peers. That is the whole of it: a service, a claim, a folder, and the peers.

2. Why it is now load-bearing

The annia pod (milner#25) mounts that same PVC at /root/.pi, whole-volume, no subPath. /root/.pi is not an annia artefact — it is the operator's folder, ~1.1 GB of pi credentials, skills, extensions and session history — and the decision is that the pod gets the real thing rather than a copy or a subset. Syncthing is the transport that keeps the pod's copy and the operator's machines one folder. Concretely:

Two pods, one claim, no sidecar. Syncthing runs only in this pod; the annia pod just mounts. That keeps the extra credential out of annia and means the annia image carries no sync daemon.
RWO ⇒ co-scheduling is mandatory. Both pods must land on the same node. node-role: worker for both; there is one worker today, so it is free — but it is a real constraint the day a second appears, and it should be written down rather than assumed.
Delivery is eventual, not live. A write inside the pod is visible to the operator within seconds. That is the accepted cost of this shape.

3. Calls the plan must settle (they are this issue's, not #25's)

Device identity comes from Vault, not from an emptyDir. Syncthing's cert.pem/key.pem are the device ID: if they are regenerated every start, every peer has to be re-approved. So the identity (and the folder/peer configuration, i.e. config.xml) is a credential → a VaultStaticSecret syncing secret/syncthing/<name> into a mount the daemon reads, per AGENTS.md's Vault-first rule — with the named-path policy and kubernetes-auth role for the namespace that the rule requires. No hand-run kubectl create secret. The GUI/API password is credential material too and belongs in the same path.
Ownership, both sides. The folder is uid 1000 on the host and the pod runs as root. Which uid owns the files in the claim — and what that means for the host's Syncthing — is a decision to make and write down here, not a surprise to discover.
Folder policy is policy, and there is nothing today. The folder has zero <ignore> entries and already carries two *.sync-conflict-* files (both session JSONLs), i.e. the conflict rate under send/receive is non-zero before a second writer exists. Decide what is ignored — the obvious candidate is anything machine-specific: agent/bin/{fd,rg} on this host are Linux aarch64 ELFs but would be mac binaries on dima, so syncing them bidirectionally is a rotation-restore problem rather than a sync. The session-log conflicts are the other candidate. Neither is #25's to decide.
Not in scope: exposing the GUI, an Ingress, or any inbound path from off-tailnet. The API is reachable in-cluster; anything else is an admin action.
Assume baseline PSA as elsewhere in annia: no added capabilities, no privileged containers, no hostPath.

4. Effect on other issues

milner#25 takes the dependency: its /root/.pi line is this issue's PVC, so #25 cannot meet its live criteria until this ships.
The layered-nix-store issue is unaffected but adjacent: it owns the node-side assembler and the shared store and deletes the hand-run seed Job; this issue's claim is an ordinary persistent claim, not a nix-class mount.

5. Verification

A file written at /root/.pi/<name> inside the pod appears under ~/.pi/<name> on the operator's host within the sync interval, and vice versa; the device ID survives a pod restart (identity from Vault), and a restart does not re-add or re-approve either peer.

Label note: with the PVC, the Vault-synced identity, the ownership decision and the folder policy in scope, effort/low looks too small — effort/medium is the honest estimate, pending the plan.

Naina: ## Scope update — the annia pod is now the first consumer (2026-09-28 session) This issue was "define the IaC code to create a pod running syncthing on Milner". That is still the core, but the pod acquires its purpose and a hard consumer, so the scope below is what the plan should cover. The body's one-line framing is superseded; the issue is not. --- ### 1. What this ships An always-on **Syncthing pod** in `annia`, running the official upstream image at a pinned tag, on a **single `persistent`-class `ReadWriteOnce` PVC** that holds one folder: the operator's `~/.pi` — Syncthing's folder id `naccs-5znau`, label `pi`. Send/receive, with the operator's `dima` (macOS) and this host (`fedora`) as the two registered peers. That is the whole of it: a service, a claim, a folder, and the peers. ### 2. Why it is now load-bearing The **annia pod (milner#25) mounts that same PVC at `/root/.pi`**, whole-volume, no subPath. `/root/.pi` is not an annia artefact — it is *the operator's folder*, ~1.1 GB of pi credentials, skills, extensions and session history — and the decision is that the pod gets the real thing rather than a copy or a subset. Syncthing is the transport that keeps the pod's copy and the operator's machines one folder. Concretely: - **Two pods, one claim, no sidecar.** Syncthing runs *only* in this pod; the annia pod just mounts. That keeps the extra credential out of annia and means the annia image carries no sync daemon. - **RWO ⇒ co-scheduling is mandatory.** Both pods must land on the same node. `node-role: worker` for both; there is one worker today, so it is free — but it is a real constraint the day a second appears, and it should be written down rather than assumed. - **Delivery is eventual, not live.** A write inside the pod is visible to the operator within seconds. That is the accepted cost of this shape. ### 3. Calls the plan must settle (they are this issue's, not #25's) - **Device identity comes from Vault, not from an emptyDir.** Syncthing's `cert.pem`/`key.pem` *are* the device ID: if they are regenerated every start, every peer has to be re-approved. So the identity (and the folder/peer configuration, i.e. `config.xml`) is a credential → a `VaultStaticSecret` syncing `secret/syncthing/<name>` into a mount the daemon reads, per `AGENTS.md`'s Vault-first rule — with the named-path policy and kubernetes-auth role for the namespace that the rule requires. No hand-run `kubectl create secret`. The GUI/API password is credential material too and belongs in the same path. - **Ownership, both sides.** The folder is uid 1000 on the host and the pod runs as root. Which uid owns the files in the claim — and what that means for the host's Syncthing — is a decision to make and write down here, not a surprise to discover. - **Folder policy is policy, and there is nothing today.** The folder has **zero `<ignore>` entries** and already carries **two `*.sync-conflict-*` files** (both session JSONLs), i.e. the conflict rate under send/receive is non-zero *before* a second writer exists. Decide what is ignored — the obvious candidate is anything machine-specific: `agent/bin/{fd,rg}` on this host are Linux aarch64 ELFs but would be mac binaries on `dima`, so syncing them bidirectionally is a rotation-restore problem rather than a sync. The session-log conflicts are the other candidate. Neither is #25's to decide. - **Not in scope**: exposing the GUI, an Ingress, or any inbound path from off-tailnet. The API is reachable in-cluster; anything else is an admin action. - **Assume `baseline` PSA** as elsewhere in `annia`: no added capabilities, no privileged containers, no hostPath. ### 4. Effect on other issues - **milner#25** takes the dependency: its `/root/.pi` line is this issue's PVC, so #25 cannot meet its live criteria until this ships. - **The layered-nix-store issue** is unaffected but adjacent: it owns the node-side assembler and the shared store and deletes the hand-run seed Job; this issue's claim is an ordinary `persistent` claim, not a `nix`-class mount. ### 5. Verification A file written at `/root/.pi/<name>` inside the pod appears under `~/.pi/<name>` on the operator's host within the sync interval, and vice versa; the device ID survives a pod restart (identity from Vault), and a restart does **not** re-add or re-approve either peer. **Label note:** with the PVC, the Vault-synced identity, the ownership decision and the folder policy in scope, `effort/low` looks too small — `effort/medium` is the honest estimate, pending the plan.

tepichord added

and removed

labels

2026-09-29 16:34:09 -07:00

tepichord commented

2026-09-29 17:56:43 -07:00

Author

Owner

Naina:

Overview

Ship an always-on Syncthing pod in annia — upstream syncthing/syncthing:2.1.5, one persistent RWO claim holding the operator's ~/.pi (folder naccs-5znau), identity + config.xml + GUI password Vault-synced from secret/syncthing/config. The "IaC" is Flux GitOps manifests under gitops/clusters/milner/, not Terraform. This unblocks milner#25, whose /root/.pi is this claim mounted whole-volume.

Technical Approach

Add the VaultStaticSecret to gitops/clusters/milner/secrets/annia-secrets.yaml (copy annia-secrets shape + irc-oper's rollout argument):¹²

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata: {name: syncthing-config, namespace: annia}
spec:
  type: kv-v2
  mount: secret
  path: syncthing/config          # fields: config.xml, cert.pem, key.pem
  refreshAfter: 1h
  vaultAuthRef: default
  # config.xml/certs are read once at start-up; a rotation that only
  # rewrote the Secret would leave the daemon on the old identity.
  rolloutRestartTargets:
    - kind: Deployment
      name: syncthing
  destination: {name: syncthing-config, create: true, overwrite: true}

Widen annia-read by one exact path in docs/vault.md §4c-ter (kv-v2 policies need the data/ segment):³

path "secret/data/annia/*" {
  capabilities = ["read"]
}
# exact path, not a prefix — the `irc-read` precedent (§141-150): the pod
# gets its identity, not a namespace's worth of credentials<sup>[4](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/docs/vault.md?display=source#L157)</sup>
path "secret/data/syncthing/config" {
  capabilities = ["read"]
}

The annia kubernetes-auth role already exists (vaultauth-annia.yaml) and already carries annia-read, so the role and set-credential.sh's single-policy SNIPPET_ROLE stay untouched.⁵⁶

New producer script gitops/clusters/milner/apps/annia/set-syncthing-credential.sh (sibling of set-credential.sh, set-herald-credential.sh precedent):⁷⁸

SYNCTHING_PATH=secret/syncthing/config   # feeds the producer-test regex<sup>[9](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/tests/test_gitops.py?display=source#L535)</sup>
./set-syncthing-credential.sh /tmp/st-gen  # reads config.xml cert.pem key.pem from dir, writes on stdin
./set-syncthing-credential.sh verify       # verdicts, never values; prints the annia-read half when ROOT_VAULT_TOKEN unset

Generate the identity offline (runbook docs/syncthing.md):

syncthing generate --home /tmp/st-gen
syncthing generate --home /tmp/st-gen --gui-user op --gui-password '<pw>'   # bcrypt inline in config.xml
# edit /tmp/st-gen/config.xml:
#   <folder id="naccs-5znau" label="pi" path="/pi" type="sendreceive"/>
#   <device id="<dima-id>" name="dima"/> <device id="<fedora-id>" name="fedora"/>

All three files are generated together (the local <device> id must match cert.pem); the Vault-held copy is the source of truth, GUI edits are transient by design.

PVC apps/annia/syncthing-pvc.yaml — mirror apps/irc/persistentvolumeclaim.yaml:¹⁰

spec:
  accessModes: [ReadWriteOnce]        # one writer; both pods pin node-role: worker (co-scheduling, written down here)
  storageClassName: persistent        # Retain + UID-free pathPattern, docs/storage.md §10a<sup>[11](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/docs/storage.md?display=source#L24)</sup>
  resources: {requests: {storage: 10Gi}}

Deployment apps/annia/syncthing.yaml:

spec:
  replicas: 1
  strategy: {type: Recreate}          # RWO single-writer, irc precedent
  template:
    spec:
      nodeSelector: {node-role: worker}
      initContainers:
        - name: seed
          image: syncthing/syncthing:2.1.5
          command: ["/bin/sh","-ec"]
          args: ["cp /etc/vault/{config.xml,cert.pem,key.pem} /var/syncthing/config/",
                 "cp /etc/ignore/stignore /pi/.stignore"]
          # mounts: secret syncthing-config (ro), stignore ConfigMap, emptyDir /var/syncthing, PVC /pi
      containers:
        - name: syncthing
          image: syncthing/syncthing:2.1.5   # 2.1.5 serves linux/arm64; pinned, never :latest
          args: [serve, --no-browser, --no-console, --no-port-probing]
          env:
            - {name: PUID, value: "0"}       # root in-pod: fsGroup cannot chown a hostPath-backed claim,
            - {name: PGID, value: "0"}       # and #25's pod mounts the same claim as root
          ports: [8384/tcp, 22000/tcp, 22000/udp, 21027/udp]
          readinessProbe: {httpGet: {path: /rest/noauth/health, port: 8384}}   # image HEALTHCHECK endpoint, unauthenticated<sup>[12](https://docs.syncthing.net/rest/noauth-health-get.html)</sup>
          resources: {requests: {cpu: 100m, memory: 256Mi}, limits: {cpu: "1", memory: 1Gi}}
          volumeMounts: [{name: config, mountPath: /var/syncthing}, {name: pi, mountPath: /pi}]
      volumes:
        - {name: pi, persistentVolumeClaim: {claimName: pi-data}}
        - {name: config, emptyDir: {}}     # writable config+index; identity re-seeded from Vault every start

No hostNetwork/hostPort/hostPath — all three are banned by the namespace's baseline PSA anyway.¹³

Two Services in apps/annia/syncthing-services.yaml:

# 1) in-cluster API/GUI reachability only — the accepted state, no Ingress
apiVersion: v1
kind: Service
metadata: {name: syncthing, namespace: annia}
spec:
  selector: {app.kubernetes.io/name: syncthing}
  ports: [{name: gui, port: 8384, targetPort: 8384}]
---
# 2) direct tailnet path for dima — exact shape of apps/irc/service.yaml<sup>[14](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/apps/irc/service.yaml?display=source#L11)</sup>
apiVersion: v1
kind: Service
metadata:
  name: syncthing-sync
  namespace: annia
  annotations:
    tailscale.com/proxy-group: ingress-proxies
    tailscale.com/hostname: syncthing      # syncthing.faun-snapper.ts.net, one admin approval (tailscale.md §2.4)
spec:
  type: LoadBalancer
  loadBalancerClass: tailscale
  selector: {app.kubernetes.io/name: syncthing}
  ports: [{name: sync, port: 22000, targetPort: sync, protocol: TCP}]   # TCP only; QUIC-over-tailnet unverified, fallback covers it

Without (2) sync still works — global discovery + relays are outbound-only from the pod — but the 1.1 GB first sync and dima traffic ride public relays. Local discovery (21027) never crosses subnets, so nothing listens for it.

Ignore policy lands before the second writer: new file apps/annia/stignore, wired via configMapGenerator (hashed name rolls the pod, irc precedent):
```
(?d)*.sync-conflict-*
/agent/bin/fd
/agent/bin/rg
```
.stignore is per-device and never synced, so dima/fedora get the same three lines once via their GUIs (runbook), and the two existing *.sync-conflict-* JSONLs are reviewed/removed before the pod's first sync. Session JSONLs stay synced — #25 needs session history; the existing conflicts are a dima↔fedora problem, not this pod's.
List the new resources in apps/annia/kustomization.yaml with the house-style comment block (placement argument + co-scheduling write-down).
Tests (counts + one new behaviour test, details below).
Docs + story: docs/syncthing.md (§19: generate → write Vault → ignore/conflict steps → verification), OPERATIONS.md index row, docs/tailscale.md autoApprovers snippet gains "svc:syncthing", and USER_STORIES.md gains story 24 under "As the operator of the cluster":

The agent's folder is one folder, everywhere it runs. What is written in ~/.pi on one device, or in the pod's copy, appears on the others within the sync interval — and restarting any copy never makes the others re-approve it.

Live rollout order (runbook — apps.yaml has wait: true, so the Vault value must exist before the Deployment merges, the failure set-credential.sh's header documents):

1. generate identity → set-syncthing-credential.sh (+ annia-read policy half)
2. host: remediate 2 sync-conflict files; .stignore on dima + fedora
3. merge → PVC bound → pod Ready
4. approve svc:syncthing in the tailscale console (once)
5. add tcp://syncthing.faun-snapper.ts.net:22000 to the pod's device entry on dima/fedora; approve the pod's device ID once
6. verify roundtrip + restart identity (issue §5)

Implementation Details

Modify

gitops/clusters/milner/apps/annia/kustomization.yaml — 4 resources + comment
gitops/clusters/milner/secrets/annia-secrets.yaml — syncthing-config VSS
docs/vault.md — §4c-ter annia-read gains the exact path
tests/test_gitops.py — 14 → 15 in test_every_vault_static_secret_path_has_a_producer and (9, 14) → (9, 15) in test_vault_crs_match_the_vso_crd_shape¹⁵¹⁶
docs/tailscale.md — autoApprovers snippet + svc:syncthing note
OPERATIONS.md — index row; USER_STORIES.md — story 24

Create

apps/annia/syncthing.yaml, syncthing-pvc.yaml, syncthing-services.yaml, stignore, set-syncthing-credential.sh, docs/syncthing.md

Constraints

Image pinned (syncthing/syncthing:2.1.5, multi-arch incl. linux/arm64)¹⁷¹⁸; resources on main and init containers (policy/k8s.rego covers both).
secrets/ stays CRs-only; VSS keys config.xml/cert.pem/key.pem are valid Secret keys and mount as exact filenames.
Pod complies with baseline PSA: no added capabilities, no privileged, no hostPath/hostNetwork/hostPort.
Producer must run before the merge (step 12); a missing key CrashLoops the pod and reddens the whole apps layer.

Out-of-scope notes

GUI exposure, Ingress, any inbound path from off-tailnet — excluded by the issue; in-cluster :8384 is the accepted state.
milner#25's own half: its mount, and the one-directional podAffinity (mutual required affinity deadlocks) that hardens co-scheduling when a second worker appears. Today nodeSelector: node-role: worker on both + WaitForFirstConsumer PV node-affinity is the actual guarantee, written into the manifest comments here.
QUIC/UDP 22000 over the tailnet Service is unverified → TCP-only for now (Syncthing falls back); revisit at implementation if wanted.
No .woodpecker/gates.yaml changes; effort/low should become effort/medium per the issue's own note (scheduler's relabel).

Acceptance Criteria

Offline gates green with the new manifests: pytest tests/, kustomize render, kubeconform, conftest — including both count assertions deliberately moved to 15 VaultStaticSecrets.
secret/syncthing/config has a tracked producer; annia-read grants exactly secret/data/syncthing/config; VSO Secret syncthing-config is Ready in annia and the pod reaches Ready on the worker.
Issue §5 verification holds: a file written at /pi/<name> in the pod appears at ~/.pi/<name> on dima and fedora and vice versa; the device ID survives a pod restart with no peer re-approval; the three .stignore copies and the conflict-file remediation are done before the pod's first sync.

Testing Plan

test_every_vault_static_secret_path_has_a_producer — bump to 15; passes via SYNCTHING_PATH= in the new script.
test_vault_crs_match_the_vso_crd_shape — bump to (9, 15); closed-key-set check covers rolloutRestartTargets.
New test_syncthing_is_recreate_rwo_and_vault_seeded (rendered apps/annia, house-style long docstring): asserts strategy: Recreate, accessModes == [ReadWriteOnce], nodeSelector: node-role: worker, init container copying exactly config.xml/cert.pem/key.pem, no hostPath/hostNetwork/hostPort, pinned tag, resources on every container.
Existing test_kustomize_build, test_kubeconform, test_all_namespaces_have_psa_enforce, test_no_secrets_with_literal_data — unchanged, must stay green.
Live smoke (runbook, not CI): restart pod → GUI shows same device ID, peers not re-added; roundtrip file check both directions.

milner#25 — annia pod mounts this claim at /root/.pi; blocked by this issue.
milner#109 — layered Nix store: adjacent (same ~/.pi story), unaffected — this is an ordinary persistent claim, not a nix-class mount.

Research Findings

Pin syncthing/syncthing:2.1.5 (2026-09-08, multi-arch incl. linux/arm64); the image defines no USER and its entrypoint su-execs to PUID:PGID (default 1000) after a root start.
A missing config.xml or cert.pem/key.pem makes Syncthing mint a fresh device ID on first start — identity must be present before every start, yet the config dir must stay writable (csrftokens.txt, rewrites), so init-copy from the Vault Secret into an emptyDir, never a read-only Secret mount.
The GUI password is a bcrypt hash inside config.xml (syncthing generate --gui-password hashes it); /rest/noauth/health is unauthenticated and maps directly to the readiness probe.
VSO copies KV fields into Secret keys verbatim, so config.xml etc. mount as exact filenames; Syncthing never hot-reloads config, so rolloutRestartTargets is the right rotation mechanism (and subPath mounts would silently skip updates).
Policies union at token time and kv-v2 paths need the data/ prefix; a role takes N policies — but the smallest change keeping secret/syncthing/<name> is one exact-path block in annia-read, per the irc-read precedent.
.stignore lives at the folder root, is per-device, never synced; config.xml's <ignores> only templates auto-accepted folders, and conflicts are resolved by manual review (three-way merge is the documented technique for JSONLs).
Global discovery and relays are outbound-only — the zero-exposure baseline works; local discovery never crosses subnets; the idiomatic direct path is a Tailscale L3 LoadBalancer Service on 22000/TCP (IRC precedent), and baseline PSA forbids hostNetwork/hostPort/hostPath outright.
fsGroup does not chown hostPath-backed claims and user namespaces are per-pod opt-in, so the pod's uid is a deliberate choice; host-side ownership is unaffected either way because Syncthing syncs content, not ownership.
Required inter-pod affinity must be one-directional (mutual required affinity deadlocks), and the persistent class's WaitForFirstConsumer PV node-affinity already co-locates any pod mounting the claim.

Naina: ## Overview Ship an always-on Syncthing pod in `annia` — upstream `syncthing/syncthing:2.1.5`, one `persistent` RWO claim holding the operator's `~/.pi` (folder `naccs-5znau`), identity + `config.xml` + GUI password Vault-synced from `secret/syncthing/config`. The "IaC" is Flux GitOps manifests under `gitops/clusters/milner/`, not Terraform. This unblocks milner#25, whose `/root/.pi` is this claim mounted whole-volume. ## Technical Approach 1. **Add the `VaultStaticSecret`** to `gitops/clusters/milner/secrets/annia-secrets.yaml` (copy `annia-secrets` shape + `irc-oper`'s rollout argument):[1](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/secrets/annia-secrets.yaml?display=source#L2)[2](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/secrets/irc-oper.yaml?display=source#L24) ```yaml apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: {name: syncthing-config, namespace: annia} spec: type: kv-v2 mount: secret path: syncthing/config # fields: config.xml, cert.pem, key.pem refreshAfter: 1h vaultAuthRef: default # config.xml/certs are read once at start-up; a rotation that only # rewrote the Secret would leave the daemon on the old identity. rolloutRestartTargets: - kind: Deployment name: syncthing destination: {name: syncthing-config, create: true, overwrite: true} ``` 2. **Widen `annia-read` by one exact path** in `docs/vault.md` §4c-ter (kv-v2 policies need the `data/` segment):[3](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/docs/vault.md?display=source#L140) ```hcl path "secret/data/annia/*" { capabilities = ["read"] } # exact path, not a prefix — the `irc-read` precedent (§141-150): the pod # gets its identity, not a namespace's worth of credentials[4](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/docs/vault.md?display=source#L157) path "secret/data/syncthing/config" { capabilities = ["read"] } ``` The `annia` kubernetes-auth role already exists (`vaultauth-annia.yaml`) and already carries `annia-read`, so the role and `set-credential.sh`'s single-policy `SNIPPET_ROLE` stay untouched.[5](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/secrets/vaultauth-annia.yaml?display=source#L10)[6](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/apps/annia/set-credential.sh?display=source#L89) 3. **New producer script** `gitops/clusters/milner/apps/annia/set-syncthing-credential.sh` (sibling of `set-credential.sh`, `set-herald-credential.sh` precedent):[7](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/apps/annia/set-credential.sh?display=source#L1)[8](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/apps/irc/set-herald-credential.sh?display=source#L1) ```bash SYNCTHING_PATH=secret/syncthing/config # feeds the producer-test regex[9](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/tests/test_gitops.py?display=source#L535) ./set-syncthing-credential.sh /tmp/st-gen # reads config.xml cert.pem key.pem from dir, writes on stdin ./set-syncthing-credential.sh verify # verdicts, never values; prints the annia-read half when ROOT_VAULT_TOKEN unset ``` 4. **Generate the identity offline** (runbook `docs/syncthing.md`): ```bash syncthing generate --home /tmp/st-gen syncthing generate --home /tmp/st-gen --gui-user op --gui-password '<pw>' # bcrypt inline in config.xml # edit /tmp/st-gen/config.xml: # <folder id="naccs-5znau" label="pi" path="/pi" type="sendreceive"/> # <device id="<dima-id>" name="dima"/> <device id="<fedora-id>" name="fedora"/> ``` All three files are generated together (the local `<device>` id must match `cert.pem`); the Vault-held copy is the source of truth, GUI edits are transient by design. 5. **PVC** `apps/annia/syncthing-pvc.yaml` — mirror `apps/irc/persistentvolumeclaim.yaml`:[10](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/apps/irc/persistentvolumeclaim.yaml?display=source#L1) ```yaml spec: accessModes: [ReadWriteOnce] # one writer; both pods pin node-role: worker (co-scheduling, written down here) storageClassName: persistent # Retain + UID-free pathPattern, docs/storage.md §10a[11](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/docs/storage.md?display=source#L24) resources: {requests: {storage: 10Gi}} ``` 6. **Deployment** `apps/annia/syncthing.yaml`: ```yaml spec: replicas: 1 strategy: {type: Recreate} # RWO single-writer, irc precedent template: spec: nodeSelector: {node-role: worker} initContainers: - name: seed image: syncthing/syncthing:2.1.5 command: ["/bin/sh","-ec"] args: ["cp /etc/vault/{config.xml,cert.pem,key.pem} /var/syncthing/config/", "cp /etc/ignore/stignore /pi/.stignore"] # mounts: secret syncthing-config (ro), stignore ConfigMap, emptyDir /var/syncthing, PVC /pi containers: - name: syncthing image: syncthing/syncthing:2.1.5 # 2.1.5 serves linux/arm64; pinned, never :latest args: [serve, --no-browser, --no-console, --no-port-probing] env: - {name: PUID, value: "0"} # root in-pod: fsGroup cannot chown a hostPath-backed claim, - {name: PGID, value: "0"} # and #25's pod mounts the same claim as root ports: [8384/tcp, 22000/tcp, 22000/udp, 21027/udp] readinessProbe: {httpGet: {path: /rest/noauth/health, port: 8384}} # image HEALTHCHECK endpoint, unauthenticated[12](https://docs.syncthing.net/rest/noauth-health-get.html) resources: {requests: {cpu: 100m, memory: 256Mi}, limits: {cpu: "1", memory: 1Gi}} volumeMounts: [{name: config, mountPath: /var/syncthing}, {name: pi, mountPath: /pi}] volumes: - {name: pi, persistentVolumeClaim: {claimName: pi-data}} - {name: config, emptyDir: {}} # writable config+index; identity re-seeded from Vault every start ``` No `hostNetwork`/`hostPort`/`hostPath` — all three are banned by the namespace's `baseline` PSA anyway.[13](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/infrastructure/namespaces/annia.yaml?display=source#L6) 7. **Two Services** in `apps/annia/syncthing-services.yaml`: ```yaml # 1) in-cluster API/GUI reachability only — the accepted state, no Ingress apiVersion: v1 kind: Service metadata: {name: syncthing, namespace: annia} spec: selector: {app.kubernetes.io/name: syncthing} ports: [{name: gui, port: 8384, targetPort: 8384}] --- # 2) direct tailnet path for dima — exact shape of apps/irc/service.yaml[14](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/gitops/clusters/milner/apps/irc/service.yaml?display=source#L11) apiVersion: v1 kind: Service metadata: name: syncthing-sync namespace: annia annotations: tailscale.com/proxy-group: ingress-proxies tailscale.com/hostname: syncthing # syncthing.faun-snapper.ts.net, one admin approval (tailscale.md §2.4) spec: type: LoadBalancer loadBalancerClass: tailscale selector: {app.kubernetes.io/name: syncthing} ports: [{name: sync, port: 22000, targetPort: sync, protocol: TCP}] # TCP only; QUIC-over-tailnet unverified, fallback covers it ``` Without (2) sync still works — global discovery + relays are outbound-only from the pod — but the 1.1 GB first sync and `dima` traffic ride public relays. Local discovery (21027) never crosses subnets, so nothing listens for it. 8. **Ignore policy lands before the second writer**: new file `apps/annia/stignore`, wired via `configMapGenerator` (hashed name rolls the pod, irc precedent): ``` (?d)*.sync-conflict-* /agent/bin/fd /agent/bin/rg ``` `.stignore` is per-device and never synced, so `dima`/`fedora` get the same three lines once via their GUIs (runbook), and the two existing `*.sync-conflict-*` JSONLs are reviewed/removed before the pod's first sync. Session JSONLs stay synced — #25 needs session history; the existing conflicts are a `dima`↔`fedora` problem, not this pod's. 9. **List the new resources** in `apps/annia/kustomization.yaml` with the house-style comment block (placement argument + co-scheduling write-down). 10. **Tests** (counts + one new behaviour test, details below). 11. **Docs + story**: `docs/syncthing.md` (§19: generate → write Vault → ignore/conflict steps → verification), `OPERATIONS.md` index row, `docs/tailscale.md` autoApprovers snippet gains `"svc:syncthing"`, and `USER_STORIES.md` gains story **24** under "As the operator of the cluster": > **The agent's folder is one folder, everywhere it runs.** What is written in `~/.pi` on one device, or in the pod's copy, appears on the others within the sync interval — and restarting any copy never makes the others re-approve it. 12. **Live rollout order** (runbook — `apps.yaml` has `wait: true`, so the Vault value must exist before the Deployment merges, the failure `set-credential.sh`'s header documents): ``` 1. generate identity → set-syncthing-credential.sh (+ annia-read policy half) 2. host: remediate 2 sync-conflict files; .stignore on dima + fedora 3. merge → PVC bound → pod Ready 4. approve svc:syncthing in the tailscale console (once) 5. add tcp://syncthing.faun-snapper.ts.net:22000 to the pod's device entry on dima/fedora; approve the pod's device ID once 6. verify roundtrip + restart identity (issue §5) ``` ## Implementation Details **Modify** - `gitops/clusters/milner/apps/annia/kustomization.yaml` — 4 resources + comment - `gitops/clusters/milner/secrets/annia-secrets.yaml` — `syncthing-config` VSS - `docs/vault.md` — §4c-ter `annia-read` gains the exact path - `tests/test_gitops.py` — `14 → 15` in `test_every_vault_static_secret_path_has_a_producer` and `(9, 14) → (9, 15)` in `test_vault_crs_match_the_vso_crd_shape`[15](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/tests/test_gitops.py?display=source#L527)[16](https://git.gloom.garden/tepichord/milner/src/commit/d30c6bd8a667634657c72719169ecff5ac55b09e/tests/test_gitops.py?display=source#L782) - `docs/tailscale.md` — autoApprovers snippet + `svc:syncthing` note - `OPERATIONS.md` — index row; `USER_STORIES.md` — story 24 **Create** - `apps/annia/syncthing.yaml`, `syncthing-pvc.yaml`, `syncthing-services.yaml`, `stignore`, `set-syncthing-credential.sh`, `docs/syncthing.md` **Constraints** - Image pinned (`syncthing/syncthing:2.1.5`, multi-arch incl. `linux/arm64`)[17](https://hub.docker.com/layers/syncthing/syncthing/2.1.5/images/sha256-a654f827a3736b48174ef3da14fe832193cde7a313860b411eb48b964fb427e)[18](https://github.com/syncthing/syncthing/blob/main/Dockerfile); resources on main **and** init containers (policy/k8s.rego covers both). - `secrets/` stays CRs-only; VSS keys `config.xml`/`cert.pem`/`key.pem` are valid Secret keys and mount as exact filenames. - Pod complies with `baseline` PSA: no added capabilities, no privileged, no hostPath/hostNetwork/hostPort. - Producer must run before the merge (step 12); a missing key CrashLoops the pod and reddens the whole `apps` layer. **Out-of-scope notes** - GUI exposure, Ingress, any inbound path from off-tailnet — excluded by the issue; in-cluster `:8384` is the accepted state. - milner#25's own half: its mount, and the one-directional `podAffinity` (mutual required affinity deadlocks) that hardens co-scheduling when a second worker appears. Today `nodeSelector: node-role: worker` on both + `WaitForFirstConsumer` PV node-affinity is the actual guarantee, written into the manifest comments here. - QUIC/UDP 22000 over the tailnet Service is unverified → TCP-only for now (Syncthing falls back); revisit at implementation if wanted. - No `.woodpecker/gates.yaml` changes; `effort/low` should become `effort/medium` per the issue's own note (scheduler's relabel). ## Acceptance Criteria - [ ] Offline gates green with the new manifests: `pytest tests/`, kustomize render, kubeconform, conftest — including both count assertions deliberately moved to 15 VaultStaticSecrets. - [ ] `secret/syncthing/config` has a tracked producer; `annia-read` grants exactly `secret/data/syncthing/config`; VSO Secret `syncthing-config` is Ready in `annia` and the pod reaches Ready on the worker. - [ ] Issue §5 verification holds: a file written at `/pi/<name>` in the pod appears at `~/.pi/<name>` on `dima` and `fedora` and vice versa; the device ID survives a pod restart with no peer re-approval; the three `.stignore` copies and the conflict-file remediation are done before the pod's first sync. ## Testing Plan - `test_every_vault_static_secret_path_has_a_producer` — bump to 15; passes via `SYNCTHING_PATH=` in the new script. - `test_vault_crs_match_the_vso_crd_shape` — bump to `(9, 15)`; closed-key-set check covers `rolloutRestartTargets`. - New `test_syncthing_is_recreate_rwo_and_vault_seeded` (rendered `apps/annia`, house-style long docstring): asserts `strategy: Recreate`, `accessModes == [ReadWriteOnce]`, `nodeSelector: node-role: worker`, init container copying exactly `config.xml`/`cert.pem`/`key.pem`, no `hostPath`/`hostNetwork`/`hostPort`, pinned tag, resources on every container. - Existing `test_kustomize_build`, `test_kubeconform`, `test_all_namespaces_have_psa_enforce`, `test_no_secrets_with_literal_data` — unchanged, must stay green. - Live smoke (runbook, not CI): restart pod → GUI shows same device ID, peers not re-added; roundtrip file check both directions. ## Related Issues/PRs - [milner#25](https://git.gloom.garden/tepichord/milner/issues/25) — annia pod mounts this claim at `/root/.pi`; blocked by this issue. - [milner#109](https://git.gloom.garden/tepichord/milner/issues/109) — layered Nix store: adjacent (same `~/.pi` story), unaffected — this is an ordinary `persistent` claim, not a `nix`-class mount. ## Research Findings - Pin `syncthing/syncthing:2.1.5` (2026-09-08, multi-arch incl. `linux/arm64`); the image defines no `USER` and its entrypoint `su-exec`s to `PUID:PGID` (default 1000) after a root start. - A missing `config.xml` **or** `cert.pem`/`key.pem` makes Syncthing mint a fresh device ID on first start — identity must be present before every start, yet the config dir must stay writable (`csrftokens.txt`, rewrites), so init-copy from the Vault Secret into an `emptyDir`, never a read-only Secret mount. - The GUI password is a bcrypt hash inside `config.xml` (`syncthing generate --gui-password` hashes it); `/rest/noauth/health` is unauthenticated and maps directly to the readiness probe. - VSO copies KV fields into Secret keys verbatim, so `config.xml` etc. mount as exact filenames; Syncthing never hot-reloads config, so `rolloutRestartTargets` is the right rotation mechanism (and `subPath` mounts would silently skip updates). - Policies union at token time and kv-v2 paths need the `data/` prefix; a role takes N policies — but the smallest change keeping `secret/syncthing/<name>` is one exact-path block in `annia-read`, per the `irc-read` precedent. - `.stignore` lives at the folder root, is per-device, never synced; `config.xml`'s `<ignores>` only templates auto-accepted folders, and conflicts are resolved by manual review (three-way merge is the documented technique for JSONLs). - Global discovery and relays are outbound-only — the zero-exposure baseline works; local discovery never crosses subnets; the idiomatic direct path is a Tailscale L3 `LoadBalancer` Service on 22000/TCP (IRC precedent), and `baseline` PSA forbids hostNetwork/hostPort/hostPath outright. - `fsGroup` does not chown hostPath-backed claims and user namespaces are per-pod opt-in, so the pod's uid is a deliberate choice; host-side ownership is unaffected either way because Syncthing syncs content, not ownership. - Required inter-pod affinity must be one-directional (mutual required affinity deadlocks), and the `persistent` class's `WaitForFirstConsumer` PV node-affinity already co-locates any pod mounting the claim.

tepichord added

and removed

labels

2026-09-29 18:03:58 -07:00

tepichord referenced this issue from a commit

2026-09-30 09:38:59 -07:00

feat(annia): syncthing service — the agent's folder, Vault-seeded and single-writer

tepichord referenced this issue from a commit