feat/20-irc-service-on-milner #105

Merged

tepichord merged 9 commits from feat/20-irc-service-on-milner into main

2026-09-25 14:41:58 -07:00

tepichord commented

2026-09-25 13:05:54 -07:00

Owner

No description provided.

tepichord added 5 commits

2026-09-25 13:05:54 -07:00

feat(irc): give the automation an account, and let a settled run say so b6da24a1d2

`herald` is the account everything unattended speaks as: one password in Vault
at `secret/irc/herald`, one 0600 copy at `~/.config/milner/irc-herald`, and
`apps/irc/herald.py` — standard library only, because its job is to work when
Vault is sealed or the cluster is down — for sending, verifying and provisioning
it. There is deliberately no policy and no VaultStaticSecret for this path: the
ircd pod reads the oper hash (§17d) and nothing else, so nothing in the cluster
can read it. The change that adds one is the change that puts a sender in the
cluster.

`host/pi-irc-notify.ts` fires on `agent_settled` — pi's "I will not continue on
my own", not `agent_end`, where pi may still retry or compact — and posts one
line: `[pi] done in milner (3m12s)`, prefixed with `@tepichord - ` when the mode
is not `tui`. The address is a constant in one file, so no notification's own
text can add, move or remove an interrupt, and the space-dash-space is
load-bearing (`irc/webpush/highlight.go` treats `-` as word-internal, so
`@tepichord-` never reaches a phone).

The sender logs in as `herald@sender` — a device id, §17e's fix, in *both* the
authcid and the authzid, which Ergo compares before it strips the id — and never
joins `#milner`: the always-on client holds the seat, which is what makes
`verify`'s check real, since it asks `WHO` from a session that never sent JOIN.
A message carrying CR or LF is refused rather than sanitised, because a text
that can end its own line can become a command.

Two copies of one password rot, so the ordering is the guard: the environment is
read before the file, and a stale export therefore fails at SASL instead of
being silently shadowed by the copy that is right. `set-herald-credential.sh
verify` compares Vault, the file and the export, and prints MATCH or DIFFERENT;
it never prints a value, and the value only ever travels on stdin.

`tests/test_irc_herald.py` covers the route of the credential (stdin, mode 600,
no argv), the SASL pair, the two tiers, the refusals and the hook's shape, with
22 mutations each proven to bite with its own message. The ruff lint now covers
every tracked Python file instead of `tests/` alone: `router/` and this sender
were outside every lint, which is where a mistake is least visible.

Stories 21-22, docs/irc.md §17h, .env.example, host/README.md.

fix(host): load the hook from the repository, not from a synced ~/.pi 267b7d1566

`~/.pi` syncs to five other devices, so a symlink into this checkout was never
going to work — and not for the reason it looks like. A missing extension path
is fatal at startup: measured against a scratch project, pi refuses the session
with `Failed to load extension … Extension path does not exist`. The symlink
would have stopped pi from starting on every other device the moment it synced,
which is worse than the notification it was meant to deliver.

A path in a settings file is resolved against that file, so `.pi/settings.json`
in this repository loads `host/pi-irc-notify.ts` in place: the extension pi runs
is the reviewed file, there is no copy to drift, and nothing is installed on any
host. Project resources load after the project is trusted, and `~/.pi/agent/`
`trust.json` already answers that for `/home/naina` — a parent-path decision
covers everything under it — so a session in this repo has the hook with no
prompt and no `-a`, and no other device's pi ever sees a path that only exists
here.

The trade is scope: this fires for sessions in this repository. Every device and
every repository would mean a copy inside the synced directory with the sender
beside it, and a credential per device — and the credential file is not synced
either (`~/.config` is not in the syncthing folder list). §17h says so rather
than leaving the next reader to reach for the symlink.

`.pi/README.md` signs the directory. The guards resolve the settings file's path
to the hook, require it to be relative, check the sender lands in the same
checkout, and check that the runbook, the host index and that README all name the
mechanism — 26 mutations in tests/test_irc_herald.py, each proven to bite with
its own message. Suite: 266 passed.

feat(host): install the hook by copy, and make the copy's drift visible 9ccbbd61eb

The operator would rather copy the file into `~/.pi/agent/extensions/` — the
synced directory they already keep current by hand — than have the repository
load it in place. So the install is one `cp`, repeated whenever this file
changes, and three things follow from it.

The copy no longer sits beside the sender, so the sender is found by looking for
a *checkout* by name: `$PI_IRC_NOTIFY_REPO`, then the repository this file is in
(the case when pi loads it in place), then `/home/naina/milner`, which
host/README.md documents. The sender is deliberately not copied with it: a
600-line program that changes for its own reasons does not belong in a synced
directory, and two copies of one script is the shape that rots. A device with no
checkout sends nothing and says so once per settle — a line in pi's log, not a
session that will not start, which is what a symlink into a checkout would have
been (measured: `Failed to load extension … Extension path does not exist` is
fatal).

The repo's own `.pi/` is gone. With the copy in the synced directory it was a
second load of the same extension — two identical lines per settle — and pi
dedupes by path, so a copy is never the path it was copied from. A guard now
fails on any file that names a project settings entry again.

A copy can fall behind, which is the one weakness of this install, so the hook
reports it: on every settle it compares itself against the checkout's file and
prints `this installed copy differs from … — re-copy it`. That check runs before
the interpreter check, and not after — the first version returned early there,
so a host without `python3` would never have heard that its copy was out of date,
which is precisely the case where re-copying is the only thing that helps.

Verified live: the copied hook loads from the synced directory with no project
settings and no `-a`, finds the sender through the `/home/naina/milner`
fallback, and printed both the drift line and the missing-interpreter line when
the checkout's file was temporarily ahead of the copy.

tests/test_irc_herald.py: 26 tests, 28 mutations each proven to bite with its own
message. Suite: 267 passed; shellcheck, ruff, yamllint, terraform fmt clean.

feat(irc): give the operator's own sessions their own channel b9395115c1

`#milner` is what the cluster says about itself and `#live-chats` is a pi session
on this host finishing, in whatever directory it ran. The hook names the second
one: a line about a laptop is never a line about the cluster, and the hook is
unscoped on purpose — it fires for any settled session here, because the gate is
a checkout, a credential and an interpreter, not the project the session happened
to run in. That is why the line names the directory.

Two channels means the account needs a seat in both, so `CHANNELS` is now one
list in `herald.py` and one array in the provisioning script: `provision` joins
every entry and `verify` checks every entry, because a seat in one channel and a
check in the other is exactly the drift a single list prevents. `send` refuses a
channel that is not in the list at all — the one caller that names a channel by
hand is a hook, and a wrong name there is a message nobody reads from a channel
that exists only because the bot joined it.

A seat can also be gone, and this is what the checks were about. Ergo writes an
always-on client's channel memberships back on a graceful shutdown
(`performAlwaysOnMaintenance`, `client.Store(0, true)`), so a restart *keeps*
them — verified by restarting the pinned image and watching `verify` pass, which
corrects what the first version of this paragraph claimed. They go when the
client is destroyed without that write-back: an always-on client that expires
(unset here, and 0 means never), always-on turned off on the account, or a
datastore restored from an older copy of the disk. `send` answers a refusal by
taking the seat again, once, because the message was aimed there anyway and
joining grants nothing (story 22); a second refusal is an error.

That "grants nothing" is now checked rather than assumed: a `352` flags field
carries the channel status prefix, so `verify` fails when the account holds
standing in a channel it speaks in, and names `/msg ChanServ DEOP`. The way to
get there is the provisioning order — Ergo makes the first joiner of an empty,
unregistered channel its operator (`newChannel := firstJoin &&
channel.registeredFounder == ""`), so the operator registers `#live-chats` from
their own client first, and the runbook and the script's closing list both say
so. Story 21 moves to the new channel and story 22 now says "any channel it
speaks in".

Checked against the pinned image on loopback, not only against the fake: both
seats taken and proven by a session that never joined, a message delivered to
`#live-chats`, `verify` naming a seat that was gone, and the repair on the wire
(`404` → one `JOIN` → delivered). The standing verdict fired on the first run
against a datastore where the bot really was the first joiner, with the DEOP
line the operator needs. The installed hook copy has been re-copied and loads
from the synced directory, stopping at the interpreter this host does not have
yet.

tests/test_irc_herald.py: 33 tests; 39 mutations, each caught by its own guard.
Suite: 274 passed; shellcheck and ruff clean.

feat(irc): let every settled run reach the phone, and every failure be read

ci/woodpecker/pr/gates Pipeline was successful

Details

05141a2632

The mention followed `ctx.mode`: a `tui` run reported without one, so the
phone stayed quiet exactly where the operator was assumed to be watching.
That is the wrong way round — the run worth walking away from is the one in
the session somebody started and left, and the terminal is not what tells
them it has finished. Every settle mentions now, and a two-second reply
buzzing too is the accepted cost (USER_STORIES.md 21).

The rest is what debugging this cost. Every failure in the hook went to
`console.error`, and a full-screen TUI is not a place that can be read: a
broken copy, a missing interpreter and a refused send all looked identical
to a hook that never fired, which is precisely how this was diagnosed the
slow way. They go through one `report` call now, which keeps the log line
for the headless modes and raises the same sentence as a `ctx.ui` banner
wherever there is a UI to raise it in.

Guards: the unconditional `--mention` in the argv, the absence of any
`ctx.mode` read, and the single `report` funnel. 41 mutations, each caught
by its own guard's own message.

tepichord added 1 commit

2026-09-25 13:46:29 -07:00

feat(ci): say in #milner when a pipeline ends, and let only a build mention d4dde38abe

A build finishing is the one thing this cluster has to say that is worth a
phone. The gates workflow now ends with a `notify` step — last, because what it
reports is what the steps above it did, and on `status: [success, failure]`
because a green-only notifier is silent in exactly the case a notification
exists for — which runs `ci/notify-build.sh`: one line out of Woodpecker's own
variables (workflow, run number, status, duration, the pull request or branch,
and the run's URL) handed to `herald.py send --source ci --channel '#milner'
--mention`. Everything else the cluster will say in that channel is a plain
line: the mention is the tier, it is one constant in the sender, and the build
is the only caller that asks for it (USER_STORIES.md 23).

Two properties of the step are deliberate, and both are about not lying. It is
`failure: ignore`, so a notifier that cannot reach IRC does not turn a green
build red — that would report the phone's own brokenness as the build's — and it
is not allowed to be silent either: the step goes red in Woodpecker's UI with
the sender's reason in its log, the contract the local hook's banner has. The
key is checked by name because it is the one somebody gets wrong by guessing:
Woodpecker's step struct has no `allow_failure`, and a YAML unmarshaller drops
what a struct does not declare, so that plausible-looking key is a step that
fails the pipeline while reading as one that cannot.

The credential is the pipeline's only `from_secret`, out of the Vault-backed
secret extension rather than a second copy in Woodpecker's own database: a rule
for tepichord/milner reading `secret/irc/herald` field `password`, which makes
the AppRole policy the pod reads through name a second path (§9k, which said
"one path and nothing else" and now says why it does not). That ordering is the
one thing here that cannot be discovered by running it: Woodpecker refuses a
pipeline whose `from_secret` names nothing the server holds — `secret "..."
not found`, at compile time, with no steps and therefore no step log — so the
rule is bootstrapped into the live ConfigMap and the extension restarted before
the pipeline that names it is pushed. §9k step 5 is that sequence.

Story 22 said the bot's password "is not something any cluster component can
read", and this change makes that false: CI has to hold the credential to speak
in the channel at all, and the alternative — a watcher on the operator's host —
is silent in the one case the notification is for, a build finishing while that
host is off. So the story says where the value does reach instead: Vault, one
file on this host, and the environment of the one step that reports a build,
never a database and never a manifest.

The step mounts the shared store and runs inside `nix develop`, which is where
the sender's `python3` comes from (docs/nix.md §14c). The three lists that have
to agree — the extension's rule events, the workflow's root `when`, the step's
`when` — are held to each other by a test that reads all three files at once.

Guards: the step's position, its status list, its failure policy, its one
`from_secret` name, the rule's repository/path/field, the three event lists, and
the tier itself — every caller of `--mention` in the tree is one of the two
known ones, and each has to name the channel it belongs in. 17 mutations, each
caught by its own guard's own message; the suite is 281 tests.

tepichord added 1 commit

2026-09-25 13:59:25 -07:00

docs(woodpecker): the bootstrap is a window, not a source of truth

ci/woodpecker/pr/gates Pipeline was successful

Details

c81d64fb71

Flux owns the extension's ConfigMap and re-applies `main` to it every
`interval` (10m), so the hand-applied rule survives at most one reconcile.
Measured 2026-09-25: applied 20:40:30Z, reverted 20:45:02Z by
kustomize-controller, the checksum change restarting the deployment — and the
pull request's pipeline, compiled 93 s later, fetched secrets with
`match_result: none` and so died as `secret "IRC_HERALD_PASSWORD" not found`:
an `error` pipeline with no steps, which is why nothing in the run explained
itself and why the agent stayed silent.

Step 5 said Flux "adopts the identical object on the merge, so this is a
bootstrap and not a second source of truth" — true of the merge, and silent
about the interval in between. What makes the window sufficient is that the
secret is read once, while the pipeline is compiled, in the second the webhook
arrives; what the window is not is durable, and a change introducing a new
secret cannot be tested by its own pull request. Both are now written down.

tepichord added 1 commit

2026-09-25 14:28:45 -07:00

fix(ci): dial the address a step pod can resolve

ci/woodpecker/pr/gates Pipeline was successful

Details

e5a2597db8

Pipeline 112 (c81d64f) ran with the secret, reported `success`, and never
reached IRC. Its notify step's whole log was

  herald: ergo.irc.svc.cluster.local:6667: [Errno -2] Name or service not known

Woodpecker runs steps as pods resolving with the node's resolver rather than the
cluster's: every step above `notify` only ever dials `zot.faun-snapper.ts.net`,
which is why nothing else ever noticed, and an in-cluster Service name is
NXDOMAIN there. The pipeline still read `success` because the step is
`failure: ignore`, so the failure existed in the step log and nowhere else —
which is the contract, and it is what the log is for.

The notification now dials irc.faun-snapper.ts.net: herald's own default, the
address the host hook already uses, and one a step pod resolves. A test holds
the two callers to herald's default so a pipeline and a laptop cannot drift onto
different addresses, and the argv test (whose rationale said the opposite — "a
step is a pod, not a tailnet client") is corrected with it.

docs/woodpecker-ci.md §9k and docs/irc.md §17h record the failure mode, because
a green pipeline is exactly what hides it.

tepichord added 1 commit

2026-09-25 14:41:20 -07:00

Merge branch 'main' into feat/20-irc-service-on-milner

ci/woodpecker/pull_request_closed/cleanup Pipeline was successful

Details

89b5192847