tepichord/milner

Fork 0

Clean up OPERATIONS.md #66

New issue

Closed

opened 2026-09-17 09:18:57 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-17 09:18:57 -07:00

Owner

OPERATIONS.md has now grown into a massive 2,000+ line runbook and is starting to become too cumbersome for agents to dig through and hold context for
I suggest trimming it down and removing unnecessary information, and then splitting it up based on potential operational domains

- OPERATIONS.md has now grown into a massive 2,000+ line runbook and is starting to become too cumbersome for agents to dig through and hold context for - I suggest trimming it down and removing unnecessary information, and then splitting it up based on potential operational domains

tepichord added the

labels

2026-09-17 09:18:57 -07:00

tepichord added this to the MVP project

2026-09-17 09:18:57 -07:00

tepichord added

and removed

labels

2026-09-17 09:19:03 -07:00

tepichord commented

2026-09-17 11:23:47 -07:00

Author

Owner

Naina:

Overview

OPERATIONS.md is 3,071 lines / 13 sections, and two of those sections are the same bring-up order (§8 and §13).¹² Split it by operational domain into docs/, keep OPERATIONS.md as the index + the one bring-up order, and delete the duplicated §13. The §N cross-reference scheme is already broken in the tree (§7h, §8=storage, teardown=§9)³⁴⁵, so the split is also a correctness fix: every reference moves to docs/<domain>.md §N and a new pytest guard makes a wrong pointer fail CI.

Technical Approach

Create docs/ and move section blocks verbatim — no prose rewrite in this change.⁶⁷⁸⁹

New file	Sections moved
`docs/tailscale.md`	§2 admin console, §3 operator bootstrap, §6 client devices
`docs/vault.md`	§4
`docs/pocket-id.md`	§5
`docs/headlamp.md`	§7
`docs/woodpecker-ci.md`	§9
`docs/storage.md`	§10
`docs/durable-volumes.md`	§11a–§11g
`docs/preview-environment.md`	§11h
`docs/logging.md`	§12
`docs/history.md`	§11f migration record, §11g open questions (records, not procedure)

Keep global §N numbers stable (decision (a) in Research Findings). A moved section keeps its number; only the file qualifier changes, so no renumbering can silently repoint a reference.
```
- # ... the AppRole (root, once per fresh Vault) per OPERATIONS.md §9k.
+ # ... the AppRole (root, once per fresh Vault) per docs/woodpecker-ci.md §9k.
```
Collapse the duplicate bring-up orders — delete §13, keep §8, which already carries the correct §10b pointer (§13 carries the stale §8b).¹⁰¹¹
```
- ## 13. Bring-up order (from a wiped cluster)
+ (deleted; §8 is the only copy)
```

Fix the stale pointers the split would otherwise multiply (verified, not exhaustive):¹²¹³¹⁴¹⁵¹⁶¹⁷

Site	Says	Actually
`cloudflare/README.md:160`, `.env.example:105,121`, `ARCHITECTURE.md:232,314`, `OPERATIONS.md:25`	`§7h`	`§9h`
`worker/README.md:94`, `ARCHITECTURE.md:325`	`§8` (storage)	`§10`
`worker/README.md:114`, `README.md:246,311,371`, `gitops/.../local-path/values.yaml:43`	`§9` (teardown)	`§11c`
`README.md:344`	preview drill `§11h`	`§11d`
`.env.example:94`	Woodpecker OAuth `§7`	`§9a`
`PLAN.md:34`	Vault token `§5`	`§4h`
`OPERATIONS.md:706`	`§8c`	`§10c`

Retarget the four documentation guards to the runbook file set.¹⁸¹⁹²⁰²¹
```
RUNBOOKS = ["OPERATIONS.md", *sorted(p.name for p in pathlib.Path("docs").glob("*.md"))]
```
test_runbook_carries_no_volatile_identity_literals and the fenced-code half of test_runbook_keeps_its_tokens_out_of_argv parametrize over RUNBOOKS. The "exactly one v()/vk() definition" assertion becomes set-wide (a second definition still silently wins).²² The vk call-site window is reworked to "a |/printf producer in the preceding two lines" — the current definition-to-body window is same-file and breaks once vk lives in docs/vault.md while call sites live in docs/pocket-id.md/docs/woodpecker-ci.md.²³²⁴ test_drill_ladder_points_at_the_harness reads docs/durable-volumes.md and ### 11d..²⁵²⁶

Add the missing cross-reference guard — this is what turns the manual audit into a failing test:

REF = re.compile(r"(?P<file>[\w./-]+\.md) §(?P<sec>\d+[a-z]?(?:-[a-z]+)?)")
# for each REF in `git ls-files '*.md'`: target file exists and has a
# `## N.` / `### Na.` heading.

Update the in-repo map — AGENTS.md repo-map rows (OPERATIONS.md, new docs/ row), README.md, ARCHITECTURE.md, .env.example, gitops/install.sh, drills/check.sh, and the gitops manifests that cite section numbers.

Implementation Details

Files created: docs/{tailscale,vault,pocket-id,headlamp,woodpecker-ci,storage,durable-volumes,preview-environment,logging,history}.md
Files modified: OPERATIONS.md, AGENTS.md, ARCHITECTURE.md, README.md, PLAN.md, USER_STORIES.md, .env.example, tests/test_docs.py, tests/test_static.py, drills/check.sh, worker/README.md, data/README.md, cloudflare/README.md, cloudflare/terraform.tfvars.example, gitops/install.sh, gitops/clusters/milner/** (manifest comments referencing OPERATIONS.md §N)
Constraints:
- Credentials stay on stdin; no new fenced code passes a token in argv.
- Exactly one v()/vk() definition in the whole runbook set.
- No volatile identity literals (hex digests, UUIDs, node names, PV names) in any runbook file.
- References use file-qualified docs/<domain>.md §N; Forgejo's auto-slugs are not GitHub's, so do not author #anchor links relying on heading text.²⁸
- Land it as one commit: some tests shell out to git ls-files, so an untracked new doc changes test behaviour.²⁷
Out-of-scope notes:
- §11e traps stay in docs/durable-volumes.md (still operationally relevant); only §11f/§11g move.
- No ADR directory is introduced; docs/history.md is the interim home and an ADR migration is a follow-up.
- §1 What lives where stays in OPERATIONS.md rather than moving beside AGENTS.md's repo map.

Acceptance Criteria

OPERATIONS.md contains only the intro, §1, §8, and a domain index; each moved section lives in exactly one docs/*.md file and appears nowhere else (no §13 duplicate).
Every §N reference in the repo resolves to a heading in the file it names; the 7 stale-pointer rows above are corrected.
All four documentation guards pass against the new file set, including exactly one v()/vk() definition across RUNBOOKS.
python3 -m pytest tests/ -v is green.

Testing Plan

New guard: a reference to a nonexistent docs/foo.md §9z fails; the current tree's stale pointers fail before the fix and pass after.
New guard: a reference to a section number absent from the named file fails.
Parametrized guard: deliberately plant a UUID in docs/vault.md and confirm test_runbook_carries_no_volatile_identity_literals fails with that filename.
Mounted helper: a vk call with no |/printf producer in any runbook file fails, including when the call and the definition are in different files.
Manual: wc -l OPERATIONS.md docs/*.md — no file approaches the original 3,071 lines; grep -rn 'OPERATIONS\.md §' --include='*.md' --include='*.yaml' --include='*.sh' . returns no references to moved sections.

Issue #66 — the requested cleanup.
No related PRs.
tests/test_docs.py, tests/test_static.py — the guards that constrain the split.

Research Findings

Diátaxis and the Architect runbook guidance both split by reader need (how-to vs reference vs explanation), which is what moves §11f/§11g (records) and §1 (reference) out of the procedure. (diataxis.fr/map, erikevenson.github.io)
The established pattern is a thin index that owns only the order, with each step a pointer into the domain file that owns the commands — so the bring-up order must collapse to one copy. (sre.google/workbook/on-call)
There is no plain-markdown way to preserve hand-written §N numbers across files; every mature system adds explicit IDs or link checking. Here the check is a pytest guard over git ls-files '*.md'. (myst-parser docs, clausul.com)
Treating the same change as mandatory is the docs-as-code norm: GitLab fails CI when a doc is renamed without handling its references. (docs.gitlab.com)
Forgejo renders with goldmark (WithAutoHeadingID, WithAttribute) and its slugs are not guaranteed to match GitHub's, so explicit file-qualified references beat #anchor links. (gitea issue #17262)

Pre-Mortem

A cross-reference is missed and the split ships stale pointers anyway — ~30+ files cite OPERATIONS.md §N, so a reference written in a form the new guard's regex or file scan does not cover silently survives, and agents follow a pointer that names both the wrong file and the wrong section. The plan's headline correctness fix then becomes a net regression rather than a cleanup.
The reworked guards pass while no longer testing what they claim — parametrizing over RUNBOOKS and converting the vk definition-to-body window into a preceding-two-lines check is the subtlest part of the change, and an empty glob or mis-scoped window makes the guard vacuous. A later regression (a second v()/vk() definition, or a token passed on argv) then ships green, while an over-strict variant fails CI on unrelated files and pressures a rushed partial revert.
The verbatim move loses or misplaces content at a section boundary — §11 is split across docs/durable-volumes.md and docs/preview-environment.md, and deleting §13 assumes §8 is a strict superset, so a boundary drawn wrong drops content or duplicates it under a new filename with no test able to detect it. Because prose rewrites are explicitly out of scope, the loss would only surface much later during a real runbook procedure.

Naina: ## Overview OPERATIONS.md is 3,071 lines / 13 sections, and two of those sections are the same bring-up order (§8 and §13).[1](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L1029)[2](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L3029) Split it by operational domain into `docs/`, keep OPERATIONS.md as the index + the one bring-up order, and delete the duplicated §13. The `§N` cross-reference scheme is already broken in the tree (§7h, §8=storage, teardown=§9)[3](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L958)[4](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L1953)[5](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L2098), so the split is also a correctness fix: every reference moves to `docs/<domain>.md §N` and a new pytest guard makes a wrong pointer fail CI. ## Technical Approach 1. **Create `docs/` and move section blocks verbatim** — no prose rewrite in this change.[6](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L51)[7](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L1953)[8](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L2060)[9](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L2852) | New file | Sections moved | |---|---| | `docs/tailscale.md` | §2 admin console, §3 operator bootstrap, §6 client devices | | `docs/vault.md` | §4 | | `docs/pocket-id.md` | §5 | | `docs/headlamp.md` | §7 | | `docs/woodpecker-ci.md` | §9 | | `docs/storage.md` | §10 | | `docs/durable-volumes.md` | §11a–§11g | | `docs/preview-environment.md` | §11h | | `docs/logging.md` | §12 | | `docs/history.md` | §11f migration record, §11g open questions (records, not procedure) | 2. **Keep global `§N` numbers stable** (decision (a) in Research Findings). A moved section keeps its number; only the file qualifier changes, so no renumbering can silently repoint a reference. ```diff - # ... the AppRole (root, once per fresh Vault) per OPERATIONS.md §9k. + # ... the AppRole (root, once per fresh Vault) per docs/woodpecker-ci.md §9k. ``` 3. **Collapse the duplicate bring-up orders** — delete §13, keep §8, which already carries the correct `§10b` pointer (§13 carries the stale `§8b`).[10](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L1041)[11](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L3045) ```diff - ## 13. Bring-up order (from a wiped cluster) + (deleted; §8 is the only copy) ``` 4. **Fix the stale pointers the split would otherwise multiply** (verified, not exhaustive):[12](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/cloudflare/README.md?display=source#L160)[13](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/ARCHITECTURE.md?display=source#L232)[14](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L25)[15](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/README.md?display=source#L344)[16](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/PLAN.md?display=source#L34)[17](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L706) | Site | Says | Actually | |---|---|---| | `cloudflare/README.md:160`, `.env.example:105,121`, `ARCHITECTURE.md:232,314`, `OPERATIONS.md:25` | `§7h` | `§9h` | | `worker/README.md:94`, `ARCHITECTURE.md:325` | `§8` (storage) | `§10` | | `worker/README.md:114`, `README.md:246,311,371`, `gitops/.../local-path/values.yaml:43` | `§9` (teardown) | `§11c` | | `README.md:344` | preview drill `§11h` | `§11d` | | `.env.example:94` | Woodpecker OAuth `§7` | `§9a` | | `PLAN.md:34` | Vault token `§5` | `§4h` | | `OPERATIONS.md:706` | `§8c` | `§10c` | 5. **Retarget the four documentation guards** to the runbook file set.[18](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_docs.py?display=source#L12)[19](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_docs.py?display=source#L97)[20](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_static.py?display=source#L427)[21](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_static.py?display=source#L444) ```python RUNBOOKS = ["OPERATIONS.md", *sorted(p.name for p in pathlib.Path("docs").glob("*.md"))] ``` `test_runbook_carries_no_volatile_identity_literals` and the fenced-code half of `test_runbook_keeps_its_tokens_out_of_argv` parametrize over `RUNBOOKS`. The "exactly one `v()`/`vk()` definition" assertion becomes set-wide (a second definition still silently wins).[22](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_static.py?display=source#L469) The `vk` call-site window is reworked to "a `|`/`printf` producer in the preceding two lines" — the current definition-to-body window is same-file and breaks once `vk` lives in `docs/vault.md` while call sites live in `docs/pocket-id.md`/`docs/woodpecker-ci.md`.[23](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_static.py?display=source#L479)[24](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_static.py?display=source#L487) `test_drill_ladder_points_at_the_harness` reads `docs/durable-volumes.md` and `### 11d.`.[25](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_docs.py?display=source#L105)[26](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/OPERATIONS.md?display=source#L2189) 6. **Add the missing cross-reference guard** — this is what turns the manual audit into a failing test: ```python REF = re.compile(r"(?P<file>[\w./-]+\.md) §(?P<sec>\d+[a-z]?(?:-[a-z]+)?)") # for each REF in `git ls-files '*.md'`: target file exists and has a # `## N.` / `### Na.` heading. ``` 7. **Update the in-repo map** — `AGENTS.md` repo-map rows (`OPERATIONS.md`, new `docs/` row), `README.md`, `ARCHITECTURE.md`, `.env.example`, `gitops/install.sh`, `drills/check.sh`, and the gitops manifests that cite section numbers. ## Implementation Details - **Files created:** `docs/{tailscale,vault,pocket-id,headlamp,woodpecker-ci,storage,durable-volumes,preview-environment,logging,history}.md` - **Files modified:** `OPERATIONS.md`, `AGENTS.md`, `ARCHITECTURE.md`, `README.md`, `PLAN.md`, `USER_STORIES.md`, `.env.example`, `tests/test_docs.py`, `tests/test_static.py`, `drills/check.sh`, `worker/README.md`, `data/README.md`, `cloudflare/README.md`, `cloudflare/terraform.tfvars.example`, `gitops/install.sh`, `gitops/clusters/milner/**` (manifest comments referencing `OPERATIONS.md §N`) - **Constraints:** - Credentials stay on stdin; no new fenced code passes a token in argv. - Exactly one `v()`/`vk()` definition in the whole runbook set. - No volatile identity literals (hex digests, UUIDs, node names, PV names) in any runbook file. - References use file-qualified `docs/<domain>.md §N`; Forgejo's auto-slugs are not GitHub's, so do **not** author `#anchor` links relying on heading text.[28](https://github.com/go-gitea/gitea/issues/17262) - Land it as one commit: some tests shell out to `git ls-files`, so an untracked new doc changes test behaviour.[27](https://git.gloom.garden/tepichord/milner/src/commit/f82a5fb50361b06772ece970017f772691ab13a4/tests/test_static.py?display=source#L2508) - **Out-of-scope notes:** - §11e traps stay in `docs/durable-volumes.md` (still operationally relevant); only §11f/§11g move. - No ADR directory is introduced; `docs/history.md` is the interim home and an ADR migration is a follow-up. - `§1 What lives where` stays in OPERATIONS.md rather than moving beside AGENTS.md's repo map. ## Acceptance Criteria - [ ] OPERATIONS.md contains only the intro, §1, §8, and a domain index; each moved section lives in exactly one `docs/*.md` file and appears nowhere else (no §13 duplicate). - [ ] Every `§N` reference in the repo resolves to a heading in the file it names; the 7 stale-pointer rows above are corrected. - [ ] All four documentation guards pass against the new file set, including exactly one `v()`/`vk()` definition across `RUNBOOKS`. - [ ] `python3 -m pytest tests/ -v` is green. ## Testing Plan - New guard: a reference to a nonexistent `docs/foo.md §9z` fails; the current tree's stale pointers fail before the fix and pass after. - New guard: a reference to a section number absent from the named file fails. - Parametrized guard: deliberately plant a UUID in `docs/vault.md` and confirm `test_runbook_carries_no_volatile_identity_literals` fails with that filename. - Mounted helper: a `vk` call with no `|`/`printf` producer in any runbook file fails, including when the call and the definition are in different files. - Manual: `wc -l OPERATIONS.md docs/*.md` — no file approaches the original 3,071 lines; `grep -rn 'OPERATIONS\.md §' --include='*.md' --include='*.yaml' --include='*.sh' .` returns no references to moved sections. ## Related Issues - Issue #66 — the requested cleanup. - No related PRs. - `tests/test_docs.py`, `tests/test_static.py` — the guards that constrain the split. ## Research Findings - Diátaxis and the Architect runbook guidance both split by reader need (how-to vs reference vs explanation), which is what moves §11f/§11g (records) and §1 (reference) out of the procedure. ([diataxis.fr/map](https://diataxis.fr/map/), [erikevenson.github.io](https://erikevenson.github.io/architect/general/operational-runbooks/)) - The established pattern is a thin index that owns only the order, with each step a pointer into the domain file that owns the commands — so the bring-up order must collapse to one copy. ([sre.google/workbook/on-call](https://sre.google/workbook/on-call/)) - There is no plain-markdown way to preserve hand-written `§N` numbers across files; every mature system adds explicit IDs or link checking. Here the check is a pytest guard over `git ls-files '*.md'`. ([myst-parser docs](https://myst-parser.readthedocs.io/en/latest/syntax/cross-referencing.html), [clausul.com](https://clausul.com/blog/cross-references-and-section-renumbering)) - Treating the same change as mandatory is the docs-as-code norm: GitLab fails CI when a doc is renamed without handling its references. ([docs.gitlab.com](https://docs.gitlab.com/development/documentation/testing/)) - Forgejo renders with goldmark (`WithAutoHeadingID`, `WithAttribute`) and its slugs are not guaranteed to match GitHub's, so explicit file-qualified references beat `#anchor` links. ([gitea issue #17262](https://github.com/go-gitea/gitea/issues/17262)) ## Pre-Mortem 1. **A cross-reference is missed and the split ships stale pointers anyway** — ~30+ files cite `OPERATIONS.md §N`, so a reference written in a form the new guard's regex or file scan does not cover silently survives, and agents follow a pointer that names both the wrong file and the wrong section. The plan's headline correctness fix then becomes a net regression rather than a cleanup. 2. **The reworked guards pass while no longer testing what they claim** — parametrizing over `RUNBOOKS` and converting the `vk` definition-to-body window into a preceding-two-lines check is the subtlest part of the change, and an empty glob or mis-scoped window makes the guard vacuous. A later regression (a second `v()`/`vk()` definition, or a token passed on argv) then ships green, while an over-strict variant fails CI on unrelated files and pressures a rushed partial revert. 3. **The verbatim move loses or misplaces content at a section boundary** — §11 is split across `docs/durable-volumes.md` and `docs/preview-environment.md`, and deleting §13 assumes §8 is a strict superset, so a boundary drawn wrong drops content or duplicates it under a new filename with no test able to detect it. Because prose rewrites are explicitly out of scope, the loss would only surface much later during a real runbook procedure.

tepichord added

and removed

labels

2026-09-17 11:41:35 -07:00

tepichord referenced this issue from a commit

2026-09-17 15:19:03 -07:00

docs(operations): split OPERATIONS.md by operational domain

tepichord referenced this issue from a commit