tepichord/milner

Fork 0

Integrate restic as a tfstate backup #41

New issue

Open

opened 2026-09-10 09:44:24 -07:00 by tepichord · 1 comment

tepichord commented

2026-09-10 09:44:24 -07:00

Owner

Tool: restic (or Kopia — equivalent; Kopia adds a repository-server + policy model and a GUI). Both give encryption, dedup, retention, --exclude, and path-preserving
restore. State is ~250 KB, so every snapshot is trivially small.
Copy set: the repo tree excluding /.terraform/ (840 MB of rebuildable providers) and .git/** (already replicated off-host on git.gloom.garden). That single rule captures
every non-reproducible file — state, gitignored tfvars, .env — in one snapshot. Verify with restic ls that you see .env and the .auto.tfvars files.
Retention: --keep-hourly 6 --keep-daily 30 --keep-monthly 12. At 250 KB per snapshot this is free.
Trigger: state only changes when someone applies, so snapshots-on-a-timer (15–60 min) plus an explicit snapshot right after each terraform apply covers it. Don't rely on the
timer alone if you apply at odd hours.
Two repos, one failure domain each: temporarily, one of the repos will live outside of the /mnt mounted drive
Key handling: the restic password lives in your password manager, never on the host (a host compromise otherwise deletes the backups too), and use a storage credential
scoped to the backup bucket only. R2 bucket locks are available if you want the offsite copy immutable against a stolen Cloudflare token.

- Tool: restic (or Kopia — equivalent; Kopia adds a repository-server + policy model and a GUI). Both give encryption, dedup, retention, --exclude, and path-preserving restore. State is ~250 KB, so every snapshot is trivially small. - Copy set: the repo tree excluding **/.terraform/** (840 MB of rebuildable providers) and .git/** (already replicated off-host on git.gloom.garden). That single rule captures every non-reproducible file — state, gitignored tfvars, .env — in one snapshot. Verify with restic ls that you see .env and the .auto.tfvars files. - Retention: --keep-hourly 6 --keep-daily 30 --keep-monthly 12. At 250 KB per snapshot this is free. - Trigger: state only changes when someone applies, so snapshots-on-a-timer (15–60 min) plus an explicit snapshot right after each terraform apply covers it. Don't rely on the timer alone if you apply at odd hours. - Two repos, one failure domain each: temporarily, one of the repos will live outside of the /mnt mounted drive - Key handling: the restic password lives in your password manager, never on the host (a host compromise otherwise deletes the backups too), and use a storage credential scoped to the backup bucket only. R2 bucket locks are available if you want the offsite copy immutable against a stolen Cloudflare token.

tepichord added the

labels

2026-09-10 09:44:24 -07:00

tepichord added this to the MVP project

2026-09-10 09:44:24 -07:00

tepichord commented

2026-09-10 11:45:22 -07:00

Author

Owner

Naina:

Overview

Add automated restic backups of the Milner repo tree (Terraform state, gitignored tfvars, .env files) to two failure-domain-separated repositories — one on the /mnt external drive, one outside it. Exclude .terraform/ and .git/ (both rebuildable/replicated). Backups run on a 30-minute systemd timer and explicitly after every terraform apply. The restic password lives in a password manager and is never stored on disk.

Technical Approach

Create backup/ directory with the following structure:

backup/
├── README.md
├── excludes.txt
├── milner-backup.sh       # main backup script
├── milner-restore.sh      # restore helper
├── milner-init.sh         # one-time repo initialization
├── restic-backup.service   # systemd unit
├── restic-backup.timer     # systemd timer (30 min)
└── post-apply-hook.sh      # sourced after terraform apply

Create backup/excludes.txt — newline-separated patterns:

# Rebuildable providers (840 MB)
.terraform/

# Already replicated via git remote
.git/

Create backup/milner-backup.sh — the core backup script:

#!/bin/bash
set -e
cd "$(dirname "$0")"

REPO_DIR="$(pwd)/.."

# Password from password manager (never on disk)
# Fallback: RESTIC_PASSWORD_FILE pointing to a temp file if no CLI available
export RESTIC_PASSWORD_COMMAND="${RESTIC_PASSWORD_COMMAND:-pass show backups/milner}"

for REPO in "${RESTIC_REPOS[@]}"; do
  echo "Backing up to $REPO ..."
  restic -r "$REPO" backup "$REPO_DIR" \
    --exclude-file "$(dirname "$0")/excludes.txt" \
    --tag milner --verbose
  restic -r "$REPO" forget --tag milner \
    --group-by "paths,tags" \
    --keep-hourly 6 --keep-daily 30 --keep-monthly 12 \
    --prune
done

Environment variable RESTIC_REPOS is an array set in a config file or exported by the caller.

Create backup/milner-init.sh — one-time repo initialization:

#!/bin/bash
set -e

# Usage: milner-init.sh <repo-url> [<repo-url> ...]
export RESTIC_PASSWORD_COMMAND="${RESTIC_PASSWORD_COMMAND:-pass show backups/milner}"

for REPO in "$@"; do
  echo "Initializing $REPO ..."
  restic -r "$REPO" init
done

Create backup/milner-restore.sh — restore helper:

#!/bin/bash
set -e

# Usage: milner-restore.sh <repo-url> [<snapshot-id>] [--target /path]
export RESTIC_PASSWORD_COMMAND="${RESTIC_PASSWORD_COMMAND:-pass show backups/milner}"

REPO="$1"; shift
SNAPSHOT="${1:-latest}"
restic -r "$REPO" restore "$SNAPSHOT" "$@"

Create backup/post-apply-hook.sh — sourced by the user after terraform apply:

# Source this after terraform apply to trigger an immediate snapshot:
#   source backup/post-apply-hook.sh
REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
exec bash "$REPO_DIR/backup/milner-backup.sh"

Create systemd user units in backup/:

restic-backup.service:

[Unit]
Description=Restic backup of Milner repo tree

[Service]
Type=oneshot
EnvironmentFile=%h/.config/milner-backup.env
ExecStart=/path/to/milner-backup.sh

restic-backup.timer:

[Unit]
Description=Milner backup every 30 minutes

[Timer]
OnCalendar=*:0/30
Persistent=true

[Install]
WantedBy=timers.target

milner-backup.env (example, user creates on host):

RESTIC_REPOS=(
  "/mnt/backup/milner"
  "/home/user/milner-backup"
)
RESTIC_PASSWORD_COMMAND=pass show backups/milner

Add restic to flake.nix dev shell — add to buildInputs:
```
# Backup
restic
```

Update tests/test_static.py — add new scripts to shellcheck list:

scripts = [
    "control-plane/cleanup.sh",
    "control-plane/use-kubectl.sh",
    "control-plane/use-talos.sh",
    "worker/cleanup.sh",
    "backup/milner-backup.sh",
    "backup/milner-restore.sh",
    "backup/milner-init.sh",
    "backup/post-apply-hook.sh",
]

Update OPERATIONS.md — add a new §7 section documenting the backup system:
- How to initialize repos (milner-init.sh)
- How to run a manual backup (milner-backup.sh)
- How to restore (milner-restore.sh)
- How to verify snapshots (restic ls)
- How to enable the systemd timer
- Password management via pass or password manager

Update AGENTS.md — add backup/ to the Repo Map table:

| `backup/`           | Restic backup scripts, excludes, systemd units                     |

Create backup/README.md — documents the backup architecture, repo URLs, retention policy, and restore procedure.

Implementation Details

Two repo locations (issue specifies "temporarily" for the second):
- Primary: /mnt/backup/milner (external drive, mounted at /mnt)
- Secondary: /home/<user>/milner-backup (on host disk, outside failure domain)
- Both initialized with restic init, password from password manager
Password handling: RESTIC_PASSWORD_COMMAND calls the password manager CLI (e.g., pass show backups/milner). The password never touches disk. If the password manager requires interactive auth, the user unlocks it before running backups.¹
R2 bucket locks (optional immutability): Cloudflare R2 supports WORM bucket locks via dashboard or Wrangler CLI. Deferred to a follow-up — not blocking for initial integration.²
Dependencies: restic (added to flake.nix), password manager (pass or equivalent), bash (already required).
Constraints:
- All new shell scripts must pass shellcheck³
- Scripts follow repo conventions: set -e, cd "$(dirname "$0")" pattern⁴
- No secrets committed — milner-backup.env is gitignored
- Systemd units are user-level (~/.config/systemd/user/), not system-level
Out-of-scope notes:
- R2 bucket lock setup (optional, deferred)
- Kopia evaluation (issue specifies restic; Kopia comparison is in research findings)⁵
- Monitoring/alerting for failed backups (out of scope for initial integration)

Acceptance Criteria

backup/excludes.txt exists and excludes .terraform/ and .git/ only
backup/milner-backup.sh runs restic backup with --exclude-file and --tag milner, then restic forget with --keep-hourly 6 --keep-daily 30 --keep-monthly 12 --prune
backup/milner-init.sh accepts repo URLs and runs restic init
backup/milner-restore.sh runs restic restore with configurable snapshot and target
backup/post-apply-hook.sh triggers milner-backup.sh (sourced or exec'd)
restic added to flake.nix buildInputs
New shell scripts pass shellcheck (added to tests/test_static.py)
AGENTS.md Repo Map includes backup/ directory
OPERATIONS.md includes §7 documenting backup/restore workflow
backup/README.md documents architecture, repo URLs, retention, restore procedure

Testing Plan

Run python -m pytest tests/test_static.py -v — shellcheck passes for all new scripts
Run python -m pytest tests/test_static.py::test_shellcheck -v — specifically verify new scripts are linted
Manual: restic init --repo /tmp/test-repo then bash backup/milner-backup.sh with a test RESTIC_REPOS array pointing to /tmp/test-repo — verify snapshot created with correct tags and excludes
Manual: restic ls latest on test repo — confirm .env and .auto.tfvars files are included, .terraform/ and .git/ are excluded
Manual: bash backup/milner-restore.sh /tmp/test-repo latest --target /tmp/restore-test — verify files restored with correct paths

#41 — the parent issue

Naina: ## Overview Add automated restic backups of the Milner repo tree (Terraform state, gitignored tfvars, `.env` files) to two failure-domain-separated repositories — one on the `/mnt` external drive, one outside it. Exclude `.terraform/` and `.git/` (both rebuildable/replicated). Backups run on a 30-minute systemd timer and explicitly after every `terraform apply`. The restic password lives in a password manager and is never stored on disk. ## Technical Approach 1. **Create `backup/` directory** with the following structure: ``` backup/ ├── README.md ├── excludes.txt ├── milner-backup.sh # main backup script ├── milner-restore.sh # restore helper ├── milner-init.sh # one-time repo initialization ├── restic-backup.service # systemd unit ├── restic-backup.timer # systemd timer (30 min) └── post-apply-hook.sh # sourced after terraform apply ``` 2. **Create `backup/excludes.txt`** — newline-separated patterns: ``` # Rebuildable providers (840 MB) .terraform/ # Already replicated via git remote .git/ ``` 3. **Create `backup/milner-backup.sh`** — the core backup script: ```bash #!/bin/bash set -e cd "$(dirname "$0")" REPO_DIR="$(pwd)/.." # Password from password manager (never on disk) # Fallback: RESTIC_PASSWORD_FILE pointing to a temp file if no CLI available export RESTIC_PASSWORD_COMMAND="${RESTIC_PASSWORD_COMMAND:-pass show backups/milner}" for REPO in "${RESTIC_REPOS[@]}"; do echo "Backing up to $REPO ..." restic -r "$REPO" backup "$REPO_DIR" \ --exclude-file "$(dirname "$0")/excludes.txt" \ --tag milner --verbose restic -r "$REPO" forget --tag milner \ --group-by "paths,tags" \ --keep-hourly 6 --keep-daily 30 --keep-monthly 12 \ --prune done ``` Environment variable `RESTIC_REPOS` is an array set in a config file or exported by the caller. 4. **Create `backup/milner-init.sh`** — one-time repo initialization: ```bash #!/bin/bash set -e # Usage: milner-init.sh <repo-url> [<repo-url> ...] export RESTIC_PASSWORD_COMMAND="${RESTIC_PASSWORD_COMMAND:-pass show backups/milner}" for REPO in "$@"; do echo "Initializing $REPO ..." restic -r "$REPO" init done ``` 5. **Create `backup/milner-restore.sh`** — restore helper: ```bash #!/bin/bash set -e # Usage: milner-restore.sh <repo-url> [<snapshot-id>] [--target /path] export RESTIC_PASSWORD_COMMAND="${RESTIC_PASSWORD_COMMAND:-pass show backups/milner}" REPO="$1"; shift SNAPSHOT="${1:-latest}" restic -r "$REPO" restore "$SNAPSHOT" "$@" ``` 6. **Create `backup/post-apply-hook.sh`** — sourced by the user after `terraform apply`: ```bash # Source this after terraform apply to trigger an immediate snapshot: # source backup/post-apply-hook.sh REPO_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)" exec bash "$REPO_DIR/backup/milner-backup.sh" ``` 7. **Create systemd user units** in `backup/`: `restic-backup.service`: ```ini [Unit] Description=Restic backup of Milner repo tree [Service] Type=oneshot EnvironmentFile=%h/.config/milner-backup.env ExecStart=/path/to/milner-backup.sh ``` `restic-backup.timer`: ```ini [Unit] Description=Milner backup every 30 minutes [Timer] OnCalendar=*:0/30 Persistent=true [Install] WantedBy=timers.target ``` `milner-backup.env` (example, user creates on host): ```ini RESTIC_REPOS=( "/mnt/backup/milner" "/home/user/milner-backup" ) RESTIC_PASSWORD_COMMAND=pass show backups/milner ``` 8. **Add `restic` to `flake.nix` dev shell** — add to `buildInputs`: ```nix # Backup restic ``` 9. **Update `tests/test_static.py`** — add new scripts to shellcheck list: ```python scripts = [ "control-plane/cleanup.sh", "control-plane/use-kubectl.sh", "control-plane/use-talos.sh", "worker/cleanup.sh", "backup/milner-backup.sh", "backup/milner-restore.sh", "backup/milner-init.sh", "backup/post-apply-hook.sh", ] ``` 10. **Update `OPERATIONS.md`** — add a new §7 section documenting the backup system: - How to initialize repos (`milner-init.sh`) - How to run a manual backup (`milner-backup.sh`) - How to restore (`milner-restore.sh`) - How to verify snapshots (`restic ls`) - How to enable the systemd timer - Password management via `pass` or password manager 11. **Update `AGENTS.md`** — add `backup/` to the Repo Map table: ``` | `backup/` | Restic backup scripts, excludes, systemd units | ``` 12. **Create `backup/README.md`** — documents the backup architecture, repo URLs, retention policy, and restore procedure. ## Implementation Details - **Two repo locations** (issue specifies "temporarily" for the second): - Primary: `/mnt/backup/milner` (external drive, mounted at `/mnt`) - Secondary: `/home/<user>/milner-backup` (on host disk, outside failure domain) - Both initialized with `restic init`, password from password manager - **Password handling**: `RESTIC_PASSWORD_COMMAND` calls the password manager CLI (e.g., `pass show backups/milner`). The password never touches disk. If the password manager requires interactive auth, the user unlocks it before running backups.[1](https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html) - **R2 bucket locks** (optional immutability): Cloudflare R2 supports WORM bucket locks via dashboard or Wrangler CLI. Deferred to a follow-up — not blocking for initial integration.[2](https://developers.cloudflare.com/r2/buckets/bucket-locks/index.md) - **Dependencies**: `restic` (added to flake.nix), password manager (`pass` or equivalent), `bash` (already required). - **Constraints**: - All new shell scripts must pass `shellcheck`[3](https://github.com/koalaman/shellcheck) - Scripts follow repo conventions: `set -e`, `cd "$(dirname "$0")"` pattern[4](https://git.gloom.garden/tepichord/milner/src/commit/2b21c2e237670b7d060c1702d05487e609dcfedc/control-plane/cleanup.sh?display=source#L1-L3) - No secrets committed — `milner-backup.env` is gitignored - Systemd units are user-level (`~/.config/systemd/user/`), not system-level - **Out-of-scope notes**: - R2 bucket lock setup (optional, deferred) - Kopia evaluation (issue specifies restic; Kopia comparison is in research findings)[5](https://selfhosting.sh/compare/kopia-vs-restic/) - Monitoring/alerting for failed backups (out of scope for initial integration) ## Acceptance Criteria - [ ] `backup/excludes.txt` exists and excludes `.terraform/` and `.git/` only - [ ] `backup/milner-backup.sh` runs `restic backup` with `--exclude-file` and `--tag milner`, then `restic forget` with `--keep-hourly 6 --keep-daily 30 --keep-monthly 12 --prune` - [ ] `backup/milner-init.sh` accepts repo URLs and runs `restic init` - [ ] `backup/milner-restore.sh` runs `restic restore` with configurable snapshot and target - [ ] `backup/post-apply-hook.sh` triggers `milner-backup.sh` (sourced or exec'd) - [ ] `restic` added to `flake.nix` `buildInputs` - [ ] New shell scripts pass `shellcheck` (added to `tests/test_static.py`) - [ ] `AGENTS.md` Repo Map includes `backup/` directory - [ ] `OPERATIONS.md` includes §7 documenting backup/restore workflow - [ ] `backup/README.md` documents architecture, repo URLs, retention, restore procedure ## Testing Plan - Run `python -m pytest tests/test_static.py -v` — shellcheck passes for all new scripts - Run `python -m pytest tests/test_static.py::test_shellcheck -v` — specifically verify new scripts are linted - Manual: `restic init --repo /tmp/test-repo` then `bash backup/milner-backup.sh` with a test `RESTIC_REPOS` array pointing to `/tmp/test-repo` — verify snapshot created with correct tags and excludes - Manual: `restic ls latest` on test repo — confirm `.env` and `.auto.tfvars` files are included, `.terraform/` and `.git/` are excluded - Manual: `bash backup/milner-restore.sh /tmp/test-repo latest --target /tmp/restore-test` — verify files restored with correct paths ## Related Issues/PRs - #41 — the parent issue

tepichord added

and removed

labels

2026-09-10 11:50:40 -07:00

tepichord referenced this issue from a commit

2026-09-10 16:28:28 -07:00

feat(backup): add restic backup scripts and excludes

tepichord referenced this issue from a commit