Plan out system architecture

tepichord commented

2026-08-17 10:02:33 -07:00

Owner

high level overview of how the evolved npcs pods will function
- A talos control plane on the strongest machine (naina) with a minimal amount of resources delegated for memory and cpu
- Worker nodes will fill the rest of the resource space on the remaining machine (naina's remainder, and all of ida)
- I want an evaluation determining the pros and cons of having small worker nodes scoped per application, vs large worker nodes that host multiple pods across different applications (for more context, naina has 24 GB of ram, while ida has 8 GB's. Each have 8 CPU cores). This is regarding more than just the evolved-npcs infrastructure
- For evolved npcs specifically, I want a pod that is orchestrating the simulation system, basically just the evolved-npcs core.
- Outside of the orchestrator, I want pods that run the simulations either in Unreal Engine or Godot. Now for resource management I want these simulations to scale to fill up essentially all of the available resources except leave a 10% headroom for the rest of the cluster. The orchestrator can basically create an unlimited amount of work for the cluster to process.
- now to process the work, the simulation pods will utilize RabbitMQ to accept work tasks.
- for scalability, we will need to limit the number of simulations based on the resource usage. I can see two potential ways to do this. Perhaps we get an upper bound on the maximum numbers of simulations that the simulation pod can handle. Then we could simply spin up the simulation pods if there are enough resources to support them so like if we know that ass simulation pod can run at max 20 simulations using 4 GB of RAM and two CPU's, then we let the simulation pod just limit itself to 20 simulations make me spin them up based on the resource usage. The other option is that we could use RabbitMQ and Kubernetes insights to handle the resources themselves in the simulation pods are just large scalable pods that consume the rest of the resources on a worker node/machine. I would like both of these options to be investigated for their feasibility, and provide one alternative option.

- high level overview of how the evolved npcs pods will function - A talos control plane on the strongest machine (naina) with a minimal amount of resources delegated for memory and cpu - Worker nodes will fill the rest of the resource space on the remaining machine (naina's remainder, and all of ida) - I want an evaluation determining the pros and cons of having small worker nodes scoped per application, vs large worker nodes that host multiple pods across different applications (for more context, naina has 24 GB of ram, while ida has 8 GB's. Each have 8 CPU cores). This is regarding more than just the evolved-npcs infrastructure - For evolved npcs specifically, I want a pod that is orchestrating the simulation system, basically just the evolved-npcs core. - Outside of the orchestrator, I want pods that run the simulations either in Unreal Engine or Godot. Now for resource management I want these simulations to scale to fill up essentially all of the available resources except leave a 10% headroom for the rest of the cluster. The orchestrator can basically create an unlimited amount of work for the cluster to process. - now to process the work, the simulation pods will utilize RabbitMQ to accept work tasks. - for scalability, we will need to limit the number of simulations based on the resource usage. I can see two potential ways to do this. Perhaps we get an upper bound on the maximum numbers of simulations that the simulation pod can handle. Then we could simply spin up the simulation pods if there are enough resources to support them so like if we know that ass simulation pod can run at max 20 simulations using 4 GB of RAM and two CPU's, then we let the simulation pod just limit itself to 20 simulations make me spin them up based on the resource usage. The other option is that we could use RabbitMQ and Kubernetes insights to handle the resources themselves in the simulation pods are just large scalable pods that consume the rest of the resources on a worker node/machine. I would like both of these options to be investigated for their feasibility, and provide one alternative option.

tepichord added the

labels

2026-08-17 10:02:33 -07:00

tepichord added this to the MVP project

2026-08-17 10:02:33 -07:00

tepichord added

and removed

labels

2026-08-18 12:57:35 -07:00

tepichord commented

2026-08-18 13:30:56 -07:00

Author

Owner

Naina:
Updated on 2026-08-18 based on maintainer feedback.

Overview

Produce a written architecture plan document (ARCHITECTURE.md) that evaluates worker node sizing strategies, simulation pod scaling approaches, and the overall pod topology for running evolved-npcs workloads on a 2-machine Talos Linux cluster (naina: 24 GB / 8 CPU, ida: 8 GB / 8 CPU). The document is the deliverable — no code changes are required in this phase.

Technical Approach

Create ARCHITECTURE.md with three main sections

# Evolved-NPCs Architecture Plan

## 1. Worker Node Sizing
## 2. Simulation Pod Scaling
## 3. Pod Topology

Worker Node Sizing — Hybrid approach (recommended by maintainer)

Option	Description	Overhead
Many small workers	2-4 GB VMs, one per application	~300-400 MB per node × N nodes
Few large workers	12-20 GB VMs, multi-tenant	~300-400 MB per node × 2
Hybrid (preferred)	1×20 GB on naina (multi-tenant), 1×7 GB on ida (simulation-only)	~700 MB total

naina (24 GB):
├── control-plane VM  ── 2 GB RAM, 2 vCPU  (existing)
└── worker-large VM   ── 20 GB RAM, 6 vCPU  (multi-tenant: CI, RabbitMQ, orchestrator)

ida (8 GB):
└── worker-sim VM     ── 7 GB RAM, 8 vCPU  (simulation pods only)

Simulation Pod Scaling — Strategy C (recommended by maintainer): KEDA with resource-based guardrails

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: simulation-scaler
spec:
  scaleTargetRef:
    name: simulation
  minReplicaCount: 1
  maxReplicaCount: 8
  cooldownPeriod: 300
  triggers:
  - type: rabbitmq
    metadata:
      protocol: http
      queueName: sim-tasks
      mode: QueueLength
      value: "10"
      host: http://guest:guest@rabbitmq.rabbitmq.svc.cluster.local:15672/
  - type: cpu
    metadata:
      type: Utilization
      value: "85"
  - type: memory
    metadata:
      type: Utilization
      value: "80"

Queue depth drives scaling, resource metrics act as a brake
Prevents resource exhaustion without external controllers
HPA picks the highest replica recommendation across all triggers

Alternative: Strategy A (fixed pod count with self-limiting simulations)

apiVersion: apps/v1
kind: Deployment
metadata:
  name: simulation
spec:
  replicas: 5  # pre-calculated from available resources
  template:
    spec:
      containers:
      - name: sim
        resources:
          requests: { cpu: "2", memory: "4Gi" }
          limits:   { cpu: "2", memory: "4Gi" }

Each pod runs ≤20 simulations internally, manages its own concurrency
Simple, predictable, no external dependencies
Not responsive to queue depth changes

Pod Topology

worker-large (naina):
├── kube-system (system pods)
├── vault         ← Vault + nginx TLS proxy (migrated from podman-compose)
├── woodpecker    ← CI agents
├── rabbitmq      ← RabbitMQ Cluster Operator (100m CPU, 128Mi RAM)
└── evolved-npcs
    └── orchestrator  ← evolved-npcs core (creates work)

worker-sim (ida):
└── evolved-npcs
    ├── simulation-0
    ├── simulation-1
    ├── ...
    └── simulation-N  ← scaled to fill ~90% of 7 GB allocatable

Resource budget calculation

ida allocatable:
  7 GB total - 400 MB Talos overhead = 6.6 GB allocatable
  90% for simulations = ~5.9 GB
  Per-pod (4 GB request) → max 1 simulation pod on ida alone
  Per-pod (2 GB request) → max 2 simulation pods on ida alone

naina worker allocatable:
  20 GB total - 400 MB Talos overhead = 19.6 GB allocatable
  Reserve for CI, Vault, RabbitMQ, orchestrator ≈ 4-6 GB
  Remaining for simulations ≈ 13.6-15.6 GB → 90% ≈ 12.2-14.0 GB

Note: Per-pod resource requirements depend on the chosen runtime. Godot is significantly lighter (256 MB – 1 GB per instance) than Unreal (2-4 GB per instance).

Runtime considerations (Godot vs Unreal)

Runtime	arm64	Per-pod memory	Notes
Godot (headless)	✅ Native	256 MB – 1 GB	`--headless` flag, no GPU needed
Unreal (dedicated server)	⚠️ Supported	2-4 GB	Requires source build, heavier

Recommendation: design for Godot first, accommodate UE as optional.

RabbitMQ deployment

Use RabbitMQ Cluster Kubernetes Operator. Minimal resource allocation for ~100-message workload:

apiVersion: rabbitmq.com/v1beta1
kind: RabbitmqCluster
metadata:
  name: rabbitmq
spec:
  replicas: 1
  resources:
    requests:
      cpu: 100m
      memory: 128Mi
    limits:
      cpu: 250m
      memory: 256Mi

Requires cert-manager for Operator v2.20+.

Vault Secrets Operator integration

Deploy VSO via Helm, sync secrets to evolved-npcs namespace:

apiVersion: secrets.hashicorp.com/v1beta1
kind: VaultStaticSecret
metadata:
  namespace: evolved-npcs
  name: app-secrets
spec:
  vaultAuthRef: vault-auth
  mount: kvv2
  type: kv-v2
  path: evolved-npcs/config
  refreshAfter: 60s
  destination:
    create: true
    name: app-secrets-k8s

Implementation Details

Files to create:

ARCHITECTURE.md — the main deliverable, containing all evaluation sections

Documentation updates (noted per maintainer request):

README.md — add reference to ARCHITECTURE.md
PLAN.md — add link to architecture plan in the relevant phase

Dependencies:

None for the document itself. Actual implementation of the architecture (Terraform, Helm charts, KEDA deployment) is out of scope for this issue.

Constraints:

arm64-only cluster — Unreal Engine has limited arm64 support
Talos Linux is immutable, API-only (no SSH)
Only 2 physical machines with asymmetric resources
Existing infra: 1 control-plane VM, standalone Vault via podman-compose
.gitignore patterns must be preserved for any new subdirectories

Out-of-scope notes:

No Terraform/Helm/Kubernetes manifests will be created in this issue
No actual deployment or testing of the architecture
Migration of Vault from podman-compose to Kubernetes is a separate concern
3-node HA control plane upgrade is mentioned but not detailed (existing infra constraint)

Acceptance Criteria

ARCHITECTURE.md exists at the repo root with all three main sections (Worker Node Sizing, Simulation Pod Scaling, Pod Topology)
Worker node sizing recommends Hybrid approach (1×20 GB on naina, 1×7 GB on ida) with pros/cons table
Simulation pod scaling recommends Strategy C (KEDA with queue depth + resource guardrails) with concrete YAML and includes Strategy A as alternative
Pod topology diagram shows the target layout across both physical machines
Resource budget calculation accounts for Talos overhead, lightweight RabbitMQ allocation (100m/128Mi), and 90% headroom target
Document addresses both Godot and Unreal Engine as simulation runtimes
Topology information referenced in README, PLAN, and ARCHITECTURE docs

Testing Plan

Review: Human reviewer validates the architecture addresses all requirements from the issue description
Review: Verify resource budget math is correct (Talos overhead, 90% headroom, per-pod allocations)
Review: Confirm arm64 compatibility notes for all referenced tools (RabbitMQ, KEDA, Godot, UE)

Issue #6 — Plan out system architecture (this issue)
Existing PLAN.md — Project vision and phased roadmap (context for the architecture)
worker/vm-setup/create-worker-vm.sh — Current worker VM creation script (to be evolved)

Naina: *Updated on 2026-08-18 based on maintainer feedback.* #### Overview Produce a written architecture plan document (`ARCHITECTURE.md`) that evaluates worker node sizing strategies, simulation pod scaling approaches, and the overall pod topology for running evolved-npcs workloads on a 2-machine Talos Linux cluster (naina: 24 GB / 8 CPU, ida: 8 GB / 8 CPU). The document is the deliverable — no code changes are required in this phase. #### Technical Approach 1. **Create `ARCHITECTURE.md` with three main sections** ``` # Evolved-NPCs Architecture Plan ## 1. Worker Node Sizing ## 2. Simulation Pod Scaling ## 3. Pod Topology ``` 2. **Worker Node Sizing — Hybrid approach (recommended by maintainer)** | Option | Description | Overhead | |--------|-------------|----------| | Many small workers | 2-4 GB VMs, one per application | ~300-400 MB per node × N nodes | | Few large workers | 12-20 GB VMs, multi-tenant | ~300-400 MB per node × 2 | | **Hybrid (preferred)** | 1×20 GB on naina (multi-tenant), 1×7 GB on ida (simulation-only) | ~700 MB total | ```text naina (24 GB): ├── control-plane VM ── 2 GB RAM, 2 vCPU (existing) └── worker-large VM ── 20 GB RAM, 6 vCPU (multi-tenant: CI, RabbitMQ, orchestrator) ida (8 GB): └── worker-sim VM ── 7 GB RAM, 8 vCPU (simulation pods only) ``` 3. **Simulation Pod Scaling — Strategy C (recommended by maintainer): KEDA with resource-based guardrails** ```yaml apiVersion: keda.sh/v1alpha1 kind: ScaledObject metadata: name: simulation-scaler spec: scaleTargetRef: name: simulation minReplicaCount: 1 maxReplicaCount: 8 cooldownPeriod: 300 triggers: - type: rabbitmq metadata: protocol: http queueName: sim-tasks mode: QueueLength value: "10" host: http://guest:guest@rabbitmq.rabbitmq.svc.cluster.local:15672/ - type: cpu metadata: type: Utilization value: "85" - type: memory metadata: type: Utilization value: "80" ``` - Queue depth drives scaling, resource metrics act as a brake - Prevents resource exhaustion without external controllers - HPA picks the highest replica recommendation across all triggers **Alternative: Strategy A (fixed pod count with self-limiting simulations)** ```yaml apiVersion: apps/v1 kind: Deployment metadata: name: simulation spec: replicas: 5 # pre-calculated from available resources template: spec: containers: - name: sim resources: requests: { cpu: "2", memory: "4Gi" } limits: { cpu: "2", memory: "4Gi" } ``` - Each pod runs ≤20 simulations internally, manages its own concurrency - Simple, predictable, no external dependencies - Not responsive to queue depth changes 4. **Pod Topology** ```text worker-large (naina): ├── kube-system (system pods) ├── vault ← Vault + nginx TLS proxy (migrated from podman-compose) ├── woodpecker ← CI agents ├── rabbitmq ← RabbitMQ Cluster Operator (100m CPU, 128Mi RAM) └── evolved-npcs └── orchestrator ← evolved-npcs core (creates work) worker-sim (ida): └── evolved-npcs ├── simulation-0 ├── simulation-1 ├── ... └── simulation-N ← scaled to fill ~90% of 7 GB allocatable ``` 5. **Resource budget calculation** ```text ida allocatable: 7 GB total - 400 MB Talos overhead = 6.6 GB allocatable 90% for simulations = ~5.9 GB Per-pod (4 GB request) → max 1 simulation pod on ida alone Per-pod (2 GB request) → max 2 simulation pods on ida alone naina worker allocatable: 20 GB total - 400 MB Talos overhead = 19.6 GB allocatable Reserve for CI, Vault, RabbitMQ, orchestrator ≈ 4-6 GB Remaining for simulations ≈ 13.6-15.6 GB → 90% ≈ 12.2-14.0 GB ``` *Note: Per-pod resource requirements depend on the chosen runtime. Godot is significantly lighter (256 MB – 1 GB per instance) than Unreal (2-4 GB per instance).* 6. **Runtime considerations (Godot vs Unreal)** | Runtime | arm64 | Per-pod memory | Notes | |---------|-------|----------------|-------| | Godot (headless) | ✅ Native | 256 MB – 1 GB | `--headless` flag, no GPU needed | | Unreal (dedicated server) | ⚠️ Supported | 2-4 GB | Requires source build, heavier | Recommendation: design for Godot first, accommodate UE as optional. 7. **RabbitMQ deployment** Use RabbitMQ Cluster Kubernetes Operator. Minimal resource allocation for ~100-message workload: ```yaml apiVersion: rabbitmq.com/v1beta1 kind: RabbitmqCluster metadata: name: rabbitmq spec: replicas: 1 resources: requests: cpu: 100m memory: 128Mi limits: cpu: 250m memory: 256Mi ``` *Requires cert-manager for Operator v2.20+.* 8. **Vault Secrets Operator integration** Deploy VSO via Helm, sync secrets to evolved-npcs namespace: ```yaml apiVersion: secrets.hashicorp.com/v1beta1 kind: VaultStaticSecret metadata: namespace: evolved-npcs name: app-secrets spec: vaultAuthRef: vault-auth mount: kvv2 type: kv-v2 path: evolved-npcs/config refreshAfter: 60s destination: create: true name: app-secrets-k8s ``` #### Implementation Details **Files to create:** - `ARCHITECTURE.md` — the main deliverable, containing all evaluation sections **Documentation updates (noted per maintainer request):** - `README.md` — add reference to `ARCHITECTURE.md` - `PLAN.md` — add link to architecture plan in the relevant phase **Dependencies:** - None for the document itself. Actual implementation of the architecture (Terraform, Helm charts, KEDA deployment) is out of scope for this issue. **Constraints:** - arm64-only cluster — Unreal Engine has limited arm64 support - Talos Linux is immutable, API-only (no SSH) - Only 2 physical machines with asymmetric resources - Existing infra: 1 control-plane VM, standalone Vault via podman-compose - `.gitignore` patterns must be preserved for any new subdirectories **Out-of-scope notes:** - No Terraform/Helm/Kubernetes manifests will be created in this issue - No actual deployment or testing of the architecture - Migration of Vault from podman-compose to Kubernetes is a separate concern - 3-node HA control plane upgrade is mentioned but not detailed (existing infra constraint) #### Acceptance Criteria - [ ] `ARCHITECTURE.md` exists at the repo root with all three main sections (Worker Node Sizing, Simulation Pod Scaling, Pod Topology) - [ ] Worker node sizing recommends Hybrid approach (1×20 GB on naina, 1×7 GB on ida) with pros/cons table - [ ] Simulation pod scaling recommends Strategy C (KEDA with queue depth + resource guardrails) with concrete YAML and includes Strategy A as alternative - [ ] Pod topology diagram shows the target layout across both physical machines - [ ] Resource budget calculation accounts for Talos overhead, lightweight RabbitMQ allocation (100m/128Mi), and 90% headroom target - [ ] Document addresses both Godot and Unreal Engine as simulation runtimes - [ ] Topology information referenced in README, PLAN, and ARCHITECTURE docs #### Testing Plan - [ ] Review: Human reviewer validates the architecture addresses all requirements from the issue description - [ ] Review: Verify resource budget math is correct (Talos overhead, 90% headroom, per-pod allocations) - [ ] Review: Confirm arm64 compatibility notes for all referenced tools (RabbitMQ, KEDA, Godot, UE) #### Related Issues/PRs - Issue #6 — Plan out system architecture (this issue) - Existing `PLAN.md` — Project vision and phased roadmap (context for the architecture) - `worker/vm-setup/create-worker-vm.sh` — Current worker VM creation script (to be evolved)

tepichord added

and removed

labels

2026-08-18 13:31:25 -07:00

tepichord commented

2026-08-18 14:13:45 -07:00

Author

Owner

I do prefer the Hybrid solution with strategy C for scaling, update the comment to reflect that
Make a note to add the topology information to documentation on the repository (README, PLAN, ARCHITECTURE, etc.)

Add section on RabbitMQ deployment ... Initial allocation: 1 CPU / 1 GB RAM on worker-large

I will say, the RabbitMQ pod will need very few resources, we're talking about maybe 100 messages in a queue max and each message is a very small payload. I honestly doubt it would use more than even a megabyte of data.

- I do prefer the Hybrid solution with strategy C for scaling, update the comment to reflect that - Make a note to add the topology information to documentation on the repository (README, PLAN, ARCHITECTURE, etc.) > Add section on RabbitMQ deployment ... Initial allocation: 1 CPU / 1 GB RAM on worker-large I will say, the RabbitMQ pod will need very few resources, we're talking about maybe 100 messages in a queue max and each message is a very small payload. I honestly doubt it would use more than even a megabyte of data.

tepichord added

and removed

labels

2026-08-18 21:14:18 -07:00

tepichord referenced this issue from a commit

2026-08-18 22:53:34 -07:00

docs(architecture): add evolved-npcs architecture plan with cross-references

tepichord referenced this issue from a pull request that will close it,