updates

2026-07-18 19:11:32 -04:00
parent 9bc7a67020
commit 1d46b7b3ad
39 changed files with 2155 additions and 67 deletions
@@ -72,6 +72,21 @@ Permissions: directory `0700`. Do **not** commit `data/` or `.env`.
 | `PEARDATA_DOCKER_SOCKET` | `/var/run/docker.sock` | Unix socket for container name enrichment |
 | `PEARDATA_PROCESSES` | off | `1` enables process top-N CPU/RSS charts (Linux `/proc`) |
 | `PEARDATA_PROCESSES_TOP` | `8` | How many processes to keep in top charts (max 32) |
+| `PEARDATA_PARENT` | off | `1` enables parent peer fleet aggregator |
+| `PEARDATA_PARENT_PEERS` | empty | Comma-separated child agent public keys (64 hex) |
+| `PEARDATA_PARENT_PEERS_FILE` | — | Optional file with one child pubkey per line |
+| `PEARDATA_PARENT_SEED` | — | Optional `SERVER_SEED` of children for admin proof dial |
+| `PEARDATA_PARENT_POLL_MS` | `5000` | How often parent polls children |
+| `PEARDATA_NGINX` | off | `1` enables nginx stub_status collector |
+| `PEARDATA_NGINX_URL` | `http://127.0.0.1/nginx_status` | stub_status URL |
+| `PEARDATA_REDIS` | off | `1` enables Redis INFO collector |
+| `PEARDATA_REDIS_URL` | `127.0.0.1:6379` | Redis host:port or `redis://` URL |
+| `PEARDATA_POSTGRES` | off | `1` enables Postgres TCP probe (+ optional stats URL) |
+| `PEARDATA_POSTGRES_HOST` | `127.0.0.1` | Postgres host |
+| `PEARDATA_POSTGRES_PORT` | `5432` | Postgres port |
+| `PEARDATA_POSTGRES_STATS_URL` | — | Optional HTTP key=value stats sidecar |
+| `PEARDATA_EXPORT_DIR` | — | Write `snapshot-*.json` from export job/RPC |
+| `PEARDATA_PUSHGATEWAY_URL` | — | POST Prometheus text (Pushgateway-compatible) |

 Storage: `$PEARDATA_DATA_DIR/corestore` (named core `peardata-meta`).

@@ -75,12 +75,37 @@ Have `pipeline.js` start each enabled collector; all emit `samples` batches into
 2. Charts: `processes.top_cpu`, `processes.top_rss` (dimensions = process comm names).
 3. Linux `/proc` only; safe no-op on other platforms.

+### Nginx stub_status plugin
+
+1. Set `PEARDATA_NGINX=1` and `PEARDATA_NGINX_URL=…`.
+2. Charts: `nginx.connections`, `nginx.requests`.
+3. Base class: `server/services/collectors/plugin.js` (`CollectorPlugin`).
+
+### Redis INFO plugin
+
+1. Set `PEARDATA_REDIS=1` and optional `PEARDATA_REDIS_URL`.
+2. Charts: `redis.memory`, `redis.clients`, `redis.stats`.
+
+### Postgres plugin
+
+1. Set `PEARDATA_POSTGRES=1` (+ host/port).
+2. Charts: `postgres.up` (TCP probe); optional `postgres.stats` via `PEARDATA_POSTGRES_STATS_URL`.
+
+### Export / Prometheus push
+
+- Job `exportSnapshot` / RPC `exportSnapshot` / REST `GET /api/v3/export`
+- Job `prometheusPush` → `PEARDATA_PUSHGATEWAY_URL`
+- Optional file write: `PEARDATA_EXPORT_DIR`
+
 ## Parent peer (fleet aggregator)

-1. Parent dials child agents with `ConnectionManager`.
-2. Subscribes to `push:metrics` / periodically `queryData`.
-3. Ingests into local store under namespaced chart ids (`childPk.system.cpu`) or labels.
-4. Exposes `/api/v3/nodes` with multiple entries + `/api/v3/data` across nodes.
+Shipped opt-in spike:
+
+1. Set `PEARDATA_PARENT=1` and `PEARDATA_PARENT_PEERS=<childPk>,…`.
+2. Parent dials children over HyperDHT (`PearDataConnection`), polls health + metrics.
+3. Emits `fleet.cpu`, `fleet.ram`, `fleet.children` into the local store.
+4. REST: `GET /api/v3/nodes` (multi), `/api/v3/fleet`, `/api/v3/stream_path`.
+5. RPC: `getFleetHealth`, `listChildPeers`.

 ## PearDock / PearVirt hooks

@@ -86,10 +86,12 @@ Auth modes at handshake: public key (viewer), capability token / `pd1.` invite,

 | Method | Role | Notes |
 |--------|------|-------|
-| `getDbInfo` | viewer | DB + discovery keys, swarm flag |
+| `getDbInfo` | viewer | DB + discovery keys, swarm flag, remotes |
 | `listPeerLinks` | viewer | Linked peers from HyperDB |
-| `linkPeer` | admin | Upsert link; optional swarm join |
+| `linkPeer` | admin | Upsert link; open remote bee + optional swarm join |
 | `unlinkPeer` | admin | Remove link |
+| `getFleetHealth` | viewer | Parent rollup (when `PEARDATA_PARENT=1`) |
+| `listChildPeers` | viewer | Dialed child agents |

 See [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md).

@@ -54,9 +54,16 @@ curl -s http://127.0.0.1:19999/api/v3/health | jq
 | GET | `/api/v2/nodes` |
 | GET | `/api/v3/nodes` |
 | GET | `/api/v3/node_instances` |
+| GET | `/api/v3/fleet` |
 | GET | `/api/v3/stream_path` |

-Single-agent MVP returns one node (this host). Parent/fleet aggregation is roadmap.
+Single-agent returns one node. With `PEARDATA_PARENT=1`, `/nodes` and `/fleet` include dialed children.
+
+### Export
+
+| Method | Path | Notes |
+|--------|------|-------|
+| GET | `/api/v3/export` | JSON snapshot; `?format=prometheus` for text; `?write=1` writes `PEARDATA_EXPORT_DIR` |

 ### Contexts & charts

@@ -49,7 +49,7 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
 | Hyperswarm mesh (`PEARDATA_SWARM=1`) | Done (opt-in) |
 | Configurable retention knobs (tier sizes) | Done (`PEARDATA_TIER*`) |
 | Documented warm history across restart (operator M4) | Done (`store-hyperdb-fallback` + STORAGE doc) |
-| Export snapshot job → JSON / Prometheus remote-write | Next |
+| Export snapshot job → JSON / Prometheus push | Done (`exportSnapshot`, `prometheusPush`) |
 | Autobase multi-writer parents | Later (Phase 2c) |
 | Rocks engine for local-only desktop cache | Optional |

@@ -57,7 +57,7 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV

 - [x] `queryData` / REST `/data` can return `source: "hyperdb-warm"`
 - [x] Soak test: empty memory + reopen Corestore → `hyperdb-warm`
- [ ] Linked peer pulls warm points over swarm without re-scraping
+- [x] Linked peer pulls warm points (`source: "hyperdb-remote"`, `test/swarm-pull.test.js`)

 ---

@@ -68,7 +68,7 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
 | Docker / container discovery | Done (opt-in `PEARDATA_DOCKER=1`) |
 | Offline banner (last-known samples) | Done |
 | Process top-N | Done (opt-in `PEARDATA_PROCESSES=1`) |
-| Service plugins | nginx, postgres, redis collectors |
+| Service plugins | nginx + redis + postgres done |
 | PearDock bridge | Container metrics from dock peers |
 | PearVirt / BareOS adapters | Thin translators into shared contexts |
 | Holesail optional REST expose | Tunneled agent API |
@@ -78,18 +78,20 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
 - [x] One container host can emit per-container CPU/mem charts (`docker.cpu.*` / `docker.mem.*`)
 - [x] Plugin docs + Docker collector in [EXTENDING.md](./EXTENDING.md)
 - [x] Process top-N opt-in collector (`processes.top_cpu` / `processes.top_rss`)
+- [x] Plugin base + nginx / redis / postgres collectors

 ---

-## Phase 4 — Smarter anomalies & fleet
+## Phase 4 — Smarter anomalies & fleet  ← **started**

-| Item | Notes |
+| Item | Status |
 |------|--------|
+| Parent peer aggregation | Done spike (`PEARDATA_PARENT=1`) |
+| Fleet REST / RPC | Done (`/api/v3/fleet`, `getFleetHealth`) |
+| Desktop fleet strip | Done (`getFleetHealth` + multi-dial peers) |
 | Anomaly scoring + UI highlight | Beyond binary warn/crit |
 | Streaming z-score / k-means job | `runJob` retrain stub exists |
-| Fleet composite views | Multi-node health strip |
 | `/api/v3/weights` depth | Real metric weights |
-| Parent peer aggregation | P2P parent without SaaS |
 | Notifications | Desktop + optional webhook |

 ---
@@ -113,8 +115,14 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
 2. ~~Docker collector spike~~ ✅ (`PEARDATA_DOCKER=1`)  
 3. ~~Offline banner~~ ✅  
 4. ~~Process top-N~~ ✅ (`PEARDATA_PROCESSES=1`)  
-5. **Parent peer prototype** — aggregate health from N agents  
-6. **Swarm pull validation** — linked peer reads warm without re-scrape  
+5. ~~Parent peer prototype~~ ✅ (`PEARDATA_PARENT=1`)  
+6. ~~Swarm / linked warm pull~~ ✅ (`hyperdb-remote`)  
+7. ~~Postgres / Redis plugins~~ ✅  
+8. ~~Desktop fleet strip~~ ✅  
+9. ~~Export snapshot / Prometheus push~~ ✅  
+10. **Anomaly scoring UI** — highlight warn/crit on charts  
+11. **Notifications** — desktop + optional webhook  
+12. **PearDock bridge** — container metrics from dock peers 

 ---

@@ -137,7 +145,7 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
 | M3 | `curl \| bash` install; rolling server+client | ✅ |
 | M4 | Historical scrub across restart (HyperDB warm) | ✅ |
 | M5 | Container charts from Docker hosts | ✅ opt-in spike |
-| M6 | Parent peer rolling up a fleet | Planned |
+| M6 | Parent peer rolling up a fleet | ✅ opt-in spike |

 ---

@@ -18,6 +18,8 @@ Live 1-second samples stay in the **memory ring** and are pushed over protomux-r
 | Query fallback (memory → HyperDB) | ✅ | `MetricStore.query()` |
 | Agent boot upsert node | ✅ | `server/server.js` |
 | RPC: `getDbInfo`, `linkPeer`, `unlinkPeer`, `listPeerLinks` | ✅ | `server/handlers/monitor.js` |
+| Open remote bee by `dbKeyHex` + query | ✅ | `server/db/remote.js` → `source: hyperdb-remote` |
+| Persist `discoveryKeyHex` on peer-link + boot rejoin | ✅ | schema v2 + `rejoinLinkedPeers()` |
 | REST: `GET /api/v3/db` | ✅ | `server/rest/routes.js` |
 | Hyperswarm replicate | ✅ stub | `server/db/replicate.js` (`PEARDATA_SWARM=1`) |
 | Autobase multi-writer parents | ⏳ Phase C | See roadmap |