Testing

2026-07-18 16:33:43 -04:00
parent f747ffbd25
commit f7e26d1aac
53 changed files with 6491 additions and 740 deletions
@@ -1,290 +1,308 @@
 # HyperDB storage & linked-node sync

-How PearData should adopt Holepunch’s **HyperDB + Corestore + Hyperswarm (+ Autobase)** stack for durable storage and P2P sync between linked agents — based on patterns in local clones under `holepunchto_repos` (`hyperdb`, `hyperdb-workshop`, `hyperdb-autobase-workshop`, `corestore`, `hyperswarm`, `autobee`, `pear-hyperdb`).
+PearData uses Holepunch **HyperDB + Corestore (+ optional Hyperswarm)** for durable metadata, warm metric history, and P2P sync between linked agents.

-## Why HyperDB (not “just SQLite”)
+Live 1-second samples stay in the **memory ring** and are pushed over protomux-rpc. HyperDB is for everything that must survive restarts and replicate.

-| Need | HyperDB fit |
-|------|-------------|
-| Typed collections + indexes | Hyperschema + `@ns/collection` + secondary indexes |
-| Local high-perf | `HyperDB.rocks(path, def)` |
-| P2P replicate | `HyperDB.bee(hypercore, def, { autoUpdate })` over Corestore |
-| Multi-writer HA parents | Autobase whose **view** is HyperDB (`extension: false`) |
-| Same query API local + remote | Workshops prove one `Registry` class works for both |
+---

-PearData’s current `server/services/store.js` is an **in-memory ring**. HyperDB replaces durability + sync; the in-memory tier stays as the **hot 1s path**.
+## Status (implemented)

-## Stack mapping (from Holepunch repos)
+| Piece | Status | Location |
+|-------|--------|----------|
+| Schema codegen | ✅ | `scripts/build-db.js` → `spec/` |
+| Collections + indexes | ✅ | `@peardata/*` in `spec/hyperdb` |
+| `PearDataModel` | ✅ | `server/db/model.js` |
+| Corestore open/close | ✅ | `server/db/index.js` |
+| Warm flush from tier1 | ✅ | `server/services/warm-flush.js` + `store` `warm` event |
+| Query fallback (memory → HyperDB) | ✅ | `MetricStore.query()` |
+| Agent boot upsert node | ✅ | `server/server.js` |
+| RPC: `getDbInfo`, `linkPeer`, `unlinkPeer`, `listPeerLinks` | ✅ | `server/handlers/monitor.js` |
+| REST: `GET /api/v3/db` | ✅ | `server/rest/routes.js` |
+| Hyperswarm replicate | ✅ stub | `server/db/replicate.js` (`PEARDATA_SWARM=1`) |
+| Autobase multi-writer parents | ⏳ Phase C | See roadmap |
+
+Disable HyperDB: `PEARDATA_HYPERDB=0`.
+
+---
+
+## Architecture

 ```mermaid
 flowchart TB
-  subgraph Agent
-    COL[collector 1s] --> HOT[Memory tier0 ring]
-    COL --> DS[Downsample]
-    DS --> HDB[(HyperDB bee/rocks)]
-    POL[peer-policy / alerts / links] --> HDB
-    HOT --> RPC[protomux-rpc + REST]
-    HDB --> RPC
-  end
-  subgraph Sync
-    CS[Corestore] --> HDB
-    SW[Hyperswarm] -->|store.replicate| CS
-    AB[Autobase optional] -->|view| HDB
-  end
-  CHILD[Linked child agent] -.->|discoveryKey| SW
-  PARENT[Parent / peer agent] -.-> SW
-  DESK[Pear desktop cache] -.-> SW
+  COL[collector 1s] --> MEM[Memory tier0 ring]
+  COL --> T1[Memory tier1 downsample]
+  T1 -->|warm event| FLUSH[warm-flush queue]
+  FLUSH --> HDB[(HyperDB.bee peardata-meta)]
+  ANO[anomaly engine] -->|alert events| HDB
+  LINK[linkPeer RPC] --> HDB
+  MEM --> PUSH[push:metrics RPC]
+  MEM --> Q[queryData / REST]
+  HDB --> Q
+  CS[Corestore data/corestore] --> HDB
+  SW[Hyperswarm PEARDATA_SWARM=1] -->|store.replicate| CS
 ```

-| Component | Repo pattern | PearData use |
-|-----------|--------------|--------------|
-| **Hyperschema + hyperdb/builder** | `hyperdb-workshop/build.js` | `spec/` codegen for collections |
-| **HyperDB.bee** | workshop `Registry` | Replicable agent DB |
-| **HyperDB.rocks** | `hyperdb` README | Optional local-only fast index |
-| **Corestore** | workshop `bin.js` | Named cores: `metrics-meta`, `alerts`, … |
-| **Hyperswarm** | `swarm.join(discoveryKey)` + `store.replicate(conn)` | Link nodes / seed DB |
-| **protomux-rpc** | already in PearData | Control plane (unchanged) |
-| **Autobase + hyperdispatch** | `hyperdb-autobase-workshop` | Multi-writer **parent** / HA registry |
-| **autobee** | experimental multiwriter bee | Alternative later; prefer Autobase+HyperDB view for now |
-| **pear-hyperdb** | Pear-shaped Model wrapper | Optional UX for desktop-local rocks |
-
-## Critical design rule: don’t put 1s samples in HyperDB txs
-
-HyperDB is an **indexable document DB** (put/get/find + flush). Writing every chart every second as HyperDB transactions will:
-
- Amplify Rocks/Bee write cost
- Create huge replication chatter
- Fight Netdata-class overhead goals
-
-**Split planes:**
+### Planes

 | Plane | Storage | Sync |
 |-------|---------|------|
-| **Hot live (≤1h @ 1s)** | Memory ring (current) | P2P `push:metrics` (current) |
-| **Warm history (downsampled)** | Hypercore append **or** HyperDB rows keyed `(chart, tsBucket)` | Corestore replicate |
-| **Metadata** (peers, alerts, labels, jobs, ACL cache) | **HyperDB** | Corestore replicate |
-| **Fleet / parent consensus** | Autobase → HyperDB view | Swarm on autobase discoveryKey |
+| Hot live (~1h @ 1s) | Memory | `push:metrics` |
+| Warm history (~1m buckets) | HyperDB `@peardata/metric-point` | Corestore replicate |
+| Metadata / links / alerts | HyperDB collections | Corestore replicate |
+| Fleet HA (future) | Autobase → HyperDB view | Swarm on autobase key |

-## Proposed HyperDB schema (`@peardata/*`)
+### Why not HyperDB for every 1s sample

-Modeled after workshop `build.js` namespaces.
+HyperDB is transactional + indexed. Flushing every chart every second would inflate write amplification and replication traffic. Tier1 downsample (default every 60 samples ≈ 1 minute) is the durable path.
+
+---
+
+## Schema (`@peardata`)
+
+Defined in `scripts/build-db.js`. **Append-only** — never delete fields from committed `spec/` (Holepunch safety rule).

 ### Collections

-```text
-@peardata/node
-  key: nodeId (string / pubkey hex)
-  fields: hostname, platform, arch, cpus, agentVersion, labels{}, updatedAt
-
-@peardata/peer-link
-  key: [localNodeId, remotePublicKey]
-  fields: role, alias, discoveryKey?, linkedAt, lastSeen, syncMode (push|pull|both)
-
-@peardata/alert-config
-  key: id
-  fields: chart, dimension, warn, crit, comparator, enabled, info
-
-@peardata/alert-event
-  key: [id, ts]   # or ulid
-  fields: severity, value, threshold, message, cleared
-
-@peardata/metric-point   # WARM tier only (e.g. 1m buckets)
-  key: [chart, ts]
-  fields: context, values{} (map), tier
-
-@peardata/job
-  key: id
-  fields: name, status, startedAt, finishedAt, result?
-```
+| Collection | Key | Purpose |
+|------------|-----|---------|
+| `@peardata/node` | `nodeId` | Agent / host inventory |
+| `@peardata/peer-link` | `localNodeId` + `remotePublicKey` | Linked peers + sync mode |
+| `@peardata/alert-config` | `id` | Threshold configs |
+| `@peardata/alert-event` | `id` + `ts` | Anomaly / alert history |
+| `@peardata/metric-point` | `chart` + `ts` | Warm downsampled samples (`valuesJson`) |
+| `@peardata/job` | `id` | Persisted job records (optional use) |

 ### Indexes

-```text
-@peardata/node-by-hostname          → node.hostname
-@peardata/peer-link-by-remote       → peer-link.remotePublicKey
-@peardata/alert-event-by-chart      → alert-event.chart + ts
-@peardata/metric-point-by-context   → metric-point.context + ts
-```
+| Index | On |
+|-------|-----|
+| `@peardata/node-by-hostname` | hostname |
+| `@peardata/peer-link-by-remote` | remotePublicKey |
+| `@peardata/alert-event-by-chart` | chart + ts |
+| `@peardata/metric-point-by-context` | context + ts |
+| `@peardata/metric-point-by-tier` | tier + chart + ts |

-Rebuild with:
+### Field notes
+
+- `valuesJson` / `labelsJson` / `resultJson` — JSON strings for open-ended maps (avoids rigid hyperschema maps).
+- `tier` on metric-point: `1` = default warm (~1m). Future coarser tiers use `2+`.
+- `syncMode` on peer-link: `push` | `pull` | `both`.
+
+### Regenerate after schema edits

 ```bash
-node scripts/build-db.js   # Hyperschema + HyperDB.toDisk → spec/
+npm run build:db
+# or: node scripts/build-db.js
 ```

-## Agent integration shape
+Then commit `spec/hyperschema/*` and `spec/hyperdb/*`.

-Follow `hyperdb-workshop` / `pear-hyperdb` Model pattern:
+---
+
+## Runtime layout

 ```text
-server/
-  db/
-    build.js              # schema codegen
-    spec/hyperschema/
-    spec/hyperdb/
-    model.js              # PearDataModel: putNode, linkPeer, queryWarm, …
-    replicate.js          # Hyperswarm join + store.replicate
-  services/
-    store.js              # HOT memory (keep)
-    store-hyperdb.js      # WARM + metadata facade used by queryData/REST
+$PEARDATA_DATA_DIR/          # default ./data
+  peer-policy.json           # existing ACL file
+  audit.log
+  corestore/                 # Corestore (HyperDB bee cores)
 ```

-### Boot (single-writer agent — Phase 2)
+Named core: **`peardata-meta`**.

-```js
-const store = new Corestore(dataDir + '/corestore')
-const swarm = new Hyperswarm({ keyPair: await store.createKeyPair('swarm') })
-swarm.on('connection', (conn) => store.replicate(conn))
+Banner fields on agent start:

-const metaCore = store.get({ name: 'peardata-meta' })
-const db = HyperDB.bee(metaCore, spec, { autoUpdate: true })
+- `hyperdb:` — DB public key (hex) for others to open a read replica
+- `swarm:` — `on` / `off`

-// announce for linked peers / desktop seeders
-swarm.join(metaCore.discoveryKey, { server: true, client: true })
-```
+---

-Collector path:
+## Configuration

-1. Ingest → memory tier0 (unchanged)
-2. Every N samples → downsample → `tx.insert('@peardata/metric-point', …); tx.flush()`
-3. Alert transitions → `alert-event` collection
-4. `queryData` / REST: memory first, then HyperDB range scan for older windows
+| Variable | Default | Meaning |
+|----------|---------|---------|
+| `PEARDATA_HYPERDB` | on | `0` / `off` disables HyperDB entirely |
+| `PEARDATA_DATA_DIR` | `./data` | Policy + corestore root |
+| `PEARDATA_SWARM` | off | `1` enables Hyperswarm `store.replicate` |
+| `PEARDATA_TIER1_EVERY` | `60` | Samples per warm bucket (~60s @ 1Hz) |
+| `PEARDATA_TIER1_POINTS` | `1440` | In-memory tier1 ring (HyperDB keeps longer) |

-### Auth note
+---

-HyperDB replication shares **capability to read the core**, not PearData RPC roles. Keep:
+## API surface

- **Noise + MethodRoles** for mutating RPC (`setAlertConfig`, `runJob`)
- Replication topic optionally gated (only invite-linked peers get discoveryKey / capability)
- Do **not** announce writable cores to the public swarm without encryption / allowlist
+### RPC

-## Linked nodes: sync modes
+| Method | Role | Description |
+|--------|------|-------------|
+| `getDbInfo` | viewer | `{ enabled, publicKeyHex, discoveryKeyHex, swarm }` |
+| `listPeerLinks` | viewer | Linked peers from HyperDB |
+| `linkPeer` | admin | Upsert link; optionally `joinRemoteTopic` if swarm on |
+| `unlinkPeer` | admin | Remove link |
+| `queryData` | viewer | Memory first; HyperDB warm if miss / `tier≥1` |

-PearData “links” are first-class `@peardata/peer-link` rows + swarm topics.
-
-| Mode | Behavior |
-|------|----------|
-| **Pull** | Local agent opens remote DB by key (`HyperDB.bee(store.get({ key }), spec, { writable: false, autoUpdate: true })`) and replicates |
-| **Push** | Remote peers allowed to replicate our meta/warm cores (seed) |
-| **Both** | Mutual swarm join (homelab mesh) |
-| **Parent aggregate** | Parent pulls many children; stores namespaced copies or Autobase fleet view |
-
-Discovery:
-
-1. Desktop/admin mints link → stores remote pubkey + optional `dbKey` (z32/hex)
-2. Agent joins `discoveryKey` of that core
-3. On connection: `store.replicate(conn)` (workshop pattern)
-4. `clone.watch` / `autoUpdate` refreshes HyperDB indexes for REST/RPC queries
-
-This is the same pattern as workshop §3.1 Lookups — **read path is swarm + HyperDB**, not constant RPC polling.
-
-## Parent / HA (Phase 4) — Autobase workshop
-
-When you need multi-writer fleet registry or HA parents:
-
- Autobase bootstrap key shared across parent instances
- `open: (store) => HyperDB.bee(store.get('db-view'), spec, { extension: false, autoUpdate: true })`
- hyperdispatch ops: `add-writer`, `put-alert`, `put-link`, `ingest-rollup`
- RPC (existing PearData / workshop) appends ops; apply mutates HyperDB view
- Clients dial **any** writer; view key stays stable
-
-Do **not** “backup” by copying Corestore folders (workshop warning). Rotate writers via Autobase instead.
-
-## What stays on protomux-rpc
-
-| Keep on RPC | Why |
-|-------------|-----|
-| Live `push:metrics` | Sub-second UX; not DB-shaped |
-| `subscribeMetrics` / handshake / ACL | Session security |
-| `runJob`, mintInvite | Mutating control |
-| On-demand `queryData` for hot window | Memory path |
-
-| Move to HyperDB (+ replicate) | Why |
-|-------------------------------|-----|
-| Peer links, aliases, labels | Shared fleet truth |
-| Alert config + history | Durable, queryable |
-| Warm/cold metric buckets | History without central server |
-| Node inventory | Parent `/api/v3/nodes` |
-
-## Phased delivery (recommended)
-
-### Phase A — Local durability (rocks or bee, no swarm yet)
-
-1. Add `hyperdb`, `hyperschema`, `corestore` deps
-2. `scripts/build-db.js` + `spec/`
-3. Persist alert configs, peer policy, warm downsample into HyperDB
-4. REST/RPC history falls back to HyperDB after memory miss
-5. Keep REST localhost semantics
-
-### Phase B — Link & replicate
-
-1. Hyperswarm alongside HyperDHT RPC (or reuse connections carefully — often **separate swarm for store.replicate**)
-2. `linkPeer` / `unlinkPeer` RPCs write `@peardata/peer-link`
-3. Desktop can seed/cache agent DB by key for offline scrubbing
-4. Document `pd1.` invite vs **db discovery key** (two layers)
-
-### Phase C — Parent fleet
-
-1. Parent process pulls N child DB keys
-2. Composite REST `/api/v3/nodes` + scoped `/api/v3/data`
-3. Optional Autobase for parent HA
-
-### Phase D — Polish
-
-1. Encryption at rest (`encryptionKey` on cores / autobee)
-2. Blind-peer seeding for always-on warm history
-3. Grafana: already have REST Prometheus export; optionally expose hypercore-stats
-
-## Concrete file plan (when implementing)
-
-```text
-peardata/
-  scripts/build-db.js
-  spec/hyperschema/
-  spec/hyperdb/
-  server/db/model.js
-  server/db/replicate.js
-  server/services/store-hyperdb.js
-  docs/STORAGE-HYPERDB.md          ← this file
-```
-
-Deps (approximate):
+`linkPeer` args:

 ```json
-"hyperdb": "^6",
-"hyperschema": "^1",
-"corestore": "^7",
-"hyperswarm": "^4",
-"autobase": "^7",
-"hyperdispatch": "^1"
+{
+  "remotePublicKey": "<64 hex>",
+  "alias": "homelab-nas",
+  "dbKeyHex": "<optional remote db key>",
+  "discoveryKeyHex": "<optional topic to pull>",
+  "syncMode": "both",
+  "role": "viewer"
+}
 ```

-(`autobee` only if you prefer that multiwriter path over Autobase+HyperDB view.)
+### REST

-## Relationship to current PearData roadmap
+| Path | Description |
+|------|-------------|
+| `GET /api/v3/db` | HyperDB keys + collection list |
+| `GET /api/v3/info` | Includes `peardata.hyperdb` block |
+| `GET /api/v3/data?...` | Uses async store query (warm fallback) |
+| Query `source` field | `memory-tier0` \| `memory-tier1` \| `hyperdb-warm` |

-| Roadmap item | HyperDB role |
-|--------------|--------------|
-| Phase 2 tiered storage | Warm/cold collections |
-| Phase 3 PearDock/containers | Extra collections / indexes |
-| Phase 4 parent peer | Autobase + replicated views |
-| REST v3 historical queries | `find` ranges on `@peardata/metric-point` |
+```bash
+curl -s http://127.0.0.1:19999/api/v3/db | jq
+curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-3600&tier=1&points=120' | jq '.source'
+```

-## References (local clones)
+---

-| Path under `holepunchto_repos` | Takeaway |
-|--------------------------------|----------|
-| `hyperdb/README.md` | rocks vs bee, find/get/tx, autoUpdate |
-| `hyperdb-workshop` | Schema builder, Corestore+Swarm replicate, RPC inserts |
-| `hyperdb-autobase-workshop` | Multi-writer view, hyperdispatch, HA |
-| `corestore/README.md` | `store.replicate(conn)`, namespacing |
-| `pear-hyperdb` | Thin Model wrapper pattern for Pear apps |
-| `autobee` | Experimental multiwriter bee alternative |
+## How linked-node sync works

-## Decision summary
+### Phase A (now) — local durability

-1. **Yes — incorporate HyperDB** for metadata + warm history + linked-node sync.
-2. **Keep memory + RPC pushes** for live 1s Netdata feel.
-3. **Sync linked nodes via Corestore replication on Hyperswarm**, not by streaming every sample over RPC.
-4. **Use Autobase+HyperDB** when you need multi-writer parents / HA — same pattern as Holepunch’s own workshops.
-5. **Treat discovery keys as sensitive** — link only invited peers; RPC AuthZ remains authoritative for mutations.
+1. Agent opens Corestore + HyperDB on boot.
+2. Warm points + alert events persist under `data/corestore`.
+3. Restart retains warm history / links / node row.
+
+### Phase B — mesh replicate (`PEARDATA_SWARM=1`)
+
+Workshop pattern (`hyperdb-workshop/bin.js`):
+
+```js
+swarm.on('connection', (conn) => store.replicate(conn))
+swarm.join(db.discoveryKey, { server: true, client: true })
+```
+
+1. Enable swarm on agents that should seed/pull.
+2. Share **db public key** + **discovery key** (`getDbInfo` / `/api/v3/db`).
+3. Admin calls `linkPeer` with `discoveryKeyHex` + `syncMode: pull|both`.
+4. Peers replicate Corestore; HyperDB `autoUpdate` refreshes indexes.
+5. Remote warm history becomes queryable locally (parent path).
+
+**Security:** replication grants read of the core to anyone who can join the topic. Prefer:
+
+- Swarm only on trusted LAN / VPN, or
+- Future: encrypted cores + allowlisted swarm joins
+- Keep **mutations** on protomux-rpc AuthZ (`linkPeer` is admin)
+
+HyperDHT RPC identity ≠ Hyperswarm topic access — treat them as two layers.
+
+### Phase C — Autobase parents (planned)
+
+Same as `hyperdb-autobase-workshop`:
+
+- Autobase bootstrap key across parent writers
+- View = `HyperDB.bee(store.get('db-view'), spec, { extension: false })`
+- hyperdispatch ops for `put-link`, `put-alert`, rollups
+- Do **not** “backup” by copying Corestore folders
+
+---
+
+## Code map
+
+| Path | Role |
+|------|------|
+| `scripts/build-db.js` | Hyperschema + HyperDB builder |
+| `spec/hyperschema/` | Generated encodings + `schema.json` |
+| `spec/hyperdb/` | Generated collections/indexes |
+| `server/db/model.js` | Typed CRUD facade |
+| `server/db/index.js` | Singleton open/close |
+| `server/db/replicate.js` | Optional Hyperswarm |
+| `server/services/warm-flush.js` | Batch writer |
+| `server/services/store.js` | Hot ring + `warm` events + query fallback |
+
+### Model usage example
+
+```js
+import { openDb, getDb, closeDb } from './server/db/index.js'
+
+await openDb()
+const db = getDb()
+
+await db.putPeerLink({
+  localNodeId: 'abc123',
+  remotePublicKey: 'ff'.repeat(32),
+  syncMode: 'pull',
+  linkedAt: Date.now(),
+})
+
+await db.putMetricPoints([
+  { chart: 'system.cpu', context: 'system.cpu', ts: Date.now(), values: { user: 12.3 }, tier: 1 },
+])
+
+const rows = await db.queryMetricPoints({
+  chart: 'system.cpu',
+  afterMs: Date.now() - 3600_000,
+  beforeMs: Date.now(),
+})
+```
+
+---
+
+## Dependencies
+
+From Holepunch stack (see `holepunchto_repos`):
+
+- `hyperdb` — DB engine (bee + rocks)
+- `hyperschema` — struct codegen
+- `corestore` — named hypercores + replicate
+- `hyperswarm` — topic discovery for store sync
+- `ready-resource` — open/close lifecycle
+
+Future: `autobase`, `hyperdispatch` for HA parents.
+
+---
+
+## Testing
+
+```bash
+npm run build:db
+SKIP_INTEGRATION=1 npm test
+# includes test/hyperdb.test.js
+```
+
+Manual:
+
+```bash
+npm run start:server
+curl -s http://127.0.0.1:19999/api/v3/db | jq
+# wait ~60s for first warm bucket, then:
+curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-7200&tier=1&points=120' | jq '.source,.points'
+```
+
+---
+
+## Operational checklist
+
+- [ ] Commit `spec/` after every `build:db`
+- [ ] Back up **keys** (`SERVER_SEED`, swarm keypair in corestore) — not by zipping live corestore while writing
+- [ ] Keep REST on localhost; swarm only when linking trusted peers
+- [ ] Prefer `pd1.` invites for RPC admin; share db discovery keys only with linked nodes
+- [ ] Monitor disk under `data/corestore` as warm retention grows
+
+---
+
+## References
+
+| Repo (local `holepunchto_repos`) | Takeaway |
+|----------------------------------|----------|
+| `hyperdb` | rocks vs bee, tx/flush, autoUpdate |
+| `hyperdb-workshop` | builder + Corestore + Swarm replicate |
+| `hyperdb-autobase-workshop` | Multi-writer view for parents |
+| `corestore` | `store.replicate(conn)` |
+| `pear-hyperdb` | Pear Model wrapper style |