Updates
This commit is contained in:
Raven Scott
@@ -0,0 +1,290 @@
|
||||
# HyperDB storage & linked-node sync
|
||||
|
||||
How PearData should adopt Holepunch’s **HyperDB + Corestore + Hyperswarm (+ Autobase)** stack for durable storage and P2P sync between linked agents — based on patterns in local clones under `holepunchto_repos` (`hyperdb`, `hyperdb-workshop`, `hyperdb-autobase-workshop`, `corestore`, `hyperswarm`, `autobee`, `pear-hyperdb`).
|
||||
|
||||
## Why HyperDB (not “just SQLite”)
|
||||
|
||||
| Need | HyperDB fit |
|
||||
|------|-------------|
|
||||
| Typed collections + indexes | Hyperschema + `@ns/collection` + secondary indexes |
|
||||
| Local high-perf | `HyperDB.rocks(path, def)` |
|
||||
| P2P replicate | `HyperDB.bee(hypercore, def, { autoUpdate })` over Corestore |
|
||||
| Multi-writer HA parents | Autobase whose **view** is HyperDB (`extension: false`) |
|
||||
| Same query API local + remote | Workshops prove one `Registry` class works for both |
|
||||
|
||||
PearData’s current `server/services/store.js` is an **in-memory ring**. HyperDB replaces durability + sync; the in-memory tier stays as the **hot 1s path**.
|
||||
|
||||
## Stack mapping (from Holepunch repos)
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Agent
|
||||
COL[collector 1s] --> HOT[Memory tier0 ring]
|
||||
COL --> DS[Downsample]
|
||||
DS --> HDB[(HyperDB bee/rocks)]
|
||||
POL[peer-policy / alerts / links] --> HDB
|
||||
HOT --> RPC[protomux-rpc + REST]
|
||||
HDB --> RPC
|
||||
end
|
||||
subgraph Sync
|
||||
CS[Corestore] --> HDB
|
||||
SW[Hyperswarm] -->|store.replicate| CS
|
||||
AB[Autobase optional] -->|view| HDB
|
||||
end
|
||||
CHILD[Linked child agent] -.->|discoveryKey| SW
|
||||
PARENT[Parent / peer agent] -.-> SW
|
||||
DESK[Pear desktop cache] -.-> SW
|
||||
```
|
||||
|
||||
| Component | Repo pattern | PearData use |
|
||||
|-----------|--------------|--------------|
|
||||
| **Hyperschema + hyperdb/builder** | `hyperdb-workshop/build.js` | `spec/` codegen for collections |
|
||||
| **HyperDB.bee** | workshop `Registry` | Replicable agent DB |
|
||||
| **HyperDB.rocks** | `hyperdb` README | Optional local-only fast index |
|
||||
| **Corestore** | workshop `bin.js` | Named cores: `metrics-meta`, `alerts`, … |
|
||||
| **Hyperswarm** | `swarm.join(discoveryKey)` + `store.replicate(conn)` | Link nodes / seed DB |
|
||||
| **protomux-rpc** | already in PearData | Control plane (unchanged) |
|
||||
| **Autobase + hyperdispatch** | `hyperdb-autobase-workshop` | Multi-writer **parent** / HA registry |
|
||||
| **autobee** | experimental multiwriter bee | Alternative later; prefer Autobase+HyperDB view for now |
|
||||
| **pear-hyperdb** | Pear-shaped Model wrapper | Optional UX for desktop-local rocks |
|
||||
|
||||
## Critical design rule: don’t put 1s samples in HyperDB txs
|
||||
|
||||
HyperDB is an **indexable document DB** (put/get/find + flush). Writing every chart every second as HyperDB transactions will:
|
||||
|
||||
- Amplify Rocks/Bee write cost
|
||||
- Create huge replication chatter
|
||||
- Fight Netdata-class overhead goals
|
||||
|
||||
**Split planes:**
|
||||
|
||||
| Plane | Storage | Sync |
|
||||
|-------|---------|------|
|
||||
| **Hot live (≤1h @ 1s)** | Memory ring (current) | P2P `push:metrics` (current) |
|
||||
| **Warm history (downsampled)** | Hypercore append **or** HyperDB rows keyed `(chart, tsBucket)` | Corestore replicate |
|
||||
| **Metadata** (peers, alerts, labels, jobs, ACL cache) | **HyperDB** | Corestore replicate |
|
||||
| **Fleet / parent consensus** | Autobase → HyperDB view | Swarm on autobase discoveryKey |
|
||||
|
||||
## Proposed HyperDB schema (`@peardata/*`)
|
||||
|
||||
Modeled after workshop `build.js` namespaces.
|
||||
|
||||
### Collections
|
||||
|
||||
```text
|
||||
@peardata/node
|
||||
key: nodeId (string / pubkey hex)
|
||||
fields: hostname, platform, arch, cpus, agentVersion, labels{}, updatedAt
|
||||
|
||||
@peardata/peer-link
|
||||
key: [localNodeId, remotePublicKey]
|
||||
fields: role, alias, discoveryKey?, linkedAt, lastSeen, syncMode (push|pull|both)
|
||||
|
||||
@peardata/alert-config
|
||||
key: id
|
||||
fields: chart, dimension, warn, crit, comparator, enabled, info
|
||||
|
||||
@peardata/alert-event
|
||||
key: [id, ts] # or ulid
|
||||
fields: severity, value, threshold, message, cleared
|
||||
|
||||
@peardata/metric-point # WARM tier only (e.g. 1m buckets)
|
||||
key: [chart, ts]
|
||||
fields: context, values{} (map), tier
|
||||
|
||||
@peardata/job
|
||||
key: id
|
||||
fields: name, status, startedAt, finishedAt, result?
|
||||
```
|
||||
|
||||
### Indexes
|
||||
|
||||
```text
|
||||
@peardata/node-by-hostname → node.hostname
|
||||
@peardata/peer-link-by-remote → peer-link.remotePublicKey
|
||||
@peardata/alert-event-by-chart → alert-event.chart + ts
|
||||
@peardata/metric-point-by-context → metric-point.context + ts
|
||||
```
|
||||
|
||||
Rebuild with:
|
||||
|
||||
```bash
|
||||
node scripts/build-db.js # Hyperschema + HyperDB.toDisk → spec/
|
||||
```
|
||||
|
||||
## Agent integration shape
|
||||
|
||||
Follow `hyperdb-workshop` / `pear-hyperdb` Model pattern:
|
||||
|
||||
```text
|
||||
server/
|
||||
db/
|
||||
build.js # schema codegen
|
||||
spec/hyperschema/
|
||||
spec/hyperdb/
|
||||
model.js # PearDataModel: putNode, linkPeer, queryWarm, …
|
||||
replicate.js # Hyperswarm join + store.replicate
|
||||
services/
|
||||
store.js # HOT memory (keep)
|
||||
store-hyperdb.js # WARM + metadata facade used by queryData/REST
|
||||
```
|
||||
|
||||
### Boot (single-writer agent — Phase 2)
|
||||
|
||||
```js
|
||||
const store = new Corestore(dataDir + '/corestore')
|
||||
const swarm = new Hyperswarm({ keyPair: await store.createKeyPair('swarm') })
|
||||
swarm.on('connection', (conn) => store.replicate(conn))
|
||||
|
||||
const metaCore = store.get({ name: 'peardata-meta' })
|
||||
const db = HyperDB.bee(metaCore, spec, { autoUpdate: true })
|
||||
|
||||
// announce for linked peers / desktop seeders
|
||||
swarm.join(metaCore.discoveryKey, { server: true, client: true })
|
||||
```
|
||||
|
||||
Collector path:
|
||||
|
||||
1. Ingest → memory tier0 (unchanged)
|
||||
2. Every N samples → downsample → `tx.insert('@peardata/metric-point', …); tx.flush()`
|
||||
3. Alert transitions → `alert-event` collection
|
||||
4. `queryData` / REST: memory first, then HyperDB range scan for older windows
|
||||
|
||||
### Auth note
|
||||
|
||||
HyperDB replication shares **capability to read the core**, not PearData RPC roles. Keep:
|
||||
|
||||
- **Noise + MethodRoles** for mutating RPC (`setAlertConfig`, `runJob`)
|
||||
- Replication topic optionally gated (only invite-linked peers get discoveryKey / capability)
|
||||
- Do **not** announce writable cores to the public swarm without encryption / allowlist
|
||||
|
||||
## Linked nodes: sync modes
|
||||
|
||||
PearData “links” are first-class `@peardata/peer-link` rows + swarm topics.
|
||||
|
||||
| Mode | Behavior |
|
||||
|------|----------|
|
||||
| **Pull** | Local agent opens remote DB by key (`HyperDB.bee(store.get({ key }), spec, { writable: false, autoUpdate: true })`) and replicates |
|
||||
| **Push** | Remote peers allowed to replicate our meta/warm cores (seed) |
|
||||
| **Both** | Mutual swarm join (homelab mesh) |
|
||||
| **Parent aggregate** | Parent pulls many children; stores namespaced copies or Autobase fleet view |
|
||||
|
||||
Discovery:
|
||||
|
||||
1. Desktop/admin mints link → stores remote pubkey + optional `dbKey` (z32/hex)
|
||||
2. Agent joins `discoveryKey` of that core
|
||||
3. On connection: `store.replicate(conn)` (workshop pattern)
|
||||
4. `clone.watch` / `autoUpdate` refreshes HyperDB indexes for REST/RPC queries
|
||||
|
||||
This is the same pattern as workshop §3.1 Lookups — **read path is swarm + HyperDB**, not constant RPC polling.
|
||||
|
||||
## Parent / HA (Phase 4) — Autobase workshop
|
||||
|
||||
When you need multi-writer fleet registry or HA parents:
|
||||
|
||||
- Autobase bootstrap key shared across parent instances
|
||||
- `open: (store) => HyperDB.bee(store.get('db-view'), spec, { extension: false, autoUpdate: true })`
|
||||
- hyperdispatch ops: `add-writer`, `put-alert`, `put-link`, `ingest-rollup`
|
||||
- RPC (existing PearData / workshop) appends ops; apply mutates HyperDB view
|
||||
- Clients dial **any** writer; view key stays stable
|
||||
|
||||
Do **not** “backup” by copying Corestore folders (workshop warning). Rotate writers via Autobase instead.
|
||||
|
||||
## What stays on protomux-rpc
|
||||
|
||||
| Keep on RPC | Why |
|
||||
|-------------|-----|
|
||||
| Live `push:metrics` | Sub-second UX; not DB-shaped |
|
||||
| `subscribeMetrics` / handshake / ACL | Session security |
|
||||
| `runJob`, mintInvite | Mutating control |
|
||||
| On-demand `queryData` for hot window | Memory path |
|
||||
|
||||
| Move to HyperDB (+ replicate) | Why |
|
||||
|-------------------------------|-----|
|
||||
| Peer links, aliases, labels | Shared fleet truth |
|
||||
| Alert config + history | Durable, queryable |
|
||||
| Warm/cold metric buckets | History without central server |
|
||||
| Node inventory | Parent `/api/v3/nodes` |
|
||||
|
||||
## Phased delivery (recommended)
|
||||
|
||||
### Phase A — Local durability (rocks or bee, no swarm yet)
|
||||
|
||||
1. Add `hyperdb`, `hyperschema`, `corestore` deps
|
||||
2. `scripts/build-db.js` + `spec/`
|
||||
3. Persist alert configs, peer policy, warm downsample into HyperDB
|
||||
4. REST/RPC history falls back to HyperDB after memory miss
|
||||
5. Keep REST localhost semantics
|
||||
|
||||
### Phase B — Link & replicate
|
||||
|
||||
1. Hyperswarm alongside HyperDHT RPC (or reuse connections carefully — often **separate swarm for store.replicate**)
|
||||
2. `linkPeer` / `unlinkPeer` RPCs write `@peardata/peer-link`
|
||||
3. Desktop can seed/cache agent DB by key for offline scrubbing
|
||||
4. Document `pd1.` invite vs **db discovery key** (two layers)
|
||||
|
||||
### Phase C — Parent fleet
|
||||
|
||||
1. Parent process pulls N child DB keys
|
||||
2. Composite REST `/api/v3/nodes` + scoped `/api/v3/data`
|
||||
3. Optional Autobase for parent HA
|
||||
|
||||
### Phase D — Polish
|
||||
|
||||
1. Encryption at rest (`encryptionKey` on cores / autobee)
|
||||
2. Blind-peer seeding for always-on warm history
|
||||
3. Grafana: already have REST Prometheus export; optionally expose hypercore-stats
|
||||
|
||||
## Concrete file plan (when implementing)
|
||||
|
||||
```text
|
||||
peardata/
|
||||
scripts/build-db.js
|
||||
spec/hyperschema/
|
||||
spec/hyperdb/
|
||||
server/db/model.js
|
||||
server/db/replicate.js
|
||||
server/services/store-hyperdb.js
|
||||
docs/STORAGE-HYPERDB.md ← this file
|
||||
```
|
||||
|
||||
Deps (approximate):
|
||||
|
||||
```json
|
||||
"hyperdb": "^6",
|
||||
"hyperschema": "^1",
|
||||
"corestore": "^7",
|
||||
"hyperswarm": "^4",
|
||||
"autobase": "^7",
|
||||
"hyperdispatch": "^1"
|
||||
```
|
||||
|
||||
(`autobee` only if you prefer that multiwriter path over Autobase+HyperDB view.)
|
||||
|
||||
## Relationship to current PearData roadmap
|
||||
|
||||
| Roadmap item | HyperDB role |
|
||||
|--------------|--------------|
|
||||
| Phase 2 tiered storage | Warm/cold collections |
|
||||
| Phase 3 PearDock/containers | Extra collections / indexes |
|
||||
| Phase 4 parent peer | Autobase + replicated views |
|
||||
| REST v3 historical queries | `find` ranges on `@peardata/metric-point` |
|
||||
|
||||
## References (local clones)
|
||||
|
||||
| Path under `holepunchto_repos` | Takeaway |
|
||||
|--------------------------------|----------|
|
||||
| `hyperdb/README.md` | rocks vs bee, find/get/tx, autoUpdate |
|
||||
| `hyperdb-workshop` | Schema builder, Corestore+Swarm replicate, RPC inserts |
|
||||
| `hyperdb-autobase-workshop` | Multi-writer view, hyperdispatch, HA |
|
||||
| `corestore/README.md` | `store.replicate(conn)`, namespacing |
|
||||
| `pear-hyperdb` | Thin Model wrapper pattern for Pear apps |
|
||||
| `autobee` | Experimental multiwriter bee alternative |
|
||||
|
||||
## Decision summary
|
||||
|
||||
1. **Yes — incorporate HyperDB** for metadata + warm history + linked-node sync.
|
||||
2. **Keep memory + RPC pushes** for live 1s Netdata feel.
|
||||
3. **Sync linked nodes via Corestore replication on Hyperswarm**, not by streaming every sample over RPC.
|
||||
4. **Use Autobase+HyperDB** when you need multi-writer parents / HA — same pattern as Holepunch’s own workshops.
|
||||
5. **Treat discovery keys as sensitive** — link only invited peers; RPC AuthZ remains authoritative for mutations.
|
||||
Reference in New Issue
Block a user