332 lines
11 KiB
Markdown
332 lines
11 KiB
Markdown
# HyperDB storage & linked-node sync
|
||
|
||
PearData uses Holepunch **HyperDB + Corestore (+ optional Hyperswarm)** for durable metadata, warm metric history, and P2P sync between linked agents.
|
||
|
||
Live 1-second samples stay in the **memory ring** and are pushed over protomux-rpc. HyperDB is for everything that must survive restarts and replicate.
|
||
|
||
---
|
||
|
||
## Status (implemented)
|
||
|
||
| Piece | Status | Location |
|
||
|-------|--------|----------|
|
||
| Schema codegen | ✅ | `scripts/build-db.js` → `spec/` |
|
||
| Collections + indexes | ✅ | `@peardata/*` in `spec/hyperdb` |
|
||
| `PearDataModel` | ✅ | `server/db/model.js` |
|
||
| Corestore open/close | ✅ | `server/db/index.js` |
|
||
| Warm flush from tier1 | ✅ | `server/services/warm-flush.js` + `store` `warm` event |
|
||
| Query fallback (memory → HyperDB) | ✅ | `MetricStore.query()` |
|
||
| Agent boot upsert node | ✅ | `server/server.js` |
|
||
| RPC: `getDbInfo`, `linkPeer`, `unlinkPeer`, `listPeerLinks` | ✅ | `server/handlers/monitor.js` |
|
||
| Open remote bee by `dbKeyHex` + query | ✅ | `server/db/remote.js` → `source: hyperdb-remote` |
|
||
| Persist `discoveryKeyHex` on peer-link + boot rejoin | ✅ | schema v2 + `rejoinLinkedPeers()` |
|
||
| REST: `GET /api/v3/db` | ✅ | `server/rest/routes.js` |
|
||
| Hyperswarm replicate | ✅ stub | `server/db/replicate.js` (`PEARDATA_SWARM=1`) |
|
||
| Autobase multi-writer parents | ⏳ Phase C | See roadmap |
|
||
|
||
Disable HyperDB: `PEARDATA_HYPERDB=0`.
|
||
|
||
---
|
||
|
||
## Architecture
|
||
|
||
```mermaid
|
||
flowchart TB
|
||
COL[collector 1s] --> MEM[Memory tier0 ring]
|
||
COL --> T1[Memory tier1 downsample]
|
||
T1 -->|warm event| FLUSH[warm-flush queue]
|
||
FLUSH --> HDB[(HyperDB.bee peardata-meta)]
|
||
ANO[anomaly engine] -->|alert events| HDB
|
||
LINK[linkPeer RPC] --> HDB
|
||
MEM --> PUSH[push:metrics RPC]
|
||
MEM --> Q[queryData / REST]
|
||
HDB --> Q
|
||
CS[Corestore data/corestore] --> HDB
|
||
SW[Hyperswarm PEARDATA_SWARM=1] -->|store.replicate| CS
|
||
```
|
||
|
||
### Planes
|
||
|
||
| Plane | Storage | Sync |
|
||
|-------|---------|------|
|
||
| Hot live (~1h @ 1s) | Memory | `push:metrics` |
|
||
| Warm history (~1m buckets) | HyperDB `@peardata/metric-point` | Corestore replicate |
|
||
| Metadata / links / alerts | HyperDB collections | Corestore replicate |
|
||
| Fleet HA (future) | Autobase → HyperDB view | Swarm on autobase key |
|
||
|
||
### Why not HyperDB for every 1s sample
|
||
|
||
HyperDB is transactional + indexed. Flushing every chart every second would inflate write amplification and replication traffic. Tier1 downsample (default every 60 samples ≈ 1 minute) is the durable path.
|
||
|
||
---
|
||
|
||
## Schema (`@peardata`)
|
||
|
||
Defined in `scripts/build-db.js`. **Append-only** — never delete fields from committed `spec/` (Holepunch safety rule).
|
||
|
||
### Collections
|
||
|
||
| Collection | Key | Purpose |
|
||
|------------|-----|---------|
|
||
| `@peardata/node` | `nodeId` | Agent / host inventory |
|
||
| `@peardata/peer-link` | `localNodeId` + `remotePublicKey` | Linked peers + sync mode |
|
||
| `@peardata/alert-config` | `id` | Threshold configs |
|
||
| `@peardata/alert-event` | `id` + `ts` | Anomaly / alert history |
|
||
| `@peardata/metric-point` | `chart` + `ts` | Warm downsampled samples (`valuesJson`) |
|
||
| `@peardata/job` | `id` | Persisted job records (optional use) |
|
||
|
||
### Indexes
|
||
|
||
| Index | On |
|
||
|-------|-----|
|
||
| `@peardata/node-by-hostname` | hostname |
|
||
| `@peardata/peer-link-by-remote` | remotePublicKey |
|
||
| `@peardata/alert-event-by-chart` | chart + ts |
|
||
|
||
Warm `@peardata/metric-point` is queried only by primary key (`chart` + `ts`). Secondary indexes `metric-point-by-context` / `metric-point-by-tier` are **deprecated** (no longer written) to cut Hyperbee write amplification (~3×) on high-volume metric inserts/deletes.
|
||
|
||
### Field notes
|
||
|
||
- `valuesJson` / `labelsJson` / `resultJson` — JSON strings for open-ended maps (avoids rigid hyperschema maps).
|
||
- `tier` on metric-point: `1` = default warm (~1m). Future coarser tiers use `2+`.
|
||
- `syncMode` on peer-link: `push` | `pull` | `both`.
|
||
|
||
### Regenerate after schema edits
|
||
|
||
```bash
|
||
npm run build:db
|
||
# or: node scripts/build-db.js
|
||
```
|
||
|
||
Then commit `spec/hyperschema/*` and `spec/hyperdb/*`.
|
||
|
||
---
|
||
|
||
## Runtime layout
|
||
|
||
```text
|
||
$PEARDATA_DATA_DIR/ # default ./data
|
||
peer-policy.json # existing ACL file
|
||
audit.log
|
||
corestore/ # Corestore (HyperDB bee cores)
|
||
```
|
||
|
||
Named core: **`peardata-meta`**.
|
||
|
||
Banner fields on agent start:
|
||
|
||
- `hyperdb:` — DB public key (hex) for others to open a read replica
|
||
- `swarm:` — `on` / `off`
|
||
|
||
---
|
||
|
||
## Configuration
|
||
|
||
| Variable | Default | Meaning |
|
||
|----------|---------|---------|
|
||
| `PEARDATA_HYPERDB` | on | `0` / `off` disables HyperDB entirely |
|
||
| `PEARDATA_DATA_DIR` | `./data` | Policy + corestore root |
|
||
| `PEARDATA_SWARM` | off | `1` enables Hyperswarm `store.replicate` |
|
||
| `PEARDATA_TIER1_EVERY` | `60` | Samples per warm bucket (~60s @ 1Hz) |
|
||
| `PEARDATA_TIER1_POINTS` | `1440` | In-memory tier1 ring (HyperDB keeps longer) |
|
||
| `PEARDATA_WARM_RETENTION_MS` | `3m` | Seeds Data Manager age rotate |
|
||
| `PEARDATA_WARM_MAX_BYTES` | `1 GiB` | Soft corestore budget; `0` = unlimited |
|
||
| `PEARDATA_WARM_MAX_POINTS` | `0` | Soft point cap; `0` = unlimited |
|
||
| `PEARDATA_AUTO_PRUNE` | on | Scheduled prune after boot |
|
||
| `PEARDATA_WARM_ALL_CHARTS` | off | `1` flushes high-cardinality instance charts to HyperDB (default: static catalog only) |
|
||
|
||
**Data Manager:** desktop **Settings → Data** (admin) calls `setRetentionConfig` / `pruneHistory`. Policy file: `$PEARDATA_DATA_DIR/retention.json`. Auto-prune runs on an interval and:
|
||
|
||
1. Deletes HyperDB `@peardata/metric-point` rows older than the age window (multi-pass)
|
||
2. Enforces optional point / soft disk budgets (multi-pass tighter cutoffs)
|
||
3. **Physical reclaim:** `hyperbee.clearUnlinked` + `hypercore.compact` so RocksDB actually frees space
|
||
|
||
HyperDB deletes alone are logical (append-only tombstones); without step 3 corestore size does not shrink.
|
||
|
||
---
|
||
|
||
## API surface
|
||
|
||
### RPC
|
||
|
||
| Method | Role | Description |
|
||
|--------|------|-------------|
|
||
| `getDbInfo` | viewer | `{ enabled, publicKeyHex, discoveryKeyHex, swarm }` |
|
||
| `listPeerLinks` | viewer | Linked peers from HyperDB |
|
||
| `linkPeer` | admin | Upsert link; optionally `joinRemoteTopic` if swarm on |
|
||
| `unlinkPeer` | admin | Remove link |
|
||
| `queryData` | viewer | Memory first; HyperDB warm if miss / `tier≥1` |
|
||
| `getStorageInfo` | viewer | Disk + memory usage for Data Manager |
|
||
| `getRetentionConfig` / `setRetentionConfig` | viewer / admin | Warm age rotate (default **3m**), **1 GiB** disk budget, auto-prune |
|
||
| `pruneHistory` | admin | Delete warm points + clearUnlinked/compact (`gcBuffers` job alias). Returns real `bytesBefore` / `bytesAfter` / `bytesFreed`. |
|
||
|
||
`linkPeer` args:
|
||
|
||
```json
|
||
{
|
||
"remotePublicKey": "<64 hex>",
|
||
"alias": "homelab-nas",
|
||
"dbKeyHex": "<optional remote db key>",
|
||
"discoveryKeyHex": "<optional topic to pull>",
|
||
"syncMode": "both",
|
||
"role": "viewer"
|
||
}
|
||
```
|
||
|
||
### REST
|
||
|
||
| Path | Description |
|
||
|------|-------------|
|
||
| `GET /api/v3/db` | HyperDB keys + collection list |
|
||
| `GET /api/v3/info` | Includes `peardata.hyperdb` block |
|
||
| `GET /api/v3/data?...` | Uses async store query (warm fallback) |
|
||
| Query `source` field | `memory-tier0` \| `memory-tier1` \| `hyperdb-warm` |
|
||
|
||
```bash
|
||
curl -s http://127.0.0.1:18888/api/v3/db | jq
|
||
curl -s 'http://127.0.0.1:18888/api/v3/data?chart=system.cpu&after=-3600&tier=1&points=120' | jq '.source'
|
||
```
|
||
|
||
---
|
||
|
||
## How linked-node sync works
|
||
|
||
### Phase A (now) — local durability
|
||
|
||
1. Agent opens Corestore + HyperDB on boot.
|
||
2. Warm points + alert events persist under `data/corestore`.
|
||
3. Restart retains warm history / links / node row.
|
||
|
||
### Phase B — mesh replicate (`PEARDATA_SWARM=1`)
|
||
|
||
Workshop pattern (`hyperdb-workshop/bin.js`):
|
||
|
||
```js
|
||
swarm.on('connection', (conn) => store.replicate(conn))
|
||
swarm.join(db.discoveryKey, { server: true, client: true })
|
||
```
|
||
|
||
1. Enable swarm on agents that should seed/pull.
|
||
2. Share **db public key** + **discovery key** (`getDbInfo` / `/api/v3/db`).
|
||
3. Admin calls `linkPeer` with `discoveryKeyHex` + `syncMode: pull|both`.
|
||
4. Peers replicate Corestore; HyperDB `autoUpdate` refreshes indexes.
|
||
5. Remote warm history becomes queryable locally (parent path).
|
||
|
||
**Security:** replication grants read of the core to anyone who can join the topic. Prefer:
|
||
|
||
- Swarm only on trusted LAN / VPN, or
|
||
- Future: encrypted cores + allowlisted swarm joins
|
||
- Keep **mutations** on protomux-rpc AuthZ (`linkPeer` is admin)
|
||
|
||
HyperDHT RPC identity ≠ Hyperswarm topic access — treat them as two layers.
|
||
|
||
### Phase C — Autobase parents (planned)
|
||
|
||
Same as `hyperdb-autobase-workshop`:
|
||
|
||
- Autobase bootstrap key across parent writers
|
||
- View = `HyperDB.bee(store.get('db-view'), spec, { extension: false })`
|
||
- hyperdispatch ops for `put-link`, `put-alert`, rollups
|
||
- Do **not** “backup” by copying Corestore folders
|
||
|
||
---
|
||
|
||
## Code map
|
||
|
||
| Path | Role |
|
||
|------|------|
|
||
| `scripts/build-db.js` | Hyperschema + HyperDB builder |
|
||
| `spec/hyperschema/` | Generated encodings + `schema.json` |
|
||
| `spec/hyperdb/` | Generated collections/indexes |
|
||
| `server/db/model.js` | Typed CRUD facade |
|
||
| `server/db/index.js` | Singleton open/close |
|
||
| `server/db/replicate.js` | Optional Hyperswarm |
|
||
| `server/services/warm-flush.js` | Batch writer |
|
||
| `server/services/store.js` | Hot ring + `warm` events + query fallback |
|
||
|
||
### Model usage example
|
||
|
||
```js
|
||
import { openDb, getDb, closeDb } from './server/db/index.js'
|
||
|
||
await openDb()
|
||
const db = getDb()
|
||
|
||
await db.putPeerLink({
|
||
localNodeId: 'abc123',
|
||
remotePublicKey: 'ff'.repeat(32),
|
||
syncMode: 'pull',
|
||
linkedAt: Date.now(),
|
||
})
|
||
|
||
await db.putMetricPoints([
|
||
{ chart: 'system.cpu', context: 'system.cpu', ts: Date.now(), values: { user: 12.3 }, tier: 1 },
|
||
])
|
||
|
||
const rows = await db.queryMetricPoints({
|
||
chart: 'system.cpu',
|
||
afterMs: Date.now() - 3600_000,
|
||
beforeMs: Date.now(),
|
||
})
|
||
```
|
||
|
||
---
|
||
|
||
## Dependencies
|
||
|
||
From Holepunch stack (see `holepunchto_repos`):
|
||
|
||
- `hyperdb` — DB engine (bee + rocks)
|
||
- `hyperschema` — struct codegen
|
||
- `corestore` — named hypercores + replicate
|
||
- `hyperswarm` — topic discovery for store sync
|
||
- `ready-resource` — open/close lifecycle
|
||
|
||
Future: `autobase`, `hyperdispatch` for HA parents.
|
||
|
||
---
|
||
|
||
## Testing
|
||
|
||
```bash
|
||
npm run build:db
|
||
SKIP_INTEGRATION=1 npm test
|
||
# includes test/hyperdb.test.js + test/store-hyperdb-fallback.test.js (M4)
|
||
```
|
||
|
||
### M4 soak — history across restart
|
||
|
||
Automated: `store-hyperdb-fallback` writes warm points, clears the memory rings, and asserts `query()` returns `source: "hyperdb-warm"` after reopening Corestore (simulates agent restart).
|
||
|
||
Manual:
|
||
|
||
```bash
|
||
npm run start:server
|
||
curl -s http://127.0.0.1:18888/api/v3/db | jq
|
||
# wait ~60s for first warm bucket, then:
|
||
curl -s 'http://127.0.0.1:18888/api/v3/data?chart=system.cpu&after=-7200&tier=1&points=120' | jq '.source,.points'
|
||
# expect "hyperdb-warm" after restarting the agent with the same PEARDATA_DATA_DIR
|
||
```
|
||
|
||
---
|
||
|
||
## Operational checklist
|
||
|
||
- [ ] Commit `spec/` after every `build:db`
|
||
- [ ] Back up **keys** (`SERVER_SEED`, swarm keypair in corestore) — not by zipping live corestore while writing
|
||
- [ ] Keep REST on localhost; swarm only when linking trusted peers
|
||
- [ ] Prefer `pd1.` invites for RPC admin; share db discovery keys only with linked nodes
|
||
- [ ] Monitor disk under `data/corestore` (or use **Settings → Data** usage cards + auto-prune)
|
||
|
||
---
|
||
|
||
## References
|
||
|
||
| Repo (local `holepunchto_repos`) | Takeaway |
|
||
|----------------------------------|----------|
|
||
| `hyperdb` | rocks vs bee, tx/flush, autoUpdate |
|
||
| `hyperdb-workshop` | builder + Corestore + Swarm replicate |
|
||
| `hyperdb-autobase-workshop` | Multi-writer view for parents |
|
||
| `corestore` | `store.replicate(conn)` |
|
||
| `pear-hyperdb` | Pear Model wrapper style |
|