peardata/docs/STORAGE-HYPERDB.md

HyperDB storage & linked-node sync

PearData uses Holepunch HyperDB + Corestore (+ optional Hyperswarm) for durable metadata, warm metric history, and P2P sync between linked agents.

Live 1-second samples stay in the memory ring and are pushed over protomux-rpc. HyperDB is for everything that must survive restarts and replicate.

---

Status (implemented)

Piece	Status	Location
Schema codegen	✅	scripts/build-db.js → spec/
Collections + indexes	✅	@peardata/* in spec/hyperdb
PearDataModel	✅	server/db/model.js
Corestore open/close	✅	server/db/index.js
Warm flush from tier1	✅	server/services/warm-flush.js + store warm event
Query fallback (memory → HyperDB)	✅	MetricStore.query()
Agent boot upsert node	✅	server/server.js
RPC: getDbInfo, linkPeer, unlinkPeer, listPeerLinks	✅	server/handlers/monitor.js
Open remote bee by dbKeyHex + query	✅	server/db/remote.js → source: hyperdb-remote
Persist discoveryKeyHex on peer-link + boot rejoin	✅	schema v2 + rejoinLinkedPeers()
REST: GET /api/v3/db	✅	server/rest/routes.js
Hyperswarm replicate	✅ stub	server/db/replicate.js (PEARDATA_SWARM=1)
Autobase multi-writer parents	⏳ Phase C	See roadmap

Disable HyperDB: PEARDATA_HYPERDB=0.

---

Architecture

flowchart TB
  COL[collector 1s] --> MEM[Memory tier0 ring]
  COL --> T1[Memory tier1 downsample]
  T1 -->|warm event| FLUSH[warm-flush queue]
  FLUSH --> HDB[(HyperDB.bee peardata-meta)]
  ANO[anomaly engine] -->|alert events| HDB
  LINK[linkPeer RPC] --> HDB
  MEM --> PUSH[push:metrics RPC]
  MEM --> Q[queryData / REST]
  HDB --> Q
  CS[Corestore data/corestore] --> HDB
  SW[Hyperswarm PEARDATA_SWARM=1] -->|store.replicate| CS

Planes

Plane	Storage	Sync
Hot live (~1h @ 1s)	Memory	push:metrics
Warm history (~1m buckets)	HyperDB @peardata/metric-point	Corestore replicate
Metadata / links / alerts	HyperDB collections	Corestore replicate
Fleet HA (future)	Autobase → HyperDB view	Swarm on autobase key

Why not HyperDB for every 1s sample

HyperDB is transactional + indexed. Flushing every chart every second would inflate write amplification and replication traffic. Tier1 downsample (default every 60 samples ≈ 1 minute) is the durable path.

---

Schema (@peardata)

Defined in scripts/build-db.js. Append-only — never delete fields from committed spec/ (Holepunch safety rule).

Collections

Collection	Key	Purpose
@peardata/node	nodeId	Agent / host inventory
@peardata/peer-link	localNodeId + remotePublicKey	Linked peers + sync mode
@peardata/alert-config	id	Threshold configs
@peardata/alert-event	id + ts	Anomaly / alert history
@peardata/metric-point	chart + ts	Warm downsampled samples (valuesJson)
@peardata/job	id	Persisted job records (optional use)

Indexes

Index	On
@peardata/node-by-hostname	hostname
@peardata/peer-link-by-remote	remotePublicKey
@peardata/alert-event-by-chart	chart + ts

Warm @peardata/metric-point is queried only by primary key (chart + ts). Secondary indexes metric-point-by-context / metric-point-by-tier are deprecated (no longer written) to cut Hyperbee write amplification (~3×) on high-volume metric inserts/deletes.

Field notes

• valuesJson / labelsJson / resultJson — JSON strings for open-ended maps (avoids rigid hyperschema maps).
• tier on metric-point: 1 = default warm (~1m). Future coarser tiers use 2+.
• syncMode on peer-link: push | pull | both.

Regenerate after schema edits

npm run build:db
# or: node scripts/build-db.js

Then commit spec/hyperschema/* and spec/hyperdb/*.

---

Runtime layout

$PEARDATA_DATA_DIR/          # default ./data
  peer-policy.json           # existing ACL file
  audit.log
  corestore/                 # Corestore (HyperDB bee cores)

Named core: peardata-meta.

Banner fields on agent start:

• hyperdb: — DB public key (hex) for others to open a read replica
• swarm: — on / off

---

Configuration

Variable	Default	Meaning
PEARDATA_HYPERDB	on	0 / off disables HyperDB entirely
PEARDATA_DATA_DIR	./data	Policy + corestore root
PEARDATA_SWARM	off	1 enables Hyperswarm store.replicate
PEARDATA_TIER1_EVERY	60	Samples per warm bucket (~60s @ 1Hz)
PEARDATA_TIER1_POINTS	1440	In-memory tier1 ring (HyperDB keeps longer)
PEARDATA_WARM_RETENTION_MS	3m	Seeds Data Manager age rotate
PEARDATA_WARM_MAX_BYTES	1 GiB	Soft corestore budget; 0 = unlimited
PEARDATA_WARM_MAX_POINTS	0	Soft point cap; 0 = unlimited
PEARDATA_AUTO_PRUNE	on	Scheduled prune after boot
PEARDATA_WARM_ALL_CHARTS	off	1 flushes high-cardinality instance charts to HyperDB (default: static catalog only)

Data Manager: desktop Settings → Data (admin) calls setRetentionConfig / pruneHistory. Policy file: $PEARDATA_DATA_DIR/retention.json. Auto-prune runs on an interval and:

1. Deletes HyperDB @peardata/metric-point rows older than the age window (multi-pass)
2. Enforces optional point / soft disk budgets (multi-pass tighter cutoffs)
3. Physical reclaim: hyperbee.clearUnlinked + hypercore.compact so RocksDB actually frees space

HyperDB deletes alone are logical (append-only tombstones); without step 3 corestore size does not shrink.

---

API surface

RPC

Method	Role	Description
getDbInfo	viewer	{ enabled, publicKeyHex, discoveryKeyHex, swarm }
listPeerLinks	viewer	Linked peers from HyperDB
linkPeer	admin	Upsert link; optionally joinRemoteTopic if swarm on
unlinkPeer	admin	Remove link
queryData	viewer	Memory first; HyperDB warm if miss / tier≥1
getStorageInfo	viewer	Disk + memory usage for Data Manager
getRetentionConfig / setRetentionConfig	viewer / admin	Warm age rotate (default 3m), 1 GiB disk budget, auto-prune
pruneHistory	admin	Delete warm points + clearUnlinked/compact (gcBuffers job alias). Returns real bytesBefore / bytesAfter / bytesFreed.

linkPeer args:

{
  "remotePublicKey": "<64 hex>",
  "alias": "homelab-nas",
  "dbKeyHex": "<optional remote db key>",
  "discoveryKeyHex": "<optional topic to pull>",
  "syncMode": "both",
  "role": "viewer"
}

REST

Path	Description
GET /api/v3/db	HyperDB keys + collection list
GET /api/v3/info	Includes peardata.hyperdb block
GET /api/v3/data?...	Uses async store query (warm fallback)
Query source field	memory-tier0 | memory-tier1 | hyperdb-warm

curl -s http://127.0.0.1:18888/api/v3/db | jq
curl -s 'http://127.0.0.1:18888/api/v3/data?chart=system.cpu&after=-3600&tier=1&points=120' | jq '.source'

---

How linked-node sync works

Phase A (now) — local durability

1. Agent opens Corestore + HyperDB on boot.
2. Warm points + alert events persist under data/corestore.
3. Restart retains warm history / links / node row.

Phase B — mesh replicate (PEARDATA_SWARM=1)

Workshop pattern (hyperdb-workshop/bin.js):

swarm.on('connection', (conn) => store.replicate(conn))
swarm.join(db.discoveryKey, { server: true, client: true })

1. Enable swarm on agents that should seed/pull.
2. Share db public key + discovery key (getDbInfo / /api/v3/db).
3. Admin calls linkPeer with discoveryKeyHex + syncMode: pull|both.
4. Peers replicate Corestore; HyperDB autoUpdate refreshes indexes.
5. Remote warm history becomes queryable locally (parent path).

Security: replication grants read of the core to anyone who can join the topic. Prefer:

• Swarm only on trusted LAN / VPN, or
• Future: encrypted cores + allowlisted swarm joins
• Keep mutations on protomux-rpc AuthZ (linkPeer is admin)

HyperDHT RPC identity ≠ Hyperswarm topic access — treat them as two layers.

Phase C — Autobase parents (planned)

Same as hyperdb-autobase-workshop:

• Autobase bootstrap key across parent writers
• View = HyperDB.bee(store.get('db-view'), spec, { extension: false })
• hyperdispatch ops for put-link, put-alert, rollups
• Do not “backup” by copying Corestore folders

---

Code map

Path	Role
scripts/build-db.js	Hyperschema + HyperDB builder
spec/hyperschema/	Generated encodings + schema.json
spec/hyperdb/	Generated collections/indexes
server/db/model.js	Typed CRUD facade
server/db/index.js	Singleton open/close
server/db/replicate.js	Optional Hyperswarm
server/services/warm-flush.js	Batch writer
server/services/store.js	Hot ring + warm events + query fallback

Model usage example

import { openDb, getDb, closeDb } from './server/db/index.js'

await openDb()
const db = getDb()

await db.putPeerLink({
  localNodeId: 'abc123',
  remotePublicKey: 'ff'.repeat(32),
  syncMode: 'pull',
  linkedAt: Date.now(),
})

await db.putMetricPoints([
  { chart: 'system.cpu', context: 'system.cpu', ts: Date.now(), values: { user: 12.3 }, tier: 1 },
])

const rows = await db.queryMetricPoints({
  chart: 'system.cpu',
  afterMs: Date.now() - 3600_000,
  beforeMs: Date.now(),
})

---

Dependencies

From Holepunch stack (see holepunchto_repos):

• hyperdb — DB engine (bee + rocks)
• hyperschema — struct codegen
• corestore — named hypercores + replicate
• hyperswarm — topic discovery for store sync
• ready-resource — open/close lifecycle

Future: autobase, hyperdispatch for HA parents.

---

Testing

npm run build:db
SKIP_INTEGRATION=1 npm test
# includes test/hyperdb.test.js + test/store-hyperdb-fallback.test.js (M4)

M4 soak — history across restart

Automated: store-hyperdb-fallback writes warm points, clears the memory rings, and asserts query() returns source: "hyperdb-warm" after reopening Corestore (simulates agent restart).

Manual:

npm run start:server
curl -s http://127.0.0.1:18888/api/v3/db | jq
# wait ~60s for first warm bucket, then:
curl -s 'http://127.0.0.1:18888/api/v3/data?chart=system.cpu&after=-7200&tier=1&points=120' | jq '.source,.points'
# expect "hyperdb-warm" after restarting the agent with the same PEARDATA_DATA_DIR

---

Operational checklist

• Commit spec/ after every build:db
• Back up keys (SERVER_SEED, swarm keypair in corestore) — not by zipping live corestore while writing
• Keep REST on localhost; swarm only when linking trusted peers
• Prefer pd1. invites for RPC admin; share db discovery keys only with linked nodes
• Monitor disk under data/corestore (or use Settings → Data usage cards + auto-prune)

---

References

Repo (local holepunchto_repos)	Takeaway
hyperdb	rocks vs bee, tx/flush, autoUpdate
hyperdb-workshop	builder + Corestore + Swarm replicate
hyperdb-autobase-workshop	Multi-writer view for parents
corestore	store.replicate(conn)
pear-hyperdb	Pear Model wrapper style