# HyperDB storage & linked-node sync

PearData uses Holepunch **HyperDB + Corestore (+ optional Hyperswarm)** for durable metadata, warm metric history, and P2P sync between linked agents.

Live 1-second samples stay in the **memory ring** and are pushed over protomux-rpc. HyperDB is for everything that must survive restarts and replicate.

---

## Status (implemented)

| Piece | Status | Location |
|-------|--------|----------|
| Schema codegen | ✅ | `scripts/build-db.js` → `spec/` |
| Collections + indexes | ✅ | `@peardata/*` in `spec/hyperdb` |
| `PearDataModel` | ✅ | `server/db/model.js` |
| Corestore open/close | ✅ | `server/db/index.js` |
| Warm flush from tier1 | ✅ | `server/services/warm-flush.js` + `store` `warm` event |
| Query fallback (memory → HyperDB) | ✅ | `MetricStore.query()` |
| Agent boot upsert node | ✅ | `server/server.js` |
| RPC: `getDbInfo`, `linkPeer`, `unlinkPeer`, `listPeerLinks` | ✅ | `server/handlers/monitor.js` |
| Open remote bee by `dbKeyHex` + query | ✅ | `server/db/remote.js` → `source: hyperdb-remote` |
| Persist `discoveryKeyHex` on peer-link + boot rejoin | ✅ | schema v2 + `rejoinLinkedPeers()` |
| REST: `GET /api/v3/db` | ✅ | `server/rest/routes.js` |
| Hyperswarm replicate | ✅ stub | `server/db/replicate.js` (`PEARDATA_SWARM=1`) |
| Autobase multi-writer parents | ⏳ Phase C | See roadmap |

Disable HyperDB: `PEARDATA_HYPERDB=0`.

---

## Architecture

```mermaid
flowchart TB
  COL[collector 1s] --> MEM[Memory tier0 ring]
  COL --> T1[Memory tier1 downsample]
  T1 -->|warm event| FLUSH[warm-flush queue]
  FLUSH --> HDB[(HyperDB.bee peardata-meta)]
  ANO[anomaly engine] -->|alert events| HDB
  LINK[linkPeer RPC] --> HDB
  MEM --> PUSH[push:metrics RPC]
  MEM --> Q[queryData / REST]
  HDB --> Q
  CS[Corestore data/corestore] --> HDB
  SW[Hyperswarm PEARDATA_SWARM=1] -->|store.replicate| CS
```

### Planes

| Plane | Storage | Sync |
|-------|---------|------|
| Hot live (~1h @ 1s) | Memory | `push:metrics` |
| Warm history (~1m buckets) | HyperDB `@peardata/metric-point` | Corestore replicate |
| Metadata / links / alerts | HyperDB collections | Corestore replicate |
| Fleet HA (future) | Autobase → HyperDB view | Swarm on autobase key |

### Why not HyperDB for every 1s sample

HyperDB is transactional + indexed. Flushing every chart every second would inflate write amplification and replication traffic. Tier1 downsample (default every 60 samples ≈ 1 minute) is the durable path.

---

## Schema (`@peardata`)

Defined in `scripts/build-db.js`. **Append-only** — never delete fields from committed `spec/` (Holepunch safety rule).

### Collections

| Collection | Key | Purpose |
|------------|-----|---------|
| `@peardata/node` | `nodeId` | Agent / host inventory |
| `@peardata/peer-link` | `localNodeId` + `remotePublicKey` | Linked peers + sync mode |
| `@peardata/alert-config` | `id` | Threshold configs |
| `@peardata/alert-event` | `id` + `ts` | Anomaly / alert history |
| `@peardata/metric-point` | `chart` + `ts` | Warm downsampled samples (`valuesJson`) |
| `@peardata/job` | `id` | Persisted job records (optional use) |

### Indexes

| Index | On |
|-------|-----|
| `@peardata/node-by-hostname` | hostname |
| `@peardata/peer-link-by-remote` | remotePublicKey |
| `@peardata/alert-event-by-chart` | chart + ts |

Warm `@peardata/metric-point` is queried only by primary key (`chart` + `ts`). Secondary indexes `metric-point-by-context` / `metric-point-by-tier` are **deprecated** (no longer written) to cut Hyperbee write amplification (~3×) on high-volume metric inserts/deletes.

### Field notes

- `valuesJson` / `labelsJson` / `resultJson` — JSON strings for open-ended maps (avoids rigid hyperschema maps).
- `tier` on metric-point: `1` = default warm (~1m). Future coarser tiers use `2+`.
- `syncMode` on peer-link: `push` | `pull` | `both`.

### Regenerate after schema edits

```bash
npm run build:db
# or: node scripts/build-db.js
```

Then commit `spec/hyperschema/*` and `spec/hyperdb/*`.

---

## Runtime layout

```text
$PEARDATA_DATA_DIR/          # default ./data
  peer-policy.json           # existing ACL file
  audit.log
  corestore/                 # Corestore (HyperDB bee cores)
```

Named core: **`peardata-meta`**.

Banner fields on agent start:

- `hyperdb:` — DB public key (hex) for others to open a read replica
- `swarm:` — `on` / `off`

---

## Configuration

| Variable | Default | Meaning |
|----------|---------|---------|
| `PEARDATA_HYPERDB` | on | `0` / `off` disables HyperDB entirely |
| `PEARDATA_DATA_DIR` | `./data` | Policy + corestore root |
| `PEARDATA_SWARM` | off | `1` enables Hyperswarm `store.replicate` |
| `PEARDATA_TIER1_EVERY` | `60` | Samples per warm bucket (~60s @ 1Hz) |
| `PEARDATA_TIER1_POINTS` | `1440` | In-memory tier1 ring (HyperDB keeps longer) |
| `PEARDATA_WARM_RETENTION_MS` | `3m` | Seeds Data Manager age rotate |
| `PEARDATA_WARM_MAX_BYTES` | `1 GiB` | Soft corestore budget; `0` = unlimited |
| `PEARDATA_WARM_MAX_POINTS` | `0` | Soft point cap; `0` = unlimited |
| `PEARDATA_AUTO_PRUNE` | on | Scheduled prune after boot |
| `PEARDATA_WARM_ALL_CHARTS` | off | `1` flushes high-cardinality instance charts to HyperDB (default: static catalog only) |

**Data Manager:** desktop **Settings → Data** (admin) calls `setRetentionConfig` / `pruneHistory`. Policy file: `$PEARDATA_DATA_DIR/retention.json`. Auto-prune runs on an interval and:

1. Deletes HyperDB `@peardata/metric-point` rows older than the age window (multi-pass)
2. Enforces optional point / soft disk budgets (multi-pass tighter cutoffs)
3. **Physical reclaim:** `hyperbee.clearUnlinked` + `hypercore.compact` so RocksDB actually frees space

HyperDB deletes alone are logical (append-only tombstones); without step 3 corestore size does not shrink.

---

## API surface

### RPC

| Method | Role | Description |
|--------|------|-------------|
| `getDbInfo` | viewer | `{ enabled, publicKeyHex, discoveryKeyHex, swarm }` |
| `listPeerLinks` | viewer | Linked peers from HyperDB |
| `linkPeer` | admin | Upsert link; optionally `joinRemoteTopic` if swarm on |
| `unlinkPeer` | admin | Remove link |
| `queryData` | viewer | Memory first; HyperDB warm if miss / `tier≥1` |
| `getStorageInfo` | viewer | Disk + memory usage for Data Manager |
| `getRetentionConfig` / `setRetentionConfig` | viewer / admin | Warm age rotate (default **3m**), **1 GiB** disk budget, auto-prune |
| `pruneHistory` | admin | Delete warm points + clearUnlinked/compact (`gcBuffers` job alias). Returns real `bytesBefore` / `bytesAfter` / `bytesFreed`. |

`linkPeer` args:

```json
{
  "remotePublicKey": "<64 hex>",
  "alias": "homelab-nas",
  "dbKeyHex": "<optional remote db key>",
  "discoveryKeyHex": "<optional topic to pull>",
  "syncMode": "both",
  "role": "viewer"
}
```

### REST

| Path | Description |
|------|-------------|
| `GET /api/v3/db` | HyperDB keys + collection list |
| `GET /api/v3/info` | Includes `peardata.hyperdb` block |
| `GET /api/v3/data?...` | Uses async store query (warm fallback) |
| Query `source` field | `memory-tier0` \| `memory-tier1` \| `hyperdb-warm` |

```bash
curl -s http://127.0.0.1:18888/api/v3/db | jq
curl -s 'http://127.0.0.1:18888/api/v3/data?chart=system.cpu&after=-3600&tier=1&points=120' | jq '.source'
```

---

## How linked-node sync works

### Phase A (now) — local durability

1. Agent opens Corestore + HyperDB on boot.
2. Warm points + alert events persist under `data/corestore`.
3. Restart retains warm history / links / node row.

### Phase B — mesh replicate (`PEARDATA_SWARM=1`)

Workshop pattern (`hyperdb-workshop/bin.js`):

```js
swarm.on('connection', (conn) => store.replicate(conn))
swarm.join(db.discoveryKey, { server: true, client: true })
```

1. Enable swarm on agents that should seed/pull.
2. Share **db public key** + **discovery key** (`getDbInfo` / `/api/v3/db`).
3. Admin calls `linkPeer` with `discoveryKeyHex` + `syncMode: pull|both`.
4. Peers replicate Corestore; HyperDB `autoUpdate` refreshes indexes.
5. Remote warm history becomes queryable locally (parent path).

**Security:** replication grants read of the core to anyone who can join the topic. Prefer:

- Swarm only on trusted LAN / VPN, or
- Future: encrypted cores + allowlisted swarm joins
- Keep **mutations** on protomux-rpc AuthZ (`linkPeer` is admin)

HyperDHT RPC identity ≠ Hyperswarm topic access — treat them as two layers.

### Phase C — Autobase parents (planned)

Same as `hyperdb-autobase-workshop`:

- Autobase bootstrap key across parent writers
- View = `HyperDB.bee(store.get('db-view'), spec, { extension: false })`
- hyperdispatch ops for `put-link`, `put-alert`, rollups
- Do **not** “backup” by copying Corestore folders

---

## Code map

| Path | Role |
|------|------|
| `scripts/build-db.js` | Hyperschema + HyperDB builder |
| `spec/hyperschema/` | Generated encodings + `schema.json` |
| `spec/hyperdb/` | Generated collections/indexes |
| `server/db/model.js` | Typed CRUD facade |
| `server/db/index.js` | Singleton open/close |
| `server/db/replicate.js` | Optional Hyperswarm |
| `server/services/warm-flush.js` | Batch writer |
| `server/services/store.js` | Hot ring + `warm` events + query fallback |

### Model usage example

```js
import { openDb, getDb, closeDb } from './server/db/index.js'

await openDb()
const db = getDb()

await db.putPeerLink({
  localNodeId: 'abc123',
  remotePublicKey: 'ff'.repeat(32),
  syncMode: 'pull',
  linkedAt: Date.now(),
})

await db.putMetricPoints([
  { chart: 'system.cpu', context: 'system.cpu', ts: Date.now(), values: { user: 12.3 }, tier: 1 },
])

const rows = await db.queryMetricPoints({
  chart: 'system.cpu',
  afterMs: Date.now() - 3600_000,
  beforeMs: Date.now(),
})
```

---

## Dependencies

From Holepunch stack (see `holepunchto_repos`):

- `hyperdb` — DB engine (bee + rocks)
- `hyperschema` — struct codegen
- `corestore` — named hypercores + replicate
- `hyperswarm` — topic discovery for store sync
- `ready-resource` — open/close lifecycle

Future: `autobase`, `hyperdispatch` for HA parents.

---

## Testing

```bash
npm run build:db
SKIP_INTEGRATION=1 npm test
# includes test/hyperdb.test.js + test/store-hyperdb-fallback.test.js (M4)
```

### M4 soak — history across restart

Automated: `store-hyperdb-fallback` writes warm points, clears the memory rings, and asserts `query()` returns `source: "hyperdb-warm"` after reopening Corestore (simulates agent restart).

Manual:

```bash
npm run start:server
curl -s http://127.0.0.1:18888/api/v3/db | jq
# wait ~60s for first warm bucket, then:
curl -s 'http://127.0.0.1:18888/api/v3/data?chart=system.cpu&after=-7200&tier=1&points=120' | jq '.source,.points'
# expect "hyperdb-warm" after restarting the agent with the same PEARDATA_DATA_DIR
```

---

## Operational checklist

- [ ] Commit `spec/` after every `build:db`
- [ ] Back up **keys** (`SERVER_SEED`, swarm keypair in corestore) — not by zipping live corestore while writing
- [ ] Keep REST on localhost; swarm only when linking trusted peers
- [ ] Prefer `pd1.` invites for RPC admin; share db discovery keys only with linked nodes
- [ ] Monitor disk under `data/corestore` (or use **Settings → Data** usage cards + auto-prune)

---

## References

| Repo (local `holepunchto_repos`) | Takeaway |
|----------------------------------|----------|
| `hyperdb` | rocks vs bee, tx/flush, autoUpdate |
| `hyperdb-workshop` | builder + Corestore + Swarm replicate |
| `hyperdb-autobase-workshop` | Multi-writer view for parents |
| `corestore` | `store.replicate(conn)` |
| `pear-hyperdb` | Pear Model wrapper style |