Testing
This commit is contained in:
Raven Scott
+13
-7
@@ -1,15 +1,16 @@
|
||||
# Architecture
|
||||
|
||||
PearData is a **decentralized, P2P clone of the Netdata real-time monitoring experience**, built on the same HyperDHT + protomux-rpc patterns as PearDock-class apps (via the pear-app template).
|
||||
PearData is a **decentralized, P2P real-time host monitoring stack**, built on the same HyperDHT + protomux-rpc patterns as PearDock-class apps (via the pear-app template).
|
||||
|
||||
## Design goals
|
||||
|
||||
1. **No central control plane** — dial an agent by Ed25519 public key.
|
||||
2. **Instant live truth** — ~1s metric push to connected desktops.
|
||||
3. **Dual API** — P2P RPC for the Pear client; Netdata-style REST for scripts/Grafana.
|
||||
3. **Dual API** — P2P RPC for the Pear client; agent-style REST for scripts/Grafana.
|
||||
4. **Clear AuthZ** — viewer (pubkey) vs operator/admin (`pd1.` invite / seed proof).
|
||||
5. **Low agent overhead** — Node/`os` + `/proc` collectors, ring buffers, hot-path RPCs.
|
||||
6. **Ecosystem-ready** — shared wire contract in `shared/` for PearDock / PearVirt adapters later.
|
||||
5. **Low agent overhead** — `os`/`bare-os` + `/proc` collectors, ring buffers, hot-path RPCs.
|
||||
6. **Bare/Pear-ready** — `package.json` import maps route builtins → `bare-*`; no Node-only core deps in the Pear path.
|
||||
7. **Ecosystem-ready** — shared wire contract in `shared/` for PearDock / PearVirt adapters later.
|
||||
|
||||
## Mapping to PearDock / template components
|
||||
|
||||
@@ -23,7 +24,7 @@ PearData is a **decentralized, P2P clone of the Netdata real-time monitoring exp
|
||||
| Connection manager / multi-peer | `client/manager.js` |
|
||||
| Job tray | `server/services/jobs.js` + desktop actions |
|
||||
| Domain service | **Metrics pipeline** (`collector` → `store` → `anomaly` → pushes) |
|
||||
| Optional HTTP surface | `server/rest/*` (Netdata v1/v2/v3) |
|
||||
| Optional HTTP surface | `server/rest/*` (REST v1/v2/v3) |
|
||||
|
||||
## System context
|
||||
|
||||
@@ -67,7 +68,7 @@ Threshold engine evaluates each batch; transitions emit `push:anomaly` / `push:a
|
||||
|
||||
### 4. Local REST (optional)
|
||||
|
||||
Netdata-compatible HTTP on `127.0.0.1:19999` by default — for local tooling without P2P. Disable with `PEARDATA_REST=0`.
|
||||
agent-compatible HTTP on `127.0.0.1:19999` by default — for local tooling without P2P. Disable with `PEARDATA_REST=0`.
|
||||
|
||||
## Layered stack
|
||||
|
||||
@@ -152,8 +153,13 @@ A heavier agent may subscribe to child agents over P2P, downsample into its own
|
||||
| `server/services/subscriptions.js` | Push fan-out |
|
||||
| `server/services/jobs.js` | On-demand jobs |
|
||||
| `server/handlers/monitor.js` | RPC surface |
|
||||
| `server/rest/*` | Netdata HTTP API |
|
||||
| `server/rest/*` | Agent HTTP API |
|
||||
| `server/pipeline.js` | Wire collector→store→push |
|
||||
| `server/db/*` | HyperDB model, Corestore, swarm replicate |
|
||||
| `spec/` | Generated Hyperschema + HyperDB defs |
|
||||
| `scripts/build-db.js` | Schema codegen |
|
||||
|
||||
HyperDB design: [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md).
|
||||
|
||||
## Related docs
|
||||
|
||||
|
||||
+21
-3
@@ -57,17 +57,34 @@ Permissions: directory `0700`. Do **not** commit `data/` or `.env`.
|
||||
| `PEARDATA_SAMPLE_MS` | `1000` | Collector interval |
|
||||
| `PEARDATA_TIER0_POINTS` | `3600` | High-res ring size (~1h @ 1s) |
|
||||
| `PEARDATA_TIER1_POINTS` | `1440` | Downsampled ring size |
|
||||
| `PEARDATA_TIER1_EVERY` | `60` | Samples per tier1 average |
|
||||
| `PEARDATA_TIER1_EVERY` | `60` | Samples per tier1 average (also HyperDB warm flush) |
|
||||
|
||||
---
|
||||
|
||||
## REST API (Netdata-style)
|
||||
## HyperDB storage & sync
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `PEARDATA_HYPERDB` | on | `0` / `off` disables HyperDB |
|
||||
| `PEARDATA_SWARM` | off | `1` enables Hyperswarm Corestore replication |
|
||||
|
||||
Storage: `$PEARDATA_DATA_DIR/corestore` (named core `peardata-meta`).
|
||||
|
||||
Full guide: [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md).
|
||||
|
||||
```bash
|
||||
npm run build:db # regenerate spec/ after schema edits
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## REST API (agent-style)
|
||||
|
||||
| Variable | Default | Description |
|
||||
|----------|---------|-------------|
|
||||
| `PEARDATA_REST` | on | Set `0` / `off` to disable HTTP API |
|
||||
| `PEARDATA_REST_HOST` | `127.0.0.1` | Bind address (`0.0.0.0` exposes LAN — firewall!) |
|
||||
| `PEARDATA_REST_PORT` | `19999` | HTTP port (Netdata classic) |
|
||||
| `PEARDATA_REST_PORT` | `19999` | HTTP port (classic local agent port) |
|
||||
| `PEARDATA_REST_CORS` | `*` | `Access-Control-Allow-Origin` |
|
||||
|
||||
See [REST-API.md](./REST-API.md).
|
||||
@@ -104,6 +121,7 @@ See [REST-API.md](./REST-API.md).
|
||||
| `npm run start:server:bin` | `bin/peardata-server.mjs` |
|
||||
| `npm test` | brittle suite |
|
||||
| `npm run mint-invite -- [role] [ttlMs]` | Offline `pd1.` invite |
|
||||
| `npm run build:db` | Regenerate HyperDB `spec/` |
|
||||
| `npm run healthcheck` | Liveness |
|
||||
| `npm run soak` | Load exercise |
|
||||
|
||||
|
||||
+23
-12
@@ -26,20 +26,31 @@ Implemented in `shared/data-model.js`, `shared/metrics.js`, and agent services.
|
||||
|
||||
## Metric contexts & charts
|
||||
|
||||
Netdata-inspired IDs:
|
||||
Canonical catalog lives in `shared/metrics.js` (`STATIC_CHART_DEFS` + runtime instance charts). Collector fills these from `/proc` + `/sys` on Linux (`os` / bare-os fallback elsewhere).
|
||||
|
||||
| Context | Chart id | Units | Dimensions (MVP) |
|
||||
|---------|----------|-------|------------------|
|
||||
| `system.cpu` | `system.cpu` | percentage | user, system, nice, iowait, irq, softirq, idle |
|
||||
| `system.ram` | `system.ram` | MiB | used, cached, buffers, free |
|
||||
| `mem.available` | `mem.available` | MiB | avail |
|
||||
| `system.load` | `system.load` | load | load1, load5, load15 |
|
||||
| `system.io` | `system.io` | KiB/s | reads, writes |
|
||||
| `system.net` | `system.net` | kilobits/s | received, sent |
|
||||
| `system.processes` | `system.processes` | processes | running, blocked, total |
|
||||
| `system.uptime` | `system.uptime` | seconds | uptime |
|
||||
### Host-wide (static)
|
||||
|
||||
Chart summary objects mirror Netdata’s `/api/v1/charts` fields (`id`, `context`, `units`, `dimensions`, `update_every`, `first_entry`, `last_entry`, …).
|
||||
| Family | Contexts / charts | Units (typical) |
|
||||
|--------|-------------------|-----------------|
|
||||
| CPU / scheduler | `system.cpu`, `system.intr`, `system.ctxt`, `system.forks`, `system.processes`, `system.active_processes`, `system.load`, `system.uptime`, `system.entropy` | %, interrupts/s, load, seconds |
|
||||
| Memory | `system.ram`, `mem.available`, `mem.swap`, `mem.swap_cached`, `mem.kernel`, `mem.slab`, `mem.writeback`, `mem.committed`, `mem.swapio`, `system.pgpgio`, `system.pgfaults` | MiB, KiB/s, faults/s |
|
||||
| Disk aggregate | `system.io` | KiB/s (`in` / `out`) |
|
||||
| Network aggregate | `system.net`, `system.ip`, `system.ipv6` | kilobits/s |
|
||||
| TCP / IPv4 | `ip.tcppackets`, `ip.tcperrors`, `ip.tcpopens`, `ip.tcpsock`, `ipv4.packets`, `ipv4.errors`, `ipv4.udppackets`, `ipv4.udperrors` | packets/s, connections |
|
||||
| PSI pressure | `system.cpu_some_pressure`, `system.memory_some_pressure`, `system.io_some_pressure` | % (avg10/60/300) |
|
||||
|
||||
`system.cpu` dimensions: `guest_nice`, `guest`, `steal`, `softirq`, `irq`, `user`, `system`, `nice`, `iowait`, `idle`.
|
||||
|
||||
### Instance charts (registered at runtime)
|
||||
|
||||
| Context | Chart id pattern | Notes |
|
||||
|---------|------------------|-------|
|
||||
| `cpu.cpu` | `cpu.cpu{N}` | Per-core utilization (same dims as `system.cpu`) |
|
||||
| `disk.io` / `disk.ops` / `disk.util` | `disk_io.{dev}`, `disk_ops.{dev}`, `disk_util.{dev}` | Per-disk from `/proc/diskstats` |
|
||||
| `disk.space` / `disk.inodes` | `disk_space.{mount}`, `disk_inodes.{mount}` | Per-mount via `statfs` |
|
||||
| `net.net` / `net.packets` / `net.errors` / `net.drops` | `net.{iface}`, … | Per-iface from `/proc/net/dev` (loopback skipped) |
|
||||
|
||||
Chart summary objects use fields: `id`, `context`, `units`, `dimensions`, `update_every`, `first_entry`, `last_entry`, …
|
||||
|
||||
## MetricSample (live push)
|
||||
|
||||
|
||||
+3
-3
@@ -2,9 +2,9 @@
|
||||
|
||||
## Add a chart / context
|
||||
|
||||
1. Define the chart in `shared/metrics.js` (`CHART_DEFS`).
|
||||
1. Define the chart in `shared/metrics.js` (`STATIC_CHART_DEFS`, or `registerChart()` for instances).
|
||||
2. Emit samples from `server/services/collector.js` (or a new collector module).
|
||||
3. Store + REST/RPC pick it up automatically via `CHART_BY_ID`.
|
||||
3. Store + REST/RPC pick it up automatically via `getAllChartDefs()` / `CHART_BY_ID`.
|
||||
4. Document dimensions in [DATA-MODEL.md](./DATA-MODEL.md).
|
||||
5. Optionally add a canvas panel in `index.html` + `app.js`.
|
||||
|
||||
@@ -40,7 +40,7 @@ Update [PROTOCOL.md](./PROTOCOL.md) and add a brittle test.
|
||||
|
||||
## Add a REST route
|
||||
|
||||
Edit `server/rest/routes.js` — keep Netdata path naming when emulating Agent APIs (`/api/v3/...`).
|
||||
Edit `server/rest/routes.js` — keep stable `/api/v3/...` path naming.
|
||||
|
||||
## Add a job
|
||||
|
||||
|
||||
@@ -44,6 +44,22 @@ curl -s 'http://127.0.0.1:19999/api/v3/allmetrics?format=prometheus' | head
|
||||
|
||||
Disable REST: `PEARDATA_REST=0`.
|
||||
|
||||
### HyperDB (warm storage)
|
||||
|
||||
On by default. Agent banner shows `hyperdb:` public key.
|
||||
|
||||
```bash
|
||||
curl -s http://127.0.0.1:19999/api/v3/db | jq
|
||||
```
|
||||
|
||||
Warm history is written about once per minute (tier1). Linked-node Hyperswarm sync:
|
||||
|
||||
```bash
|
||||
PEARDATA_SWARM=1 npm run start:server
|
||||
```
|
||||
|
||||
See [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md).
|
||||
|
||||
## Mint an invite
|
||||
|
||||
```bash
|
||||
|
||||
+13
-2
@@ -48,12 +48,12 @@ Auth modes at handshake: public key (viewer), capability token / `pd1.` invite,
|
||||
|--------|------|------|--------|
|
||||
| `listContexts` | viewer | — | Context catalog |
|
||||
| `getContext` | viewer | `{ id }` | Charts in context |
|
||||
| `listCharts` | viewer | — | Netdata-ish chart map |
|
||||
| `listCharts` | viewer | — | agent-style chart map |
|
||||
| `getChart` | viewer | `{ id }` | Chart summary |
|
||||
| `queryData` | viewer | `{ chart, after, before, points, group, tier }` | Time series |
|
||||
| `getAllMetrics` | viewer | `{ format: json\|prometheus\|shell }` | Latest export |
|
||||
|
||||
`after` / `before`: absolute unix seconds, or relative (negative = relative to `before`/`now`), Netdata-style.
|
||||
`after` / `before`: absolute unix seconds, or relative (negative = relative to `before`/`now`), agent-style.
|
||||
|
||||
### Live subscriptions
|
||||
|
||||
@@ -82,6 +82,17 @@ Auth modes at handshake: public key (viewer), capability token / `pd1.` invite,
|
||||
| `runJob` | operator | `collectOnce`, `snapshot`, `gcBuffers` |
|
||||
| `cancelJob` | operator | By job id |
|
||||
|
||||
### HyperDB / peer links
|
||||
|
||||
| Method | Role | Notes |
|
||||
|--------|------|-------|
|
||||
| `getDbInfo` | viewer | DB + discovery keys, swarm flag |
|
||||
| `listPeerLinks` | viewer | Linked peers from HyperDB |
|
||||
| `linkPeer` | admin | Upsert link; optional swarm join |
|
||||
| `unlinkPeer` | admin | Remove link |
|
||||
|
||||
See [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md).
|
||||
|
||||
### Admin
|
||||
|
||||
| Method | Role |
|
||||
|
||||
+2
-2
@@ -3,11 +3,11 @@
|
||||
| Doc | Audience | Contents |
|
||||
|-----|----------|----------|
|
||||
| [GETTING-STARTED.md](./GETTING-STARTED.md) | Operators | Install, run agent, REST, desktop, systemd |
|
||||
| [ROADMAP.md](./ROADMAP.md) | Everyone | Phased MVP → advanced Netdata-class features |
|
||||
| [ROADMAP.md](./ROADMAP.md) | Everyone | Phased MVP → advanced real-time features |
|
||||
| [ARCHITECTURE.md](./ARCHITECTURE.md) | Engineers | Planes, PearDock mapping, module map |
|
||||
| [PROTOCOL.md](./PROTOCOL.md) | Protocol owners | RPC methods, pushes, versioning |
|
||||
| [DATA-MODEL.md](./DATA-MODEL.md) | Engineers | Metrics, anomalies, alerts, jobs |
|
||||
| [REST-API.md](./REST-API.md) | Integrators | Netdata-style `/api/v1\|v2\|v3` |
|
||||
| [REST-API.md](./REST-API.md) | Integrators | agent-style `/api/v1\|v2\|v3` |
|
||||
| [TECH-CHOICES.md](./TECH-CHOICES.md) | Engineers | Collector, charts, libraries |
|
||||
| [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | Engineers | HyperDB / Corestore / Autobase sync design |
|
||||
| [DESKTOP.md](./DESKTOP.md) | UI developers | Pear shell, titlebar, identity |
|
||||
|
||||
+16
-15
@@ -1,12 +1,12 @@
|
||||
# REST API (Netdata-compatible)
|
||||
# REST API (agent-compatible)
|
||||
|
||||
PearMonitor agents expose an optional HTTP API modeled on **Netdata Agent** endpoints (`/api/v1`, `/api/v2`, `/api/v3`).
|
||||
PearMonitor agents expose an optional HTTP API for local tooling (`/api/v1`, `/api/v2`, `/api/v3`).
|
||||
|
||||
Default bind: `http://127.0.0.1:19999` (Netdata’s classic port).
|
||||
Default bind: `http://127.0.0.1:19999`.
|
||||
Disable: `PEARDATA_REST=0`.
|
||||
Bind all interfaces (careful): `PEARDATA_REST_HOST=0.0.0.0`.
|
||||
|
||||
This is a **compatibility layer**, not a byte-for-byte Netdata clone. Core query/metadata paths are implemented for scripts, Grafana, and Prometheus scrapers.
|
||||
Core query/metadata paths are implemented for scripts, Grafana, and Prometheus scrapers.
|
||||
|
||||
## Quick examples
|
||||
|
||||
@@ -36,10 +36,11 @@ curl -s http://127.0.0.1:19999/api/v3/health | jq
|
||||
|
||||
| Method | Path | Notes |
|
||||
|--------|------|-------|
|
||||
| GET | `/` or `/api` | Service index + P2P pubkey |
|
||||
| GET | `/` or `/api` | Service index + P2P pubkey + HyperDB keys |
|
||||
| GET | `/api/v1/info` | Agent info |
|
||||
| GET | `/api/v2/info` | same |
|
||||
| GET | `/api/v3/info` | **preferred** |
|
||||
| GET | `/api/v3/info` | **preferred** (includes `peardata.hyperdb`) |
|
||||
| GET | `/api/v3/db` | HyperDB public/discovery keys + collections |
|
||||
| GET | `/api/v3/versions` | Agent / protocol / API versions |
|
||||
| GET | `/api/v3/me` | Anonymous REST identity note |
|
||||
| GET | `/api/v3/settings` | Runtime knobs |
|
||||
@@ -75,7 +76,7 @@ Single-agent MVP returns one node (this host). Parent/fleet aggregation is roadm
|
||||
| GET | `/api/v2/data` | same |
|
||||
| GET | `/api/v1/data` | same (legacy) |
|
||||
|
||||
**Params (Netdata-style)**
|
||||
**Params (agent-style)**
|
||||
|
||||
| Param | Default | Description |
|
||||
|-------|---------|-------------|
|
||||
@@ -121,7 +122,7 @@ Single-agent MVP returns one node (this host). Parent/fleet aggregation is roadm
|
||||
|
||||
## Auth model (REST)
|
||||
|
||||
- **Default:** localhost-only, no bearer required (like a typical Netdata agent bind).
|
||||
- **Default:** localhost-only, no bearer required (like a typical local agent bind).
|
||||
- **P2P remains the secure remote path** (Noise + roles).
|
||||
- If you bind `0.0.0.0`, put REST behind a firewall, reverse proxy, or Holesail tunnel — do not expose raw metrics to the internet.
|
||||
- Future: optional bearer gate (`/api/v3/bearer_protection` parity).
|
||||
@@ -130,15 +131,15 @@ Single-agent MVP returns one node (this host). Parent/fleet aggregation is roadm
|
||||
|
||||
`Access-Control-Allow-Origin` defaults to `*` (override with `PEARDATA_REST_CORS`).
|
||||
|
||||
## Compatibility notes
|
||||
## Scope notes
|
||||
|
||||
| Netdata | PearData MVP |
|
||||
|---------|--------------|
|
||||
| Full ML weights / metric correlations | Simplified health weights |
|
||||
| Area | PearData |
|
||||
|------|----------|
|
||||
| ML weights / metric correlations | Simplified health weights |
|
||||
| Multi-node parent streaming | Single node; parent planned |
|
||||
| Cloud POST `/api/v3/spaces/.../data` | Not implemented (agent GET style only) |
|
||||
| Every chart Netdata ships | Core system charts (expanding) |
|
||||
| Functions execute via HTTP | Listed; run via P2P jobs |
|
||||
| Cloud POST spaces APIs | Not implemented (agent GET style only) |
|
||||
| App/plugin charts (nginx, DB, …) | System/OS charts; plugins later |
|
||||
| Functions via HTTP | Listed; run via P2P jobs |
|
||||
|
||||
## Implementation
|
||||
|
||||
|
||||
+7
-8
@@ -1,13 +1,13 @@
|
||||
# PearData Roadmap
|
||||
|
||||
Phased plan from MVP agent → Netdata-class fleet observability on pure P2P.
|
||||
Phased plan from MVP agent → real-time fleet observability on pure P2P.
|
||||
|
||||
## Guiding principles
|
||||
|
||||
1. **Instant value** — connect a pubkey, see live charts in seconds.
|
||||
2. **Agent efficiency** — stay in Netdata’s ballpark for CPU/RAM overhead.
|
||||
2. **Agent efficiency** — stay in a tight ballpark for CPU/RAM overhead.
|
||||
3. **PearDock patterns** — HyperDHT identity, protomux-rpc, roles, `pd1.` invites.
|
||||
4. **Dual access** — P2P desktop + Netdata-style REST (`/api/v1|v2|v3`).
|
||||
4. **Dual access** — P2P desktop + agent-style REST (`/api/v1|v2|v3`).
|
||||
5. **Ecosystem glue** — ready for PearDock / PearVirt / HoneyPeer / BareOS later.
|
||||
|
||||
---
|
||||
@@ -18,11 +18,11 @@ Phased plan from MVP agent → Netdata-class fleet observability on pure P2P.
|
||||
|------|--------|
|
||||
| Copy pear-app-template → PearData rebrand (`pd1.`, `PEARDATA_*`, `peardata/rpc`) | Done |
|
||||
| Shared protocol + schema + metrics catalog | Done |
|
||||
| Agent collector (CPU/RAM/load/net/disk/processes) @ ~1s | Done |
|
||||
| Agent collector (full system/OS charts + per-cpu/disk/iface/mount) @ ~1s | Done |
|
||||
| In-memory tiered store (1s + downsample) | Done |
|
||||
| Threshold anomaly engine + health | Done |
|
||||
| P2P RPC surface (query, subscribe, alerts, jobs, ACL) | Done |
|
||||
| Netdata-compatible REST v1/v2/v3 (local :19999) | Done |
|
||||
| agent-compatible REST v1/v2/v3 (local :19999) | Done |
|
||||
| Pear desktop fleet overview + live canvas charts | Done |
|
||||
| Docs: architecture, protocol, data model, REST, roadmap, tech choices | Done |
|
||||
| systemd unit + CI skeletons | Done |
|
||||
@@ -43,7 +43,6 @@ Phased plan from MVP agent → Netdata-class fleet observability on pure P2P.
|
||||
- Multi-peer compare mode (overlay 2–4 nodes on one chart)
|
||||
- Reconnection / offline banners with last-known samples
|
||||
- Agent process title / `peardata-agent` binary naming polish
|
||||
- Expand collectors: per-core CPU, per-iface net, mount disk space
|
||||
- Alert silence TTL auto-reenable
|
||||
- brittle tests for collector, store query, REST routes
|
||||
- One-line install script (`curl | bash`) for Linux agents
|
||||
@@ -96,8 +95,8 @@ See **[STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md)** for the full design (from Hol
|
||||
|
||||
## Non-goals (for now)
|
||||
|
||||
- Replacing Netdata Cloud SaaS multi-tenant product
|
||||
- Full byte-identical Netdata internal DB format
|
||||
- Replacing centralized multi-tenant SaaS monitoring products
|
||||
- Third-party proprietary on-disk metric DB formats
|
||||
- Shipping a browser-only public dashboard without auth by default (REST stays localhost unless explicitly bound)
|
||||
|
||||
---
|
||||
|
||||
+1
-1
@@ -31,7 +31,7 @@
|
||||
## REST API exposure
|
||||
|
||||
- Default bind is **localhost only** (`127.0.0.1:19999`).
|
||||
- REST is intentionally open on that bind (Netdata-agent style) — **do not** set `PEARDATA_REST_HOST=0.0.0.0` without a firewall, reverse proxy, or Holesail tunnel.
|
||||
- REST is intentionally open on that bind (local-agent style) — **do not** set `PEARDATA_REST_HOST=0.0.0.0` without a firewall, reverse proxy, or Holesail tunnel.
|
||||
- Prefer **P2P + roles** for remote multi-operator access; use REST for local scrapers/Grafana.
|
||||
|
||||
## Production checklist
|
||||
|
||||
+256
-238
@@ -1,290 +1,308 @@
|
||||
# HyperDB storage & linked-node sync
|
||||
|
||||
How PearData should adopt Holepunch’s **HyperDB + Corestore + Hyperswarm (+ Autobase)** stack for durable storage and P2P sync between linked agents — based on patterns in local clones under `holepunchto_repos` (`hyperdb`, `hyperdb-workshop`, `hyperdb-autobase-workshop`, `corestore`, `hyperswarm`, `autobee`, `pear-hyperdb`).
|
||||
PearData uses Holepunch **HyperDB + Corestore (+ optional Hyperswarm)** for durable metadata, warm metric history, and P2P sync between linked agents.
|
||||
|
||||
## Why HyperDB (not “just SQLite”)
|
||||
Live 1-second samples stay in the **memory ring** and are pushed over protomux-rpc. HyperDB is for everything that must survive restarts and replicate.
|
||||
|
||||
| Need | HyperDB fit |
|
||||
|------|-------------|
|
||||
| Typed collections + indexes | Hyperschema + `@ns/collection` + secondary indexes |
|
||||
| Local high-perf | `HyperDB.rocks(path, def)` |
|
||||
| P2P replicate | `HyperDB.bee(hypercore, def, { autoUpdate })` over Corestore |
|
||||
| Multi-writer HA parents | Autobase whose **view** is HyperDB (`extension: false`) |
|
||||
| Same query API local + remote | Workshops prove one `Registry` class works for both |
|
||||
---
|
||||
|
||||
PearData’s current `server/services/store.js` is an **in-memory ring**. HyperDB replaces durability + sync; the in-memory tier stays as the **hot 1s path**.
|
||||
## Status (implemented)
|
||||
|
||||
## Stack mapping (from Holepunch repos)
|
||||
| Piece | Status | Location |
|
||||
|-------|--------|----------|
|
||||
| Schema codegen | ✅ | `scripts/build-db.js` → `spec/` |
|
||||
| Collections + indexes | ✅ | `@peardata/*` in `spec/hyperdb` |
|
||||
| `PearDataModel` | ✅ | `server/db/model.js` |
|
||||
| Corestore open/close | ✅ | `server/db/index.js` |
|
||||
| Warm flush from tier1 | ✅ | `server/services/warm-flush.js` + `store` `warm` event |
|
||||
| Query fallback (memory → HyperDB) | ✅ | `MetricStore.query()` |
|
||||
| Agent boot upsert node | ✅ | `server/server.js` |
|
||||
| RPC: `getDbInfo`, `linkPeer`, `unlinkPeer`, `listPeerLinks` | ✅ | `server/handlers/monitor.js` |
|
||||
| REST: `GET /api/v3/db` | ✅ | `server/rest/routes.js` |
|
||||
| Hyperswarm replicate | ✅ stub | `server/db/replicate.js` (`PEARDATA_SWARM=1`) |
|
||||
| Autobase multi-writer parents | ⏳ Phase C | See roadmap |
|
||||
|
||||
Disable HyperDB: `PEARDATA_HYPERDB=0`.
|
||||
|
||||
---
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart TB
|
||||
subgraph Agent
|
||||
COL[collector 1s] --> HOT[Memory tier0 ring]
|
||||
COL --> DS[Downsample]
|
||||
DS --> HDB[(HyperDB bee/rocks)]
|
||||
POL[peer-policy / alerts / links] --> HDB
|
||||
HOT --> RPC[protomux-rpc + REST]
|
||||
HDB --> RPC
|
||||
end
|
||||
subgraph Sync
|
||||
CS[Corestore] --> HDB
|
||||
SW[Hyperswarm] -->|store.replicate| CS
|
||||
AB[Autobase optional] -->|view| HDB
|
||||
end
|
||||
CHILD[Linked child agent] -.->|discoveryKey| SW
|
||||
PARENT[Parent / peer agent] -.-> SW
|
||||
DESK[Pear desktop cache] -.-> SW
|
||||
COL[collector 1s] --> MEM[Memory tier0 ring]
|
||||
COL --> T1[Memory tier1 downsample]
|
||||
T1 -->|warm event| FLUSH[warm-flush queue]
|
||||
FLUSH --> HDB[(HyperDB.bee peardata-meta)]
|
||||
ANO[anomaly engine] -->|alert events| HDB
|
||||
LINK[linkPeer RPC] --> HDB
|
||||
MEM --> PUSH[push:metrics RPC]
|
||||
MEM --> Q[queryData / REST]
|
||||
HDB --> Q
|
||||
CS[Corestore data/corestore] --> HDB
|
||||
SW[Hyperswarm PEARDATA_SWARM=1] -->|store.replicate| CS
|
||||
```
|
||||
|
||||
| Component | Repo pattern | PearData use |
|
||||
|-----------|--------------|--------------|
|
||||
| **Hyperschema + hyperdb/builder** | `hyperdb-workshop/build.js` | `spec/` codegen for collections |
|
||||
| **HyperDB.bee** | workshop `Registry` | Replicable agent DB |
|
||||
| **HyperDB.rocks** | `hyperdb` README | Optional local-only fast index |
|
||||
| **Corestore** | workshop `bin.js` | Named cores: `metrics-meta`, `alerts`, … |
|
||||
| **Hyperswarm** | `swarm.join(discoveryKey)` + `store.replicate(conn)` | Link nodes / seed DB |
|
||||
| **protomux-rpc** | already in PearData | Control plane (unchanged) |
|
||||
| **Autobase + hyperdispatch** | `hyperdb-autobase-workshop` | Multi-writer **parent** / HA registry |
|
||||
| **autobee** | experimental multiwriter bee | Alternative later; prefer Autobase+HyperDB view for now |
|
||||
| **pear-hyperdb** | Pear-shaped Model wrapper | Optional UX for desktop-local rocks |
|
||||
|
||||
## Critical design rule: don’t put 1s samples in HyperDB txs
|
||||
|
||||
HyperDB is an **indexable document DB** (put/get/find + flush). Writing every chart every second as HyperDB transactions will:
|
||||
|
||||
- Amplify Rocks/Bee write cost
|
||||
- Create huge replication chatter
|
||||
- Fight Netdata-class overhead goals
|
||||
|
||||
**Split planes:**
|
||||
### Planes
|
||||
|
||||
| Plane | Storage | Sync |
|
||||
|-------|---------|------|
|
||||
| **Hot live (≤1h @ 1s)** | Memory ring (current) | P2P `push:metrics` (current) |
|
||||
| **Warm history (downsampled)** | Hypercore append **or** HyperDB rows keyed `(chart, tsBucket)` | Corestore replicate |
|
||||
| **Metadata** (peers, alerts, labels, jobs, ACL cache) | **HyperDB** | Corestore replicate |
|
||||
| **Fleet / parent consensus** | Autobase → HyperDB view | Swarm on autobase discoveryKey |
|
||||
| Hot live (~1h @ 1s) | Memory | `push:metrics` |
|
||||
| Warm history (~1m buckets) | HyperDB `@peardata/metric-point` | Corestore replicate |
|
||||
| Metadata / links / alerts | HyperDB collections | Corestore replicate |
|
||||
| Fleet HA (future) | Autobase → HyperDB view | Swarm on autobase key |
|
||||
|
||||
## Proposed HyperDB schema (`@peardata/*`)
|
||||
### Why not HyperDB for every 1s sample
|
||||
|
||||
Modeled after workshop `build.js` namespaces.
|
||||
HyperDB is transactional + indexed. Flushing every chart every second would inflate write amplification and replication traffic. Tier1 downsample (default every 60 samples ≈ 1 minute) is the durable path.
|
||||
|
||||
---
|
||||
|
||||
## Schema (`@peardata`)
|
||||
|
||||
Defined in `scripts/build-db.js`. **Append-only** — never delete fields from committed `spec/` (Holepunch safety rule).
|
||||
|
||||
### Collections
|
||||
|
||||
```text
|
||||
@peardata/node
|
||||
key: nodeId (string / pubkey hex)
|
||||
fields: hostname, platform, arch, cpus, agentVersion, labels{}, updatedAt
|
||||
|
||||
@peardata/peer-link
|
||||
key: [localNodeId, remotePublicKey]
|
||||
fields: role, alias, discoveryKey?, linkedAt, lastSeen, syncMode (push|pull|both)
|
||||
|
||||
@peardata/alert-config
|
||||
key: id
|
||||
fields: chart, dimension, warn, crit, comparator, enabled, info
|
||||
|
||||
@peardata/alert-event
|
||||
key: [id, ts] # or ulid
|
||||
fields: severity, value, threshold, message, cleared
|
||||
|
||||
@peardata/metric-point # WARM tier only (e.g. 1m buckets)
|
||||
key: [chart, ts]
|
||||
fields: context, values{} (map), tier
|
||||
|
||||
@peardata/job
|
||||
key: id
|
||||
fields: name, status, startedAt, finishedAt, result?
|
||||
```
|
||||
| Collection | Key | Purpose |
|
||||
|------------|-----|---------|
|
||||
| `@peardata/node` | `nodeId` | Agent / host inventory |
|
||||
| `@peardata/peer-link` | `localNodeId` + `remotePublicKey` | Linked peers + sync mode |
|
||||
| `@peardata/alert-config` | `id` | Threshold configs |
|
||||
| `@peardata/alert-event` | `id` + `ts` | Anomaly / alert history |
|
||||
| `@peardata/metric-point` | `chart` + `ts` | Warm downsampled samples (`valuesJson`) |
|
||||
| `@peardata/job` | `id` | Persisted job records (optional use) |
|
||||
|
||||
### Indexes
|
||||
|
||||
```text
|
||||
@peardata/node-by-hostname → node.hostname
|
||||
@peardata/peer-link-by-remote → peer-link.remotePublicKey
|
||||
@peardata/alert-event-by-chart → alert-event.chart + ts
|
||||
@peardata/metric-point-by-context → metric-point.context + ts
|
||||
```
|
||||
| Index | On |
|
||||
|-------|-----|
|
||||
| `@peardata/node-by-hostname` | hostname |
|
||||
| `@peardata/peer-link-by-remote` | remotePublicKey |
|
||||
| `@peardata/alert-event-by-chart` | chart + ts |
|
||||
| `@peardata/metric-point-by-context` | context + ts |
|
||||
| `@peardata/metric-point-by-tier` | tier + chart + ts |
|
||||
|
||||
Rebuild with:
|
||||
### Field notes
|
||||
|
||||
- `valuesJson` / `labelsJson` / `resultJson` — JSON strings for open-ended maps (avoids rigid hyperschema maps).
|
||||
- `tier` on metric-point: `1` = default warm (~1m). Future coarser tiers use `2+`.
|
||||
- `syncMode` on peer-link: `push` | `pull` | `both`.
|
||||
|
||||
### Regenerate after schema edits
|
||||
|
||||
```bash
|
||||
node scripts/build-db.js # Hyperschema + HyperDB.toDisk → spec/
|
||||
npm run build:db
|
||||
# or: node scripts/build-db.js
|
||||
```
|
||||
|
||||
## Agent integration shape
|
||||
Then commit `spec/hyperschema/*` and `spec/hyperdb/*`.
|
||||
|
||||
Follow `hyperdb-workshop` / `pear-hyperdb` Model pattern:
|
||||
---
|
||||
|
||||
## Runtime layout
|
||||
|
||||
```text
|
||||
server/
|
||||
db/
|
||||
build.js # schema codegen
|
||||
spec/hyperschema/
|
||||
spec/hyperdb/
|
||||
model.js # PearDataModel: putNode, linkPeer, queryWarm, …
|
||||
replicate.js # Hyperswarm join + store.replicate
|
||||
services/
|
||||
store.js # HOT memory (keep)
|
||||
store-hyperdb.js # WARM + metadata facade used by queryData/REST
|
||||
$PEARDATA_DATA_DIR/ # default ./data
|
||||
peer-policy.json # existing ACL file
|
||||
audit.log
|
||||
corestore/ # Corestore (HyperDB bee cores)
|
||||
```
|
||||
|
||||
### Boot (single-writer agent — Phase 2)
|
||||
Named core: **`peardata-meta`**.
|
||||
|
||||
```js
|
||||
const store = new Corestore(dataDir + '/corestore')
|
||||
const swarm = new Hyperswarm({ keyPair: await store.createKeyPair('swarm') })
|
||||
swarm.on('connection', (conn) => store.replicate(conn))
|
||||
Banner fields on agent start:
|
||||
|
||||
const metaCore = store.get({ name: 'peardata-meta' })
|
||||
const db = HyperDB.bee(metaCore, spec, { autoUpdate: true })
|
||||
- `hyperdb:` — DB public key (hex) for others to open a read replica
|
||||
- `swarm:` — `on` / `off`
|
||||
|
||||
// announce for linked peers / desktop seeders
|
||||
swarm.join(metaCore.discoveryKey, { server: true, client: true })
|
||||
```
|
||||
---
|
||||
|
||||
Collector path:
|
||||
## Configuration
|
||||
|
||||
1. Ingest → memory tier0 (unchanged)
|
||||
2. Every N samples → downsample → `tx.insert('@peardata/metric-point', …); tx.flush()`
|
||||
3. Alert transitions → `alert-event` collection
|
||||
4. `queryData` / REST: memory first, then HyperDB range scan for older windows
|
||||
| Variable | Default | Meaning |
|
||||
|----------|---------|---------|
|
||||
| `PEARDATA_HYPERDB` | on | `0` / `off` disables HyperDB entirely |
|
||||
| `PEARDATA_DATA_DIR` | `./data` | Policy + corestore root |
|
||||
| `PEARDATA_SWARM` | off | `1` enables Hyperswarm `store.replicate` |
|
||||
| `PEARDATA_TIER1_EVERY` | `60` | Samples per warm bucket (~60s @ 1Hz) |
|
||||
| `PEARDATA_TIER1_POINTS` | `1440` | In-memory tier1 ring (HyperDB keeps longer) |
|
||||
|
||||
### Auth note
|
||||
---
|
||||
|
||||
HyperDB replication shares **capability to read the core**, not PearData RPC roles. Keep:
|
||||
## API surface
|
||||
|
||||
- **Noise + MethodRoles** for mutating RPC (`setAlertConfig`, `runJob`)
|
||||
- Replication topic optionally gated (only invite-linked peers get discoveryKey / capability)
|
||||
- Do **not** announce writable cores to the public swarm without encryption / allowlist
|
||||
### RPC
|
||||
|
||||
## Linked nodes: sync modes
|
||||
| Method | Role | Description |
|
||||
|--------|------|-------------|
|
||||
| `getDbInfo` | viewer | `{ enabled, publicKeyHex, discoveryKeyHex, swarm }` |
|
||||
| `listPeerLinks` | viewer | Linked peers from HyperDB |
|
||||
| `linkPeer` | admin | Upsert link; optionally `joinRemoteTopic` if swarm on |
|
||||
| `unlinkPeer` | admin | Remove link |
|
||||
| `queryData` | viewer | Memory first; HyperDB warm if miss / `tier≥1` |
|
||||
|
||||
PearData “links” are first-class `@peardata/peer-link` rows + swarm topics.
|
||||
|
||||
| Mode | Behavior |
|
||||
|------|----------|
|
||||
| **Pull** | Local agent opens remote DB by key (`HyperDB.bee(store.get({ key }), spec, { writable: false, autoUpdate: true })`) and replicates |
|
||||
| **Push** | Remote peers allowed to replicate our meta/warm cores (seed) |
|
||||
| **Both** | Mutual swarm join (homelab mesh) |
|
||||
| **Parent aggregate** | Parent pulls many children; stores namespaced copies or Autobase fleet view |
|
||||
|
||||
Discovery:
|
||||
|
||||
1. Desktop/admin mints link → stores remote pubkey + optional `dbKey` (z32/hex)
|
||||
2. Agent joins `discoveryKey` of that core
|
||||
3. On connection: `store.replicate(conn)` (workshop pattern)
|
||||
4. `clone.watch` / `autoUpdate` refreshes HyperDB indexes for REST/RPC queries
|
||||
|
||||
This is the same pattern as workshop §3.1 Lookups — **read path is swarm + HyperDB**, not constant RPC polling.
|
||||
|
||||
## Parent / HA (Phase 4) — Autobase workshop
|
||||
|
||||
When you need multi-writer fleet registry or HA parents:
|
||||
|
||||
- Autobase bootstrap key shared across parent instances
|
||||
- `open: (store) => HyperDB.bee(store.get('db-view'), spec, { extension: false, autoUpdate: true })`
|
||||
- hyperdispatch ops: `add-writer`, `put-alert`, `put-link`, `ingest-rollup`
|
||||
- RPC (existing PearData / workshop) appends ops; apply mutates HyperDB view
|
||||
- Clients dial **any** writer; view key stays stable
|
||||
|
||||
Do **not** “backup” by copying Corestore folders (workshop warning). Rotate writers via Autobase instead.
|
||||
|
||||
## What stays on protomux-rpc
|
||||
|
||||
| Keep on RPC | Why |
|
||||
|-------------|-----|
|
||||
| Live `push:metrics` | Sub-second UX; not DB-shaped |
|
||||
| `subscribeMetrics` / handshake / ACL | Session security |
|
||||
| `runJob`, mintInvite | Mutating control |
|
||||
| On-demand `queryData` for hot window | Memory path |
|
||||
|
||||
| Move to HyperDB (+ replicate) | Why |
|
||||
|-------------------------------|-----|
|
||||
| Peer links, aliases, labels | Shared fleet truth |
|
||||
| Alert config + history | Durable, queryable |
|
||||
| Warm/cold metric buckets | History without central server |
|
||||
| Node inventory | Parent `/api/v3/nodes` |
|
||||
|
||||
## Phased delivery (recommended)
|
||||
|
||||
### Phase A — Local durability (rocks or bee, no swarm yet)
|
||||
|
||||
1. Add `hyperdb`, `hyperschema`, `corestore` deps
|
||||
2. `scripts/build-db.js` + `spec/`
|
||||
3. Persist alert configs, peer policy, warm downsample into HyperDB
|
||||
4. REST/RPC history falls back to HyperDB after memory miss
|
||||
5. Keep REST localhost semantics
|
||||
|
||||
### Phase B — Link & replicate
|
||||
|
||||
1. Hyperswarm alongside HyperDHT RPC (or reuse connections carefully — often **separate swarm for store.replicate**)
|
||||
2. `linkPeer` / `unlinkPeer` RPCs write `@peardata/peer-link`
|
||||
3. Desktop can seed/cache agent DB by key for offline scrubbing
|
||||
4. Document `pd1.` invite vs **db discovery key** (two layers)
|
||||
|
||||
### Phase C — Parent fleet
|
||||
|
||||
1. Parent process pulls N child DB keys
|
||||
2. Composite REST `/api/v3/nodes` + scoped `/api/v3/data`
|
||||
3. Optional Autobase for parent HA
|
||||
|
||||
### Phase D — Polish
|
||||
|
||||
1. Encryption at rest (`encryptionKey` on cores / autobee)
|
||||
2. Blind-peer seeding for always-on warm history
|
||||
3. Grafana: already have REST Prometheus export; optionally expose hypercore-stats
|
||||
|
||||
## Concrete file plan (when implementing)
|
||||
|
||||
```text
|
||||
peardata/
|
||||
scripts/build-db.js
|
||||
spec/hyperschema/
|
||||
spec/hyperdb/
|
||||
server/db/model.js
|
||||
server/db/replicate.js
|
||||
server/services/store-hyperdb.js
|
||||
docs/STORAGE-HYPERDB.md ← this file
|
||||
```
|
||||
|
||||
Deps (approximate):
|
||||
`linkPeer` args:
|
||||
|
||||
```json
|
||||
"hyperdb": "^6",
|
||||
"hyperschema": "^1",
|
||||
"corestore": "^7",
|
||||
"hyperswarm": "^4",
|
||||
"autobase": "^7",
|
||||
"hyperdispatch": "^1"
|
||||
{
|
||||
"remotePublicKey": "<64 hex>",
|
||||
"alias": "homelab-nas",
|
||||
"dbKeyHex": "<optional remote db key>",
|
||||
"discoveryKeyHex": "<optional topic to pull>",
|
||||
"syncMode": "both",
|
||||
"role": "viewer"
|
||||
}
|
||||
```
|
||||
|
||||
(`autobee` only if you prefer that multiwriter path over Autobase+HyperDB view.)
|
||||
### REST
|
||||
|
||||
## Relationship to current PearData roadmap
|
||||
| Path | Description |
|
||||
|------|-------------|
|
||||
| `GET /api/v3/db` | HyperDB keys + collection list |
|
||||
| `GET /api/v3/info` | Includes `peardata.hyperdb` block |
|
||||
| `GET /api/v3/data?...` | Uses async store query (warm fallback) |
|
||||
| Query `source` field | `memory-tier0` \| `memory-tier1` \| `hyperdb-warm` |
|
||||
|
||||
| Roadmap item | HyperDB role |
|
||||
|--------------|--------------|
|
||||
| Phase 2 tiered storage | Warm/cold collections |
|
||||
| Phase 3 PearDock/containers | Extra collections / indexes |
|
||||
| Phase 4 parent peer | Autobase + replicated views |
|
||||
| REST v3 historical queries | `find` ranges on `@peardata/metric-point` |
|
||||
```bash
|
||||
curl -s http://127.0.0.1:19999/api/v3/db | jq
|
||||
curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-3600&tier=1&points=120' | jq '.source'
|
||||
```
|
||||
|
||||
## References (local clones)
|
||||
---
|
||||
|
||||
| Path under `holepunchto_repos` | Takeaway |
|
||||
|--------------------------------|----------|
|
||||
| `hyperdb/README.md` | rocks vs bee, find/get/tx, autoUpdate |
|
||||
| `hyperdb-workshop` | Schema builder, Corestore+Swarm replicate, RPC inserts |
|
||||
| `hyperdb-autobase-workshop` | Multi-writer view, hyperdispatch, HA |
|
||||
| `corestore/README.md` | `store.replicate(conn)`, namespacing |
|
||||
| `pear-hyperdb` | Thin Model wrapper pattern for Pear apps |
|
||||
| `autobee` | Experimental multiwriter bee alternative |
|
||||
## How linked-node sync works
|
||||
|
||||
## Decision summary
|
||||
### Phase A (now) — local durability
|
||||
|
||||
1. **Yes — incorporate HyperDB** for metadata + warm history + linked-node sync.
|
||||
2. **Keep memory + RPC pushes** for live 1s Netdata feel.
|
||||
3. **Sync linked nodes via Corestore replication on Hyperswarm**, not by streaming every sample over RPC.
|
||||
4. **Use Autobase+HyperDB** when you need multi-writer parents / HA — same pattern as Holepunch’s own workshops.
|
||||
5. **Treat discovery keys as sensitive** — link only invited peers; RPC AuthZ remains authoritative for mutations.
|
||||
1. Agent opens Corestore + HyperDB on boot.
|
||||
2. Warm points + alert events persist under `data/corestore`.
|
||||
3. Restart retains warm history / links / node row.
|
||||
|
||||
### Phase B — mesh replicate (`PEARDATA_SWARM=1`)
|
||||
|
||||
Workshop pattern (`hyperdb-workshop/bin.js`):
|
||||
|
||||
```js
|
||||
swarm.on('connection', (conn) => store.replicate(conn))
|
||||
swarm.join(db.discoveryKey, { server: true, client: true })
|
||||
```
|
||||
|
||||
1. Enable swarm on agents that should seed/pull.
|
||||
2. Share **db public key** + **discovery key** (`getDbInfo` / `/api/v3/db`).
|
||||
3. Admin calls `linkPeer` with `discoveryKeyHex` + `syncMode: pull|both`.
|
||||
4. Peers replicate Corestore; HyperDB `autoUpdate` refreshes indexes.
|
||||
5. Remote warm history becomes queryable locally (parent path).
|
||||
|
||||
**Security:** replication grants read of the core to anyone who can join the topic. Prefer:
|
||||
|
||||
- Swarm only on trusted LAN / VPN, or
|
||||
- Future: encrypted cores + allowlisted swarm joins
|
||||
- Keep **mutations** on protomux-rpc AuthZ (`linkPeer` is admin)
|
||||
|
||||
HyperDHT RPC identity ≠ Hyperswarm topic access — treat them as two layers.
|
||||
|
||||
### Phase C — Autobase parents (planned)
|
||||
|
||||
Same as `hyperdb-autobase-workshop`:
|
||||
|
||||
- Autobase bootstrap key across parent writers
|
||||
- View = `HyperDB.bee(store.get('db-view'), spec, { extension: false })`
|
||||
- hyperdispatch ops for `put-link`, `put-alert`, rollups
|
||||
- Do **not** “backup” by copying Corestore folders
|
||||
|
||||
---
|
||||
|
||||
## Code map
|
||||
|
||||
| Path | Role |
|
||||
|------|------|
|
||||
| `scripts/build-db.js` | Hyperschema + HyperDB builder |
|
||||
| `spec/hyperschema/` | Generated encodings + `schema.json` |
|
||||
| `spec/hyperdb/` | Generated collections/indexes |
|
||||
| `server/db/model.js` | Typed CRUD facade |
|
||||
| `server/db/index.js` | Singleton open/close |
|
||||
| `server/db/replicate.js` | Optional Hyperswarm |
|
||||
| `server/services/warm-flush.js` | Batch writer |
|
||||
| `server/services/store.js` | Hot ring + `warm` events + query fallback |
|
||||
|
||||
### Model usage example
|
||||
|
||||
```js
|
||||
import { openDb, getDb, closeDb } from './server/db/index.js'
|
||||
|
||||
await openDb()
|
||||
const db = getDb()
|
||||
|
||||
await db.putPeerLink({
|
||||
localNodeId: 'abc123',
|
||||
remotePublicKey: 'ff'.repeat(32),
|
||||
syncMode: 'pull',
|
||||
linkedAt: Date.now(),
|
||||
})
|
||||
|
||||
await db.putMetricPoints([
|
||||
{ chart: 'system.cpu', context: 'system.cpu', ts: Date.now(), values: { user: 12.3 }, tier: 1 },
|
||||
])
|
||||
|
||||
const rows = await db.queryMetricPoints({
|
||||
chart: 'system.cpu',
|
||||
afterMs: Date.now() - 3600_000,
|
||||
beforeMs: Date.now(),
|
||||
})
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Dependencies
|
||||
|
||||
From Holepunch stack (see `holepunchto_repos`):
|
||||
|
||||
- `hyperdb` — DB engine (bee + rocks)
|
||||
- `hyperschema` — struct codegen
|
||||
- `corestore` — named hypercores + replicate
|
||||
- `hyperswarm` — topic discovery for store sync
|
||||
- `ready-resource` — open/close lifecycle
|
||||
|
||||
Future: `autobase`, `hyperdispatch` for HA parents.
|
||||
|
||||
---
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
npm run build:db
|
||||
SKIP_INTEGRATION=1 npm test
|
||||
# includes test/hyperdb.test.js
|
||||
```
|
||||
|
||||
Manual:
|
||||
|
||||
```bash
|
||||
npm run start:server
|
||||
curl -s http://127.0.0.1:19999/api/v3/db | jq
|
||||
# wait ~60s for first warm bucket, then:
|
||||
curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-7200&tier=1&points=120' | jq '.source,.points'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Operational checklist
|
||||
|
||||
- [ ] Commit `spec/` after every `build:db`
|
||||
- [ ] Back up **keys** (`SERVER_SEED`, swarm keypair in corestore) — not by zipping live corestore while writing
|
||||
- [ ] Keep REST on localhost; swarm only when linking trusted peers
|
||||
- [ ] Prefer `pd1.` invites for RPC admin; share db discovery keys only with linked nodes
|
||||
- [ ] Monitor disk under `data/corestore` as warm retention grows
|
||||
|
||||
---
|
||||
|
||||
## References
|
||||
|
||||
| Repo (local `holepunchto_repos`) | Takeaway |
|
||||
|----------------------------------|----------|
|
||||
| `hyperdb` | rocks vs bee, tx/flush, autoUpdate |
|
||||
| `hyperdb-workshop` | builder + Corestore + Swarm replicate |
|
||||
| `hyperdb-autobase-workshop` | Multi-writer view for parents |
|
||||
| `corestore` | `store.replicate(conn)` |
|
||||
| `pear-hyperdb` | Pear Model wrapper style |
|
||||
|
||||
+22
-6
@@ -23,13 +23,29 @@ If fleet scale demands it later:
|
||||
|
||||
For MVP, JSON pushes keep the stack simple and debuggable.
|
||||
|
||||
## Bare / Pear runtime (no Node builtins)
|
||||
|
||||
Pear’s native runtime is **Bare**. Do not rely on Node core modules existing at runtime.
|
||||
|
||||
| Pattern | Use |
|
||||
|---------|-----|
|
||||
| `package.json` `imports` map | `fs` / `os` / `path` / `crypto` / `events` / `http` / … → `bare-*` under `"bare"` condition (same idea as PearDock / pear-docs node-compat) |
|
||||
| `bare-node-runtime/global` | Loaded first in Bare entrypoints (`bin/peardata-server.mjs`, `index.js`) for `process` / `Buffer` / `fetch` |
|
||||
| `b4a` | Buffers in `shared/` and wire code (prefer over Node `Buffer`) |
|
||||
| Direct `bare-*` | Optional for new Bare-first modules |
|
||||
|
||||
**Keep bare-safe:** `shared/`, `client/`, Pear `app.js` / `index.js` / `ui/`.
|
||||
**Agent (`server/`):** may keep Node-style import names; under Bare they resolve via the imports map. Boot via `bin/peardata-server.mjs`.
|
||||
|
||||
Do **not** use `if (isBare) require('bare-fs') else require('fs')` branches for packable apps — Bare’s packer walks both sides. Prefer import maps.
|
||||
|
||||
## Collection
|
||||
|
||||
| Option | Verdict |
|
||||
|--------|---------|
|
||||
| **Node `os` + `/proc` (chosen)** | Zero native deps, good enough for MVP, low overhead |
|
||||
| **`os` + `/proc` via bare-os / bare-fs (chosen)** | Hybrid Node+Bare; low overhead |
|
||||
| `node-os-utils` | Convenient but extra dep / less control |
|
||||
| Native bindings (netdata collectors, `systeminformation`) | Higher fidelity; consider Phase 2 for Windows depth |
|
||||
| Native bindings (`systeminformation`, etc.) | Higher fidelity; consider Phase 2 for Windows depth |
|
||||
| Shell out to `vmstat`/`iostat` | Avoid on hot path |
|
||||
|
||||
**Linux:** `/proc/meminfo`, `/proc/loadavg`, `/proc/net/dev`, `/proc/diskstats`
|
||||
@@ -58,20 +74,20 @@ Target overhead: single timer, no child processes per tick, ring buffers only.
|
||||
| Option | Verdict |
|
||||
|--------|---------|
|
||||
| **Canvas sparklines (chosen MVP)** | No extra deps inside Pear; snappy for 1s updates |
|
||||
| **uPlot** | Best next step for Netdata-like interactive charts (tiny, fast) |
|
||||
| **uPlot** | Best next step for interactive charts (tiny, fast) |
|
||||
| Chart.js / ECharts | Heavier; fine for secondary views |
|
||||
| Grafana via REST | External; use `/api/v3/data` + Prometheus export |
|
||||
|
||||
## REST
|
||||
|
||||
Built-in Node `http` (no Express) — small attack surface, enough for Netdata-style GET APIs.
|
||||
`http` module (no Express) — under Bare this is `bare-http1` via import maps. Small attack surface for agent-style GET APIs.
|
||||
|
||||
## Packaging
|
||||
|
||||
| Piece | Approach |
|
||||
|-------|----------|
|
||||
| Agent | Node 20+ , systemd unit `deploy/peardata.service` |
|
||||
| Desktop | Pear (`pear-electron` + `pear-bridge`) |
|
||||
| Agent | Node 20+ or Bare (`bin/peardata-server.mjs`), systemd unit `deploy/peardata.service` |
|
||||
| Desktop | Pear (`pear-electron` + `pear-bridge`) with Bare-ready imports |
|
||||
| Invites | `pd1.` tokens (PearDock-style) |
|
||||
| Installer | Phase 1 one-liner script |
|
||||
|
||||
|
||||
+2
-1
@@ -16,7 +16,8 @@
|
||||
| `test/crypto-auth.test.js` | MAC key, capabilities, admin proof, invites, classify input |
|
||||
| `test/protocol.test.js` | Constants, monitoring `MethodRoles`, schema validators |
|
||||
| `test/store.test.js` | Metric ring buffer ingest + query |
|
||||
| `test/rest.test.js` | Netdata-style `/api/v3` routes |
|
||||
| `test/hyperdb.test.js` | HyperDB model (node, links, warm points, alerts) |
|
||||
| `test/rest.test.js` | agent-style `/api/v3` routes |
|
||||
| `test/integration.test.js` | Live HyperDHT agent + handshake + metrics query |
|
||||
|
||||
```bash
|
||||
|
||||
Reference in New Issue
Block a user