snxraven/peardata
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Updates

Browse Source
This commit is contained in:
Raven Scott
2026-07-18 16:22:56 -04:00
parent 015d92a257
commit f747ffbd25
36 changed files with 1975 additions and 873 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+123 -151
View File
@@ -1,193 +1,165 @@
# Architecture
PearData is a **decentralized, P2P clone of the Netdata real-time monitoring experience**, built on the same HyperDHT + protomux-rpc patterns as PearDock-class apps (via the pear-app template).
## Design goals
1. **No central control plane** — peers dial a public key, not a SaaS tenant.
2. **Cryptographic identity** — HyperDHT Noise streams authenticate both ends.
3. **Clear AuthZ** — roles, rate limits, audit, optional allowlist / revoke.
4. **Shared wire contract** — `shared/*` is the single source of truth for client + server.
5. **Replaceable domain** — demo room is a thin layer over the session stack.
6. **Pear-native desktop** — `pear-electron` shell with `<pear-ctrl>` window chrome.
1. **No central control plane** — dial an agent by Ed25519 public key.
2. **Instant live truth** — ~1s metric push to connected desktops.
3. **Dual API** — P2P RPC for the Pear client; Netdata-style REST for scripts/Grafana.
4. **Clear AuthZ** — viewer (pubkey) vs operator/admin (`pd1.` invite / seed proof).
5. **Low agent overhead** — Node/`os` + `/proc` collectors, ring buffers, hot-path RPCs.
6. **Ecosystem-ready** — shared wire contract in `shared/` for PearDock / PearVirt adapters later.
## Mapping to PearDock / template components
| PearDock-class concept | PearData component |
|------------------------|--------------------|
| HyperDHT secret stream | `server/server.js` + `client/connection.js` |
| protomux-rpc methods | `shared/protocol.js` + `server/handlers/monitor.js` |
| Capability invites (`pd1.`) | `shared/crypto-auth.js` |
| Roles viewer/operator/admin | `shared/protocol.js` `MethodRoles` + `server/core/acl.js` |
| Peer policy / revoke | `server/core/peer-policy.js` |
| Connection manager / multi-peer | `client/manager.js` |
| Job tray | `server/services/jobs.js` + desktop actions |
| Domain service | **Metrics pipeline** (`collector` → `store` → `anomaly` → pushes) |
| Optional HTTP surface | `server/rest/*` (Netdata v1/v2/v3) |
## System context
```mermaid
flowchart LR
UI[Pear desktop / scripts] -->|HyperDHT Noise| SRV[Node server]
SRV --> STATE[Room / your domain]
UI -.->|bootstrap / punch| NET[HyperDHT network]
SRV -.-> NET
```
## Layered stack
```mermaid
flowchart TB
subgraph Presentation
HTML[index.html + app.js + ui/styles.css]
PEAR[index.js pear-electron + bridge]
end
subgraph ClientCore
MGR[client/manager.js]
CON[client/connection.js]
ID[client/identity.js]
end
subgraph Wire
PROT[shared/protocol.js]
ENC[shared/encodings.js]
AUTH[shared/crypto-auth.js]
SCH[shared/schema.js]
end
subgraph ServerCore
BOOT[server/server.js]
SESS[server/rpc/session.js]
ACL[server/core/acl.js]
HAND[server/handlers/*]
SVC[server/services/*]
end
PEAR --> HTML
HTML --> MGR --> CON
CON --> PROT
CON --> AUTH
SESS --> PROT
SESS --> ACL
SESS --> SCH
HAND --> SVC
BOOT --> SESS
CON <-->|secret stream| SESS
UI[PearData desktop] -->|HyperDHT Noise + RPC| AG[PearMonitor agent]
SCRIPTS[curl / Grafana / Prometheus scrapers] -->|HTTP REST :19999| AG
AG --> COL[Collector 1s]
COL --> STORE[Tiered ring buffers]
COL --> ANO[Anomaly engine]
AG -.->|bootstrap / punch| DHT[HyperDHT]
UI -.-> DHT
```
## Process model
| Process | Entry | Responsibility |
|---------|-------|----------------|
| **Server** | `server/server.js` or `bin/peardata-server.mjs` | HyperDHT listen, RPC, domain state |
| **Desktop** | `index.js` → Pear Runtime | Window + HTML UI; dials servers as a client |
| **Scripts** | `scripts/*` | mint-invite, healthcheck, soak (use client stack) |
| **Agent** | `server/server.js` / `bin/peardata-server.mjs` | Collect, store, P2P RPC, optional REST |
| **Desktop** | `index.js` → Pear Runtime | Fleet UI, multi-peer dial, live charts |
| **Scripts** | `scripts/*` | mint-invite, healthcheck, soak |
Server and desktop are **independent**. You can run many clients against one server, or headless scripts with no UI.
Many desktops may dial one agent; one desktop may dial many agents.
## Server boot
## Planes
1. `loadOrCreateKeyPair()` → persist `SERVER_SEED` / `SERVER_PUBLIC_KEY` in `.env`
2. `initAuthKeys()` → HKDF MAC key for capabilities
3. `loadPeerPolicy()` → roles / revocations / spent JTIs from `PEARDATA_DATA_DIR`
4. `dht.createServer().listen(keyPair)`
5. On connection → revoke check → `PeerSession` → `registerAllHandlers` → peer registry
6. Banner logs public key + secure/insecure mode
7. `graceful-goodbye` / SIGINT / SIGTERM drain peers and destroy DHT
### 1. Control / metadata (RPC)
## Session middleware (every non-hot RPC)
Handshake, node info, chart/context catalog, alert config, invites, jobs, ACL.
### 2. High-frequency metrics
- **Ingest:** collector emits sample batches every `PEARDATA_SAMPLE_MS` (default 1000).
- **Store:** tier0 (1s) + tier1 (downsampled averages).
- **Push:** `push:metrics` to subscribed peers (protomux-rpc events).
- **Pull:** `queryData` / REST `/api/v3/data` for history windows.
### 3. Anomaly / health
Threshold engine evaluates each batch; transitions emit `push:anomaly` / `push:alert`; health snapshot on an interval via `push:health`.
### 4. Local REST (optional)
Netdata-compatible HTTP on `127.0.0.1:19999` by default — for local tooling without P2P. Disable with `PEARDATA_REST=0`.
## Layered stack
```mermaid
flowchart TD
IN[method + args] --> RL{Rate limit}
RL -->|deny| E1[RATE_LIMIT_EXCEEDED]
RL -->|ok| ACL{roleAllows MethodRoles}
ACL -->|deny| E2[PERMISSION_DENIED + audit]
ACL -->|ok| VAL{validateMethodArgs}
VAL -->|fail| E3[INVALID_ARGS]
VAL -->|ok| H[Handler]
H --> OK[Result + optional audit]
flowchart TB
subgraph Presentation
HTML[index.html + app.js]
PEAR[index.js pear-electron]
end
subgraph ClientCore
MGR[client/manager.js]
CON[client/connection.js]
end
subgraph Wire
PROT[shared/protocol.js]
MET[shared/metrics.js]
SCH[shared/schema.js]
AUTH[shared/crypto-auth.js]
end
subgraph Agent
BOOT[server/server.js]
PIPE[server/pipeline.js]
COL[collector]
STORE[store]
ANO[anomaly]
HAND[handlers/monitor.js]
REST[rest/http-server.js]
end
PEAR --> HTML --> MGR --> CON
CON <-->|Noise + protomux-rpc| HAND
BOOT --> PIPE --> COL --> STORE
COL --> ANO
REST --> STORE
HAND --> STORE
```
**Hot path** (`session.respond(method, handler, { hot: true })` or stream method names):
## Agent boot sequence
- Still rate-limited and ACL-checked
- Skips full schema validation / success audit (for high-frequency streams)
1. Load/create `SERVER_SEED` / public key → `initAuthKeys`
2. Load peer policy from `PEARDATA_DATA_DIR`
3. `startPipeline()` — collector + store + anomaly fan-out
4. `startRestServer()` — unless disabled
5. HyperDHT `createServer().listen(keyPair)`
6. On connection → revoke check → `PeerSession` → register monitor handlers
7. Banner prints pubkey + REST URL
## Client connection states
## Session middleware
```
idle → dialing → handshaking → ready
↘ closed → (manager reconnect timer)
```
| State | Meaning |
|-------|---------|
| `idle` | Constructed, not dialing |
| `dialing` | `dht.connect(serverPk)` in flight |
| `handshaking` | Stream open; `handshake` RPC |
| `ready` | Authenticated; RPCs allowed |
| `closed` | Torn down |
`ConnectionManager` tracks multiple peers, active selection, and reconnect (max `PEARDATA_MAX_RECONNECT`).
Same as the template: rate limit → ACL (`MethodRoles`) → schema validate → handler.
Hot methods (`queryData`, `subscribeMetrics`, `ping`, …) skip heavy audit.
## Identity planes
| Plane | Storage | Purpose |
|-------|---------|---------|
| **Server keypair** | `.env` (`SERVER_SEED`) | DHT listen address + HMAC root |
| **Client keypair** | `~/.config/peardata/identity.json` | Stable peerId for AuthZ / revoke |
| **Capabilities** | Issued as tokens / `pd1.` invites | Role grants with optional expiry & peer bind |
| **Peer policy** | `data/peer-policy.json` | Registered roles, revocations, spent JTIs |
| **Audit** | `data/audit.log` | Mutating RPC trail |
| Agent keypair | `.env` `SERVER_SEED` | DHT address + HMAC root |
| Client keypair | `~/.config/peardata/identity.json` | Stable peerId |
| Capabilities | `pd1.` invites / raw tokens | Role grants |
| Peer policy | `data/peer-policy.json` | Roles, revokes, spent JTIs |
| Audit | `data/audit.log` | Mutating RPC trail |
## Parent peer (future)
A heavier agent may subscribe to child agents over P2P, downsample into its own store, and expose fleet REST — still no central SaaS. Design leaves room via `stream_path` REST stub and multi-peer desktop manager.
## Module map
### `shared/`
### Keep from template
| File | Role |
|------|------|
| `protocol.js` | `PROTOCOL`, roles, `MethodRoles`, `Methods`, `Pushes` |
| `encodings.js` | compact-encoding JSON for protomux-rpc |
| `crypto-auth.js` | MAC key, capabilities, admin proof, invites |
| `schema.js` | Lightweight request validation |
`server/core/*`, `server/rpc/session.js`, `client/*`, `shared/crypto-auth.js`, `shared/encodings.js`, Pear titlebar patterns.
### `server/`
### PearData domain
| Path | Role |
|------|------|
| `server.js` | Boot + DHT accept loop |
| `core/keys.js` | Seed load / generate |
| `core/auth-keys.js` | Process-wide MAC key |
| `core/acl.js` | Role resolution + assert |
| `core/peer-policy.js` | File-backed policy |
| `core/peer-registry.js` | Live sessions |
| `core/audit.js` | Audit log writer |
| `rpc/session.js` | ProtomuxRPC + middleware |
| `rpc/register.js` | Wire handlers per session |
| `handlers/demo.js` | **Replace** — domain RPCs |
| `services/room.js` | **Replace** — domain state |
| `utils/logger.js` | Structured / pretty logs |
| `utils/rateLimiter.js` | Per-peer RPM |
### `client/`
| File | Role |
|------|------|
| `identity.js` | Persistent client seed |
| `connection.js` | Single peer RPC client |
| `manager.js` | Multi-peer + reconnect |
| `errors.js` | Error normalization |
| `index.js` | Public re-exports |
### Desktop shell
| File | Role |
|------|------|
| `index.js` | Pear Runtime + Bridge |
| `index.html` | Titlebar (`pear-ctrl`) + layout |
| `app.js` | UI → manager |
| `ui/styles.css` | Drag regions + theme |
See [DESKTOP.md](./DESKTOP.md).
## What to keep vs replace
| Keep | Replace when productizing |
|------|---------------------------|
| `shared/*` wire + crypto | Method names / schema for your domain |
| `server/rpc/session.js` | Rarely — middleware is generic |
| `server/core/*` | Peer policy storage backend if needed |
| `client/connection.js` + `manager.js` | UI-specific multi-peer UX |
| Titlebar / `pear-ctrl` patterns | Visual design only — keep drag + controls |
| `server/handlers/demo.js` + `services/room.js` | **Your product** |
| `shared/metrics.js` | Chart/context catalog |
| `shared/data-model.js` | Typed shapes |
| `server/services/collector.js` | System sampling |
| `server/services/store.js` | Tiered buffers + query |
| `server/services/anomaly.js` | Thresholds |
| `server/services/alerts.js` | Alert CRUD helpers |
| `server/services/subscriptions.js` | Push fan-out |
| `server/services/jobs.js` | On-demand jobs |
| `server/handlers/monitor.js` | RPC surface |
| `server/rest/*` | Netdata HTTP API |
| `server/pipeline.js` | Wire collector→store→push |
## Related docs
- [PROTOCOL.md](./PROTOCOL.md)
- [SECURITY.md](./SECURITY.md)
- [DESKTOP.md](./DESKTOP.md)
- [CONFIGURATION.md](./CONFIGURATION.md)
- [EXTENDING.md](./EXTENDING.md)
- [PROTOCOL.md](./PROTOCOL.md) — RPC methods & pushes
- [DATA-MODEL.md](./DATA-MODEL.md) — metrics / anomalies / health
- [REST-API.md](./REST-API.md) — `/api/v1|v2|v3`
- [TECH-CHOICES.md](./TECH-CHOICES.md) — collector & charts
- [ROADMAP.md](./ROADMAP.md) — phases
- [SECURITY.md](./SECURITY.md) — threat model
docs/CI.md
+1 -1
View File
@@ -20,7 +20,7 @@
**lint-docs**
- Asserts required documentation files exist (README, architecture, protocol, getting started, security, desktop, configuration, LICENSE, etc.)
- Asserts required documentation files exist (README, architecture, protocol, data model, REST API, roadmap, tech choices, getting started, security, desktop, configuration, LICENSE, etc.)
### Release job details (`release.yml`)
docs/CONFIGURATION.md
+54 -84
View File
@@ -1,6 +1,6 @@
# Configuration reference
All knobs can be set via environment variables. The server loads `.env` through `dotenv` on boot (`server/core/keys.js`). Copy `.env.example` to get started.
All knobs can be set via environment variables. The agent loads `.env` through `dotenv` on boot (`server/core/keys.js`). Copy `.env.example` to get started.
```bash
cp .env.example .env
@@ -8,7 +8,7 @@ cp .env.example .env
---
## Server identity
## Agent identity
| Variable | Default | Description |
|----------|---------|-------------|
@@ -16,8 +16,6 @@ cp .env.example .env
| `SERVER_PUBLIC_KEY` | *derived* | 32-byte public key as **64 hex**. Clients dial this. Auto-synced to `.env` when seed loads. |
| `SERVER_KEY` | — | Alias accepted for `SERVER_SEED` (legacy). Prefer `SERVER_SEED`. |
On first boot without a seed, the server appends both values to `.env`.
Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators.
---
@@ -26,10 +24,10 @@ Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators.
| Variable | Default | Description |
|----------|---------|-------------|
| `PEARDATA_DEFAULT_ROLE` | `viewer` | Baseline role for unknown peers: `viewer` \| `operator` \| `admin` |
| `PEARDATA_ADMIN_KEYS` | empty | Comma-separated peer public keys always elevated to **admin** |
| `PEARDATA_ALLOWLIST` | empty | If **non-empty**, only listed peer pubs (plus already-registered policy peers) may connect |
| `PEARDATA_INSECURE_OPEN_ADMIN` | off | `1` / `true` / `yes` → every peer is admin. **Dev only.** |
| `PEARDATA_DEFAULT_ROLE` | `viewer` | Baseline role: `viewer` \| `operator` \| `admin` |
| `PEARDATA_ADMIN_KEYS` | empty | Comma-separated peer pubs always elevated to **admin** |
| `PEARDATA_ALLOWLIST` | empty | If **non-empty**, only listed peers may connect |
| `PEARDATA_INSECURE_OPEN_ADMIN` | off | `1` → every peer is admin. **Dev only.** |
---
@@ -38,20 +36,41 @@ Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators.
| Variable | Default | Description |
|----------|---------|-------------|
| `PEARDATA_DATA_DIR` | `./data` | Peer policy JSON + `audit.log` |
| `PEARDATA_HOME` | OS home | Root for client identity path construction (`client/identity.js`) |
| `PEARDATA_MAX_MESSAGES` | `500` | In-memory demo room ring buffer size |
| `PEARDATA_HOME` | OS home | Root for client identity (`~/.config/peardata/identity.json`) |
| `PEARDATA_RATE_LIMIT_RPM` | `120` | Per-peer RPC requests per minute |
| `PEARDATA_MAX_RECONNECT` | `20` | Client manager reconnect attempts per peer |
### Data directory layout
```
data/
├── peer-policy.json # registered peers, roles, revocations, spent JTIs
└── audit.log # JSON lines for mutating RPCs + failures
├── peer-policy.json
└── audit.log
```
Recommended permissions: directory `0700`. Do **not** commit `data/` or `.env`.
Permissions: directory `0700`. Do **not** commit `data/` or `.env`.
---
## Metrics pipeline
| Variable | Default | Description |
|----------|---------|-------------|
| `PEARDATA_SAMPLE_MS` | `1000` | Collector interval |
| `PEARDATA_TIER0_POINTS` | `3600` | High-res ring size (~1h @ 1s) |
| `PEARDATA_TIER1_POINTS` | `1440` | Downsampled ring size |
| `PEARDATA_TIER1_EVERY` | `60` | Samples per tier1 average |
---
## REST API (Netdata-style)
| Variable | Default | Description |
|----------|---------|-------------|
| `PEARDATA_REST` | on | Set `0` / `off` to disable HTTP API |
| `PEARDATA_REST_HOST` | `127.0.0.1` | Bind address (`0.0.0.0` exposes LAN — firewall!) |
| `PEARDATA_REST_PORT` | `19999` | HTTP port (Netdata classic) |
| `PEARDATA_REST_CORS` | `*` | `Access-Control-Allow-Origin` |
See [REST-API.md](./REST-API.md).
---
@@ -60,87 +79,38 @@ Recommended permissions: directory `0700`. Do **not** commit `data/` or `.env`.
| Variable | Default | Description |
|----------|---------|-------------|
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` |
| `LOG_JSON` | off | `1` → structured JSON logs (good for journald) |
| `LOG_JSON` | off | `1` → JSON lines |
---
## Healthcheck & soak
## Healthcheck / soak / tests
| Variable | Default | Description |
|----------|---------|-------------|
| `PEARDATA_HEALTH_KEY` | — | Public key for remote health dial (falls back to `SERVER_PUBLIC_KEY`) |
| `HEALTHCHECK_TIMEOUT_MS` | `8000` | Healthcheck hard timeout (ms) |
| `SOAK_DURATION_MS` | `60000` | Soak test run length |
| `SOAK_INTERVAL_MS` | `500` | Delay between soak posts |
```bash
# Liveness only (no dial) when no key is set
npm run healthcheck
# Full dial + ping (server must be running)
SERVER_PUBLIC_KEY=… SERVER_SEED=… npm run healthcheck
# Load soak (admin seed recommended so postMessage is allowed)
SERVER_PUBLIC_KEY=… SERVER_SEED=… npm run soak
```
---
## Testing
| Variable | Default | Description |
|----------|---------|-------------|
| `SKIP_INTEGRATION` | unset | `1` skips live HyperDHT integration test |
---
## Pear GUI (package.json)
Not environment variables — set under `package.json` → `pear.gui`. See [DESKTOP.md](./DESKTOP.md).
| Field | Template |
|-------|----------|
| `width` × `height` | 1100 × 780 |
| `minWidth` × `minHeight` | 720 × 480 |
| `resizable` / `movable` | `true` |
| `backgroundColor` | `#0b1020` |
---
## systemd
`deploy/peardata.service` expects:
| Path | Purpose |
|------|---------|
| `/opt/peardata` | WorkingDirectory |
| `/opt/peardata/.env` | `EnvironmentFile` |
| `/opt/peardata/data` | Writable data dir |
Edit unit paths before enabling. See [GETTING-STARTED.md](./GETTING-STARTED.md#systemd).
| `PEARDATA_HEALTH_KEY` | `SERVER_PUBLIC_KEY` | Key for remote ping |
| `HEALTHCHECK_TIMEOUT_MS` | `8000` | Dial timeout |
| `SOAK_DURATION_MS` | `60000` | Soak length |
| `SOAK_INTERVAL_MS` | `500` | Soak RPC interval |
| `SKIP_INTEGRATION` | off | `1` skips DHT integration test |
---
## npm scripts
| Script | Command | Purpose |
|--------|---------|---------|
| `npm start` / `npm run dev` | `pear run -d .` | Pear desktop UI |
| `npm run start:server` / `server` | `node server/server.js` | P2P server |
| `npm run start:server:bin` | `node bin/peardata-server.mjs` | Alternate server entry |
| `npm test` | brittle suite | Unit + integration |
| `npm run test:integration` | integration only | Live DHT test |
| `npm run healthcheck` | dial or liveness | Process / network check |
| `npm run soak` | long connect loop | Stability exercise |
| `npm run mint-invite -- [role] [ttlMs]` | mint `pd1.` invite | Offline invite tooling |
| `npm run rename -- <slug> <Product>` | rebrand tree | New product from template |
| `bash scripts/release.sh` | source tarball | Local release artifacts |
| Script | Purpose |
|--------|---------|
| `npm start` / `npm run dev` | Pear desktop |
| `npm run start:server` | Agent (P2P + REST) |
| `npm run start:server:bin` | `bin/peardata-server.mjs` |
| `npm test` | brittle suite |
| `npm run mint-invite -- [role] [ttlMs]` | Offline `pd1.` invite |
| `npm run healthcheck` | Liveness |
| `npm run soak` | Load exercise |
---
## Security notes
## systemd
- Never ship `SERVER_SEED` in client bundles or public repos.
- Prefer invites over seed distribution.
- Rotate seed = new public key → all clients re-dial and re-invite.
- See [SECURITY.md](./SECURITY.md) for the full checklist.
Unit: `deploy/peardata.service`
WorkingDirectory: `/opt/peardata`
EnvironmentFile: `/opt/peardata/.env`
docs/DATA-MODEL.md
+141
View File
@@ -0,0 +1,141 @@
# Data model
Canonical shapes for metrics, anomalies, alerts, health, and jobs.
Implemented in `shared/data-model.js`, `shared/metrics.js`, and agent services.
## NodeInfo
```json
{
"nodeId": "a1b2c3…",
"hostname": "homelab-1",
"publicKeyHex": "…64 hex…",
"platform": "linux",
"arch": "x64",
"release": "6.x",
"cpus": 8,
"totalMemMiB": 32000,
"agentVersion": "0.1.0",
"protocolVersion": 1,
"startedAt": 1710000000000,
"charts": ["system.cpu", "system.ram", "…"],
"sampleIntervalMs": 1000,
"sampleCount": 42
}
```
## Metric contexts & charts
Netdata-inspired IDs:
| Context | Chart id | Units | Dimensions (MVP) |
|---------|----------|-------|------------------|
| `system.cpu` | `system.cpu` | percentage | user, system, nice, iowait, irq, softirq, idle |
| `system.ram` | `system.ram` | MiB | used, cached, buffers, free |
| `mem.available` | `mem.available` | MiB | avail |
| `system.load` | `system.load` | load | load1, load5, load15 |
| `system.io` | `system.io` | KiB/s | reads, writes |
| `system.net` | `system.net` | kilobits/s | received, sent |
| `system.processes` | `system.processes` | processes | running, blocked, total |
| `system.uptime` | `system.uptime` | seconds | uptime |
Chart summary objects mirror Netdata’s `/api/v1/charts` fields (`id`, `context`, `units`, `dimensions`, `update_every`, `first_entry`, `last_entry`, …).
## MetricSample (live push)
```json
{
"chart": "system.cpu",
"context": "system.cpu",
"ts": 1710000000123,
"values": { "user": 12.4, "system": 3.1, "idle": 84.0 }
}
```
## QueryResult (history)
```json
{
"chart": "system.cpu",
"context": "system.cpu",
"labels": ["time", "user", "system", "nice", "iowait", "irq", "softirq", "idle"],
"data": [[1710000000, 12.4, 3.1, 0, 0, 0, 0, 84.0]],
"view_update_every": 1,
"after": 1709999940,
"before": 1710000000,
"points": 60,
"format": "json"
}
```
`time` is unix seconds. Nulls allowed for missing dimensions.
## Storage tiers
| Tier | Resolution | Default retention | Env |
|------|------------|-------------------|-----|
| 0 | 1s | 3600 points (~1h) | `PEARDATA_TIER0_POINTS` |
| 1 | avg every N samples (default 60 → ~1m) | 1440 points (~24h) | `PEARDATA_TIER1_POINTS`, `PEARDATA_TIER1_EVERY` |
## AnomalyEvent
```json
{
"id": "cpu_user_high:1710000000123",
"chart": "system.cpu",
"context": "system.cpu",
"dimension": "user",
"severity": "warning",
"score": 0.6,
"value": 88.2,
"threshold": 80,
"comparator": ">",
"message": "CPU user time high: user=88.2 > 80",
"ts": 1710000000123,
"cleared": false
}
```
## AlertConfig / AlertState
Config fields: `id`, `chart`, `dimension`, `warn`, `crit`, `comparator`, `lookbackSec`, `enabled`, `info`.
State adds: `status` (`CLEAR`|`WARNING`|`CRITICAL`|`UNDEFINED`), `value`, `lastStatusChange`.
## HealthSnapshot
```json
{
"status": "ok",
"score": 1,
"checks": [{ "id": "cpu_user_high", "ok": true, "detail": "…" }],
"ts": 1710000000123
}
```
Aggregate: any CRITICAL → `critical`; else any WARNING → `degraded`; else `ok`.
## JobRecord
```json
{
"id": "uuid",
"name": "snapshot",
"status": "done",
"startedAt": 1710000000000,
"finishedAt": 1710000000500,
"result": { "ok": true }
}
```
## REST ↔ RPC parity
| Concept | RPC | REST |
|---------|-----|------|
| Charts | `listCharts` | `GET /api/v1/charts` |
| Data | `queryData` | `GET /api/v3/data` |
| Contexts | `listContexts` | `GET /api/v3/contexts` |
| Nodes | `getNodeInfo` | `GET /api/v3/nodes` |
| Alerts | `listAlerts` | `GET /api/v3/alerts` |
| Export | `getAllMetrics` | `GET /api/v3/allmetrics` |
| Health | `getHealth` | `GET /api/v3/health` |
docs/DESKTOP.md
+3 -3
View File
@@ -8,7 +8,7 @@ The client shell is a **Pear desktop application** built with `pear-electron` +
|------|------|
| `index.js` | Pear process entry — starts `pear-electron` Runtime + `pear-bridge` |
| `index.html` | GUI main (`pear.gui.main`) — titlebar + panels |
| `app.js` | UI logic (connect, room, presence, invites) |
| `app.js` | UI logic (connect, fleet, live charts, invites) |
| `ui/styles.css` | Layout, theme, **titlebar drag regions** |
| `client/*` | HyperDHT connection stack used by the UI |
@@ -25,8 +25,8 @@ Requires the [Pear](https://docs.pears.com) CLI installed and bootstrapped (`pea
| Field | Template default | Purpose |
|-------|------------------|---------|
| `main` | `index.html` | HTML entry |
| `width` / `height` | `1100` / `780` | Initial size |
| `minWidth` / `minHeight` | `720` / `480` | Resize floor |
| `width` / `height` | `1280` / `860` | Initial size |
| `minWidth` / `minHeight` | `900` / `560` | Resize floor |
| `resizable` | `true` | Edge/corner resize |
| `movable` | `true` | Allow OS move (with drag region) |
| `minimizable` / `maximizable` / `closable` | `true` | Window buttons |
docs/EXTENDING.md
+48 -119
View File
@@ -1,159 +1,88 @@
# Extending the template
# Extending PearData
## 1. Rebrand
## Add a chart / context
```bash
npm run rename -- notes-mesh NotesMesh
```
1. Define the chart in `shared/metrics.js` (`CHART_DEFS`).
2. Emit samples from `server/services/collector.js` (or a new collector module).
3. Store + REST/RPC pick it up automatically via `CHART_BY_ID`.
4. Document dimensions in [DATA-MODEL.md](./DATA-MODEL.md).
5. Optionally add a canvas panel in `index.html` + `app.js`.
This rewrites:
- package name / product name
- protocol id (`notes-mesh/rpc`)
- env prefix (`NOTES_MESH_`)
- invite prefix (derived, e.g. `no1.`)
- binary + systemd unit filenames
Then:
```bash
npm install
npm test
git diff # review
```
## 1b. Titlebar / window chrome
Keep these when restyling:
| Piece | Role |
|-------|------|
| `<pear-ctrl>` in `#titlebar` | Close / minimize / maximize (Pear runtime custom element) |
| `#titlebar { -webkit-app-region: drag }` | Drag the window |
| Interactive children `no-drag` | Buttons, chips, inputs stay clickable |
| `pear.gui.resizable: true` | Edge/corner resize |
| `pear.gui.minWidth` / `minHeight` | Floor size while resizing |
Do not remove `<pear-ctrl>` unless you intentionally want a frame without in-content controls (and understand platform differences). Details: [DESKTOP.md](./DESKTOP.md).
## 2. Add an RPC method
## Add an RPC method
### `shared/protocol.js`
```js
export const MethodRoles = Object.freeze({
// ...
listNotes: Roles.viewer,
createNote: Roles.operator,
listContainers: Roles.viewer,
restartCollector: Roles.admin,
})
```
`Methods` is derived automatically from `MethodRoles` keys.
### `shared/schema.js`
Validate `createNote` args (required fields, max lengths).
Validate args.
### `server/services/notes.js`
### `server/services/…` + `server/handlers/monitor.js`
Domain logic / storage (keep IO out of handlers when possible).
Register with `session.respond(...)`. Use `{ hot: true }` for high-frequency paths.
### `server/handlers/notes.js`
### Client
```js
export function registerNoteHandlers(session) {
session.respond('listNotes', async () => ({ notes: [] }))
session.respond('createNote', async (args, s) => { /* ... */ })
}
await manager.request(Methods.listContainers, {})
```
### `server/rpc/register.js`
### Docs + tests
Call `registerNoteHandlers(session)`.
Update [PROTOCOL.md](./PROTOCOL.md) and add a brittle test.
### Client / UI
## Add a REST route
Edit `server/rest/routes.js` — keep Netdata path naming when emulating Agent APIs (`/api/v3/...`).
## Add a job
Register in `server/services/jobs.js` `JOB_HANDLERS`.
```js
await manager.request(Methods.createNote, { title: '…' })
retrainAnomaly: async () => { /* … */ return { ok: true } }
```
### Tests
Operators run via `runJob` or future REST function execute.
- Unit for pure helpers + schema
- Integration for the happy path if AuthZ/wire matter
## Add a collector plugin (pattern)
## 3. Add a push channel
1. Add to `Pushes` in `shared/protocol.js`
2. Optionally map in `PushToType`
3. `session.push(Pushes.foo, payload)` or broadcast via peer registry
4. Listen in UI: `manager.on('push', …)` or `conn.on(Pushes.foo, …)`
(`connection.js` auto-registers all `Pushes` values)
## 4. Persistence
Demo room is in-memory (`server/services/room.js`). Swap for:
| Store | Good for |
|-------|----------|
| **Corestore / Hypercore** | Append-only logs, P2P replication |
| **SQLite** | Structured queries |
| **JSON files** under `PEARDATA_DATA_DIR` | Small config (peer-policy already does this) |
Keep RPC handlers thin; put IO in `services/`.
## 5. Binary streams
Peardock-class apps use chunked binary RPC for uploads. Pattern:
- methods `binaryStreamOpen` / `Chunk` / `Close` marked `hot: true` in `session.respond`
- skip heavy schema/audit on the hot path
- still enforce ACL + rate limits
Stub hooks exist via `rateLimiter.isStreamMethod`.
## 6. Multi-server fleet
`client/manager.js` already holds many connections + active selection + reconnect:
```js
await manager.connect(keyOrInviteA, { autoReconnect: true })
await manager.connect(keyOrInviteB, { autoReconnect: true })
manager.setActive(keyA)
await manager.request(Methods.ping, {})
```
server/services/collectors/
system.js # default
docker.js # Phase 3
peardock.js # bridge
```
Point UI at a peer list stored in `localStorage` or a file cache for a fuller fleet UX.
Have `pipeline.js` start each enabled collector; all emit `samples` batches into the same store.
## 7. Custom desktop UX
## Parent peer (fleet aggregator)
| Goal | Touch |
|------|-------|
| New screens | `index.html` + `app.js` + `ui/styles.css` |
| Window size | `package.json` → `pear.gui` |
| Branding in titlebar | `.app-brand` markup/CSS |
| Persist UI prefs | replace demo `localStorage` keys |
1. Parent dials child agents with `ConnectionManager`.
2. Subscribes to `push:metrics` / periodically `queryData`.
3. Ingests into local store under namespaced chart ids (`childPk.system.cpu`) or labels.
4. Exposes `/api/v3/nodes` with multiple entries + `/api/v3/data` across nodes.
Preserve drag / `pear-ctrl` behavior — [DESKTOP.md](./DESKTOP.md).
## PearDock / PearVirt hooks
## 8. Desktop packaging
- Prefer **RPC adapters** over scraping: call dock/virt methods, map to PearData contexts.
- Keep AGPL boundaries clean — depend on public APIs / your own MIT bridges.
This template ships the **Pear** GUI path (`pear run`). For Electron-forge / bare-standalone multi-arch releases, copy more complex packaging scripts from a production app (forge config, make scripts) once your protocol stabilizes.
## Desktop chrome
## 9. Checklist for a new product
Keep `<pear-ctrl>`, titlebar drag regions, and `pear.gui.resizable`. See [DESKTOP.md](./DESKTOP.md).
- [ ] `npm run rename -- …`
- [ ] Replace demo handlers + services
- [ ] Update PROTOCOL.md method table
- [ ] Update SECURITY / CONFIG if new secrets
- [ ] Tests green (`npm test`)
- [ ] README product description
- [ ] CI still green
- [ ] First release tag ([RELEASE.md](./RELEASE.md))
## Rebrand (fork)
## Related
```bash
npm run rename -- my-monitor MyMonitor
```
- [ARCHITECTURE.md](./ARCHITECTURE.md)
- [PROTOCOL.md](./PROTOCOL.md)
- [TESTING.md](./TESTING.md)
- [DESKTOP.md](./DESKTOP.md)
Review invite prefix / env prefix, then `npm test`.
docs/GETTING-STARTED.md
+63 -146
View File
@@ -1,188 +1,105 @@
# Getting started
## Prerequisites
## Requirements
| Tool | Required | Notes |
|------|----------|--------|
| **Node.js ≥ 20** | Yes | Server + tests |
| **npm** | Yes | Install deps |
| **[Pear](https://docs.pears.com) CLI** | For desktop UI | Run `pear` once to bootstrap the runtime |
| UDP / network | For real peers | HyperDHT hole-punching |
- Node.js **≥ 20**
- Pear runtime (for desktop): install via Holepunch/Pear docs
- Linux recommended for full net/disk `/proc` collectors (macOS gets CPU/RAM/load)
## Install
```bash
git clone <your-fork-or-template-url> my-app
cd my-app
cd peardata
npm install
cp .env.example .env # optional; server auto-writes seed on first boot
```
## Run the server
Copy env template if needed:
```bash
cp .env.example .env
```
## Run the agent
```bash
npm run start:server
# alias: npm run server
```
On first boot the server appends to `.env`:
Banner shows:
```
SERVER_SEED=<64 hex secret>
SERVER_PUBLIC_KEY=<64 hex public>
- **publicKey** — dial this from the desktop (viewer)
- **rest** — e.g. `http://127.0.0.1:19999/api/v3/info`
- **admin** — use `SERVER_SEED` from `.env` or mint `pd1.` invites
Agent persists identity in `.env` on first boot.
### REST smoke tests
```bash
curl -s http://127.0.0.1:19999/api/v3/info
curl -s http://127.0.0.1:19999/api/v1/charts | head
curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-60&points=60'
curl -s 'http://127.0.0.1:19999/api/v3/allmetrics?format=prometheus' | head
```
**Treat `SERVER_SEED` like a root password.** Anyone with it can mint admin proofs and capabilities.
Disable REST: `PEARDATA_REST=0`.
Banner output shows the public key clients dial, for example:
## Mint an invite
```
Client → dial <SERVER_PUBLIC_KEY>
```bash
npm run mint-invite -- operator
# → pd1.…
```
Leave this process running while clients connect.
Share the invite string. Recipients paste it into the desktop connect box.
## Run the desktop UI
## Run the desktop
```bash
npm start
# or: npm run dev
# or: pear run -d .
# pear run -d .
```
### Desktop window chrome
1. Paste agent public key or `pd1.` invite
2. Optional: paste `SERVER_SEED` for admin
3. Connect → live overview + charts
4. Admin: **Mint invite**
The Pear UI uses a custom titlebar:
| Piece | Behavior |
|-------|----------|
| `<pear-ctrl>` | Platform window controls (macOS traffic lights / Windows & Linux min·max·close) |
| `#titlebar` | `-webkit-app-region: drag` — drag to move the window |
| Interactive children | `no-drag` so buttons and chips stay clickable |
| `pear.gui.resizable` | Edge/corner resize (`true` by default) |
| `minWidth` / `minHeight` | 720 × 480 floor |
Full details: [DESKTOP.md](./DESKTOP.md).
## Connect as admin (dev)
1. Start the Pear UI: `npm start`
2. Paste `SERVER_PUBLIC_KEY` into **Server / invite**
3. Paste `SERVER_SEED` into **Admin seed**
4. Optionally set a **Display name**
5. Click **Connect** → role badge should show `admin`
The seed never goes over the wire as plaintext — the client sends an HMAC **admin proof**.
## Connect via invite
## systemd (Linux)
```bash
# Persistent operator invite (default)
npm run mint-invite -- operator
# 7-day operator invite (ttl in ms)
npm run mint-invite -- operator 604800000
# Admin invite
npm run mint-invite -- admin
```
Stdout prints a `pd1.…` string. Paste it into the UI connect field (no seed needed).
You can also mint from a connected admin session with **Mint invite** in the UI.
## Viewer-only
Paste only the public key. You can:
- `listMessages`, `getPresence`, `getServerInfo`, `setDisplayName`, `ping`
You cannot:
- `postMessage` (needs `operator+`)
- `clearMessages`, `mintInvite`, `listPeers`, `revokePeer` (needs `admin`)
Unless you raise `PEARDATA_DEFAULT_ROLE` (not recommended for multi-user hosts).
## Quick verification
```bash
npm test
SKIP_INTEGRATION=1 npm test # unit only
# With server running:
export SERVER_PUBLIC_KEY=… # from .env
export SERVER_SEED=… # optional but enables admin dial
npm run healthcheck
npm run soak # optional load exercise
```
## Environment knobs (summary)
| Variable | Purpose |
|----------|---------|
| `SERVER_SEED` / `SERVER_PUBLIC_KEY` | Server identity |
| `PEARDATA_DEFAULT_ROLE` | Baseline role (`viewer` default) |
| `PEARDATA_ADMIN_KEYS` | Peer pubs always admin |
| `PEARDATA_INSECURE_OPEN_ADMIN` | Dev only — all peers admin |
| `PEARDATA_ALLOWLIST` | If set, only listed / registered peers |
| `PEARDATA_DATA_DIR` | Peer policy + audit log directory |
| `PEARDATA_RATE_LIMIT_RPM` | Per-peer RPC budget |
| `PEARDATA_MAX_MESSAGES` | Demo room history cap |
| `PEARDATA_MAX_RECONNECT` | Client reconnect tries |
| `LOG_LEVEL` / `LOG_JSON` | Logging |
Full table: [CONFIGURATION.md](./CONFIGURATION.md).
## Rebrand for a new product
```bash
npm run rename -- my-app MyApp
npm install
npm test
```
Rewrites package name, protocol id, env prefixes, invite prefix (`pd1.` → derived), product strings, and renames the server binary / systemd unit. Review `git diff` after.
## systemd
```bash
# Install tree to /opt/peardata (example)
sudo mkdir -p /opt/peardata
sudo rsync -a --exclude node_modules --exclude .git ./ /opt/peardata/
cd /opt/peardata && sudo npm install --omit=dev
sudo rsync -a ./ /opt/peardata/ --exclude node_modules
cd /opt/peardata && sudo npm ci --omit=dev
sudo cp deploy/peardata.service /etc/systemd/system/
# Edit WorkingDirectory, EnvironmentFile, ReadWritePaths if paths differ
sudo systemctl daemon-reload
sudo systemctl enable --now peardata
sudo journalctl -u peardata -f
journalctl -u peardata -f
```
The unit sets `NoNewPrivileges`, `ProtectSystem=strict`, and writable paths for `data/` + `.env`.
Ensure `ReadWritePaths` can write `/opt/peardata/data` and `.env`.
## Roles cheat sheet
| Goal | Action |
|------|--------|
| Read-only monitoring | Dial pubkey only |
| Ack alerts / run jobs | `pd1.` operator invite |
| Mint invites / revoke | Admin seed or admin invite |
## Troubleshooting
| Symptom | What to check |
|---------|----------------|
| `Connection timeout` | Server running? Correct 64-hex key? Firewall / UDP? |
| `PERMISSION_DENIED` on send | Role is viewer — use invite or admin seed |
| `Rate limit exceeded` | Raise `PEARDATA_RATE_LIMIT_RPM` or slow clients |
| Window won’t drag | Titlebar drag CSS; don’t cover bar with full-screen `no-drag` overlay |
| Window won’t resize | `pear.gui.resizable` must be true; try edges not just corners |
| No `<pear-ctrl>` buttons | Running under Pear (`pear run`)? Element is runtime-provided |
| Integration test fails in CI | Set `SKIP_INTEGRATION=1` or allow UDP |
| Seed regenerated every boot | `.env` not writable / wrong cwd |
| Symptom | Check |
|---------|-------|
| REST connection refused | Agent running? `PEARDATA_REST` not `0`? Port free? |
| Desktop can’t dial | Firewall / DHT; same machine should work; wait for punch |
| Empty net/disk charts | Non-Linux host — expected until platform collectors land |
| `PERMISSION_DENIED` | Need higher role invite or admin seed |
| Integration tests hang | `SKIP_INTEGRATION=1 npm test` on restricted networks |
## Next steps
## Next reading
| Doc | When |
|-----|------|
| [DESKTOP.md](./DESKTOP.md) | Titlebar, pear-ctrl, packaging |
| [ARCHITECTURE.md](./ARCHITECTURE.md) | Stack & session pipeline |
| [PROTOCOL.md](./PROTOCOL.md) | Methods, pushes, versioning |
| [SECURITY.md](./SECURITY.md) | Production hardening |
| [CONFIGURATION.md](./CONFIGURATION.md) | Full env reference |
| [EXTENDING.md](./EXTENDING.md) | Replace the demo room |
| [TESTING.md](./TESTING.md) | Tests & soak |
| [CI.md](./CI.md) / [RELEASE.md](./RELEASE.md) | Pipelines & shipping |
- [REST-API.md](./REST-API.md)
- [ARCHITECTURE.md](./ARCHITECTURE.md)
- [ROADMAP.md](./ROADMAP.md)
- [CONFIGURATION.md](./CONFIGURATION.md)
docs/PROTOCOL.md
+104 -131
View File
@@ -1,158 +1,131 @@
# Protocol
## Constants
Wire contract for PearData P2P RPC. Source of truth: `shared/protocol.js`, `shared/schema.js`, `shared/metrics.js`.
## Identity
| Constant | Value |
|----------|--------|
| `PROTOCOL` | `peardata/rpc` |
| `PROTOCOL_VERSION` | `1` |
| `APP_NAME` | `peardata` |
| `APP_VERSION` | `0.1.0` (keep in sync with package where useful) |
| Encoding | compact-encoding JSON (`shared/encodings.js`) |
| Schema | lightweight validators (`shared/schema.js`) `SCHEMA_VERSION=1` |
|----------|-------|
| Protocol id | `peardata/rpc` |
| Protocol version | `1` |
| Invite prefix | `pd1.` |
| Default role | `viewer` |
Bump `PROTOCOL_VERSION` on breaking request/response shapes. Additive methods may land without a bump if clients ignore unknown methods.
After `npm run rename`, `PROTOCOL` becomes `<slug>/rpc` and invite prefix is regenerated.
## Transport
1. Client opens HyperDHT secret stream to server public key (Noise, mutual key auth).
2. `ProtomuxRPC` is attached with `protocol: PROTOCOL` and shared encodings.
3. Client calls `handshake` before other RPCs (connection helper does this automatically).
4. Server may `push` events on named channels.
Bump `PROTOCOL_VERSION` on breaking argument/result shapes.
## Roles
| Role | Intent |
|------|--------|
| `viewer` | Read-only |
| `operator` | Mutate domain data |
| `admin` | Invite mint, clear, revoke, config |
| Role | Rank | Typical access |
|------|------|----------------|
| `viewer` | 1 | Read metrics, subscribe, list alerts |
| `operator` | 2 | Ack/silence alerts, run jobs, set alert config |
| `admin` | 3 | Mint invites, revoke peers, export snapshot |
Hierarchy: `admin > operator > viewer` (`roleAllows`).
Unknown methods default to **admin** required in `assertAllowed` if missing from `MethodRoles` — always register new methods.
Auth modes at handshake: public key (viewer), capability token / `pd1.` invite, admin seed proof, allowlist.
## Methods
| Method | Min role | Request args | Response (summary) |
|--------|----------|--------------|--------------------|
| `handshake` | viewer | `clientName`, `clientVersion`, optional `capability`, `adminProof` | role, auth, versions, features |
| `ping` | viewer | `{}` | `{ ok, pong, peerId }` |
| `getServerInfo` | viewer | `{}` | app, versions, host, peer count |
| `getAuthStatus` | viewer | `{}` | peerId, role, authMode, displayName |
| `listMessages` | viewer | `{ limit? }` | `{ messages: [...] }` |
| `getPresence` | viewer | `{}` | `{ peers: [...] }` |
| `postMessage` | operator | `{ text }` (1–2000 chars) | created message |
| `setDisplayName` | viewer | `{ name }` (1–40 chars) | updated label |
| `clearMessages` | admin | `{}` | success + system push |
| `mintInvite` | admin | `{ role?, ttlMs?, peerId?, alias? }` | `invite` (`pd1.…`), jti, exp |
| `listPeers` | admin | `{}` | live + policy peers |
| `revokePeer` | admin | `{ peerId }` (64 hex) | success; target dropped |
### Session
### Handshake request
| Method | Role | Notes |
|--------|------|-------|
| `handshake` | viewer | Negotiate version + elevate role |
| `ping` | viewer | Hot |
| `getServerInfo` | viewer | Agent metadata |
| `getAuthStatus` | viewer | Peer role / auth mode |
| `setDisplayName` | viewer | Label for audit / UI |
```json
{
"clientName": "peardata",
"clientVersion": "0.1.0",
"capability": "<optional HMAC token>",
"adminProof": { "nonce": "<hex>", "mac": "<hex>" }
}
```
### Node
### Handshake response
| Method | Role | Result |
|--------|------|--------|
| `getNodeInfo` | viewer | Hostname, CPUs, charts, sample interval |
| `getHealth` | viewer | `{ status, score, checks }` |
```json
{
"success": true,
"protocol": "peardata/rpc",
"protocolVersion": 1,
"schemaVersion": 1,
"role": "operator",
"peerId": "<64 hex>",
"serverTime": 0,
"auth": { "mode": "capability", "role": "operator" },
"features": { "hmacAuth": true, "invites": true, "room": true }
}
```
### Metrics discovery & query
### Auth modes (`auth.mode` / `session.authMode`)
| Method | Role | Args | Result |
|--------|------|------|--------|
| `listContexts` | viewer | — | Context catalog |
| `getContext` | viewer | `{ id }` | Charts in context |
| `listCharts` | viewer | — | Netdata-ish chart map |
| `getChart` | viewer | `{ id }` | Chart summary |
| `queryData` | viewer | `{ chart, after, before, points, group, tier }` | Time series |
| `getAllMetrics` | viewer | `{ format: json\|prometheus\|shell }` | Latest export |
| Mode | How obtained |
|------|----------------|
| `viewer` | Default after connect with no grant |
| `capability` | Valid capability / invite |
| `seed` | Valid admin proof from `SERVER_SEED` |
| `allowlist` / registered | Elevated via policy / admin keys (implementation in ACL + policy) |
`after` / `before`: absolute unix seconds, or relative (negative = relative to `before`/`now`), Netdata-style.
Exact labels depend on handshake path; UI shows `authMode` from `getAuthStatus`.
### Live subscriptions
## Pushes (server → client events)
| Method | Role | Args |
|--------|------|------|
| `subscribeMetrics` | viewer | `{ charts: string[]\|['*'], intervalMs }` |
| `unsubscribeMetrics` | viewer | — |
| `subscribeAnomalies` | viewer | — |
| `unsubscribeAnomalies` | viewer | — |
| Push | Payload |
### Anomalies & alerts
| Method | Role | Notes |
|--------|------|-------|
| `listAnomalies` | viewer | Recent events |
| `listAlerts` / `getAlert` | viewer | State + config |
| `setAlertConfig` | operator | Upsert threshold |
| `ackAlert` | operator | Clear until next breach |
| `silenceAlert` | operator | Disable temporarily |
### Jobs
| Method | Role | Known jobs |
|--------|------|------------|
| `listJobs` | viewer | — |
| `runJob` | operator | `collectOnce`, `snapshot`, `gcBuffers` |
| `cancelJob` | operator | By job id |
### Admin
| Method | Role |
|--------|------|
| `mintInvite` | admin |
| `listPeers` | admin |
| `revokePeer` | admin |
| `exportSnapshot` | admin |
## Pushes (server → client)
| Event | Payload |
|-------|---------|
| `push:metrics` | `{ samples: MetricSample[] }` |
| `push:anomaly` | `AnomalyEvent` |
| `push:alert` | Alert transition |
| `push:health` | `HealthSnapshot` |
| `push:job` | `JobRecord` |
| `push:system` | Generic notices |
## Errors
Handlers throw `Error` with `.code`:
| Code | Meaning |
|------|---------|
| `push:message` | `{ id, peerId, displayName, text, ts }` |
| `push:presence` | `{ peers: [...] }` |
| `push:system` | `{ type, ... }` e.g. `{ type: "cleared" }` |
Registered via `rpc.event` / `session.push`. Client `connection.js` binds all `Pushes` values.
## Invites
Envelope: `pd1.` + base64url(JSON):
```json
{
"v": 1,
"publicKeyHex": "<server>",
"capability": "<token>",
"role": "operator",
"jti": "...",
"expiresAt": null
}
```
Capability token: `base64url(payload).base64url(HMAC-SHA256)`.
Payload fields (canonical order for MAC): `v`, `role`, `peerId`, `exp`, `jti`, `iat`.
### Connection input classification
`classifyConnectionInput(string)` accepts:
| Input | Kind |
|-------|------|
| 64 hex chars | `publicKey` |
| `pd1.…` | `invite` (extracts key + capability) |
| other | error |
## Error codes
| Code | When |
|------|------|
| `RATE_LIMIT_EXCEEDED` | Peer over RPM budget |
| `PERMISSION_DENIED` | Role too low for method |
| `INVALID_ARGS` | Schema validation failed |
| `CONNECTION_TIMEOUT` | Client dial timeout |
| `RPC_ERROR` | Generic client-normalized failure |
| `UNKNOWN_ERROR` | Unclassified server handler error |
Clients should read `error.code` when present (`client/errors.js` preserves codes).
| `PERMISSION_DENIED` | Role too low |
| `RATE_LIMIT_EXCEEDED` | RPM exceeded |
| `INVALID_ARGS` | Schema failure |
| `NOT_CONNECTED` | Client-side |
| `CAPABILITY_*` | Invite/token issues |
## Versioning policy
1. Document every method in this file.
2. Add `MethodRoles` entry before implementing handlers.
3. Add `validateMethodArgs` case for mutating methods.
4. Add brittle tests for pure helpers; integration test for critical paths.
5. Bump `PROTOCOL_VERSION` when existing response shapes break.
6. Bump `SCHEMA_VERSION` when validation semantics change meaningfully.
1. Additive methods/fields: no version bump if old clients ignore unknowns.
2. Rename/remove/change meaning: bump `PROTOCOL_VERSION`; reject or compat-negotiate in handshake.
3. REST API versions (`v1`/`v2`/`v3`) are independent of RPC version but share the store.
## Related
## Example session
- [ARCHITECTURE.md](./ARCHITECTURE.md)
- [SECURITY.md](./SECURITY.md)
- [EXTENDING.md](./EXTENDING.md)
- [TESTING.md](./TESTING.md)
```text
client → handshake { clientName, clientVersion, capability? }
server → { role, protocolVersion, auth }
client → subscribeMetrics { charts: ['*'], intervalMs: 1000 }
server → push:metrics { samples: [...] } # ~1 Hz
client → queryData { chart: 'system.cpu', after: -300, points: 300 }
```
docs/README.md
+14 -11
View File
@@ -2,17 +2,20 @@
| Doc | Audience | Contents |
|-----|----------|----------|
| [GETTING-STARTED.md](./GETTING-STARTED.md) | New operators | Install, run, connect, systemd, troubleshooting |
| [DESKTOP.md](./DESKTOP.md) | UI developers | Pear shell, `pear-ctrl`, drag/resize, identity |
| [ARCHITECTURE.md](./ARCHITECTURE.md) | Engineers | Planes, boot, middleware, module map |
| [PROTOCOL.md](./PROTOCOL.md) | Protocol owners | Methods, pushes, errors, versioning |
| [SECURITY.md](./SECURITY.md) | Operators / security | Trust model, checklist, crypto, incidents |
| [GETTING-STARTED.md](./GETTING-STARTED.md) | Operators | Install, run agent, REST, desktop, systemd |
| [ROADMAP.md](./ROADMAP.md) | Everyone | Phased MVP → advanced Netdata-class features |
| [ARCHITECTURE.md](./ARCHITECTURE.md) | Engineers | Planes, PearDock mapping, module map |
| [PROTOCOL.md](./PROTOCOL.md) | Protocol owners | RPC methods, pushes, versioning |
| [DATA-MODEL.md](./DATA-MODEL.md) | Engineers | Metrics, anomalies, alerts, jobs |
| [REST-API.md](./REST-API.md) | Integrators | Netdata-style `/api/v1\|v2\|v3` |
| [TECH-CHOICES.md](./TECH-CHOICES.md) | Engineers | Collector, charts, libraries |
| [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | Engineers | HyperDB / Corestore / Autobase sync design |
| [DESKTOP.md](./DESKTOP.md) | UI developers | Pear shell, titlebar, identity |
| [SECURITY.md](./SECURITY.md) | Operators / security | Trust model, secrets, hardening |
| [CONFIGURATION.md](./CONFIGURATION.md) | Operators | Full env vars + npm scripts |
| [TESTING.md](./TESTING.md) | Contributors | brittle suite, soak, manual QA |
| [TESTING.md](./TESTING.md) | Contributors | brittle, soak, manual QA |
| [CI.md](./CI.md) | Maintainers | GitHub/Gitea pipelines |
| [RELEASE.md](./RELEASE.md) | Maintainers | Version, tag, tarball, rollback |
| [EXTENDING.md](./EXTENDING.md) | Product builders | Rebrand, new RPCs, persistence, fleet |
| [RELEASE.md](./RELEASE.md) | Maintainers | Version, tag, tarball |
| [EXTENDING.md](./EXTENDING.md) | Product builders | New charts, collectors, parents |
Start here if you are new: **[GETTING-STARTED.md](./GETTING-STARTED.md)**.
Root overview: **[../README.md](../README.md)**.
Start here: **[GETTING-STARTED.md](./GETTING-STARTED.md)** · Product overview: **[../README.md](../README.md)**.
docs/REST-API.md
+147
View File
@@ -0,0 +1,147 @@
# REST API (Netdata-compatible)
PearMonitor agents expose an optional HTTP API modeled on **Netdata Agent** endpoints (`/api/v1`, `/api/v2`, `/api/v3`).
Default bind: `http://127.0.0.1:19999` (Netdata’s classic port).
Disable: `PEARDATA_REST=0`.
Bind all interfaces (careful): `PEARDATA_REST_HOST=0.0.0.0`.
This is a **compatibility layer**, not a byte-for-byte Netdata clone. Core query/metadata paths are implemented for scripts, Grafana, and Prometheus scrapers.
## Quick examples
```bash
# Agent info
curl -s http://127.0.0.1:19999/api/v3/info | jq
# Chart catalog (v1 style)
curl -s http://127.0.0.1:19999/api/v1/charts | jq '.charts | keys'
# Last 60s of CPU (v3)
curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-60&points=60' | jq
# Contexts
curl -s http://127.0.0.1:19999/api/v3/contexts | jq
# Prometheus export
curl -s 'http://127.0.0.1:19999/api/v3/allmetrics?format=prometheus'
# Health
curl -s http://127.0.0.1:19999/api/v3/health | jq
```
## Endpoint matrix
### Info & identity
| Method | Path | Notes |
|--------|------|-------|
| GET | `/` or `/api` | Service index + P2P pubkey |
| GET | `/api/v1/info` | Agent info |
| GET | `/api/v2/info` | same |
| GET | `/api/v3/info` | **preferred** |
| GET | `/api/v3/versions` | Agent / protocol / API versions |
| GET | `/api/v3/me` | Anonymous REST identity note |
| GET | `/api/v3/settings` | Runtime knobs |
| GET | `/api/v3/config` | alias of settings |
| GET | `/health`, `/api/v1/health`, `/api/v3/health` | Aggregate health |
### Nodes
| Method | Path |
|--------|------|
| GET | `/api/v2/nodes` |
| GET | `/api/v3/nodes` |
| GET | `/api/v3/node_instances` |
| GET | `/api/v3/stream_path` |
Single-agent MVP returns one node (this host). Parent/fleet aggregation is roadmap.
### Contexts & charts
| Method | Path | Notes |
|--------|------|-------|
| GET | `/api/v3/contexts` | Context map |
| GET | `/api/v2/contexts` | same |
| GET | `/api/v3/context?context=` | One context |
| GET | `/api/v1/charts` | Full chart summary (legacy but useful) |
| GET | `/api/v1/chart?chart=` | One chart |
### Data queries
| Method | Path | Query params |
|--------|------|--------------|
| GET | `/api/v3/data` | `chart` or `context`, `after`, `before`, `points`, `group`, `tier`, `format` |
| GET | `/api/v2/data` | same |
| GET | `/api/v1/data` | same (legacy) |
**Params (Netdata-style)**
| Param | Default | Description |
|-------|---------|-------------|
| `chart` / `context` | required | Chart id or context id |
| `after` | `-60` | Absolute unix sec, or relative (negative) |
| `before` | `0` (now) | Absolute or relative |
| `points` | `60` | Max points returned (downsampled) |
| `group` | `average` | `average` \| `min` \| `max` \| `sum` |
| `tier` | `0` | `0` = 1s buffer, `1` = downsampled |
| `format` | `json` | `json` \| `csv` \| `array` |
### Search & weights
| Method | Path | Notes |
|--------|------|-------|
| GET | `/api/v3/q?q=` | Full-text over chart ids/titles |
| GET | `/api/v3/weights` | MVP: health-derived scores |
### Alerts
| Method | Path |
|--------|------|
| GET | `/api/v3/alerts` |
| GET | `/api/v2/alerts` |
| GET | `/api/v1/alarms` |
| GET | `/api/v3/alert_transitions` |
| GET | `/api/v3/alert_config` |
| GET | `/api/v3/variable` |
### Export & badges
| Method | Path | Params |
|--------|------|--------|
| GET | `/api/v3/allmetrics` | `format=json\|prometheus\|shell` |
| GET | `/api/v1/allmetrics` | same |
| GET | `/api/v3/badge.svg` | `chart`, `dimensions`, `label` |
### Functions (stub)
| Method | Path | Notes |
|--------|------|-------|
| GET | `/api/v3/functions` | Lists job names; execution remains P2P `runJob` for auth |
## Auth model (REST)
- **Default:** localhost-only, no bearer required (like a typical Netdata agent bind).
- **P2P remains the secure remote path** (Noise + roles).
- If you bind `0.0.0.0`, put REST behind a firewall, reverse proxy, or Holesail tunnel — do not expose raw metrics to the internet.
- Future: optional bearer gate (`/api/v3/bearer_protection` parity).
## CORS
`Access-Control-Allow-Origin` defaults to `*` (override with `PEARDATA_REST_CORS`).
## Compatibility notes
| Netdata | PearData MVP |
|---------|--------------|
| Full ML weights / metric correlations | Simplified health weights |
| Multi-node parent streaming | Single node; parent planned |
| Cloud POST `/api/v3/spaces/.../data` | Not implemented (agent GET style only) |
| Every chart Netdata ships | Core system charts (expanding) |
| Functions execute via HTTP | Listed; run via P2P jobs |
## Implementation
- Router: `server/rest/routes.js`
- Server: `server/rest/http-server.js`
- Exporters: `server/rest/formatters.js`
docs/ROADMAP.md
+114
View File
@@ -0,0 +1,114 @@
# PearData Roadmap
Phased plan from MVP agent → Netdata-class fleet observability on pure P2P.
## Guiding principles
1. **Instant value** — connect a pubkey, see live charts in seconds.
2. **Agent efficiency** — stay in Netdata’s ballpark for CPU/RAM overhead.
3. **PearDock patterns** — HyperDHT identity, protomux-rpc, roles, `pd1.` invites.
4. **Dual access** — P2P desktop + Netdata-style REST (`/api/v1|v2|v3`).
5. **Ecosystem glue** — ready for PearDock / PearVirt / HoneyPeer / BareOS later.
---
## Phase 0 — Foundation ✅ (this repo)
| Item | Status |
|------|--------|
| Copy pear-app-template → PearData rebrand (`pd1.`, `PEARDATA_*`, `peardata/rpc`) | Done |
| Shared protocol + schema + metrics catalog | Done |
| Agent collector (CPU/RAM/load/net/disk/processes) @ ~1s | Done |
| In-memory tiered store (1s + downsample) | Done |
| Threshold anomaly engine + health | Done |
| P2P RPC surface (query, subscribe, alerts, jobs, ACL) | Done |
| Netdata-compatible REST v1/v2/v3 (local :19999) | Done |
| Pear desktop fleet overview + live canvas charts | Done |
| Docs: architecture, protocol, data model, REST, roadmap, tech choices | Done |
| systemd unit + CI skeletons | Done |
**MVP acceptance criteria**
- [x] Agent exposes metrics over P2P by public key
- [x] Desktop connects to one+ agents and shows live CPU/RAM/net/disk
- [x] REST `/api/v3/data?chart=system.cpu&after=-60` returns series
- [x] Viewer vs admin (pubkey / seed / `pd1.` invite)
- [x] Simple threshold anomalies pushed to clients
---
## Phase 1 — Hardening & UX (next)
- Persistent client peer bookmarks + aliases in desktop
- Multi-peer compare mode (overlay 2–4 nodes on one chart)
- Reconnection / offline banners with last-known samples
- Agent process title / `peardata-agent` binary naming polish
- Expand collectors: per-core CPU, per-iface net, mount disk space
- Alert silence TTL auto-reenable
- brittle tests for collector, store query, REST routes
- One-line install script (`curl | bash`) for Linux agents
---
## Phase 2 — Storage & history (HyperDB)
See **[STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md)** for the full design (from Holepunch `hyperdb` / workshop / Autobase patterns).
- HyperDB (bee + Corestore) for metadata, alerts, peer-links, **warm** downsampled points
- Keep memory ring for hot 1s path; do **not** tx every sample into HyperDB
- Hyperswarm `store.replicate` for linked-node / desktop seed sync
- Configurable retention (hours@1s memory, days@1m HyperDB, weeks@1h)
- Historical query: memory miss → HyperDB range
- Optional later: Autobase multi-writer parents; Rocks engine for local-only speed
- Export snapshot job → JSON / Prometheus remote write (optional)
---
## Phase 3 — Discovery & apps
- Docker / container auto-discovery (cgroup + Docker API)
- Common service collectors (nginx, postgres, redis) as plugins
- PearDock integration: container metrics from dock peers
- PearVirt / BareOS node metric adapters
- Holesail optional expose of REST UI per agent
---
## Phase 4 — Smarter anomalies & fleet
- Anomaly scoring + chart highlighting in UI
- Lightweight ML job (`runJob` retrain) — start with streaming z-score / k-means
- Fleet-wide composite views and correlation (`/api/v3/weights` depth)
- Parent peer aggregation (P2P “parent” without central SaaS)
- Push notifications (desktop + optional webhook)
---
## Phase 5 — Product polish
- Signed release artifacts (agent + Pear desktop)
- Role templates (viewer / SRE operator / admin)
- Plugin SDK documentation
- Grafana datasource (REST) cookbook
- HoneyPeer presence for agent directory (opt-in)
---
## Non-goals (for now)
- Replacing Netdata Cloud SaaS multi-tenant product
- Full byte-identical Netdata internal DB format
- Shipping a browser-only public dashboard without auth by default (REST stays localhost unless explicitly bound)
---
## Milestone checklist (operators)
| Milestone | You can… |
|-----------|----------|
| M0 | `npm run start:server` + `curl localhost:19999/api/v3/info` |
| M1 | Pear UI live charts from agent pubkey |
| M2 | Mint `pd1.` operator invite; revoke peer |
| M3 | Historical scrub 1h@1s via REST + RPC |
| M4 | Container charts from Docker hosts |
| M5 | Parent peer rolling up a homelab fleet |
docs/SECURITY.md
+8 -1
View File
@@ -28,13 +28,20 @@
| Identity file mode | `0600` |
| Data directory | local `./data` (not committed) |
## REST API exposure
- Default bind is **localhost only** (`127.0.0.1:19999`).
- REST is intentionally open on that bind (Netdata-agent style) — **do not** set `PEARDATA_REST_HOST=0.0.0.0` without a firewall, reverse proxy, or Holesail tunnel.
- Prefer **P2P + roles** for remote multi-operator access; use REST for local scrapers/Grafana.
## Production checklist
- [ ] Never set `PEARDATA_INSECURE_OPEN_ADMIN` outside local demos
- [ ] Keep `PEARDATA_DEFAULT_ROLE=viewer`
- [ ] Prefer invites over sharing `SERVER_SEED`
- [ ] Prefer `pd1.` invites over sharing `SERVER_SEED`
- [ ] Use short `ttlMs` for high-privilege invites when practical
- [ ] Set `PEARDATA_ALLOWLIST` if only known operators should dial
- [ ] Keep REST on localhost unless explicitly secured
- [ ] Back up `SERVER_SEED` offline; rotate by redeploying a new keypair (clients must re-dial)
- [ ] Persist `data/` with mode `0700`; `audit.log` may contain peer ids
- [ ] Run under systemd with `ProtectSystem` / `NoNewPrivileges` (see `deploy/`)
docs/STORAGE-HYPERDB.md
+290
View File
@@ -0,0 +1,290 @@
# HyperDB storage & linked-node sync
How PearData should adopt Holepunch’s **HyperDB + Corestore + Hyperswarm (+ Autobase)** stack for durable storage and P2P sync between linked agents — based on patterns in local clones under `holepunchto_repos` (`hyperdb`, `hyperdb-workshop`, `hyperdb-autobase-workshop`, `corestore`, `hyperswarm`, `autobee`, `pear-hyperdb`).
## Why HyperDB (not “just SQLite”)
| Need | HyperDB fit |
|------|-------------|
| Typed collections + indexes | Hyperschema + `@ns/collection` + secondary indexes |
| Local high-perf | `HyperDB.rocks(path, def)` |
| P2P replicate | `HyperDB.bee(hypercore, def, { autoUpdate })` over Corestore |
| Multi-writer HA parents | Autobase whose **view** is HyperDB (`extension: false`) |
| Same query API local + remote | Workshops prove one `Registry` class works for both |
PearData’s current `server/services/store.js` is an **in-memory ring**. HyperDB replaces durability + sync; the in-memory tier stays as the **hot 1s path**.
## Stack mapping (from Holepunch repos)
```mermaid
flowchart TB
subgraph Agent
COL[collector 1s] --> HOT[Memory tier0 ring]
COL --> DS[Downsample]
DS --> HDB[(HyperDB bee/rocks)]
POL[peer-policy / alerts / links] --> HDB
HOT --> RPC[protomux-rpc + REST]
HDB --> RPC
end
subgraph Sync
CS[Corestore] --> HDB
SW[Hyperswarm] -->|store.replicate| CS
AB[Autobase optional] -->|view| HDB
end
CHILD[Linked child agent] -.->|discoveryKey| SW
PARENT[Parent / peer agent] -.-> SW
DESK[Pear desktop cache] -.-> SW
```
| Component | Repo pattern | PearData use |
|-----------|--------------|--------------|
| **Hyperschema + hyperdb/builder** | `hyperdb-workshop/build.js` | `spec/` codegen for collections |
| **HyperDB.bee** | workshop `Registry` | Replicable agent DB |
| **HyperDB.rocks** | `hyperdb` README | Optional local-only fast index |
| **Corestore** | workshop `bin.js` | Named cores: `metrics-meta`, `alerts`, … |
| **Hyperswarm** | `swarm.join(discoveryKey)` + `store.replicate(conn)` | Link nodes / seed DB |
| **protomux-rpc** | already in PearData | Control plane (unchanged) |
| **Autobase + hyperdispatch** | `hyperdb-autobase-workshop` | Multi-writer **parent** / HA registry |
| **autobee** | experimental multiwriter bee | Alternative later; prefer Autobase+HyperDB view for now |
| **pear-hyperdb** | Pear-shaped Model wrapper | Optional UX for desktop-local rocks |
## Critical design rule: don’t put 1s samples in HyperDB txs
HyperDB is an **indexable document DB** (put/get/find + flush). Writing every chart every second as HyperDB transactions will:
- Amplify Rocks/Bee write cost
- Create huge replication chatter
- Fight Netdata-class overhead goals
**Split planes:**
| Plane | Storage | Sync |
|-------|---------|------|
| **Hot live (≤1h @ 1s)** | Memory ring (current) | P2P `push:metrics` (current) |
| **Warm history (downsampled)** | Hypercore append **or** HyperDB rows keyed `(chart, tsBucket)` | Corestore replicate |
| **Metadata** (peers, alerts, labels, jobs, ACL cache) | **HyperDB** | Corestore replicate |
| **Fleet / parent consensus** | Autobase → HyperDB view | Swarm on autobase discoveryKey |
## Proposed HyperDB schema (`@peardata/*`)
Modeled after workshop `build.js` namespaces.
### Collections
```text
@peardata/node
key: nodeId (string / pubkey hex)
fields: hostname, platform, arch, cpus, agentVersion, labels{}, updatedAt
@peardata/peer-link
key: [localNodeId, remotePublicKey]
fields: role, alias, discoveryKey?, linkedAt, lastSeen, syncMode (push|pull|both)
@peardata/alert-config
key: id
fields: chart, dimension, warn, crit, comparator, enabled, info
@peardata/alert-event
key: [id, ts] # or ulid
fields: severity, value, threshold, message, cleared
@peardata/metric-point # WARM tier only (e.g. 1m buckets)
key: [chart, ts]
fields: context, values{} (map), tier
@peardata/job
key: id
fields: name, status, startedAt, finishedAt, result?
```
### Indexes
```text
@peardata/node-by-hostname → node.hostname
@peardata/peer-link-by-remote → peer-link.remotePublicKey
@peardata/alert-event-by-chart → alert-event.chart + ts
@peardata/metric-point-by-context → metric-point.context + ts
```
Rebuild with:
```bash
node scripts/build-db.js # Hyperschema + HyperDB.toDisk → spec/
```
## Agent integration shape
Follow `hyperdb-workshop` / `pear-hyperdb` Model pattern:
```text
server/
db/
build.js # schema codegen
spec/hyperschema/
spec/hyperdb/
model.js # PearDataModel: putNode, linkPeer, queryWarm, …
replicate.js # Hyperswarm join + store.replicate
services/
store.js # HOT memory (keep)
store-hyperdb.js # WARM + metadata facade used by queryData/REST
```
### Boot (single-writer agent — Phase 2)
```js
const store = new Corestore(dataDir + '/corestore')
const swarm = new Hyperswarm({ keyPair: await store.createKeyPair('swarm') })
swarm.on('connection', (conn) => store.replicate(conn))
const metaCore = store.get({ name: 'peardata-meta' })
const db = HyperDB.bee(metaCore, spec, { autoUpdate: true })
// announce for linked peers / desktop seeders
swarm.join(metaCore.discoveryKey, { server: true, client: true })
```
Collector path:
1. Ingest → memory tier0 (unchanged)
2. Every N samples → downsample → `tx.insert('@peardata/metric-point', …); tx.flush()`
3. Alert transitions → `alert-event` collection
4. `queryData` / REST: memory first, then HyperDB range scan for older windows
### Auth note
HyperDB replication shares **capability to read the core**, not PearData RPC roles. Keep:
- **Noise + MethodRoles** for mutating RPC (`setAlertConfig`, `runJob`)
- Replication topic optionally gated (only invite-linked peers get discoveryKey / capability)
- Do **not** announce writable cores to the public swarm without encryption / allowlist
## Linked nodes: sync modes
PearData “links” are first-class `@peardata/peer-link` rows + swarm topics.
| Mode | Behavior |
|------|----------|
| **Pull** | Local agent opens remote DB by key (`HyperDB.bee(store.get({ key }), spec, { writable: false, autoUpdate: true })`) and replicates |
| **Push** | Remote peers allowed to replicate our meta/warm cores (seed) |
| **Both** | Mutual swarm join (homelab mesh) |
| **Parent aggregate** | Parent pulls many children; stores namespaced copies or Autobase fleet view |
Discovery:
1. Desktop/admin mints link → stores remote pubkey + optional `dbKey` (z32/hex)
2. Agent joins `discoveryKey` of that core
3. On connection: `store.replicate(conn)` (workshop pattern)
4. `clone.watch` / `autoUpdate` refreshes HyperDB indexes for REST/RPC queries
This is the same pattern as workshop §3.1 Lookups — **read path is swarm + HyperDB**, not constant RPC polling.
## Parent / HA (Phase 4) — Autobase workshop
When you need multi-writer fleet registry or HA parents:
- Autobase bootstrap key shared across parent instances
- `open: (store) => HyperDB.bee(store.get('db-view'), spec, { extension: false, autoUpdate: true })`
- hyperdispatch ops: `add-writer`, `put-alert`, `put-link`, `ingest-rollup`
- RPC (existing PearData / workshop) appends ops; apply mutates HyperDB view
- Clients dial **any** writer; view key stays stable
Do **not** “backup” by copying Corestore folders (workshop warning). Rotate writers via Autobase instead.
## What stays on protomux-rpc
| Keep on RPC | Why |
|-------------|-----|
| Live `push:metrics` | Sub-second UX; not DB-shaped |
| `subscribeMetrics` / handshake / ACL | Session security |
| `runJob`, mintInvite | Mutating control |
| On-demand `queryData` for hot window | Memory path |
| Move to HyperDB (+ replicate) | Why |
|-------------------------------|-----|
| Peer links, aliases, labels | Shared fleet truth |
| Alert config + history | Durable, queryable |
| Warm/cold metric buckets | History without central server |
| Node inventory | Parent `/api/v3/nodes` |
## Phased delivery (recommended)
### Phase A — Local durability (rocks or bee, no swarm yet)
1. Add `hyperdb`, `hyperschema`, `corestore` deps
2. `scripts/build-db.js` + `spec/`
3. Persist alert configs, peer policy, warm downsample into HyperDB
4. REST/RPC history falls back to HyperDB after memory miss
5. Keep REST localhost semantics
### Phase B — Link & replicate
1. Hyperswarm alongside HyperDHT RPC (or reuse connections carefully — often **separate swarm for store.replicate**)
2. `linkPeer` / `unlinkPeer` RPCs write `@peardata/peer-link`
3. Desktop can seed/cache agent DB by key for offline scrubbing
4. Document `pd1.` invite vs **db discovery key** (two layers)
### Phase C — Parent fleet
1. Parent process pulls N child DB keys
2. Composite REST `/api/v3/nodes` + scoped `/api/v3/data`
3. Optional Autobase for parent HA
### Phase D — Polish
1. Encryption at rest (`encryptionKey` on cores / autobee)
2. Blind-peer seeding for always-on warm history
3. Grafana: already have REST Prometheus export; optionally expose hypercore-stats
## Concrete file plan (when implementing)
```text
peardata/
scripts/build-db.js
spec/hyperschema/
spec/hyperdb/
server/db/model.js
server/db/replicate.js
server/services/store-hyperdb.js
docs/STORAGE-HYPERDB.md ← this file
```
Deps (approximate):
```json
"hyperdb": "^6",
"hyperschema": "^1",
"corestore": "^7",
"hyperswarm": "^4",
"autobase": "^7",
"hyperdispatch": "^1"
```
(`autobee` only if you prefer that multiwriter path over Autobase+HyperDB view.)
## Relationship to current PearData roadmap
| Roadmap item | HyperDB role |
|--------------|--------------|
| Phase 2 tiered storage | Warm/cold collections |
| Phase 3 PearDock/containers | Extra collections / indexes |
| Phase 4 parent peer | Autobase + replicated views |
| REST v3 historical queries | `find` ranges on `@peardata/metric-point` |
## References (local clones)
| Path under `holepunchto_repos` | Takeaway |
|--------------------------------|----------|
| `hyperdb/README.md` | rocks vs bee, find/get/tx, autoUpdate |
| `hyperdb-workshop` | Schema builder, Corestore+Swarm replicate, RPC inserts |
| `hyperdb-autobase-workshop` | Multi-writer view, hyperdispatch, HA |
| `corestore/README.md` | `store.replicate(conn)`, namespacing |
| `pear-hyperdb` | Thin Model wrapper pattern for Pear apps |
| `autobee` | Experimental multiwriter bee alternative |
## Decision summary
1. **Yes — incorporate HyperDB** for metadata + warm history + linked-node sync.
2. **Keep memory + RPC pushes** for live 1s Netdata feel.
3. **Sync linked nodes via Corestore replication on Hyperswarm**, not by streaming every sample over RPC.
4. **Use Autobase+HyperDB** when you need multi-writer parents / HA — same pattern as Holepunch’s own workshops.
5. **Treat discovery keys as sensitive** — link only invited peers; RPC AuthZ remains authoritative for mutations.
docs/TECH-CHOICES.md
+98
View File
@@ -0,0 +1,98 @@
# Tech choices
Recommendations for collection, streaming, visualization, and reusable libraries.
## Transport & identity
| Choice | Why |
|--------|-----|
| **HyperDHT** | Same peer identity + hole-punching model as PearDock |
| **protomux-rpc** | Multiplexed request/response + server push events |
| **compact-encoding JSON** | Fast enough for control + 1s metric batches; shared with template |
| **Ed25519 seeds** | Agent address = pubkey; admin proof via HMAC from seed |
### Streaming metrics
MVP uses **protomux-rpc `event` pushes** (`push:metrics`) with per-session subscribe + throttle.
If fleet scale demands it later:
1. Dedicated protomux channel with binary packs (Float64 arrays)
2. Hypercore / Hyperbee for durable streams a parent can replicate
3. Extract a shared **`pearrpc`** package (session middleware + hot-path helpers) from this template
For MVP, JSON pushes keep the stack simple and debuggable.
## Collection
| Option | Verdict |
|--------|---------|
| **Node `os` + `/proc` (chosen)** | Zero native deps, good enough for MVP, low overhead |
| `node-os-utils` | Convenient but extra dep / less control |
| Native bindings (netdata collectors, `systeminformation`) | Higher fidelity; consider Phase 2 for Windows depth |
| Shell out to `vmstat`/`iostat` | Avoid on hot path |
**Linux:** `/proc/meminfo`, `/proc/loadavg`, `/proc/net/dev`, `/proc/diskstats`
**macOS/Windows:** `os` fallbacks (CPU%, freemem, loadavg where available); I/O/net rates may be zero until platform collectors land.
Target overhead: single timer, no child processes per tick, ring buffers only.
## Storage
| Tier | Implementation |
|------|----------------|
| Now | In-memory ring (`server/services/store.js`) |
| Next | **HyperDB** + Corestore (+ Hyperswarm replicate) — see [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) |
| HA / multi-writer parents | Autobase with HyperDB view (`extension: false`) |
| Avoid | SQLite as primary P2P store; copying corestore folders as “backup” |
### HyperDB engines
| Engine | Sync | Use |
|--------|------|-----|
| `HyperDB.bee(core, spec)` | P2P via Hypercore | Agent meta + warm history + linked peers |
| `HyperDB.rocks(path, spec)` | Local only | Fast local index / desktop cache (`pear-hyperdb` style) |
## Visualization (desktop)
| Option | Verdict |
|--------|---------|
| **Canvas sparklines (chosen MVP)** | No extra deps inside Pear; snappy for 1s updates |
| **uPlot** | Best next step for Netdata-like interactive charts (tiny, fast) |
| Chart.js / ECharts | Heavier; fine for secondary views |
| Grafana via REST | External; use `/api/v3/data` + Prometheus export |
## REST
Built-in Node `http` (no Express) — small attack surface, enough for Netdata-style GET APIs.
## Packaging
| Piece | Approach |
|-------|----------|
| Agent | Node 20+ , systemd unit `deploy/peardata.service` |
| Desktop | Pear (`pear-electron` + `pear-bridge`) |
| Invites | `pd1.` tokens (PearDock-style) |
| Installer | Phase 1 one-liner script |
## Reusable library extraction (recommended later)
From this codebase / PearDock patterns:
| Package | Contents |
|---------|----------|
| `@pear/rpc-session` or `pearrpc` | PeerSession middleware, ACL, rate limit, audit hooks |
| `pear-metrics-wire` | Context/chart catalog types + query args |
| `pear-invite` | `pd1.` encode/decode + capability HMAC |
Keep MIT-clean relative to AGPL PearDock sources (template already does).
## Integration seams
| Ecosystem | Seam |
|-----------|------|
| PearDock | Container chart plugin reading dock RPC |
| PearVirt | VM CPU/mem contexts |
| HoneyPeer | Optional agent directory announcements |
| BareOS | Bare-compatible collector build |
| Holesail | Tunnel REST or future agent web UI |
docs/TESTING.md
+8 -6
View File
@@ -14,8 +14,10 @@
|------|----------|
| `test/acl.test.js` | Role hierarchy, `assertAllowed` |
| `test/crypto-auth.test.js` | MAC key, capabilities, admin proof, invites, classify input |
| `test/protocol.test.js` | Constants, `MethodRoles`, schema validators |
| `test/integration.test.js` | Live HyperDHT server + client handshake, post, push |
| `test/protocol.test.js` | Constants, monitoring `MethodRoles`, schema validators |
| `test/store.test.js` | Metric ring buffer ingest + query |
| `test/rest.test.js` | Netdata-style `/api/v3` routes |
| `test/integration.test.js` | Live HyperDHT agent + handshake + metrics query |
```bash
npm test
@@ -27,10 +29,10 @@ SKIP_INTEGRATION=1 npm test
## Integration test behavior
1. Starts an ephemeral HyperDHT server in-process
1. Starts metrics pipeline + ephemeral HyperDHT agent in-process
2. Sets `PEARDATA_INSECURE_OPEN_ADMIN=1` for the process (restored in teardown)
3. Dials as a client, handshakes, posts a message, asserts push delivery
4. Tears down sockets / DHT
3. Dials as a client, handshakes, lists charts, queries `system.cpu`
4. Tears down collector / sockets / DHT
Requires outbound/inbound UDP for HyperDHT. If the test hangs or fails on a locked-down network, use `SKIP_INTEGRATION=1`.
@@ -78,7 +80,7 @@ When adding an RPC method:
| Desktop chrome | `npm start` — drag titlebar, resize edges, min/max/close |
| Admin connect | Paste key + `SERVER_SEED` in UI |
| Invite connect | Paste `pd1.…` without seed |
| Viewer denial | Public key only → `postMessage` fails with permission error |
| Viewer denial | Public key only → `runJob` / `mintInvite` fail with permission error |
| Health | With server up: `SERVER_PUBLIC_KEY=… npm run healthcheck` |
| Soak | `SERVER_PUBLIC_KEY=… SERVER_SEED=… npm run soak` |