This commit is contained in:
Raven Scott
2026-07-18 16:22:56 -04:00
parent 015d92a257
commit f747ffbd25
36 changed files with 1975 additions and 873 deletions
+13 -20
View File
@@ -1,45 +1,38 @@
# ── Server identity ────────────────────────────────────────── # ── Agent identity ──────────────────────────────────────────
# 32-byte seed as 64 hex chars. Auto-generated into .env on first boot if missing. # 32-byte seed as 64 hex chars. Auto-generated into .env on first boot if missing.
# Treat SERVER_SEED like a root password (capabilities + admin proofs). # Treat SERVER_SEED like a root password (capabilities + admin proofs).
# SERVER_SEED= # SERVER_SEED=
# SERVER_PUBLIC_KEY= # SERVER_PUBLIC_KEY=
# ── Roles ──────────────────────────────────────────────────── # ── Roles ────────────────────────────────────────────────────
# Baseline role for unknown peers: viewer | operator | admin
PEARDATA_DEFAULT_ROLE=viewer PEARDATA_DEFAULT_ROLE=viewer
# Comma-separated peer public keys that always get admin
# PEARDATA_ADMIN_KEYS= # PEARDATA_ADMIN_KEYS=
# DEV ONLY — every peer is admin (never enable in production)
# PEARDATA_INSECURE_OPEN_ADMIN=1 # PEARDATA_INSECURE_OPEN_ADMIN=1
# Optional allowlist (empty = all non-revoked peers accepted)
# PEARDATA_ALLOWLIST= # PEARDATA_ALLOWLIST=
# ── Runtime paths & limits ─────────────────────────────────── # ── Runtime paths & limits ───────────────────────────────────
# Replaces OS home for client identity path:
# $PEARDATA_HOME/.config/peardata/identity.json
# PEARDATA_HOME= # PEARDATA_HOME=
# Peer policy + audit.log (default ./data)
# PEARDATA_DATA_DIR=./data # PEARDATA_DATA_DIR=./data
# Demo room ring buffer size
# PEARDATA_MAX_MESSAGES=500
# Per-peer RPC requests per minute
# PEARDATA_RATE_LIMIT_RPM=120 # PEARDATA_RATE_LIMIT_RPM=120
# Client ConnectionManager reconnect attempts
# PEARDATA_MAX_RECONNECT=20 # PEARDATA_MAX_RECONNECT=20
# ── Metrics pipeline ─────────────────────────────────────────
# PEARDATA_SAMPLE_MS=1000
# PEARDATA_TIER0_POINTS=3600
# PEARDATA_TIER1_POINTS=1440
# PEARDATA_TIER1_EVERY=60
# ── Netdata-style REST API ───────────────────────────────────
# PEARDATA_REST=1
# PEARDATA_REST_HOST=127.0.0.1
# PEARDATA_REST_PORT=19999
# PEARDATA_REST_CORS=*
# ── Logging ────────────────────────────────────────────────── # ── Logging ──────────────────────────────────────────────────
# LOG_LEVEL=info # LOG_LEVEL=info
# LOG_JSON=1 # LOG_JSON=1
# ── Healthcheck / soak ─────────────────────────────────────── # ── Healthcheck / soak ───────────────────────────────────────
# Remote dial key (falls back to SERVER_PUBLIC_KEY)
# PEARDATA_HEALTH_KEY= # PEARDATA_HEALTH_KEY=
# HEALTHCHECK_TIMEOUT_MS=8000 # HEALTHCHECK_TIMEOUT_MS=8000
# SOAK_DURATION_MS=60000 # SOAK_DURATION_MS=60000
+5
View File
@@ -61,6 +61,11 @@ jobs:
LICENSE \ LICENSE \
docs/ARCHITECTURE.md \ docs/ARCHITECTURE.md \
docs/PROTOCOL.md \ docs/PROTOCOL.md \
docs/DATA-MODEL.md \
docs/REST-API.md \
docs/ROADMAP.md \
docs/TECH-CHOICES.md \
docs/STORAGE-HYPERDB.md \
docs/GETTING-STARTED.md \ docs/GETTING-STARTED.md \
docs/SECURITY.md \ docs/SECURITY.md \
docs/DESKTOP.md \ docs/DESKTOP.md \
+51 -90
View File
@@ -1,43 +1,45 @@
# peardata # PearData
**Production-oriented boilerplate for Holepunch / HyperDHT P2P apps.** **Decentralized, P2P, Netdata-class real-time monitoring for the Pear / Holepunch ecosystem.**
Distilled from patterns used in [peardock](https://github.com/snxraven/peardock)-class apps (MIT template — not a copy of peardocks AGPL sources): Each machine runs a lightweight **PearMonitor agent**. A Pear desktop client dials agents by public key and shows fleet + per-node live dashboards. Agents also expose a **Netdata-compatible REST API** (`/api/v1`, `/api/v2`, `/api/v3`) for scripts, Grafana, and Prometheus.
Built from the [pear-app-template](https://github.com/snxraven) HyperDHT + protomux-rpc patterns used by PearDock-class apps (MIT template — not a copy of PearDocks AGPL sources).
| Plane | Stack | | Plane | Stack |
|-------|--------| |-------|--------|
| Transport | **HyperDHT** secret streams (Noise) | | Transport | **HyperDHT** secret streams (Noise) |
| RPC | **protomux-rpc** + compact-encoding JSON | | RPC | **protomux-rpc** + compact-encoding JSON |
| Identity | Ed25519 keypairs (server seed + client identity file) | | Metrics | 1s collector → tiered buffers → push + query |
| AuthZ | Roles (`viewer` / `operator` / `admin`) + HMAC capabilities + admin seed proof | | AuthZ | Roles (`viewer` / `operator` / `admin`) + `pd1.` invites + admin seed |
| Desktop | **Pear** (`pear-electron` + `pear-bridge` + `<pear-ctrl>` titlebar) | | HTTP | Netdata-style REST on `127.0.0.1:19999` |
| Server | Node 20+ (systemd unit included) | | Desktop | **Pear** (`pear-electron` + `pear-bridge` + `<pear-ctrl>`) |
The demo product is **PearData**: a multi-peer room with messages, presence, and invite minting. Swap `server/handlers/demo.js` + `server/services/room.js` for your domain.
--- ---
## Quick start ## Quick start
```bash ```bash
cd pear_app_template # or your clone path cd peardata
npm install npm install
# Terminal A — server (prints public key) # Terminal A — agent (prints public key + REST URL)
npm run start:server npm run start:server
# Terminal B — mint an operator invite (optional) # Terminal B — REST smoke test
curl -s http://127.0.0.1:19999/api/v3/info | jq
curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-30&points=30' | jq
# Terminal C — mint an operator invite (optional)
npm run mint-invite -- operator npm run mint-invite -- operator
# Terminal C — Pear desktop UI # Terminal D — Pear desktop
npm start npm start
# or: pear run -d . # or: pear run -d .
``` ```
In the UI, paste the **server public key** (viewer) or a **`pd1.` invite** (elevated role). In the UI, paste the **agent public key** (viewer) or a **`pd1.` invite**.
Paste `SERVER_SEED` from `.env` into “Admin seed” for full admin without an invite. Paste `SERVER_SEED` for admin without an invite.
Drag the **titlebar** to move the window; resize from the edges (`pear.gui.resizable`).
```bash ```bash
npm test npm test
@@ -49,26 +51,20 @@ npm run healthcheck
## Repository layout ## Repository layout
``` ```
pear_app_template/ peardata/
├── app.js # Desktop UI logic ├── app.js / index.html / ui/ # Pear fleet dashboard
├── index.html / index.js # Pear shell (titlebar + pear-ctrl + drag) ├── shared/ # Protocol, metrics catalog, schema, crypto
├── ui/styles.css # Titlebar drag regions + layout ├── server/ # PearMonitor agent
├── shared/ # Protocol, encodings, schema, crypto-auth ├── server.js # DHT + pipeline + REST boot
├── server/ # HyperDHT listener + RPC middleware + demo domain │ ├── pipeline.js # collector → store → anomaly → push
│ ├── server.js │ ├── handlers/monitor.js # Domain RPCs
│ ├── core/ # keys, acl, audit, peer-policy, registry │ ├── services/ # collector, store, anomaly, jobs, …
│ ├── rpc/ # PeerSession + register │ ├── rest/ # Netdata-compatible HTTP API
── handlers/ # Domain RPCs (replace demo.js) ── core/ rpc/ utils/ # Auth, ACL, sessions (template)
│ ├── services/ # Domain state (replace room.js) ├── client/ # Multi-peer connection manager
│ └── utils/ ├── docs/ # Full documentation set
├── client/ # Connection, manager, identity ├── deploy/peardata.service # systemd
── bin/ # Server binary entry ── test/
├── scripts/ # healthcheck, soak, mint-invite, rename, release
├── test/ # brittle unit + integration
├── docs/ # Full documentation set
├── deploy/ # systemd unit
├── .github/workflows/ # GitHub CI + release
└── .gitea/workflows/ # Gitea CI + rolling release skeleton
``` ```
--- ---
@@ -77,57 +73,23 @@ pear_app_template/
| Script | Purpose | | Script | Purpose |
|--------|---------| |--------|---------|
| `npm start` / `npm run dev` | Pear desktop UI | | `npm start` | Pear desktop UI |
| `npm run start:server` | HyperDHT server | | `npm run start:server` | PearMonitor agent (P2P + REST) |
| `npm test` | brittle unit + integration | | `npm test` | brittle unit + integration |
| `npm run mint-invite -- [role] [ttlMs]` | Offline `pd1.` invite | | `npm run mint-invite -- [role]` | Offline `pd1.` invite |
| `npm run healthcheck` | Liveness / remote ping | | `npm run healthcheck` | Liveness / remote ping |
| `npm run soak` | Load exercise (needs env keys) | | `npm run soak` | Load exercise |
| `npm run rename -- <slug> <Product>` | Rebrand the tree |
| `bash scripts/release.sh` | Source tarball + checksum |
Full env reference: [docs/CONFIGURATION.md](./docs/CONFIGURATION.md).
--- ---
## Rebrand for a new app ## Auth model
```bash
npm run rename -- my-app MyApp
# → package name, protocol id, env prefixes, invite prefix, product strings
```
Then implement your domain:
1. Extend `shared/protocol.js` (`MethodRoles`, `Pushes`, `Methods`)
2. Validate args in `shared/schema.js`
3. Add handlers under `server/handlers/`
4. Register them in `server/rpc/register.js`
5. Call from `client/` + UI
6. Update `docs/PROTOCOL.md` and tests
See [docs/EXTENDING.md](./docs/EXTENDING.md).
---
## Auth model (secure defaults)
| Mode | How | Role | | Mode | How | Role |
|------|-----|------| |------|-----|------|
| Viewer | Dial public key only | `viewer` (read) | | Viewer | Dial public key only | `viewer` (read + subscribe) |
| Capability | `pd1.` invite or raw HMAC token | grant role | | Capability | `pd1.` invite or HMAC token | granted role |
| Admin seed | HMAC proof from `SERVER_SEED` | `admin` | | Admin seed | HMAC proof from `SERVER_SEED` | `admin` |
| Allowlist | `PEARDATA_ADMIN_KEYS` | admin for listed peers | | Allowlist | `PEARDATA_ADMIN_KEYS` | admin for listed peers |
| Dev escape | `PEARDATA_INSECURE_OPEN_ADMIN=1` | everyone admin |
See [docs/SECURITY.md](./docs/SECURITY.md).
---
## CI
- **GitHub**: `.github/workflows/ci.yml` (Node 20/22 matrix + docs presence), `release.yml` on `v*` tags
- **Gitea**: `.gitea/workflows/ci.yml`, `release-rolling.yml` (every push to `main``rolling` release; needs `RELEASE_TOKEN`)
--- ---
@@ -135,20 +97,19 @@ See [docs/SECURITY.md](./docs/SECURITY.md).
| Doc | Contents | | Doc | Contents |
|-----|----------| |-----|----------|
| [Getting started](./docs/GETTING-STARTED.md) | Install, run, connect, systemd, troubleshooting | | [Getting started](./docs/GETTING-STARTED.md) | Install, agent, REST, desktop, systemd |
| [Desktop](./docs/DESKTOP.md) | Pear shell, `pear-ctrl`, drag/resize, identity | | [Roadmap](./docs/ROADMAP.md) | MVP → advanced phases |
| [Architecture](./docs/ARCHITECTURE.md) | Planes, boot, session pipeline, module map | | [Architecture](./docs/ARCHITECTURE.md) | PearDock mapping, planes, modules |
| [Protocol](./docs/PROTOCOL.md) | Methods, pushes, errors, versioning | | [Protocol](./docs/PROTOCOL.md) | RPC methods & pushes |
| [Security](./docs/SECURITY.md) | Threat model, secrets, hardening checklist | | [Data model](./docs/DATA-MODEL.md) | Metrics, anomalies, health |
| [Configuration](./docs/CONFIGURATION.md) | Full environment + scripts reference | | [REST API](./docs/REST-API.md) | `/api/v1\|v2\|v3` |
| [Testing](./docs/TESTING.md) | brittle suite, soak, manual checks | | [Tech choices](./docs/TECH-CHOICES.md) | Collector, charts, libraries |
| [CI](./docs/CI.md) | Pipelines and required docs | | [Security](./docs/SECURITY.md) | Threat model & hardening |
| [Release](./docs/RELEASE.md) | Version, tag, tarball, rollback | | [Configuration](./docs/CONFIGURATION.md) | Environment reference |
| [Extending](./docs/EXTENDING.md) | Grow past the demo room |
--- ---
## License ## License
MIT — use this as a starting point for proprietary or open apps. MIT — Pear ecosystem tooling starter.
(Peardock itself is AGPL; this template does **not** copy peardock source verbatim and is intentionally MIT.) (Peardock itself is AGPL; this project does **not** copy peardock source verbatim.)
+19 -32
View File
@@ -112,7 +112,7 @@ function drawChart(canvasId, values, color = '#5b8cff') {
} }
function renderPeers() { function renderPeers() {
const list = manager.list?.() || [] const list = manager.list() || []
els.peerList.innerHTML = '' els.peerList.innerHTML = ''
if (!list.length) { if (!list.length) {
const li = document.createElement('li') const li = document.createElement('li')
@@ -123,11 +123,12 @@ function renderPeers() {
} }
for (const p of list) { for (const p of list) {
const li = document.createElement('li') const li = document.createElement('li')
const active = manager.activeId === p.id const id = p.publicKeyHex || p.id
const active = manager.active?.publicKeyHex === id
li.className = active ? 'active' : '' li.className = active ? 'active' : ''
li.innerHTML = `<span>${escapeHtml(p.id)}</span><span class="muted">${p.connected ? 'live' : '…'}</span>` li.innerHTML = `<span>${escapeHtml(String(id).slice(0, 12))}</span><span class="muted">${p.connected ? 'live' : '…'}</span>`
li.addEventListener('click', () => { li.addEventListener('click', () => {
manager.setActive?.(p.id) manager.setActive(id)
renderPeers() renderPeers()
}) })
els.peerList.appendChild(li) els.peerList.appendChild(li)
@@ -189,18 +190,6 @@ async function refreshMeta() {
renderPeers() renderPeers()
} }
function parseConnectInput(raw) {
const input = raw.trim()
if (input.startsWith('pd1.')) {
const inv = decodeInvite(input)
return {
publicKeyHex: inv.publicKeyHex,
capability: inv.capability,
}
}
return { publicKeyHex: input.toLowerCase(), capability: null }
}
els.btnConnect.addEventListener('click', async () => { els.btnConnect.addEventListener('click', async () => {
const raw = els.connectInput.value.trim() const raw = els.connectInput.value.trim()
const adminSeed = els.adminSeed.value.trim() || null const adminSeed = els.adminSeed.value.trim() || null
@@ -210,9 +199,9 @@ els.btnConnect.addEventListener('click', async () => {
} }
els.btnConnect.disabled = true els.btnConnect.disabled = true
try { try {
const { publicKeyHex, capability } = parseConnectInput(raw) log(`Dialing…`)
log(`Dialing ${publicKeyHex.slice(0, 16)}`) const conn = await manager.connect(raw, { adminSeed })
await manager.connect(publicKeyHex, { capability, adminSeed }) log(`Connected ${conn.publicKeyHex.slice(0, 16)}`)
setOnline(true) setOnline(true)
await manager.request(Methods.subscribeMetrics, { charts: ['*'], intervalMs: 1000 }) await manager.request(Methods.subscribeMetrics, { charts: ['*'], intervalMs: 1000 })
await manager.request(Methods.subscribeAnomalies, {}) await manager.request(Methods.subscribeAnomalies, {})
@@ -250,7 +239,7 @@ els.btnConnect.addEventListener('click', async () => {
}) })
els.btnDisconnect.addEventListener('click', async () => { els.btnDisconnect.addEventListener('click', async () => {
await manager.disconnectAll?.() await manager.disconnect()
setOnline(false) setOnline(false)
renderPeers() renderPeers()
log('Disconnected') log('Disconnected')
@@ -267,7 +256,7 @@ els.btnInvite.addEventListener('click', async () => {
} }
}) })
manager.on?.('push', (ev) => { manager.on('push', (ev) => {
if (ev.type === Pushes.metrics) onSamples(ev.data?.samples) if (ev.type === Pushes.metrics) onSamples(ev.data?.samples)
if (ev.type === Pushes.anomaly) { if (ev.type === Pushes.anomaly) {
prependAnomaly(ev.data) prependAnomaly(ev.data)
@@ -278,17 +267,15 @@ manager.on?.('push', (ev) => {
} }
}) })
// Compatibility if manager emits per-connection manager.on('connected', () => {
manager.on?.('connection', (conn) => { setOnline(true)
conn.on?.(Pushes.metrics, (data) => onSamples(data?.samples)) renderPeers()
conn.on?.(Pushes.anomaly, (data) => prependAnomaly(data)) })
conn.on?.(Pushes.health, (data) => {
els.statHealth.textContent = data?.status || '—' manager.on('disconnected', () => {
}) if (!manager.active?.connected) setOnline(false)
conn.on?.('disconnected', () => { renderPeers()
setOnline(false) log('Agent disconnected')
log('Agent disconnected')
})
}) })
setOnline(false) setOnline(false)
+1 -1
View File
@@ -20,7 +20,7 @@ export class ConnectionManager extends EventEmitter {
} }
/** /**
* @param {string} input - public key, pa1 invite, or capability+key object fields * @param {string} input - public key, pd1 invite, or capability+key object fields
* @param {{ adminSeed?: string, alias?: string, autoReconnect?: boolean }} [opts] * @param {{ adminSeed?: string, alias?: string, autoReconnect?: boolean }} [opts]
*/ */
async connect(input, opts = {}) { async connect(input, opts = {}) {
+1 -1
View File
@@ -1,5 +1,5 @@
[Unit] [Unit]
Description=Pear App P2P server (HyperDHT + protomux-rpc) Description=PearData PearMonitor agent (HyperDHT + REST metrics)
After=network-online.target After=network-online.target
Wants=network-online.target Wants=network-online.target
+123 -151
View File
@@ -1,193 +1,165 @@
# Architecture # Architecture
PearData is a **decentralized, P2P clone of the Netdata real-time monitoring experience**, built on the same HyperDHT + protomux-rpc patterns as PearDock-class apps (via the pear-app template).
## Design goals ## Design goals
1. **No central control plane**peers dial a public key, not a SaaS tenant. 1. **No central control plane**dial an agent by Ed25519 public key.
2. **Cryptographic identity** — HyperDHT Noise streams authenticate both ends. 2. **Instant live truth** — ~1s metric push to connected desktops.
3. **Clear AuthZ** — roles, rate limits, audit, optional allowlist / revoke. 3. **Dual API** — P2P RPC for the Pear client; Netdata-style REST for scripts/Grafana.
4. **Shared wire contract**`shared/*` is the single source of truth for client + server. 4. **Clear AuthZ** — viewer (pubkey) vs operator/admin (`pd1.` invite / seed proof).
5. **Replaceable domain** — demo room is a thin layer over the session stack. 5. **Low agent overhead** — Node/`os` + `/proc` collectors, ring buffers, hot-path RPCs.
6. **Pear-native desktop**`pear-electron` shell with `<pear-ctrl>` window chrome. 6. **Ecosystem-ready** — shared wire contract in `shared/` for PearDock / PearVirt adapters later.
## Mapping to PearDock / template components
| PearDock-class concept | PearData component |
|------------------------|--------------------|
| HyperDHT secret stream | `server/server.js` + `client/connection.js` |
| protomux-rpc methods | `shared/protocol.js` + `server/handlers/monitor.js` |
| Capability invites (`pd1.`) | `shared/crypto-auth.js` |
| Roles viewer/operator/admin | `shared/protocol.js` `MethodRoles` + `server/core/acl.js` |
| Peer policy / revoke | `server/core/peer-policy.js` |
| Connection manager / multi-peer | `client/manager.js` |
| Job tray | `server/services/jobs.js` + desktop actions |
| Domain service | **Metrics pipeline** (`collector``store``anomaly` → pushes) |
| Optional HTTP surface | `server/rest/*` (Netdata v1/v2/v3) |
## System context ## System context
```mermaid ```mermaid
flowchart LR flowchart LR
UI[Pear desktop / scripts] -->|HyperDHT Noise| SRV[Node server] UI[PearData desktop] -->|HyperDHT Noise + RPC| AG[PearMonitor agent]
SRV --> STATE[Room / your domain] SCRIPTS[curl / Grafana / Prometheus scrapers] -->|HTTP REST :19999| AG
UI -.->|bootstrap / punch| NET[HyperDHT network] AG --> COL[Collector 1s]
SRV -.-> NET COL --> STORE[Tiered ring buffers]
``` COL --> ANO[Anomaly engine]
AG -.->|bootstrap / punch| DHT[HyperDHT]
## Layered stack UI -.-> DHT
```mermaid
flowchart TB
subgraph Presentation
HTML[index.html + app.js + ui/styles.css]
PEAR[index.js pear-electron + bridge]
end
subgraph ClientCore
MGR[client/manager.js]
CON[client/connection.js]
ID[client/identity.js]
end
subgraph Wire
PROT[shared/protocol.js]
ENC[shared/encodings.js]
AUTH[shared/crypto-auth.js]
SCH[shared/schema.js]
end
subgraph ServerCore
BOOT[server/server.js]
SESS[server/rpc/session.js]
ACL[server/core/acl.js]
HAND[server/handlers/*]
SVC[server/services/*]
end
PEAR --> HTML
HTML --> MGR --> CON
CON --> PROT
CON --> AUTH
SESS --> PROT
SESS --> ACL
SESS --> SCH
HAND --> SVC
BOOT --> SESS
CON <-->|secret stream| SESS
``` ```
## Process model ## Process model
| Process | Entry | Responsibility | | Process | Entry | Responsibility |
|---------|-------|----------------| |---------|-------|----------------|
| **Server** | `server/server.js` or `bin/peardata-server.mjs` | HyperDHT listen, RPC, domain state | | **Agent** | `server/server.js` / `bin/peardata-server.mjs` | Collect, store, P2P RPC, optional REST |
| **Desktop** | `index.js` → Pear Runtime | Window + HTML UI; dials servers as a client | | **Desktop** | `index.js` → Pear Runtime | Fleet UI, multi-peer dial, live charts |
| **Scripts** | `scripts/*` | mint-invite, healthcheck, soak (use client stack) | | **Scripts** | `scripts/*` | mint-invite, healthcheck, soak |
Server and desktop are **independent**. You can run many clients against one server, or headless scripts with no UI. Many desktops may dial one agent; one desktop may dial many agents.
## Server boot ## Planes
1. `loadOrCreateKeyPair()` → persist `SERVER_SEED` / `SERVER_PUBLIC_KEY` in `.env` ### 1. Control / metadata (RPC)
2. `initAuthKeys()` → HKDF MAC key for capabilities
3. `loadPeerPolicy()` → roles / revocations / spent JTIs from `PEARDATA_DATA_DIR`
4. `dht.createServer().listen(keyPair)`
5. On connection → revoke check → `PeerSession``registerAllHandlers` → peer registry
6. Banner logs public key + secure/insecure mode
7. `graceful-goodbye` / SIGINT / SIGTERM drain peers and destroy DHT
## Session middleware (every non-hot RPC) Handshake, node info, chart/context catalog, alert config, invites, jobs, ACL.
### 2. High-frequency metrics
- **Ingest:** collector emits sample batches every `PEARDATA_SAMPLE_MS` (default 1000).
- **Store:** tier0 (1s) + tier1 (downsampled averages).
- **Push:** `push:metrics` to subscribed peers (protomux-rpc events).
- **Pull:** `queryData` / REST `/api/v3/data` for history windows.
### 3. Anomaly / health
Threshold engine evaluates each batch; transitions emit `push:anomaly` / `push:alert`; health snapshot on an interval via `push:health`.
### 4. Local REST (optional)
Netdata-compatible HTTP on `127.0.0.1:19999` by default — for local tooling without P2P. Disable with `PEARDATA_REST=0`.
## Layered stack
```mermaid ```mermaid
flowchart TD flowchart TB
IN[method + args] --> RL{Rate limit} subgraph Presentation
RL -->|deny| E1[RATE_LIMIT_EXCEEDED] HTML[index.html + app.js]
RL -->|ok| ACL{roleAllows MethodRoles} PEAR[index.js pear-electron]
ACL -->|deny| E2[PERMISSION_DENIED + audit] end
ACL -->|ok| VAL{validateMethodArgs} subgraph ClientCore
VAL -->|fail| E3[INVALID_ARGS] MGR[client/manager.js]
VAL -->|ok| H[Handler] CON[client/connection.js]
H --> OK[Result + optional audit] end
subgraph Wire
PROT[shared/protocol.js]
MET[shared/metrics.js]
SCH[shared/schema.js]
AUTH[shared/crypto-auth.js]
end
subgraph Agent
BOOT[server/server.js]
PIPE[server/pipeline.js]
COL[collector]
STORE[store]
ANO[anomaly]
HAND[handlers/monitor.js]
REST[rest/http-server.js]
end
PEAR --> HTML --> MGR --> CON
CON <-->|Noise + protomux-rpc| HAND
BOOT --> PIPE --> COL --> STORE
COL --> ANO
REST --> STORE
HAND --> STORE
``` ```
**Hot path** (`session.respond(method, handler, { hot: true })` or stream method names): ## Agent boot sequence
- Still rate-limited and ACL-checked 1. Load/create `SERVER_SEED` / public key → `initAuthKeys`
- Skips full schema validation / success audit (for high-frequency streams) 2. Load peer policy from `PEARDATA_DATA_DIR`
3. `startPipeline()` — collector + store + anomaly fan-out
4. `startRestServer()` — unless disabled
5. HyperDHT `createServer().listen(keyPair)`
6. On connection → revoke check → `PeerSession` → register monitor handlers
7. Banner prints pubkey + REST URL
## Client connection states ## Session middleware
``` Same as the template: rate limit → ACL (`MethodRoles`) → schema validate → handler.
idle → dialing → handshaking → ready Hot methods (`queryData`, `subscribeMetrics`, `ping`, …) skip heavy audit.
↘ closed → (manager reconnect timer)
```
| State | Meaning |
|-------|---------|
| `idle` | Constructed, not dialing |
| `dialing` | `dht.connect(serverPk)` in flight |
| `handshaking` | Stream open; `handshake` RPC |
| `ready` | Authenticated; RPCs allowed |
| `closed` | Torn down |
`ConnectionManager` tracks multiple peers, active selection, and reconnect (max `PEARDATA_MAX_RECONNECT`).
## Identity planes ## Identity planes
| Plane | Storage | Purpose | | Plane | Storage | Purpose |
|-------|---------|---------| |-------|---------|---------|
| **Server keypair** | `.env` (`SERVER_SEED`) | DHT listen address + HMAC root | | Agent keypair | `.env` `SERVER_SEED` | DHT address + HMAC root |
| **Client keypair** | `~/.config/peardata/identity.json` | Stable peerId for AuthZ / revoke | | Client keypair | `~/.config/peardata/identity.json` | Stable peerId |
| **Capabilities** | Issued as tokens / `pd1.` invites | Role grants with optional expiry & peer bind | | Capabilities | `pd1.` invites / raw tokens | Role grants |
| **Peer policy** | `data/peer-policy.json` | Registered roles, revocations, spent JTIs | | Peer policy | `data/peer-policy.json` | Roles, revokes, spent JTIs |
| **Audit** | `data/audit.log` | Mutating RPC trail | | Audit | `data/audit.log` | Mutating RPC trail |
## Parent peer (future)
A heavier agent may subscribe to child agents over P2P, downsample into its own store, and expose fleet REST — still no central SaaS. Design leaves room via `stream_path` REST stub and multi-peer desktop manager.
## Module map ## Module map
### `shared/` ### Keep from template
| File | Role | `server/core/*`, `server/rpc/session.js`, `client/*`, `shared/crypto-auth.js`, `shared/encodings.js`, Pear titlebar patterns.
|------|------|
| `protocol.js` | `PROTOCOL`, roles, `MethodRoles`, `Methods`, `Pushes` |
| `encodings.js` | compact-encoding JSON for protomux-rpc |
| `crypto-auth.js` | MAC key, capabilities, admin proof, invites |
| `schema.js` | Lightweight request validation |
### `server/` ### PearData domain
| Path | Role | | Path | Role |
|------|------| |------|------|
| `server.js` | Boot + DHT accept loop | | `shared/metrics.js` | Chart/context catalog |
| `core/keys.js` | Seed load / generate | | `shared/data-model.js` | Typed shapes |
| `core/auth-keys.js` | Process-wide MAC key | | `server/services/collector.js` | System sampling |
| `core/acl.js` | Role resolution + assert | | `server/services/store.js` | Tiered buffers + query |
| `core/peer-policy.js` | File-backed policy | | `server/services/anomaly.js` | Thresholds |
| `core/peer-registry.js` | Live sessions | | `server/services/alerts.js` | Alert CRUD helpers |
| `core/audit.js` | Audit log writer | | `server/services/subscriptions.js` | Push fan-out |
| `rpc/session.js` | ProtomuxRPC + middleware | | `server/services/jobs.js` | On-demand jobs |
| `rpc/register.js` | Wire handlers per session | | `server/handlers/monitor.js` | RPC surface |
| `handlers/demo.js` | **Replace** — domain RPCs | | `server/rest/*` | Netdata HTTP API |
| `services/room.js` | **Replace** — domain state | | `server/pipeline.js` | Wire collector→store→push |
| `utils/logger.js` | Structured / pretty logs |
| `utils/rateLimiter.js` | Per-peer RPM |
### `client/`
| File | Role |
|------|------|
| `identity.js` | Persistent client seed |
| `connection.js` | Single peer RPC client |
| `manager.js` | Multi-peer + reconnect |
| `errors.js` | Error normalization |
| `index.js` | Public re-exports |
### Desktop shell
| File | Role |
|------|------|
| `index.js` | Pear Runtime + Bridge |
| `index.html` | Titlebar (`pear-ctrl`) + layout |
| `app.js` | UI → manager |
| `ui/styles.css` | Drag regions + theme |
See [DESKTOP.md](./DESKTOP.md).
## What to keep vs replace
| Keep | Replace when productizing |
|------|---------------------------|
| `shared/*` wire + crypto | Method names / schema for your domain |
| `server/rpc/session.js` | Rarely — middleware is generic |
| `server/core/*` | Peer policy storage backend if needed |
| `client/connection.js` + `manager.js` | UI-specific multi-peer UX |
| Titlebar / `pear-ctrl` patterns | Visual design only — keep drag + controls |
| `server/handlers/demo.js` + `services/room.js` | **Your product** |
## Related docs ## Related docs
- [PROTOCOL.md](./PROTOCOL.md) - [PROTOCOL.md](./PROTOCOL.md) — RPC methods & pushes
- [SECURITY.md](./SECURITY.md) - [DATA-MODEL.md](./DATA-MODEL.md) — metrics / anomalies / health
- [DESKTOP.md](./DESKTOP.md) - [REST-API.md](./REST-API.md) — `/api/v1|v2|v3`
- [CONFIGURATION.md](./CONFIGURATION.md) - [TECH-CHOICES.md](./TECH-CHOICES.md) — collector & charts
- [EXTENDING.md](./EXTENDING.md) - [ROADMAP.md](./ROADMAP.md) — phases
- [SECURITY.md](./SECURITY.md) — threat model
+1 -1
View File
@@ -20,7 +20,7 @@
**lint-docs** **lint-docs**
- Asserts required documentation files exist (README, architecture, protocol, getting started, security, desktop, configuration, LICENSE, etc.) - Asserts required documentation files exist (README, architecture, protocol, data model, REST API, roadmap, tech choices, getting started, security, desktop, configuration, LICENSE, etc.)
### Release job details (`release.yml`) ### Release job details (`release.yml`)
+54 -84
View File
@@ -1,6 +1,6 @@
# Configuration reference # Configuration reference
All knobs can be set via environment variables. The server loads `.env` through `dotenv` on boot (`server/core/keys.js`). Copy `.env.example` to get started. All knobs can be set via environment variables. The agent loads `.env` through `dotenv` on boot (`server/core/keys.js`). Copy `.env.example` to get started.
```bash ```bash
cp .env.example .env cp .env.example .env
@@ -8,7 +8,7 @@ cp .env.example .env
--- ---
## Server identity ## Agent identity
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
@@ -16,8 +16,6 @@ cp .env.example .env
| `SERVER_PUBLIC_KEY` | *derived* | 32-byte public key as **64 hex**. Clients dial this. Auto-synced to `.env` when seed loads. | | `SERVER_PUBLIC_KEY` | *derived* | 32-byte public key as **64 hex**. Clients dial this. Auto-synced to `.env` when seed loads. |
| `SERVER_KEY` | — | Alias accepted for `SERVER_SEED` (legacy). Prefer `SERVER_SEED`. | | `SERVER_KEY` | — | Alias accepted for `SERVER_SEED` (legacy). Prefer `SERVER_SEED`. |
On first boot without a seed, the server appends both values to `.env`.
Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators. Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators.
--- ---
@@ -26,10 +24,10 @@ Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators.
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `PEARDATA_DEFAULT_ROLE` | `viewer` | Baseline role for unknown peers: `viewer` \| `operator` \| `admin` | | `PEARDATA_DEFAULT_ROLE` | `viewer` | Baseline role: `viewer` \| `operator` \| `admin` |
| `PEARDATA_ADMIN_KEYS` | empty | Comma-separated peer public keys always elevated to **admin** | | `PEARDATA_ADMIN_KEYS` | empty | Comma-separated peer pubs always elevated to **admin** |
| `PEARDATA_ALLOWLIST` | empty | If **non-empty**, only listed peer pubs (plus already-registered policy peers) may connect | | `PEARDATA_ALLOWLIST` | empty | If **non-empty**, only listed peers may connect |
| `PEARDATA_INSECURE_OPEN_ADMIN` | off | `1` / `true` / `yes` → every peer is admin. **Dev only.** | | `PEARDATA_INSECURE_OPEN_ADMIN` | off | `1` → every peer is admin. **Dev only.** |
--- ---
@@ -38,20 +36,41 @@ Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators.
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `PEARDATA_DATA_DIR` | `./data` | Peer policy JSON + `audit.log` | | `PEARDATA_DATA_DIR` | `./data` | Peer policy JSON + `audit.log` |
| `PEARDATA_HOME` | OS home | Root for client identity path construction (`client/identity.js`) | | `PEARDATA_HOME` | OS home | Root for client identity (`~/.config/peardata/identity.json`) |
| `PEARDATA_MAX_MESSAGES` | `500` | In-memory demo room ring buffer size |
| `PEARDATA_RATE_LIMIT_RPM` | `120` | Per-peer RPC requests per minute | | `PEARDATA_RATE_LIMIT_RPM` | `120` | Per-peer RPC requests per minute |
| `PEARDATA_MAX_RECONNECT` | `20` | Client manager reconnect attempts per peer | | `PEARDATA_MAX_RECONNECT` | `20` | Client manager reconnect attempts per peer |
### Data directory layout
``` ```
data/ data/
├── peer-policy.json # registered peers, roles, revocations, spent JTIs ├── peer-policy.json
└── audit.log # JSON lines for mutating RPCs + failures └── audit.log
``` ```
Recommended permissions: directory `0700`. Do **not** commit `data/` or `.env`. Permissions: directory `0700`. Do **not** commit `data/` or `.env`.
---
## Metrics pipeline
| Variable | Default | Description |
|----------|---------|-------------|
| `PEARDATA_SAMPLE_MS` | `1000` | Collector interval |
| `PEARDATA_TIER0_POINTS` | `3600` | High-res ring size (~1h @ 1s) |
| `PEARDATA_TIER1_POINTS` | `1440` | Downsampled ring size |
| `PEARDATA_TIER1_EVERY` | `60` | Samples per tier1 average |
---
## REST API (Netdata-style)
| Variable | Default | Description |
|----------|---------|-------------|
| `PEARDATA_REST` | on | Set `0` / `off` to disable HTTP API |
| `PEARDATA_REST_HOST` | `127.0.0.1` | Bind address (`0.0.0.0` exposes LAN — firewall!) |
| `PEARDATA_REST_PORT` | `19999` | HTTP port (Netdata classic) |
| `PEARDATA_REST_CORS` | `*` | `Access-Control-Allow-Origin` |
See [REST-API.md](./REST-API.md).
--- ---
@@ -60,87 +79,38 @@ Recommended permissions: directory `0700`. Do **not** commit `data/` or `.env`.
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` | | `LOG_LEVEL` | `info` | `debug` \| `info` \| `warn` \| `error` |
| `LOG_JSON` | off | `1`structured JSON logs (good for journald) | | `LOG_JSON` | off | `1`JSON lines |
--- ---
## Healthcheck & soak ## Healthcheck / soak / tests
| Variable | Default | Description | | Variable | Default | Description |
|----------|---------|-------------| |----------|---------|-------------|
| `PEARDATA_HEALTH_KEY` | — | Public key for remote health dial (falls back to `SERVER_PUBLIC_KEY`) | | `PEARDATA_HEALTH_KEY` | `SERVER_PUBLIC_KEY` | Key for remote ping |
| `HEALTHCHECK_TIMEOUT_MS` | `8000` | Healthcheck hard timeout (ms) | | `HEALTHCHECK_TIMEOUT_MS` | `8000` | Dial timeout |
| `SOAK_DURATION_MS` | `60000` | Soak test run length | | `SOAK_DURATION_MS` | `60000` | Soak length |
| `SOAK_INTERVAL_MS` | `500` | Delay between soak posts | | `SOAK_INTERVAL_MS` | `500` | Soak RPC interval |
| `SKIP_INTEGRATION` | off | `1` skips DHT integration test |
```bash
# Liveness only (no dial) when no key is set
npm run healthcheck
# Full dial + ping (server must be running)
SERVER_PUBLIC_KEY=SERVER_SEED=… npm run healthcheck
# Load soak (admin seed recommended so postMessage is allowed)
SERVER_PUBLIC_KEY=SERVER_SEED=… npm run soak
```
---
## Testing
| Variable | Default | Description |
|----------|---------|-------------|
| `SKIP_INTEGRATION` | unset | `1` skips live HyperDHT integration test |
---
## Pear GUI (package.json)
Not environment variables — set under `package.json``pear.gui`. See [DESKTOP.md](./DESKTOP.md).
| Field | Template |
|-------|----------|
| `width` × `height` | 1100 × 780 |
| `minWidth` × `minHeight` | 720 × 480 |
| `resizable` / `movable` | `true` |
| `backgroundColor` | `#0b1020` |
---
## systemd
`deploy/peardata.service` expects:
| Path | Purpose |
|------|---------|
| `/opt/peardata` | WorkingDirectory |
| `/opt/peardata/.env` | `EnvironmentFile` |
| `/opt/peardata/data` | Writable data dir |
Edit unit paths before enabling. See [GETTING-STARTED.md](./GETTING-STARTED.md#systemd).
--- ---
## npm scripts ## npm scripts
| Script | Command | Purpose | | Script | Purpose |
|--------|---------|---------| |--------|---------|
| `npm start` / `npm run dev` | `pear run -d .` | Pear desktop UI | | `npm start` / `npm run dev` | Pear desktop |
| `npm run start:server` / `server` | `node server/server.js` | P2P server | | `npm run start:server` | Agent (P2P + REST) |
| `npm run start:server:bin` | `node bin/peardata-server.mjs` | Alternate server entry | | `npm run start:server:bin` | `bin/peardata-server.mjs` |
| `npm test` | brittle suite | Unit + integration | | `npm test` | brittle suite |
| `npm run test:integration` | integration only | Live DHT test | | `npm run mint-invite -- [role] [ttlMs]` | Offline `pd1.` invite |
| `npm run healthcheck` | dial or liveness | Process / network check | | `npm run healthcheck` | Liveness |
| `npm run soak` | long connect loop | Stability exercise | | `npm run soak` | Load exercise |
| `npm run mint-invite -- [role] [ttlMs]` | mint `pd1.` invite | Offline invite tooling |
| `npm run rename -- <slug> <Product>` | rebrand tree | New product from template |
| `bash scripts/release.sh` | source tarball | Local release artifacts |
--- ---
## Security notes ## systemd
- Never ship `SERVER_SEED` in client bundles or public repos. Unit: `deploy/peardata.service`
- Prefer invites over seed distribution. WorkingDirectory: `/opt/peardata`
- Rotate seed = new public key → all clients re-dial and re-invite. EnvironmentFile: `/opt/peardata/.env`
- See [SECURITY.md](./SECURITY.md) for the full checklist.
+141
View File
@@ -0,0 +1,141 @@
# Data model
Canonical shapes for metrics, anomalies, alerts, health, and jobs.
Implemented in `shared/data-model.js`, `shared/metrics.js`, and agent services.
## NodeInfo
```json
{
"nodeId": "a1b2c3…",
"hostname": "homelab-1",
"publicKeyHex": "…64 hex…",
"platform": "linux",
"arch": "x64",
"release": "6.x",
"cpus": 8,
"totalMemMiB": 32000,
"agentVersion": "0.1.0",
"protocolVersion": 1,
"startedAt": 1710000000000,
"charts": ["system.cpu", "system.ram", "…"],
"sampleIntervalMs": 1000,
"sampleCount": 42
}
```
## Metric contexts & charts
Netdata-inspired IDs:
| Context | Chart id | Units | Dimensions (MVP) |
|---------|----------|-------|------------------|
| `system.cpu` | `system.cpu` | percentage | user, system, nice, iowait, irq, softirq, idle |
| `system.ram` | `system.ram` | MiB | used, cached, buffers, free |
| `mem.available` | `mem.available` | MiB | avail |
| `system.load` | `system.load` | load | load1, load5, load15 |
| `system.io` | `system.io` | KiB/s | reads, writes |
| `system.net` | `system.net` | kilobits/s | received, sent |
| `system.processes` | `system.processes` | processes | running, blocked, total |
| `system.uptime` | `system.uptime` | seconds | uptime |
Chart summary objects mirror Netdatas `/api/v1/charts` fields (`id`, `context`, `units`, `dimensions`, `update_every`, `first_entry`, `last_entry`, …).
## MetricSample (live push)
```json
{
"chart": "system.cpu",
"context": "system.cpu",
"ts": 1710000000123,
"values": { "user": 12.4, "system": 3.1, "idle": 84.0 }
}
```
## QueryResult (history)
```json
{
"chart": "system.cpu",
"context": "system.cpu",
"labels": ["time", "user", "system", "nice", "iowait", "irq", "softirq", "idle"],
"data": [[1710000000, 12.4, 3.1, 0, 0, 0, 0, 84.0]],
"view_update_every": 1,
"after": 1709999940,
"before": 1710000000,
"points": 60,
"format": "json"
}
```
`time` is unix seconds. Nulls allowed for missing dimensions.
## Storage tiers
| Tier | Resolution | Default retention | Env |
|------|------------|-------------------|-----|
| 0 | 1s | 3600 points (~1h) | `PEARDATA_TIER0_POINTS` |
| 1 | avg every N samples (default 60 → ~1m) | 1440 points (~24h) | `PEARDATA_TIER1_POINTS`, `PEARDATA_TIER1_EVERY` |
## AnomalyEvent
```json
{
"id": "cpu_user_high:1710000000123",
"chart": "system.cpu",
"context": "system.cpu",
"dimension": "user",
"severity": "warning",
"score": 0.6,
"value": 88.2,
"threshold": 80,
"comparator": ">",
"message": "CPU user time high: user=88.2 > 80",
"ts": 1710000000123,
"cleared": false
}
```
## AlertConfig / AlertState
Config fields: `id`, `chart`, `dimension`, `warn`, `crit`, `comparator`, `lookbackSec`, `enabled`, `info`.
State adds: `status` (`CLEAR`|`WARNING`|`CRITICAL`|`UNDEFINED`), `value`, `lastStatusChange`.
## HealthSnapshot
```json
{
"status": "ok",
"score": 1,
"checks": [{ "id": "cpu_user_high", "ok": true, "detail": "…" }],
"ts": 1710000000123
}
```
Aggregate: any CRITICAL → `critical`; else any WARNING → `degraded`; else `ok`.
## JobRecord
```json
{
"id": "uuid",
"name": "snapshot",
"status": "done",
"startedAt": 1710000000000,
"finishedAt": 1710000000500,
"result": { "ok": true }
}
```
## REST ↔ RPC parity
| Concept | RPC | REST |
|---------|-----|------|
| Charts | `listCharts` | `GET /api/v1/charts` |
| Data | `queryData` | `GET /api/v3/data` |
| Contexts | `listContexts` | `GET /api/v3/contexts` |
| Nodes | `getNodeInfo` | `GET /api/v3/nodes` |
| Alerts | `listAlerts` | `GET /api/v3/alerts` |
| Export | `getAllMetrics` | `GET /api/v3/allmetrics` |
| Health | `getHealth` | `GET /api/v3/health` |
+3 -3
View File
@@ -8,7 +8,7 @@ The client shell is a **Pear desktop application** built with `pear-electron` +
|------|------| |------|------|
| `index.js` | Pear process entry — starts `pear-electron` Runtime + `pear-bridge` | | `index.js` | Pear process entry — starts `pear-electron` Runtime + `pear-bridge` |
| `index.html` | GUI main (`pear.gui.main`) — titlebar + panels | | `index.html` | GUI main (`pear.gui.main`) — titlebar + panels |
| `app.js` | UI logic (connect, room, presence, invites) | | `app.js` | UI logic (connect, fleet, live charts, invites) |
| `ui/styles.css` | Layout, theme, **titlebar drag regions** | | `ui/styles.css` | Layout, theme, **titlebar drag regions** |
| `client/*` | HyperDHT connection stack used by the UI | | `client/*` | HyperDHT connection stack used by the UI |
@@ -25,8 +25,8 @@ Requires the [Pear](https://docs.pears.com) CLI installed and bootstrapped (`pea
| Field | Template default | Purpose | | Field | Template default | Purpose |
|-------|------------------|---------| |-------|------------------|---------|
| `main` | `index.html` | HTML entry | | `main` | `index.html` | HTML entry |
| `width` / `height` | `1100` / `780` | Initial size | | `width` / `height` | `1280` / `860` | Initial size |
| `minWidth` / `minHeight` | `720` / `480` | Resize floor | | `minWidth` / `minHeight` | `900` / `560` | Resize floor |
| `resizable` | `true` | Edge/corner resize | | `resizable` | `true` | Edge/corner resize |
| `movable` | `true` | Allow OS move (with drag region) | | `movable` | `true` | Allow OS move (with drag region) |
| `minimizable` / `maximizable` / `closable` | `true` | Window buttons | | `minimizable` / `maximizable` / `closable` | `true` | Window buttons |
+48 -119
View File
@@ -1,159 +1,88 @@
# Extending the template # Extending PearData
## 1. Rebrand ## Add a chart / context
```bash 1. Define the chart in `shared/metrics.js` (`CHART_DEFS`).
npm run rename -- notes-mesh NotesMesh 2. Emit samples from `server/services/collector.js` (or a new collector module).
``` 3. Store + REST/RPC pick it up automatically via `CHART_BY_ID`.
4. Document dimensions in [DATA-MODEL.md](./DATA-MODEL.md).
5. Optionally add a canvas panel in `index.html` + `app.js`.
This rewrites: ## Add an RPC method
- package name / product name
- protocol id (`notes-mesh/rpc`)
- env prefix (`NOTES_MESH_`)
- invite prefix (derived, e.g. `no1.`)
- binary + systemd unit filenames
Then:
```bash
npm install
npm test
git diff # review
```
## 1b. Titlebar / window chrome
Keep these when restyling:
| Piece | Role |
|-------|------|
| `<pear-ctrl>` in `#titlebar` | Close / minimize / maximize (Pear runtime custom element) |
| `#titlebar { -webkit-app-region: drag }` | Drag the window |
| Interactive children `no-drag` | Buttons, chips, inputs stay clickable |
| `pear.gui.resizable: true` | Edge/corner resize |
| `pear.gui.minWidth` / `minHeight` | Floor size while resizing |
Do not remove `<pear-ctrl>` unless you intentionally want a frame without in-content controls (and understand platform differences). Details: [DESKTOP.md](./DESKTOP.md).
## 2. Add an RPC method
### `shared/protocol.js` ### `shared/protocol.js`
```js ```js
export const MethodRoles = Object.freeze({ export const MethodRoles = Object.freeze({
// ... // ...
listNotes: Roles.viewer, listContainers: Roles.viewer,
createNote: Roles.operator, restartCollector: Roles.admin,
}) })
``` ```
`Methods` is derived automatically from `MethodRoles` keys.
### `shared/schema.js` ### `shared/schema.js`
Validate `createNote` args (required fields, max lengths). Validate args.
### `server/services/notes.js` ### `server/services/…` + `server/handlers/monitor.js`
Domain logic / storage (keep IO out of handlers when possible). Register with `session.respond(...)`. Use `{ hot: true }` for high-frequency paths.
### `server/handlers/notes.js` ### Client
```js ```js
export function registerNoteHandlers(session) { await manager.request(Methods.listContainers, {})
session.respond('listNotes', async () => ({ notes: [] }))
session.respond('createNote', async (args, s) => { /* ... */ })
}
``` ```
### `server/rpc/register.js` ### Docs + tests
Call `registerNoteHandlers(session)`. Update [PROTOCOL.md](./PROTOCOL.md) and add a brittle test.
### Client / UI ## Add a REST route
Edit `server/rest/routes.js` — keep Netdata path naming when emulating Agent APIs (`/api/v3/...`).
## Add a job
Register in `server/services/jobs.js` `JOB_HANDLERS`.
```js ```js
await manager.request(Methods.createNote, { title: '…' }) retrainAnomaly: async () => { /* … */ return { ok: true } }
``` ```
### Tests Operators run via `runJob` or future REST function execute.
- Unit for pure helpers + schema ## Add a collector plugin (pattern)
- Integration for the happy path if AuthZ/wire matter
## 3. Add a push channel ```
server/services/collectors/
1. Add to `Pushes` in `shared/protocol.js` system.js # default
2. Optionally map in `PushToType` docker.js # Phase 3
3. `session.push(Pushes.foo, payload)` or broadcast via peer registry peardock.js # bridge
4. Listen in UI: `manager.on('push', …)` or `conn.on(Pushes.foo, …)`
(`connection.js` auto-registers all `Pushes` values)
## 4. Persistence
Demo room is in-memory (`server/services/room.js`). Swap for:
| Store | Good for |
|-------|----------|
| **Corestore / Hypercore** | Append-only logs, P2P replication |
| **SQLite** | Structured queries |
| **JSON files** under `PEARDATA_DATA_DIR` | Small config (peer-policy already does this) |
Keep RPC handlers thin; put IO in `services/`.
## 5. Binary streams
Peardock-class apps use chunked binary RPC for uploads. Pattern:
- methods `binaryStreamOpen` / `Chunk` / `Close` marked `hot: true` in `session.respond`
- skip heavy schema/audit on the hot path
- still enforce ACL + rate limits
Stub hooks exist via `rateLimiter.isStreamMethod`.
## 6. Multi-server fleet
`client/manager.js` already holds many connections + active selection + reconnect:
```js
await manager.connect(keyOrInviteA, { autoReconnect: true })
await manager.connect(keyOrInviteB, { autoReconnect: true })
manager.setActive(keyA)
await manager.request(Methods.ping, {})
``` ```
Point UI at a peer list stored in `localStorage` or a file cache for a fuller fleet UX. Have `pipeline.js` start each enabled collector; all emit `samples` batches into the same store.
## 7. Custom desktop UX ## Parent peer (fleet aggregator)
| Goal | Touch | 1. Parent dials child agents with `ConnectionManager`.
|------|-------| 2. Subscribes to `push:metrics` / periodically `queryData`.
| New screens | `index.html` + `app.js` + `ui/styles.css` | 3. Ingests into local store under namespaced chart ids (`childPk.system.cpu`) or labels.
| Window size | `package.json``pear.gui` | 4. Exposes `/api/v3/nodes` with multiple entries + `/api/v3/data` across nodes.
| Branding in titlebar | `.app-brand` markup/CSS |
| Persist UI prefs | replace demo `localStorage` keys |
Preserve drag / `pear-ctrl` behavior — [DESKTOP.md](./DESKTOP.md). ## PearDock / PearVirt hooks
## 8. Desktop packaging - Prefer **RPC adapters** over scraping: call dock/virt methods, map to PearData contexts.
- Keep AGPL boundaries clean — depend on public APIs / your own MIT bridges.
This template ships the **Pear** GUI path (`pear run`). For Electron-forge / bare-standalone multi-arch releases, copy more complex packaging scripts from a production app (forge config, make scripts) once your protocol stabilizes. ## Desktop chrome
## 9. Checklist for a new product Keep `<pear-ctrl>`, titlebar drag regions, and `pear.gui.resizable`. See [DESKTOP.md](./DESKTOP.md).
- [ ] `npm run rename -- …` ## Rebrand (fork)
- [ ] Replace demo handlers + services
- [ ] Update PROTOCOL.md method table
- [ ] Update SECURITY / CONFIG if new secrets
- [ ] Tests green (`npm test`)
- [ ] README product description
- [ ] CI still green
- [ ] First release tag ([RELEASE.md](./RELEASE.md))
## Related ```bash
npm run rename -- my-monitor MyMonitor
```
- [ARCHITECTURE.md](./ARCHITECTURE.md) Review invite prefix / env prefix, then `npm test`.
- [PROTOCOL.md](./PROTOCOL.md)
- [TESTING.md](./TESTING.md)
- [DESKTOP.md](./DESKTOP.md)
+63 -146
View File
@@ -1,188 +1,105 @@
# Getting started # Getting started
## Prerequisites ## Requirements
| Tool | Required | Notes | - Node.js **≥ 20**
|------|----------|--------| - Pear runtime (for desktop): install via Holepunch/Pear docs
| **Node.js ≥ 20** | Yes | Server + tests | - Linux recommended for full net/disk `/proc` collectors (macOS gets CPU/RAM/load)
| **npm** | Yes | Install deps |
| **[Pear](https://docs.pears.com) CLI** | For desktop UI | Run `pear` once to bootstrap the runtime |
| UDP / network | For real peers | HyperDHT hole-punching |
## Install ## Install
```bash ```bash
git clone <your-fork-or-template-url> my-app cd peardata
cd my-app
npm install npm install
cp .env.example .env # optional; server auto-writes seed on first boot
``` ```
## Run the server Copy env template if needed:
```bash
cp .env.example .env
```
## Run the agent
```bash ```bash
npm run start:server npm run start:server
# alias: npm run server
``` ```
On first boot the server appends to `.env`: Banner shows:
``` - **publicKey** — dial this from the desktop (viewer)
SERVER_SEED=<64 hex secret> - **rest** — e.g. `http://127.0.0.1:19999/api/v3/info`
SERVER_PUBLIC_KEY=<64 hex public> - **admin** — use `SERVER_SEED` from `.env` or mint `pd1.` invites
Agent persists identity in `.env` on first boot.
### REST smoke tests
```bash
curl -s http://127.0.0.1:19999/api/v3/info
curl -s http://127.0.0.1:19999/api/v1/charts | head
curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-60&points=60'
curl -s 'http://127.0.0.1:19999/api/v3/allmetrics?format=prometheus' | head
``` ```
**Treat `SERVER_SEED` like a root password.** Anyone with it can mint admin proofs and capabilities. Disable REST: `PEARDATA_REST=0`.
Banner output shows the public key clients dial, for example: ## Mint an invite
``` ```bash
Client → dial <SERVER_PUBLIC_KEY> npm run mint-invite -- operator
# → pd1.…
``` ```
Leave this process running while clients connect. Share the invite string. Recipients paste it into the desktop connect box.
## Run the desktop UI ## Run the desktop
```bash ```bash
npm start npm start
# or: npm run dev # pear run -d .
# or: pear run -d .
``` ```
### Desktop window chrome 1. Paste agent public key or `pd1.` invite
2. Optional: paste `SERVER_SEED` for admin
3. Connect → live overview + charts
4. Admin: **Mint invite**
The Pear UI uses a custom titlebar: ## systemd (Linux)
| Piece | Behavior |
|-------|----------|
| `<pear-ctrl>` | Platform window controls (macOS traffic lights / Windows & Linux min·max·close) |
| `#titlebar` | `-webkit-app-region: drag` — drag to move the window |
| Interactive children | `no-drag` so buttons and chips stay clickable |
| `pear.gui.resizable` | Edge/corner resize (`true` by default) |
| `minWidth` / `minHeight` | 720 × 480 floor |
Full details: [DESKTOP.md](./DESKTOP.md).
## Connect as admin (dev)
1. Start the Pear UI: `npm start`
2. Paste `SERVER_PUBLIC_KEY` into **Server / invite**
3. Paste `SERVER_SEED` into **Admin seed**
4. Optionally set a **Display name**
5. Click **Connect** → role badge should show `admin`
The seed never goes over the wire as plaintext — the client sends an HMAC **admin proof**.
## Connect via invite
```bash ```bash
# Persistent operator invite (default)
npm run mint-invite -- operator
# 7-day operator invite (ttl in ms)
npm run mint-invite -- operator 604800000
# Admin invite
npm run mint-invite -- admin
```
Stdout prints a `pd1.…` string. Paste it into the UI connect field (no seed needed).
You can also mint from a connected admin session with **Mint invite** in the UI.
## Viewer-only
Paste only the public key. You can:
- `listMessages`, `getPresence`, `getServerInfo`, `setDisplayName`, `ping`
You cannot:
- `postMessage` (needs `operator+`)
- `clearMessages`, `mintInvite`, `listPeers`, `revokePeer` (needs `admin`)
Unless you raise `PEARDATA_DEFAULT_ROLE` (not recommended for multi-user hosts).
## Quick verification
```bash
npm test
SKIP_INTEGRATION=1 npm test # unit only
# With server running:
export SERVER_PUBLIC_KEY=# from .env
export SERVER_SEED=# optional but enables admin dial
npm run healthcheck
npm run soak # optional load exercise
```
## Environment knobs (summary)
| Variable | Purpose |
|----------|---------|
| `SERVER_SEED` / `SERVER_PUBLIC_KEY` | Server identity |
| `PEARDATA_DEFAULT_ROLE` | Baseline role (`viewer` default) |
| `PEARDATA_ADMIN_KEYS` | Peer pubs always admin |
| `PEARDATA_INSECURE_OPEN_ADMIN` | Dev only — all peers admin |
| `PEARDATA_ALLOWLIST` | If set, only listed / registered peers |
| `PEARDATA_DATA_DIR` | Peer policy + audit log directory |
| `PEARDATA_RATE_LIMIT_RPM` | Per-peer RPC budget |
| `PEARDATA_MAX_MESSAGES` | Demo room history cap |
| `PEARDATA_MAX_RECONNECT` | Client reconnect tries |
| `LOG_LEVEL` / `LOG_JSON` | Logging |
Full table: [CONFIGURATION.md](./CONFIGURATION.md).
## Rebrand for a new product
```bash
npm run rename -- my-app MyApp
npm install
npm test
```
Rewrites package name, protocol id, env prefixes, invite prefix (`pd1.` → derived), product strings, and renames the server binary / systemd unit. Review `git diff` after.
## systemd
```bash
# Install tree to /opt/peardata (example)
sudo mkdir -p /opt/peardata sudo mkdir -p /opt/peardata
sudo rsync -a --exclude node_modules --exclude .git ./ /opt/peardata/ sudo rsync -a ./ /opt/peardata/ --exclude node_modules
cd /opt/peardata && sudo npm install --omit=dev cd /opt/peardata && sudo npm ci --omit=dev
sudo cp deploy/peardata.service /etc/systemd/system/ sudo cp deploy/peardata.service /etc/systemd/system/
# Edit WorkingDirectory, EnvironmentFile, ReadWritePaths if paths differ
sudo systemctl daemon-reload sudo systemctl daemon-reload
sudo systemctl enable --now peardata sudo systemctl enable --now peardata
sudo journalctl -u peardata -f journalctl -u peardata -f
``` ```
The unit sets `NoNewPrivileges`, `ProtectSystem=strict`, and writable paths for `data/` + `.env`. Ensure `ReadWritePaths` can write `/opt/peardata/data` and `.env`.
## Roles cheat sheet
| Goal | Action |
|------|--------|
| Read-only monitoring | Dial pubkey only |
| Ack alerts / run jobs | `pd1.` operator invite |
| Mint invites / revoke | Admin seed or admin invite |
## Troubleshooting ## Troubleshooting
| Symptom | What to check | | Symptom | Check |
|---------|----------------| |---------|-------|
| `Connection timeout` | Server running? Correct 64-hex key? Firewall / UDP? | | REST connection refused | Agent running? `PEARDATA_REST` not `0`? Port free? |
| `PERMISSION_DENIED` on send | Role is viewer — use invite or admin seed | | Desktop cant dial | Firewall / DHT; same machine should work; wait for punch |
| `Rate limit exceeded` | Raise `PEARDATA_RATE_LIMIT_RPM` or slow clients | | Empty net/disk charts | Non-Linux host — expected until platform collectors land |
| Window wont drag | Titlebar drag CSS; dont cover bar with full-screen `no-drag` overlay | | `PERMISSION_DENIED` | Need higher role invite or admin seed |
| Window wont resize | `pear.gui.resizable` must be true; try edges not just corners | | Integration tests hang | `SKIP_INTEGRATION=1 npm test` on restricted networks |
| No `<pear-ctrl>` buttons | Running under Pear (`pear run`)? Element is runtime-provided |
| Integration test fails in CI | Set `SKIP_INTEGRATION=1` or allow UDP |
| Seed regenerated every boot | `.env` not writable / wrong cwd |
## Next steps ## Next reading
| Doc | When | - [REST-API.md](./REST-API.md)
|-----|------| - [ARCHITECTURE.md](./ARCHITECTURE.md)
| [DESKTOP.md](./DESKTOP.md) | Titlebar, pear-ctrl, packaging | - [ROADMAP.md](./ROADMAP.md)
| [ARCHITECTURE.md](./ARCHITECTURE.md) | Stack & session pipeline | - [CONFIGURATION.md](./CONFIGURATION.md)
| [PROTOCOL.md](./PROTOCOL.md) | Methods, pushes, versioning |
| [SECURITY.md](./SECURITY.md) | Production hardening |
| [CONFIGURATION.md](./CONFIGURATION.md) | Full env reference |
| [EXTENDING.md](./EXTENDING.md) | Replace the demo room |
| [TESTING.md](./TESTING.md) | Tests & soak |
| [CI.md](./CI.md) / [RELEASE.md](./RELEASE.md) | Pipelines & shipping |
+104 -131
View File
@@ -1,158 +1,131 @@
# Protocol # Protocol
## Constants Wire contract for PearData P2P RPC. Source of truth: `shared/protocol.js`, `shared/schema.js`, `shared/metrics.js`.
## Identity
| Constant | Value | | Constant | Value |
|----------|--------| |----------|-------|
| `PROTOCOL` | `peardata/rpc` | | Protocol id | `peardata/rpc` |
| `PROTOCOL_VERSION` | `1` | | Protocol version | `1` |
| `APP_NAME` | `peardata` | | Invite prefix | `pd1.` |
| `APP_VERSION` | `0.1.0` (keep in sync with package where useful) | | Default role | `viewer` |
| Encoding | compact-encoding JSON (`shared/encodings.js`) |
| Schema | lightweight validators (`shared/schema.js`) `SCHEMA_VERSION=1` |
Bump `PROTOCOL_VERSION` on breaking request/response shapes. Additive methods may land without a bump if clients ignore unknown methods. Bump `PROTOCOL_VERSION` on breaking argument/result shapes.
After `npm run rename`, `PROTOCOL` becomes `<slug>/rpc` and invite prefix is regenerated.
## Transport
1. Client opens HyperDHT secret stream to server public key (Noise, mutual key auth).
2. `ProtomuxRPC` is attached with `protocol: PROTOCOL` and shared encodings.
3. Client calls `handshake` before other RPCs (connection helper does this automatically).
4. Server may `push` events on named channels.
## Roles ## Roles
| Role | Intent | | Role | Rank | Typical access |
|------|--------| |------|------|----------------|
| `viewer` | Read-only | | `viewer` | 1 | Read metrics, subscribe, list alerts |
| `operator` | Mutate domain data | | `operator` | 2 | Ack/silence alerts, run jobs, set alert config |
| `admin` | Invite mint, clear, revoke, config | | `admin` | 3 | Mint invites, revoke peers, export snapshot |
Hierarchy: `admin > operator > viewer` (`roleAllows`). Auth modes at handshake: public key (viewer), capability token / `pd1.` invite, admin seed proof, allowlist.
Unknown methods default to **admin** required in `assertAllowed` if missing from `MethodRoles` — always register new methods.
## Methods ## Methods
| Method | Min role | Request args | Response (summary) | ### Session
|--------|----------|--------------|--------------------|
| `handshake` | viewer | `clientName`, `clientVersion`, optional `capability`, `adminProof` | role, auth, versions, features |
| `ping` | viewer | `{}` | `{ ok, pong, peerId }` |
| `getServerInfo` | viewer | `{}` | app, versions, host, peer count |
| `getAuthStatus` | viewer | `{}` | peerId, role, authMode, displayName |
| `listMessages` | viewer | `{ limit? }` | `{ messages: [...] }` |
| `getPresence` | viewer | `{}` | `{ peers: [...] }` |
| `postMessage` | operator | `{ text }` (12000 chars) | created message |
| `setDisplayName` | viewer | `{ name }` (140 chars) | updated label |
| `clearMessages` | admin | `{}` | success + system push |
| `mintInvite` | admin | `{ role?, ttlMs?, peerId?, alias? }` | `invite` (`pd1.…`), jti, exp |
| `listPeers` | admin | `{}` | live + policy peers |
| `revokePeer` | admin | `{ peerId }` (64 hex) | success; target dropped |
### Handshake request | Method | Role | Notes |
|--------|------|-------|
| `handshake` | viewer | Negotiate version + elevate role |
| `ping` | viewer | Hot |
| `getServerInfo` | viewer | Agent metadata |
| `getAuthStatus` | viewer | Peer role / auth mode |
| `setDisplayName` | viewer | Label for audit / UI |
```json ### Node
{
"clientName": "peardata",
"clientVersion": "0.1.0",
"capability": "<optional HMAC token>",
"adminProof": { "nonce": "<hex>", "mac": "<hex>" }
}
```
### Handshake response | Method | Role | Result |
|--------|------|--------|
| `getNodeInfo` | viewer | Hostname, CPUs, charts, sample interval |
| `getHealth` | viewer | `{ status, score, checks }` |
```json ### Metrics discovery & query
{
"success": true,
"protocol": "peardata/rpc",
"protocolVersion": 1,
"schemaVersion": 1,
"role": "operator",
"peerId": "<64 hex>",
"serverTime": 0,
"auth": { "mode": "capability", "role": "operator" },
"features": { "hmacAuth": true, "invites": true, "room": true }
}
```
### Auth modes (`auth.mode` / `session.authMode`) | Method | Role | Args | Result |
|--------|------|------|--------|
| `listContexts` | viewer | — | Context catalog |
| `getContext` | viewer | `{ id }` | Charts in context |
| `listCharts` | viewer | — | Netdata-ish chart map |
| `getChart` | viewer | `{ id }` | Chart summary |
| `queryData` | viewer | `{ chart, after, before, points, group, tier }` | Time series |
| `getAllMetrics` | viewer | `{ format: json\|prometheus\|shell }` | Latest export |
| Mode | How obtained | `after` / `before`: absolute unix seconds, or relative (negative = relative to `before`/`now`), Netdata-style.
|------|----------------|
| `viewer` | Default after connect with no grant |
| `capability` | Valid capability / invite |
| `seed` | Valid admin proof from `SERVER_SEED` |
| `allowlist` / registered | Elevated via policy / admin keys (implementation in ACL + policy) |
Exact labels depend on handshake path; UI shows `authMode` from `getAuthStatus`. ### Live subscriptions
## Pushes (server → client events) | Method | Role | Args |
|--------|------|------|
| `subscribeMetrics` | viewer | `{ charts: string[]\|['*'], intervalMs }` |
| `unsubscribeMetrics` | viewer | — |
| `subscribeAnomalies` | viewer | — |
| `unsubscribeAnomalies` | viewer | — |
| Push | Payload | ### Anomalies & alerts
| Method | Role | Notes |
|--------|------|-------|
| `listAnomalies` | viewer | Recent events |
| `listAlerts` / `getAlert` | viewer | State + config |
| `setAlertConfig` | operator | Upsert threshold |
| `ackAlert` | operator | Clear until next breach |
| `silenceAlert` | operator | Disable temporarily |
### Jobs
| Method | Role | Known jobs |
|--------|------|------------|
| `listJobs` | viewer | — |
| `runJob` | operator | `collectOnce`, `snapshot`, `gcBuffers` |
| `cancelJob` | operator | By job id |
### Admin
| Method | Role |
|--------|------|
| `mintInvite` | admin |
| `listPeers` | admin |
| `revokePeer` | admin |
| `exportSnapshot` | admin |
## Pushes (server → client)
| Event | Payload |
|-------|---------|
| `push:metrics` | `{ samples: MetricSample[] }` |
| `push:anomaly` | `AnomalyEvent` |
| `push:alert` | Alert transition |
| `push:health` | `HealthSnapshot` |
| `push:job` | `JobRecord` |
| `push:system` | Generic notices |
## Errors
Handlers throw `Error` with `.code`:
| Code | Meaning |
|------|---------| |------|---------|
| `push:message` | `{ id, peerId, displayName, text, ts }` | | `PERMISSION_DENIED` | Role too low |
| `push:presence` | `{ peers: [...] }` | | `RATE_LIMIT_EXCEEDED` | RPM exceeded |
| `push:system` | `{ type, ... }` e.g. `{ type: "cleared" }` | | `INVALID_ARGS` | Schema failure |
| `NOT_CONNECTED` | Client-side |
Registered via `rpc.event` / `session.push`. Client `connection.js` binds all `Pushes` values. | `CAPABILITY_*` | Invite/token issues |
## Invites
Envelope: `pd1.` + base64url(JSON):
```json
{
"v": 1,
"publicKeyHex": "<server>",
"capability": "<token>",
"role": "operator",
"jti": "...",
"expiresAt": null
}
```
Capability token: `base64url(payload).base64url(HMAC-SHA256)`.
Payload fields (canonical order for MAC): `v`, `role`, `peerId`, `exp`, `jti`, `iat`.
### Connection input classification
`classifyConnectionInput(string)` accepts:
| Input | Kind |
|-------|------|
| 64 hex chars | `publicKey` |
| `pd1.…` | `invite` (extracts key + capability) |
| other | error |
## Error codes
| Code | When |
|------|------|
| `RATE_LIMIT_EXCEEDED` | Peer over RPM budget |
| `PERMISSION_DENIED` | Role too low for method |
| `INVALID_ARGS` | Schema validation failed |
| `CONNECTION_TIMEOUT` | Client dial timeout |
| `RPC_ERROR` | Generic client-normalized failure |
| `UNKNOWN_ERROR` | Unclassified server handler error |
Clients should read `error.code` when present (`client/errors.js` preserves codes).
## Versioning policy ## Versioning policy
1. Document every method in this file. 1. Additive methods/fields: no version bump if old clients ignore unknowns.
2. Add `MethodRoles` entry before implementing handlers. 2. Rename/remove/change meaning: bump `PROTOCOL_VERSION`; reject or compat-negotiate in handshake.
3. Add `validateMethodArgs` case for mutating methods. 3. REST API versions (`v1`/`v2`/`v3`) are independent of RPC version but share the store.
4. Add brittle tests for pure helpers; integration test for critical paths.
5. Bump `PROTOCOL_VERSION` when existing response shapes break.
6. Bump `SCHEMA_VERSION` when validation semantics change meaningfully.
## Related ## Example session
- [ARCHITECTURE.md](./ARCHITECTURE.md) ```text
- [SECURITY.md](./SECURITY.md) client → handshake { clientName, clientVersion, capability? }
- [EXTENDING.md](./EXTENDING.md) server → { role, protocolVersion, auth }
- [TESTING.md](./TESTING.md) client → subscribeMetrics { charts: ['*'], intervalMs: 1000 }
server → push:metrics { samples: [...] } # ~1 Hz
client → queryData { chart: 'system.cpu', after: -300, points: 300 }
```
+14 -11
View File
@@ -2,17 +2,20 @@
| Doc | Audience | Contents | | Doc | Audience | Contents |
|-----|----------|----------| |-----|----------|----------|
| [GETTING-STARTED.md](./GETTING-STARTED.md) | New operators | Install, run, connect, systemd, troubleshooting | | [GETTING-STARTED.md](./GETTING-STARTED.md) | Operators | Install, run agent, REST, desktop, systemd |
| [DESKTOP.md](./DESKTOP.md) | UI developers | Pear shell, `pear-ctrl`, drag/resize, identity | | [ROADMAP.md](./ROADMAP.md) | Everyone | Phased MVP → advanced Netdata-class features |
| [ARCHITECTURE.md](./ARCHITECTURE.md) | Engineers | Planes, boot, middleware, module map | | [ARCHITECTURE.md](./ARCHITECTURE.md) | Engineers | Planes, PearDock mapping, module map |
| [PROTOCOL.md](./PROTOCOL.md) | Protocol owners | Methods, pushes, errors, versioning | | [PROTOCOL.md](./PROTOCOL.md) | Protocol owners | RPC methods, pushes, versioning |
| [SECURITY.md](./SECURITY.md) | Operators / security | Trust model, checklist, crypto, incidents | | [DATA-MODEL.md](./DATA-MODEL.md) | Engineers | Metrics, anomalies, alerts, jobs |
| [REST-API.md](./REST-API.md) | Integrators | Netdata-style `/api/v1\|v2\|v3` |
| [TECH-CHOICES.md](./TECH-CHOICES.md) | Engineers | Collector, charts, libraries |
| [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | Engineers | HyperDB / Corestore / Autobase sync design |
| [DESKTOP.md](./DESKTOP.md) | UI developers | Pear shell, titlebar, identity |
| [SECURITY.md](./SECURITY.md) | Operators / security | Trust model, secrets, hardening |
| [CONFIGURATION.md](./CONFIGURATION.md) | Operators | Full env vars + npm scripts | | [CONFIGURATION.md](./CONFIGURATION.md) | Operators | Full env vars + npm scripts |
| [TESTING.md](./TESTING.md) | Contributors | brittle suite, soak, manual QA | | [TESTING.md](./TESTING.md) | Contributors | brittle, soak, manual QA |
| [CI.md](./CI.md) | Maintainers | GitHub/Gitea pipelines | | [CI.md](./CI.md) | Maintainers | GitHub/Gitea pipelines |
| [RELEASE.md](./RELEASE.md) | Maintainers | Version, tag, tarball, rollback | | [RELEASE.md](./RELEASE.md) | Maintainers | Version, tag, tarball |
| [EXTENDING.md](./EXTENDING.md) | Product builders | Rebrand, new RPCs, persistence, fleet | | [EXTENDING.md](./EXTENDING.md) | Product builders | New charts, collectors, parents |
Start here if you are new: **[GETTING-STARTED.md](./GETTING-STARTED.md)**. Start here: **[GETTING-STARTED.md](./GETTING-STARTED.md)** · Product overview: **[../README.md](../README.md)**.
Root overview: **[../README.md](../README.md)**.
+147
View File
@@ -0,0 +1,147 @@
# REST API (Netdata-compatible)
PearMonitor agents expose an optional HTTP API modeled on **Netdata Agent** endpoints (`/api/v1`, `/api/v2`, `/api/v3`).
Default bind: `http://127.0.0.1:19999` (Netdatas classic port).
Disable: `PEARDATA_REST=0`.
Bind all interfaces (careful): `PEARDATA_REST_HOST=0.0.0.0`.
This is a **compatibility layer**, not a byte-for-byte Netdata clone. Core query/metadata paths are implemented for scripts, Grafana, and Prometheus scrapers.
## Quick examples
```bash
# Agent info
curl -s http://127.0.0.1:19999/api/v3/info | jq
# Chart catalog (v1 style)
curl -s http://127.0.0.1:19999/api/v1/charts | jq '.charts | keys'
# Last 60s of CPU (v3)
curl -s 'http://127.0.0.1:19999/api/v3/data?chart=system.cpu&after=-60&points=60' | jq
# Contexts
curl -s http://127.0.0.1:19999/api/v3/contexts | jq
# Prometheus export
curl -s 'http://127.0.0.1:19999/api/v3/allmetrics?format=prometheus'
# Health
curl -s http://127.0.0.1:19999/api/v3/health | jq
```
## Endpoint matrix
### Info & identity
| Method | Path | Notes |
|--------|------|-------|
| GET | `/` or `/api` | Service index + P2P pubkey |
| GET | `/api/v1/info` | Agent info |
| GET | `/api/v2/info` | same |
| GET | `/api/v3/info` | **preferred** |
| GET | `/api/v3/versions` | Agent / protocol / API versions |
| GET | `/api/v3/me` | Anonymous REST identity note |
| GET | `/api/v3/settings` | Runtime knobs |
| GET | `/api/v3/config` | alias of settings |
| GET | `/health`, `/api/v1/health`, `/api/v3/health` | Aggregate health |
### Nodes
| Method | Path |
|--------|------|
| GET | `/api/v2/nodes` |
| GET | `/api/v3/nodes` |
| GET | `/api/v3/node_instances` |
| GET | `/api/v3/stream_path` |
Single-agent MVP returns one node (this host). Parent/fleet aggregation is roadmap.
### Contexts & charts
| Method | Path | Notes |
|--------|------|-------|
| GET | `/api/v3/contexts` | Context map |
| GET | `/api/v2/contexts` | same |
| GET | `/api/v3/context?context=` | One context |
| GET | `/api/v1/charts` | Full chart summary (legacy but useful) |
| GET | `/api/v1/chart?chart=` | One chart |
### Data queries
| Method | Path | Query params |
|--------|------|--------------|
| GET | `/api/v3/data` | `chart` or `context`, `after`, `before`, `points`, `group`, `tier`, `format` |
| GET | `/api/v2/data` | same |
| GET | `/api/v1/data` | same (legacy) |
**Params (Netdata-style)**
| Param | Default | Description |
|-------|---------|-------------|
| `chart` / `context` | required | Chart id or context id |
| `after` | `-60` | Absolute unix sec, or relative (negative) |
| `before` | `0` (now) | Absolute or relative |
| `points` | `60` | Max points returned (downsampled) |
| `group` | `average` | `average` \| `min` \| `max` \| `sum` |
| `tier` | `0` | `0` = 1s buffer, `1` = downsampled |
| `format` | `json` | `json` \| `csv` \| `array` |
### Search & weights
| Method | Path | Notes |
|--------|------|-------|
| GET | `/api/v3/q?q=` | Full-text over chart ids/titles |
| GET | `/api/v3/weights` | MVP: health-derived scores |
### Alerts
| Method | Path |
|--------|------|
| GET | `/api/v3/alerts` |
| GET | `/api/v2/alerts` |
| GET | `/api/v1/alarms` |
| GET | `/api/v3/alert_transitions` |
| GET | `/api/v3/alert_config` |
| GET | `/api/v3/variable` |
### Export & badges
| Method | Path | Params |
|--------|------|--------|
| GET | `/api/v3/allmetrics` | `format=json\|prometheus\|shell` |
| GET | `/api/v1/allmetrics` | same |
| GET | `/api/v3/badge.svg` | `chart`, `dimensions`, `label` |
### Functions (stub)
| Method | Path | Notes |
|--------|------|-------|
| GET | `/api/v3/functions` | Lists job names; execution remains P2P `runJob` for auth |
## Auth model (REST)
- **Default:** localhost-only, no bearer required (like a typical Netdata agent bind).
- **P2P remains the secure remote path** (Noise + roles).
- If you bind `0.0.0.0`, put REST behind a firewall, reverse proxy, or Holesail tunnel — do not expose raw metrics to the internet.
- Future: optional bearer gate (`/api/v3/bearer_protection` parity).
## CORS
`Access-Control-Allow-Origin` defaults to `*` (override with `PEARDATA_REST_CORS`).
## Compatibility notes
| Netdata | PearData MVP |
|---------|--------------|
| Full ML weights / metric correlations | Simplified health weights |
| Multi-node parent streaming | Single node; parent planned |
| Cloud POST `/api/v3/spaces/.../data` | Not implemented (agent GET style only) |
| Every chart Netdata ships | Core system charts (expanding) |
| Functions execute via HTTP | Listed; run via P2P jobs |
## Implementation
- Router: `server/rest/routes.js`
- Server: `server/rest/http-server.js`
- Exporters: `server/rest/formatters.js`
+114
View File
@@ -0,0 +1,114 @@
# PearData Roadmap
Phased plan from MVP agent → Netdata-class fleet observability on pure P2P.
## Guiding principles
1. **Instant value** — connect a pubkey, see live charts in seconds.
2. **Agent efficiency** — stay in Netdatas ballpark for CPU/RAM overhead.
3. **PearDock patterns** — HyperDHT identity, protomux-rpc, roles, `pd1.` invites.
4. **Dual access** — P2P desktop + Netdata-style REST (`/api/v1|v2|v3`).
5. **Ecosystem glue** — ready for PearDock / PearVirt / HoneyPeer / BareOS later.
---
## Phase 0 — Foundation ✅ (this repo)
| Item | Status |
|------|--------|
| Copy pear-app-template → PearData rebrand (`pd1.`, `PEARDATA_*`, `peardata/rpc`) | Done |
| Shared protocol + schema + metrics catalog | Done |
| Agent collector (CPU/RAM/load/net/disk/processes) @ ~1s | Done |
| In-memory tiered store (1s + downsample) | Done |
| Threshold anomaly engine + health | Done |
| P2P RPC surface (query, subscribe, alerts, jobs, ACL) | Done |
| Netdata-compatible REST v1/v2/v3 (local :19999) | Done |
| Pear desktop fleet overview + live canvas charts | Done |
| Docs: architecture, protocol, data model, REST, roadmap, tech choices | Done |
| systemd unit + CI skeletons | Done |
**MVP acceptance criteria**
- [x] Agent exposes metrics over P2P by public key
- [x] Desktop connects to one+ agents and shows live CPU/RAM/net/disk
- [x] REST `/api/v3/data?chart=system.cpu&after=-60` returns series
- [x] Viewer vs admin (pubkey / seed / `pd1.` invite)
- [x] Simple threshold anomalies pushed to clients
---
## Phase 1 — Hardening & UX (next)
- Persistent client peer bookmarks + aliases in desktop
- Multi-peer compare mode (overlay 24 nodes on one chart)
- Reconnection / offline banners with last-known samples
- Agent process title / `peardata-agent` binary naming polish
- Expand collectors: per-core CPU, per-iface net, mount disk space
- Alert silence TTL auto-reenable
- brittle tests for collector, store query, REST routes
- One-line install script (`curl | bash`) for Linux agents
---
## Phase 2 — Storage & history (HyperDB)
See **[STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md)** for the full design (from Holepunch `hyperdb` / workshop / Autobase patterns).
- HyperDB (bee + Corestore) for metadata, alerts, peer-links, **warm** downsampled points
- Keep memory ring for hot 1s path; do **not** tx every sample into HyperDB
- Hyperswarm `store.replicate` for linked-node / desktop seed sync
- Configurable retention (hours@1s memory, days@1m HyperDB, weeks@1h)
- Historical query: memory miss → HyperDB range
- Optional later: Autobase multi-writer parents; Rocks engine for local-only speed
- Export snapshot job → JSON / Prometheus remote write (optional)
---
## Phase 3 — Discovery & apps
- Docker / container auto-discovery (cgroup + Docker API)
- Common service collectors (nginx, postgres, redis) as plugins
- PearDock integration: container metrics from dock peers
- PearVirt / BareOS node metric adapters
- Holesail optional expose of REST UI per agent
---
## Phase 4 — Smarter anomalies & fleet
- Anomaly scoring + chart highlighting in UI
- Lightweight ML job (`runJob` retrain) — start with streaming z-score / k-means
- Fleet-wide composite views and correlation (`/api/v3/weights` depth)
- Parent peer aggregation (P2P “parent” without central SaaS)
- Push notifications (desktop + optional webhook)
---
## Phase 5 — Product polish
- Signed release artifacts (agent + Pear desktop)
- Role templates (viewer / SRE operator / admin)
- Plugin SDK documentation
- Grafana datasource (REST) cookbook
- HoneyPeer presence for agent directory (opt-in)
---
## Non-goals (for now)
- Replacing Netdata Cloud SaaS multi-tenant product
- Full byte-identical Netdata internal DB format
- Shipping a browser-only public dashboard without auth by default (REST stays localhost unless explicitly bound)
---
## Milestone checklist (operators)
| Milestone | You can… |
|-----------|----------|
| M0 | `npm run start:server` + `curl localhost:19999/api/v3/info` |
| M1 | Pear UI live charts from agent pubkey |
| M2 | Mint `pd1.` operator invite; revoke peer |
| M3 | Historical scrub 1h@1s via REST + RPC |
| M4 | Container charts from Docker hosts |
| M5 | Parent peer rolling up a homelab fleet |
+8 -1
View File
@@ -28,13 +28,20 @@
| Identity file mode | `0600` | | Identity file mode | `0600` |
| Data directory | local `./data` (not committed) | | Data directory | local `./data` (not committed) |
## REST API exposure
- Default bind is **localhost only** (`127.0.0.1:19999`).
- REST is intentionally open on that bind (Netdata-agent style) — **do not** set `PEARDATA_REST_HOST=0.0.0.0` without a firewall, reverse proxy, or Holesail tunnel.
- Prefer **P2P + roles** for remote multi-operator access; use REST for local scrapers/Grafana.
## Production checklist ## Production checklist
- [ ] Never set `PEARDATA_INSECURE_OPEN_ADMIN` outside local demos - [ ] Never set `PEARDATA_INSECURE_OPEN_ADMIN` outside local demos
- [ ] Keep `PEARDATA_DEFAULT_ROLE=viewer` - [ ] Keep `PEARDATA_DEFAULT_ROLE=viewer`
- [ ] Prefer invites over sharing `SERVER_SEED` - [ ] Prefer `pd1.` invites over sharing `SERVER_SEED`
- [ ] Use short `ttlMs` for high-privilege invites when practical - [ ] Use short `ttlMs` for high-privilege invites when practical
- [ ] Set `PEARDATA_ALLOWLIST` if only known operators should dial - [ ] Set `PEARDATA_ALLOWLIST` if only known operators should dial
- [ ] Keep REST on localhost unless explicitly secured
- [ ] Back up `SERVER_SEED` offline; rotate by redeploying a new keypair (clients must re-dial) - [ ] Back up `SERVER_SEED` offline; rotate by redeploying a new keypair (clients must re-dial)
- [ ] Persist `data/` with mode `0700`; `audit.log` may contain peer ids - [ ] Persist `data/` with mode `0700`; `audit.log` may contain peer ids
- [ ] Run under systemd with `ProtectSystem` / `NoNewPrivileges` (see `deploy/`) - [ ] Run under systemd with `ProtectSystem` / `NoNewPrivileges` (see `deploy/`)
+290
View File
@@ -0,0 +1,290 @@
# HyperDB storage & linked-node sync
How PearData should adopt Holepunchs **HyperDB + Corestore + Hyperswarm (+ Autobase)** stack for durable storage and P2P sync between linked agents — based on patterns in local clones under `holepunchto_repos` (`hyperdb`, `hyperdb-workshop`, `hyperdb-autobase-workshop`, `corestore`, `hyperswarm`, `autobee`, `pear-hyperdb`).
## Why HyperDB (not “just SQLite”)
| Need | HyperDB fit |
|------|-------------|
| Typed collections + indexes | Hyperschema + `@ns/collection` + secondary indexes |
| Local high-perf | `HyperDB.rocks(path, def)` |
| P2P replicate | `HyperDB.bee(hypercore, def, { autoUpdate })` over Corestore |
| Multi-writer HA parents | Autobase whose **view** is HyperDB (`extension: false`) |
| Same query API local + remote | Workshops prove one `Registry` class works for both |
PearDatas current `server/services/store.js` is an **in-memory ring**. HyperDB replaces durability + sync; the in-memory tier stays as the **hot 1s path**.
## Stack mapping (from Holepunch repos)
```mermaid
flowchart TB
subgraph Agent
COL[collector 1s] --> HOT[Memory tier0 ring]
COL --> DS[Downsample]
DS --> HDB[(HyperDB bee/rocks)]
POL[peer-policy / alerts / links] --> HDB
HOT --> RPC[protomux-rpc + REST]
HDB --> RPC
end
subgraph Sync
CS[Corestore] --> HDB
SW[Hyperswarm] -->|store.replicate| CS
AB[Autobase optional] -->|view| HDB
end
CHILD[Linked child agent] -.->|discoveryKey| SW
PARENT[Parent / peer agent] -.-> SW
DESK[Pear desktop cache] -.-> SW
```
| Component | Repo pattern | PearData use |
|-----------|--------------|--------------|
| **Hyperschema + hyperdb/builder** | `hyperdb-workshop/build.js` | `spec/` codegen for collections |
| **HyperDB.bee** | workshop `Registry` | Replicable agent DB |
| **HyperDB.rocks** | `hyperdb` README | Optional local-only fast index |
| **Corestore** | workshop `bin.js` | Named cores: `metrics-meta`, `alerts`, … |
| **Hyperswarm** | `swarm.join(discoveryKey)` + `store.replicate(conn)` | Link nodes / seed DB |
| **protomux-rpc** | already in PearData | Control plane (unchanged) |
| **Autobase + hyperdispatch** | `hyperdb-autobase-workshop` | Multi-writer **parent** / HA registry |
| **autobee** | experimental multiwriter bee | Alternative later; prefer Autobase+HyperDB view for now |
| **pear-hyperdb** | Pear-shaped Model wrapper | Optional UX for desktop-local rocks |
## Critical design rule: dont put 1s samples in HyperDB txs
HyperDB is an **indexable document DB** (put/get/find + flush). Writing every chart every second as HyperDB transactions will:
- Amplify Rocks/Bee write cost
- Create huge replication chatter
- Fight Netdata-class overhead goals
**Split planes:**
| Plane | Storage | Sync |
|-------|---------|------|
| **Hot live (≤1h @ 1s)** | Memory ring (current) | P2P `push:metrics` (current) |
| **Warm history (downsampled)** | Hypercore append **or** HyperDB rows keyed `(chart, tsBucket)` | Corestore replicate |
| **Metadata** (peers, alerts, labels, jobs, ACL cache) | **HyperDB** | Corestore replicate |
| **Fleet / parent consensus** | Autobase → HyperDB view | Swarm on autobase discoveryKey |
## Proposed HyperDB schema (`@peardata/*`)
Modeled after workshop `build.js` namespaces.
### Collections
```text
@peardata/node
key: nodeId (string / pubkey hex)
fields: hostname, platform, arch, cpus, agentVersion, labels{}, updatedAt
@peardata/peer-link
key: [localNodeId, remotePublicKey]
fields: role, alias, discoveryKey?, linkedAt, lastSeen, syncMode (push|pull|both)
@peardata/alert-config
key: id
fields: chart, dimension, warn, crit, comparator, enabled, info
@peardata/alert-event
key: [id, ts] # or ulid
fields: severity, value, threshold, message, cleared
@peardata/metric-point # WARM tier only (e.g. 1m buckets)
key: [chart, ts]
fields: context, values{} (map), tier
@peardata/job
key: id
fields: name, status, startedAt, finishedAt, result?
```
### Indexes
```text
@peardata/node-by-hostname → node.hostname
@peardata/peer-link-by-remote → peer-link.remotePublicKey
@peardata/alert-event-by-chart → alert-event.chart + ts
@peardata/metric-point-by-context → metric-point.context + ts
```
Rebuild with:
```bash
node scripts/build-db.js # Hyperschema + HyperDB.toDisk → spec/
```
## Agent integration shape
Follow `hyperdb-workshop` / `pear-hyperdb` Model pattern:
```text
server/
db/
build.js # schema codegen
spec/hyperschema/
spec/hyperdb/
model.js # PearDataModel: putNode, linkPeer, queryWarm, …
replicate.js # Hyperswarm join + store.replicate
services/
store.js # HOT memory (keep)
store-hyperdb.js # WARM + metadata facade used by queryData/REST
```
### Boot (single-writer agent — Phase 2)
```js
const store = new Corestore(dataDir + '/corestore')
const swarm = new Hyperswarm({ keyPair: await store.createKeyPair('swarm') })
swarm.on('connection', (conn) => store.replicate(conn))
const metaCore = store.get({ name: 'peardata-meta' })
const db = HyperDB.bee(metaCore, spec, { autoUpdate: true })
// announce for linked peers / desktop seeders
swarm.join(metaCore.discoveryKey, { server: true, client: true })
```
Collector path:
1. Ingest → memory tier0 (unchanged)
2. Every N samples → downsample → `tx.insert('@peardata/metric-point', …); tx.flush()`
3. Alert transitions → `alert-event` collection
4. `queryData` / REST: memory first, then HyperDB range scan for older windows
### Auth note
HyperDB replication shares **capability to read the core**, not PearData RPC roles. Keep:
- **Noise + MethodRoles** for mutating RPC (`setAlertConfig`, `runJob`)
- Replication topic optionally gated (only invite-linked peers get discoveryKey / capability)
- Do **not** announce writable cores to the public swarm without encryption / allowlist
## Linked nodes: sync modes
PearData “links” are first-class `@peardata/peer-link` rows + swarm topics.
| Mode | Behavior |
|------|----------|
| **Pull** | Local agent opens remote DB by key (`HyperDB.bee(store.get({ key }), spec, { writable: false, autoUpdate: true })`) and replicates |
| **Push** | Remote peers allowed to replicate our meta/warm cores (seed) |
| **Both** | Mutual swarm join (homelab mesh) |
| **Parent aggregate** | Parent pulls many children; stores namespaced copies or Autobase fleet view |
Discovery:
1. Desktop/admin mints link → stores remote pubkey + optional `dbKey` (z32/hex)
2. Agent joins `discoveryKey` of that core
3. On connection: `store.replicate(conn)` (workshop pattern)
4. `clone.watch` / `autoUpdate` refreshes HyperDB indexes for REST/RPC queries
This is the same pattern as workshop §3.1 Lookups — **read path is swarm + HyperDB**, not constant RPC polling.
## Parent / HA (Phase 4) — Autobase workshop
When you need multi-writer fleet registry or HA parents:
- Autobase bootstrap key shared across parent instances
- `open: (store) => HyperDB.bee(store.get('db-view'), spec, { extension: false, autoUpdate: true })`
- hyperdispatch ops: `add-writer`, `put-alert`, `put-link`, `ingest-rollup`
- RPC (existing PearData / workshop) appends ops; apply mutates HyperDB view
- Clients dial **any** writer; view key stays stable
Do **not** “backup” by copying Corestore folders (workshop warning). Rotate writers via Autobase instead.
## What stays on protomux-rpc
| Keep on RPC | Why |
|-------------|-----|
| Live `push:metrics` | Sub-second UX; not DB-shaped |
| `subscribeMetrics` / handshake / ACL | Session security |
| `runJob`, mintInvite | Mutating control |
| On-demand `queryData` for hot window | Memory path |
| Move to HyperDB (+ replicate) | Why |
|-------------------------------|-----|
| Peer links, aliases, labels | Shared fleet truth |
| Alert config + history | Durable, queryable |
| Warm/cold metric buckets | History without central server |
| Node inventory | Parent `/api/v3/nodes` |
## Phased delivery (recommended)
### Phase A — Local durability (rocks or bee, no swarm yet)
1. Add `hyperdb`, `hyperschema`, `corestore` deps
2. `scripts/build-db.js` + `spec/`
3. Persist alert configs, peer policy, warm downsample into HyperDB
4. REST/RPC history falls back to HyperDB after memory miss
5. Keep REST localhost semantics
### Phase B — Link & replicate
1. Hyperswarm alongside HyperDHT RPC (or reuse connections carefully — often **separate swarm for store.replicate**)
2. `linkPeer` / `unlinkPeer` RPCs write `@peardata/peer-link`
3. Desktop can seed/cache agent DB by key for offline scrubbing
4. Document `pd1.` invite vs **db discovery key** (two layers)
### Phase C — Parent fleet
1. Parent process pulls N child DB keys
2. Composite REST `/api/v3/nodes` + scoped `/api/v3/data`
3. Optional Autobase for parent HA
### Phase D — Polish
1. Encryption at rest (`encryptionKey` on cores / autobee)
2. Blind-peer seeding for always-on warm history
3. Grafana: already have REST Prometheus export; optionally expose hypercore-stats
## Concrete file plan (when implementing)
```text
peardata/
scripts/build-db.js
spec/hyperschema/
spec/hyperdb/
server/db/model.js
server/db/replicate.js
server/services/store-hyperdb.js
docs/STORAGE-HYPERDB.md ← this file
```
Deps (approximate):
```json
"hyperdb": "^6",
"hyperschema": "^1",
"corestore": "^7",
"hyperswarm": "^4",
"autobase": "^7",
"hyperdispatch": "^1"
```
(`autobee` only if you prefer that multiwriter path over Autobase+HyperDB view.)
## Relationship to current PearData roadmap
| Roadmap item | HyperDB role |
|--------------|--------------|
| Phase 2 tiered storage | Warm/cold collections |
| Phase 3 PearDock/containers | Extra collections / indexes |
| Phase 4 parent peer | Autobase + replicated views |
| REST v3 historical queries | `find` ranges on `@peardata/metric-point` |
## References (local clones)
| Path under `holepunchto_repos` | Takeaway |
|--------------------------------|----------|
| `hyperdb/README.md` | rocks vs bee, find/get/tx, autoUpdate |
| `hyperdb-workshop` | Schema builder, Corestore+Swarm replicate, RPC inserts |
| `hyperdb-autobase-workshop` | Multi-writer view, hyperdispatch, HA |
| `corestore/README.md` | `store.replicate(conn)`, namespacing |
| `pear-hyperdb` | Thin Model wrapper pattern for Pear apps |
| `autobee` | Experimental multiwriter bee alternative |
## Decision summary
1. **Yes — incorporate HyperDB** for metadata + warm history + linked-node sync.
2. **Keep memory + RPC pushes** for live 1s Netdata feel.
3. **Sync linked nodes via Corestore replication on Hyperswarm**, not by streaming every sample over RPC.
4. **Use Autobase+HyperDB** when you need multi-writer parents / HA — same pattern as Holepunchs own workshops.
5. **Treat discovery keys as sensitive** — link only invited peers; RPC AuthZ remains authoritative for mutations.
+98
View File
@@ -0,0 +1,98 @@
# Tech choices
Recommendations for collection, streaming, visualization, and reusable libraries.
## Transport & identity
| Choice | Why |
|--------|-----|
| **HyperDHT** | Same peer identity + hole-punching model as PearDock |
| **protomux-rpc** | Multiplexed request/response + server push events |
| **compact-encoding JSON** | Fast enough for control + 1s metric batches; shared with template |
| **Ed25519 seeds** | Agent address = pubkey; admin proof via HMAC from seed |
### Streaming metrics
MVP uses **protomux-rpc `event` pushes** (`push:metrics`) with per-session subscribe + throttle.
If fleet scale demands it later:
1. Dedicated protomux channel with binary packs (Float64 arrays)
2. Hypercore / Hyperbee for durable streams a parent can replicate
3. Extract a shared **`pearrpc`** package (session middleware + hot-path helpers) from this template
For MVP, JSON pushes keep the stack simple and debuggable.
## Collection
| Option | Verdict |
|--------|---------|
| **Node `os` + `/proc` (chosen)** | Zero native deps, good enough for MVP, low overhead |
| `node-os-utils` | Convenient but extra dep / less control |
| Native bindings (netdata collectors, `systeminformation`) | Higher fidelity; consider Phase 2 for Windows depth |
| Shell out to `vmstat`/`iostat` | Avoid on hot path |
**Linux:** `/proc/meminfo`, `/proc/loadavg`, `/proc/net/dev`, `/proc/diskstats`
**macOS/Windows:** `os` fallbacks (CPU%, freemem, loadavg where available); I/O/net rates may be zero until platform collectors land.
Target overhead: single timer, no child processes per tick, ring buffers only.
## Storage
| Tier | Implementation |
|------|----------------|
| Now | In-memory ring (`server/services/store.js`) |
| Next | **HyperDB** + Corestore (+ Hyperswarm replicate) — see [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) |
| HA / multi-writer parents | Autobase with HyperDB view (`extension: false`) |
| Avoid | SQLite as primary P2P store; copying corestore folders as “backup” |
### HyperDB engines
| Engine | Sync | Use |
|--------|------|-----|
| `HyperDB.bee(core, spec)` | P2P via Hypercore | Agent meta + warm history + linked peers |
| `HyperDB.rocks(path, spec)` | Local only | Fast local index / desktop cache (`pear-hyperdb` style) |
## Visualization (desktop)
| Option | Verdict |
|--------|---------|
| **Canvas sparklines (chosen MVP)** | No extra deps inside Pear; snappy for 1s updates |
| **uPlot** | Best next step for Netdata-like interactive charts (tiny, fast) |
| Chart.js / ECharts | Heavier; fine for secondary views |
| Grafana via REST | External; use `/api/v3/data` + Prometheus export |
## REST
Built-in Node `http` (no Express) — small attack surface, enough for Netdata-style GET APIs.
## Packaging
| Piece | Approach |
|-------|----------|
| Agent | Node 20+ , systemd unit `deploy/peardata.service` |
| Desktop | Pear (`pear-electron` + `pear-bridge`) |
| Invites | `pd1.` tokens (PearDock-style) |
| Installer | Phase 1 one-liner script |
## Reusable library extraction (recommended later)
From this codebase / PearDock patterns:
| Package | Contents |
|---------|----------|
| `@pear/rpc-session` or `pearrpc` | PeerSession middleware, ACL, rate limit, audit hooks |
| `pear-metrics-wire` | Context/chart catalog types + query args |
| `pear-invite` | `pd1.` encode/decode + capability HMAC |
Keep MIT-clean relative to AGPL PearDock sources (template already does).
## Integration seams
| Ecosystem | Seam |
|-----------|------|
| PearDock | Container chart plugin reading dock RPC |
| PearVirt | VM CPU/mem contexts |
| HoneyPeer | Optional agent directory announcements |
| BareOS | Bare-compatible collector build |
| Holesail | Tunnel REST or future agent web UI |
+8 -6
View File
@@ -14,8 +14,10 @@
|------|----------| |------|----------|
| `test/acl.test.js` | Role hierarchy, `assertAllowed` | | `test/acl.test.js` | Role hierarchy, `assertAllowed` |
| `test/crypto-auth.test.js` | MAC key, capabilities, admin proof, invites, classify input | | `test/crypto-auth.test.js` | MAC key, capabilities, admin proof, invites, classify input |
| `test/protocol.test.js` | Constants, `MethodRoles`, schema validators | | `test/protocol.test.js` | Constants, monitoring `MethodRoles`, schema validators |
| `test/integration.test.js` | Live HyperDHT server + client handshake, post, push | | `test/store.test.js` | Metric ring buffer ingest + query |
| `test/rest.test.js` | Netdata-style `/api/v3` routes |
| `test/integration.test.js` | Live HyperDHT agent + handshake + metrics query |
```bash ```bash
npm test npm test
@@ -27,10 +29,10 @@ SKIP_INTEGRATION=1 npm test
## Integration test behavior ## Integration test behavior
1. Starts an ephemeral HyperDHT server in-process 1. Starts metrics pipeline + ephemeral HyperDHT agent in-process
2. Sets `PEARDATA_INSECURE_OPEN_ADMIN=1` for the process (restored in teardown) 2. Sets `PEARDATA_INSECURE_OPEN_ADMIN=1` for the process (restored in teardown)
3. Dials as a client, handshakes, posts a message, asserts push delivery 3. Dials as a client, handshakes, lists charts, queries `system.cpu`
4. Tears down sockets / DHT 4. Tears down collector / sockets / DHT
Requires outbound/inbound UDP for HyperDHT. If the test hangs or fails on a locked-down network, use `SKIP_INTEGRATION=1`. Requires outbound/inbound UDP for HyperDHT. If the test hangs or fails on a locked-down network, use `SKIP_INTEGRATION=1`.
@@ -78,7 +80,7 @@ When adding an RPC method:
| Desktop chrome | `npm start` — drag titlebar, resize edges, min/max/close | | Desktop chrome | `npm start` — drag titlebar, resize edges, min/max/close |
| Admin connect | Paste key + `SERVER_SEED` in UI | | Admin connect | Paste key + `SERVER_SEED` in UI |
| Invite connect | Paste `pd1.…` without seed | | Invite connect | Paste `pd1.…` without seed |
| Viewer denial | Public key only → `postMessage` fails with permission error | | Viewer denial | Public key only → `runJob` / `mintInvite` fail with permission error |
| Health | With server up: `SERVER_PUBLIC_KEY=… npm run healthcheck` | | Health | With server up: `SERVER_PUBLIC_KEY=… npm run healthcheck` |
| Soak | `SERVER_PUBLIC_KEY=… SERVER_SEED=… npm run soak` | | Soak | `SERVER_PUBLIC_KEY=… SERVER_SEED=… npm run soak` |
+240 -3
View File
@@ -11,16 +11,21 @@
"dependencies": { "dependencies": {
"b4a": "^1.8.1", "b4a": "^1.8.1",
"compact-encoding": "^3.3.0", "compact-encoding": "^3.3.0",
"corestore": "^7.11.1",
"dotenv": "^17.4.2", "dotenv": "^17.4.2",
"graceful-goodbye": "^1.3.3", "graceful-goodbye": "^1.3.3",
"hypercore-crypto": "^3.7.0", "hypercore-crypto": "^3.7.0",
"hypercore-id-encoding": "^1.3.0",
"hyperdb": "^6.7.0",
"hyperdht": "^6.33.0", "hyperdht": "^6.33.0",
"hyperschema": "^1.21.0",
"pear-bridge": "^1.2.5", "pear-bridge": "^1.2.5",
"pear-electron": "^1.7.28", "pear-electron": "^1.7.28",
"pear-pipe": "^1.0.6", "pear-pipe": "^1.0.6",
"pear-run": "^1.0.8", "pear-run": "^1.0.8",
"protomux": "^3.11.0", "protomux": "^3.11.0",
"protomux-rpc": "^1.10.0", "protomux-rpc": "^1.10.0",
"ready-resource": "^1.2.0",
"safety-catch": "^1.0.3", "safety-catch": "^1.0.3",
"z32": "^1.1.0" "z32": "^1.1.0"
}, },
@@ -825,6 +830,12 @@
} }
} }
}, },
"node_modules/big-sparse-array": {
"version": "1.0.3",
"resolved": "https://registry.npmjs.org/big-sparse-array/-/big-sparse-array-1.0.3.tgz",
"integrity": "sha512-6RjV/3mSZORlMdpUaQ6rUSpG637cZm0//E54YYGtQg1c1O+AbZP8UTdJ/TchsDZcTVLmyWZcseBfp2HBeXUXOQ==",
"license": "MIT"
},
"node_modules/binary-stream-equals": { "node_modules/binary-stream-equals": {
"version": "1.0.0", "version": "1.0.0",
"resolved": "https://registry.npmjs.org/binary-stream-equals/-/binary-stream-equals-1.0.0.tgz", "resolved": "https://registry.npmjs.org/binary-stream-equals/-/binary-stream-equals-1.0.0.tgz",
@@ -929,6 +940,15 @@
"integrity": "sha512-9z8TZaGM1pfswYeXrUpzPrkx8UnWYdhJclsiYMm6x/w5+nN+8Tf/LnAgfLGQCm59qAOxU8WwHEq2vNwF6i4j+Q==", "integrity": "sha512-9z8TZaGM1pfswYeXrUpzPrkx8UnWYdhJclsiYMm6x/w5+nN+8Tf/LnAgfLGQCm59qAOxU8WwHEq2vNwF6i4j+Q==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/codecs": {
"version": "3.1.0",
"resolved": "https://registry.npmjs.org/codecs/-/codecs-3.1.0.tgz",
"integrity": "sha512-Dqx8NwvBvnMeuPQdVKy/XEF71igjR5apxBvCGeV0pP1tXadOiaLvDTXt7xh+/5wI1ASB195mXQGJbw3Ml4YDWQ==",
"license": "MIT",
"dependencies": {
"b4a": "^1.6.3"
}
},
"node_modules/compact-encoding": { "node_modules/compact-encoding": {
"version": "3.3.0", "version": "3.3.0",
"resolved": "https://registry.npmjs.org/compact-encoding/-/compact-encoding-3.3.0.tgz", "resolved": "https://registry.npmjs.org/compact-encoding/-/compact-encoding-3.3.0.tgz",
@@ -963,6 +983,24 @@
"dev": true, "dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/corestore": {
"version": "7.11.1",
"resolved": "https://registry.npmjs.org/corestore/-/corestore-7.11.1.tgz",
"integrity": "sha512-X6UsGFyHeAcqffTqgoDV7WDZjhfvolWabwzws6GbdqeXQY0CTM6gNpNMGEi6RGvAdZDPnFaboF5jFBy1lw1rDA==",
"license": "MIT",
"dependencies": {
"b4a": "^1.6.7",
"bare-events": "^2.8.3",
"hypercore": "^11.32.0",
"hypercore-crypto": "^3.4.2",
"hypercore-errors": "^1.4.0",
"hypercore-id-encoding": "^1.3.0",
"ready-resource": "^1.1.1",
"sodium-universal": "^5.0.1",
"streamx": "^2.26.0",
"which-runtime": "^1.2.1"
}
},
"node_modules/crc-native": { "node_modules/crc-native": {
"version": "1.1.8", "version": "1.1.8",
"resolved": "https://registry.npmjs.org/crc-native/-/crc-native-1.1.8.tgz", "resolved": "https://registry.npmjs.org/crc-native/-/crc-native-1.1.8.tgz",
@@ -982,6 +1020,26 @@
"crc-native": "^1.0.3" "crc-native": "^1.0.3"
} }
}, },
"node_modules/debounceify": {
"version": "1.1.0",
"resolved": "https://registry.npmjs.org/debounceify/-/debounceify-1.1.0.tgz",
"integrity": "sha512-eKuHDVfJVg+u/0nPy8P+fhnLgbyuTgVxuCRrS/R7EpDSMMkBDgSes41MJtSAY1F1hcqfHz3Zy/qpqHHIp/EhdA==",
"license": "MIT"
},
"node_modules/device-file": {
"version": "2.3.1",
"resolved": "https://registry.npmjs.org/device-file/-/device-file-2.3.1.tgz",
"integrity": "sha512-bmON44lwxJPle9N2OcH4tqM44pMGZKT8G6OkzXkz0urvqQ9LKkoQdTS6w0ztYfmLdUE27R4UUmx0+y2gEz4Jug==",
"license": "Apache-2.0",
"dependencies": {
"b4a": "^1.6.7",
"bare-fs": "^4.0.1",
"bare-path": "^3.0.0",
"fd-lock": "^2.1.0",
"fs-native-extensions": "^1.4.0",
"ready-resource": "^1.2.0"
}
},
"node_modules/dht-rpc": { "node_modules/dht-rpc": {
"version": "6.27.0", "version": "6.27.0",
"resolved": "https://registry.npmjs.org/dht-rpc/-/dht-rpc-6.27.0.tgz", "resolved": "https://registry.npmjs.org/dht-rpc/-/dht-rpc-6.27.0.tgz",
@@ -1056,6 +1114,24 @@
"integrity": "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==", "integrity": "sha512-/d9sfos4yxzpwkDkuN7k2SqFKtYNmCTzgfEpz82x34IM9/zc8KGxQoXg1liNC/izpRM/MBdt44Nmx41ZWqk+FQ==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/fd-lock": {
"version": "2.2.0",
"resolved": "https://registry.npmjs.org/fd-lock/-/fd-lock-2.2.0.tgz",
"integrity": "sha512-Il4jWBhjjgvUg+z8d0bC7ncXqB42caqKTUpnZ9jpcqiPmw8bkIixt7Kic1czbZPMxAQD/9kEqfZ5Dq77nSll0w==",
"license": "Apache-2.0",
"dependencies": {
"bare-fs": "^4.5.0",
"fs-native-extensions": "^1.4.4",
"ready-resource": "^1.2.0",
"resource-on-exit": "^1.0.0"
}
},
"node_modules/flat-tree": {
"version": "1.13.0",
"resolved": "https://registry.npmjs.org/flat-tree/-/flat-tree-1.13.0.tgz",
"integrity": "sha512-fT3HIuCPwHhFgJ20QYzDHgUG0zMmFg5cHvFiFo5h+QMSJ28TihsEVY0f8HGliuO+pOzmvjMx1odToeaEWkTnyQ==",
"license": "MIT"
},
"node_modules/framed-stream": { "node_modules/framed-stream": {
"version": "1.0.1", "version": "1.0.1",
"resolved": "https://registry.npmjs.org/framed-stream/-/framed-stream-1.0.1.tgz", "resolved": "https://registry.npmjs.org/framed-stream/-/framed-stream-1.0.1.tgz",
@@ -1132,6 +1208,55 @@
"safety-catch": "^1.0.2" "safety-catch": "^1.0.2"
} }
}, },
"node_modules/hyperbee": {
"version": "2.27.3",
"resolved": "https://registry.npmjs.org/hyperbee/-/hyperbee-2.27.3.tgz",
"integrity": "sha512-PXURH2U4juUZyJRKHTrY5z1zX851pmI1Q0jfv5F/hCIErDt/ND8jOZuxc3hfOLM9f0W3qJEDTMlV5AJBkVPy8w==",
"license": "MIT",
"dependencies": {
"b4a": "^1.6.0",
"codecs": "^3.0.0",
"debounceify": "^1.0.0",
"hypercore-errors": "^1.0.0",
"mutexify": "^1.4.0",
"protocol-buffers-encodings": "^1.2.0",
"rache": "^1.0.0",
"ready-resource": "^1.0.0",
"resolve-reject-promise": "^1.1.0",
"safety-catch": "^1.0.2",
"streamx": "^2.12.4",
"unslab": "^1.2.0"
}
},
"node_modules/hypercore": {
"version": "11.34.1",
"resolved": "https://registry.npmjs.org/hypercore/-/hypercore-11.34.1.tgz",
"integrity": "sha512-nlrbXjI4x59akT8AwVEFLN54OicB53JDnH1xYP4j9zKDksHUVgqJFXnFwOkRr4Nr3kwCU0b9Rh3eIsU0JX5HEA==",
"license": "MIT",
"dependencies": {
"@hyperswarm/secret-stream": "^6.0.0",
"b4a": "^1.1.0",
"bare-events": "^2.2.0",
"big-sparse-array": "^1.0.3",
"compact-encoding": "^3.0.0",
"fast-fifo": "^1.3.0",
"flat-tree": "^1.9.0",
"hypercore-crypto": "^3.2.1",
"hypercore-errors": "^1.5.0",
"hypercore-id-encoding": "^1.2.0",
"hypercore-storage": "^3.2.0",
"is-options": "^1.0.1",
"nanoassert": "^2.0.0",
"protomux": "^3.5.0",
"quickbit-universal": "^2.2.0",
"random-array-iterator": "^1.0.0",
"safety-catch": "^1.0.1",
"sodium-universal": "^5.0.1",
"streamx": "^2.12.4",
"unslab": "^1.3.0",
"z32": "^1.0.0"
}
},
"node_modules/hypercore-crypto": { "node_modules/hypercore-crypto": {
"version": "3.7.0", "version": "3.7.0",
"resolved": "https://registry.npmjs.org/hypercore-crypto/-/hypercore-crypto-3.7.0.tgz", "resolved": "https://registry.npmjs.org/hypercore-crypto/-/hypercore-crypto-3.7.0.tgz",
@@ -1143,6 +1268,15 @@
"sodium-universal": "^5.0.0" "sodium-universal": "^5.0.0"
} }
}, },
"node_modules/hypercore-errors": {
"version": "1.5.0",
"resolved": "https://registry.npmjs.org/hypercore-errors/-/hypercore-errors-1.5.0.tgz",
"integrity": "sha512-5KQ/SuDxsvet+7qWA35Ay6zdD9WyAHQoyWHGcPUTbmJBd300gvNIJoi3oma7kp4TTCSzii6qYumNZe/s0j/saQ==",
"license": "Apache-2.0",
"dependencies": {
"hypercore-id-encoding": "^1.3.0"
}
},
"node_modules/hypercore-id-encoding": { "node_modules/hypercore-id-encoding": {
"version": "1.3.0", "version": "1.3.0",
"resolved": "https://registry.npmjs.org/hypercore-id-encoding/-/hypercore-id-encoding-1.3.0.tgz", "resolved": "https://registry.npmjs.org/hypercore-id-encoding/-/hypercore-id-encoding-1.3.0.tgz",
@@ -1153,6 +1287,47 @@
"z32": "^1.0.0" "z32": "^1.0.0"
} }
}, },
"node_modules/hypercore-storage": {
"version": "3.2.0",
"resolved": "https://registry.npmjs.org/hypercore-storage/-/hypercore-storage-3.2.0.tgz",
"integrity": "sha512-i5O5ZMkdZaNAWGuNRXUZjuzv7B41OIhDHUwLCZPjucgVmywY0ZE5PDafu6e5oPuhtghjl6S8q6Jn+POVKN2BvQ==",
"license": "Apache-2.0",
"dependencies": {
"b4a": "^1.6.7",
"bare-path": "^3.0.0",
"compact-encoding": "^3.1.0",
"device-file": "^2.1.2",
"flat-tree": "^1.12.1",
"hypercore-crypto": "^3.4.2",
"hyperschema": "^1.21.0",
"index-encoder": "^3.3.2",
"resolve-reject-promise": "^1.0.0",
"rocksdb-native": "^3.11.0",
"scope-lock": "^1.2.4",
"streamx": "^2.21.1",
"xache": "^1.2.1"
}
},
"node_modules/hyperdb": {
"version": "6.7.0",
"resolved": "https://registry.npmjs.org/hyperdb/-/hyperdb-6.7.0.tgz",
"integrity": "sha512-SVzV1mWNxD8c28sDlAXalOsac5ZPhSSknYwwVdMZyFz4vteXgANkq6pGDkvI2UUH2ZkF0KvcsGrBgLB+pbX41w==",
"license": "Apache-2.0",
"dependencies": {
"b4a": "^1.6.6",
"compact-encoding": "^3.0.0",
"generate-object-property": "^2.0.0",
"generate-string": "^1.0.1",
"hyperbee": "^2.24.2",
"hypercore": "^11.29.0",
"hyperschema": "^1.9.2",
"index-encoder": "^3.4.0",
"refcounter": "^1.0.0",
"rocksdb-native": "^3.0.0",
"scope-lock": "^1.2.4",
"streamx": "^2.20.0"
}
},
"node_modules/hyperdht": { "node_modules/hyperdht": {
"version": "6.33.0", "version": "6.33.0",
"resolved": "https://registry.npmjs.org/hyperdht/-/hyperdht-6.33.0.tgz", "resolved": "https://registry.npmjs.org/hyperdht/-/hyperdht-6.33.0.tgz",
@@ -1205,6 +1380,24 @@
"generate-string": "^1.0.1" "generate-string": "^1.0.1"
} }
}, },
"node_modules/index-encoder": {
"version": "3.5.0",
"resolved": "https://registry.npmjs.org/index-encoder/-/index-encoder-3.5.0.tgz",
"integrity": "sha512-idZ1cxtZz2dRV6rUiaP9Xo99UjXbSzjcMacoQmxUMu/A7fEQcNPngvwDJYeWelQUS5XFlY71/or70lKn6XnwbQ==",
"license": "Apache-2.0",
"dependencies": {
"b4a": "^1.6.4"
}
},
"node_modules/is-options": {
"version": "1.0.2",
"resolved": "https://registry.npmjs.org/is-options/-/is-options-1.0.2.tgz",
"integrity": "sha512-u+Ai74c8Q74aS8BuHwPdI1jptGOT1FQXgCq8/zv0xRuE+wRgSMEJLj8lVO8Zp9BeGb29BXY6AsNPinfqjkr7Fg==",
"license": "MIT",
"dependencies": {
"b4a": "^1.1.1"
}
},
"node_modules/is-property": { "node_modules/is-property": {
"version": "1.0.2", "version": "1.0.2",
"resolved": "https://registry.npmjs.org/is-property/-/is-property-1.0.2.tgz", "resolved": "https://registry.npmjs.org/is-property/-/is-property-1.0.2.tgz",
@@ -1345,9 +1538,9 @@
"pear-wakeups": "^1.0.0" "pear-wakeups": "^1.0.0"
} }
}, },
"node_modules/peardatadrive": { "node_modules/pear-appdrive": {
"version": "1.1.2", "version": "1.1.2",
"resolved": "https://registry.npmjs.org/peardatadrive/-/peardatadrive-1.1.2.tgz", "resolved": "https://registry.npmjs.org/pear-appdrive/-/pear-appdrive-1.1.2.tgz",
"integrity": "sha512-AmCrZaNGFU/BvHR+oFP/cNEUCH1vbzezsJV9wqASoggyg8d4sSb0L6kr99ugGkAuUFlOBI2uHni71hBTaO2rDA==", "integrity": "sha512-AmCrZaNGFU/BvHR+oFP/cNEUCH1vbzezsJV9wqASoggyg8d4sSb0L6kr99ugGkAuUFlOBI2uHni71hBTaO2rDA==",
"license": "Apache-2.0", "license": "Apache-2.0",
"dependencies": { "dependencies": {
@@ -1365,7 +1558,7 @@
"bare-http1": "^4.0.2", "bare-http1": "^4.0.2",
"get-mime-type": "^2.0.0", "get-mime-type": "^2.0.0",
"listen-async": "^1.0.0", "listen-async": "^1.0.0",
"peardatadrive": "^1.0.0", "pear-appdrive": "^1.0.0",
"pear-errors": "^1.0.0", "pear-errors": "^1.0.0",
"pear-gunk": "^1.0.0", "pear-gunk": "^1.0.0",
"pear-stamp": "^1.0.1", "pear-stamp": "^1.0.1",
@@ -1754,6 +1947,17 @@
"url": "https://github.com/sponsors/jonschlinkert" "url": "https://github.com/sponsors/jonschlinkert"
} }
}, },
"node_modules/protocol-buffers-encodings": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/protocol-buffers-encodings/-/protocol-buffers-encodings-1.2.0.tgz",
"integrity": "sha512-daeNPuKh1NlLD1uDfbLpD+xyUTc07nEtfHwmBZmt/vH0B7VOM+JOCOpDcx9ZRpqHjAiIkGqyTDi+wfGSl17R9w==",
"license": "MIT",
"dependencies": {
"b4a": "^1.6.0",
"signed-varint": "^2.0.1",
"varint": "5.0.0"
}
},
"node_modules/protomux": { "node_modules/protomux": {
"version": "3.11.0", "version": "3.11.0",
"resolved": "https://registry.npmjs.org/protomux/-/protomux-3.11.0.tgz", "resolved": "https://registry.npmjs.org/protomux/-/protomux-3.11.0.tgz",
@@ -1840,6 +2044,18 @@
"streamx": "^2.23.0" "streamx": "^2.23.0"
} }
}, },
"node_modules/rache": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/rache/-/rache-1.0.0.tgz",
"integrity": "sha512-e0k0g0w/8jOCB+7YqCIlOa+OJ38k0wrYS4x18pMSmqOvLKoyhmMhmQyCcvfY6VaP8D75cqkEnlakXs+RYYLqNg==",
"license": "Apache-2.0"
},
"node_modules/random-array-iterator": {
"version": "1.0.0",
"resolved": "https://registry.npmjs.org/random-array-iterator/-/random-array-iterator-1.0.0.tgz",
"integrity": "sha512-u7xCM93XqKEvPTP6xZp2ehttcAemKnh73oKNf1FvzuVCfpt6dILDt1Kxl1LeBjm2iNIeR49VGFhy4Iz3yOun+Q==",
"license": "MIT"
},
"node_modules/read-write-mutexify": { "node_modules/read-write-mutexify": {
"version": "2.1.0", "version": "2.1.0",
"resolved": "https://registry.npmjs.org/read-write-mutexify/-/read-write-mutexify-2.1.0.tgz", "resolved": "https://registry.npmjs.org/read-write-mutexify/-/read-write-mutexify-2.1.0.tgz",
@@ -1931,6 +2147,12 @@
"dev": true, "dev": true,
"license": "MIT" "license": "MIT"
}, },
"node_modules/scope-lock": {
"version": "1.2.4",
"resolved": "https://registry.npmjs.org/scope-lock/-/scope-lock-1.2.4.tgz",
"integrity": "sha512-BpSd8VCuCxW9ZitcdIC/vjs3gMaP9bRBL5nkHcyfX2VrS52n13/rHuBA2xJ/S/4DPuRdAO/Bk8pWd8eD/gHCIA==",
"license": "Apache-2.0"
},
"node_modules/script-linker": { "node_modules/script-linker": {
"version": "2.5.4", "version": "2.5.4",
"resolved": "https://registry.npmjs.org/script-linker/-/script-linker-2.5.4.tgz", "resolved": "https://registry.npmjs.org/script-linker/-/script-linker-2.5.4.tgz",
@@ -1954,6 +2176,15 @@
"integrity": "sha512-WBgv0UnIq2C+Aeh0/n+IRpP6967eIx9WpynTUoiW3isPpfe1zu2LJzyfXdo9Tgef8yR/sGjcMvoUXD7EYdiz+g==", "integrity": "sha512-WBgv0UnIq2C+Aeh0/n+IRpP6967eIx9WpynTUoiW3isPpfe1zu2LJzyfXdo9Tgef8yR/sGjcMvoUXD7EYdiz+g==",
"license": "MIT" "license": "MIT"
}, },
"node_modules/signed-varint": {
"version": "2.0.1",
"resolved": "https://registry.npmjs.org/signed-varint/-/signed-varint-2.0.1.tgz",
"integrity": "sha512-abgDPg1106vuZZOvw7cFwdCABddfJRz5akcCcchzTbhyhYnsG31y4AlZEgp315T7W3nQq5P4xeOm186ZiPVFzw==",
"license": "MIT",
"dependencies": {
"varint": "~5.0.0"
}
},
"node_modules/simdle-native": { "node_modules/simdle-native": {
"version": "1.3.9", "version": "1.3.9",
"resolved": "https://registry.npmjs.org/simdle-native/-/simdle-native-1.3.9.tgz", "resolved": "https://registry.npmjs.org/simdle-native/-/simdle-native-1.3.9.tgz",
@@ -2188,6 +2419,12 @@
"node": ">=10.12.0" "node": ">=10.12.0"
} }
}, },
"node_modules/varint": {
"version": "5.0.0",
"resolved": "https://registry.npmjs.org/varint/-/varint-5.0.0.tgz",
"integrity": "sha512-gC13b/bWrqQoKY2EmROCZ+AR0jitc6DnDGaQ6Ls9QpKmuSgJB1eQ7H3KETtQm7qSdMWMKCmsshyCmUwMLh3OAA==",
"license": "MIT"
},
"node_modules/webidl-conversions": { "node_modules/webidl-conversions": {
"version": "4.0.2", "version": "4.0.2",
"resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-4.0.2.tgz", "resolved": "https://registry.npmjs.org/webidl-conversions/-/webidl-conversions-4.0.2.tgz",
+10 -5
View File
@@ -1,7 +1,7 @@
{ {
"name": "peardata", "name": "peardata",
"version": "0.1.0", "version": "0.1.0",
"description": "Production-ready HyperDHT + protomux-rpc P2P app template (demo room + presence)", "description": "Decentralized P2P Netdata-class monitoring — PearMonitor agent + Pear desktop + REST v3",
"type": "module", "type": "module",
"license": "MIT", "license": "MIT",
"main": "index.js", "main": "index.js",
@@ -15,10 +15,10 @@
"gui": { "gui": {
"main": "index.html", "main": "index.html",
"backgroundColor": "#0b1020", "backgroundColor": "#0b1020",
"height": 780, "height": 860,
"width": 1100, "width": 1280,
"minWidth": 720, "minWidth": 900,
"minHeight": 480, "minHeight": 560,
"resizable": true, "resizable": true,
"movable": true, "movable": true,
"minimizable": true, "minimizable": true,
@@ -50,16 +50,21 @@
"dependencies": { "dependencies": {
"b4a": "^1.8.1", "b4a": "^1.8.1",
"compact-encoding": "^3.3.0", "compact-encoding": "^3.3.0",
"corestore": "^7.11.1",
"dotenv": "^17.4.2", "dotenv": "^17.4.2",
"graceful-goodbye": "^1.3.3", "graceful-goodbye": "^1.3.3",
"hypercore-crypto": "^3.7.0", "hypercore-crypto": "^3.7.0",
"hypercore-id-encoding": "^1.3.0",
"hyperdb": "^6.7.0",
"hyperdht": "^6.33.0", "hyperdht": "^6.33.0",
"hyperschema": "^1.21.0",
"pear-bridge": "^1.2.5", "pear-bridge": "^1.2.5",
"pear-electron": "^1.7.28", "pear-electron": "^1.7.28",
"pear-pipe": "^1.0.6", "pear-pipe": "^1.0.6",
"pear-run": "^1.0.8", "pear-run": "^1.0.8",
"protomux": "^3.11.0", "protomux": "^3.11.0",
"protomux-rpc": "^1.10.0", "protomux-rpc": "^1.10.0",
"ready-resource": "^1.2.0",
"safety-catch": "^1.0.3", "safety-catch": "^1.0.3",
"z32": "^1.1.0" "z32": "^1.1.0"
}, },
+1 -1
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env node #!/usr/bin/env node
/** /**
* Mint a pa1 invite from SERVER_SEED without starting the full server process. * Mint a pd1 invite from SERVER_SEED without starting the full server process.
* *
* Usage: * Usage:
* node scripts/mint-invite.js [role=operator] [ttlMs] * node scripts/mint-invite.js [role=operator] [ttlMs]
+30 -16
View File
@@ -27,7 +27,6 @@ fi
ROOT="$(cd "$(dirname "$0")/.." && pwd)" ROOT="$(cd "$(dirname "$0")/.." && pwd)"
cd "$ROOT" cd "$ROOT"
# Derive invite prefix from first letters + 1.
PREFIX="$(echo "$NEW_SLUG" | tr -cd 'a-z' | cut -c1-2)1." PREFIX="$(echo "$NEW_SLUG" | tr -cd 'a-z' | cut -c1-2)1."
ENV_PREFIX="$(echo "$NEW_SLUG" | tr 'a-z-' 'A-Z_')_" ENV_PREFIX="$(echo "$NEW_SLUG" | tr 'a-z-' 'A-Z_')_"
ENV_PREFIX="${ENV_PREFIX//__/_}" ENV_PREFIX="${ENV_PREFIX//__/_}"
@@ -40,24 +39,39 @@ echo " invite: $OLD_PREFIX → $PREFIX"
echo " env: $OLD_ENV$ENV_PREFIX" echo " env: $OLD_ENV$ENV_PREFIX"
export LC_ALL=C export LC_ALL=C
find . -type f \ # Use python for safe replacements (perl struggles with slashes in protocol id)
\( -name '*.js' -o -name '*.mjs' -o -name '*.cjs' -o -name '*.json' -o -name '*.md' -o -name '*.html' -o -name '*.css' -o -name '*.yml' -o -name '*.yaml' -o -name '*.service' -o -name '*.example' -o -name '*.sh' \) \ python3 - "$OLD_NAME" "$NEW_SLUG" "$OLD_PRODUCT" "$NEW_PRODUCT" "$OLD_PROTOCOL" "${NEW_SLUG}/rpc" "$OLD_SLUG" "$NEW_SLUG" "$OLD_PREFIX" "$PREFIX" "$OLD_ENV" "$ENV_PREFIX" <<'PY'
! -path './node_modules/*' ! -path './.git/*' ! -path './data/*' \ import os, sys
-print0 | while IFS= read -r -d '' f; do from pathlib import Path
perl -pi -e "
s/\Q$OLD_NAME\E/$NEW_SLUG/g;
s/\Q$OLD_PRODUCT\E/$NEW_PRODUCT/g;
s/\Q$OLD_PROTOCOL\E/${NEW_SLUG}\\/rpc/g;
s/\Q$OLD_SLUG\E/$NEW_SLUG/g;
s/\Q$OLD_PREFIX\E/$PREFIX/g;
s/\Q$OLD_ENV\E/$ENV_PREFIX/g;
" "$f"
done
if [[ -f bin/peardata-server.mjs ]]; then pairs = list(zip(sys.argv[1::2], sys.argv[2::2]))
# longer keys first
pairs.sort(key=lambda x: -len(x[0]))
SKIP = {'node_modules', '.git', 'data'}
EXTS = {'.js','.mjs','.cjs','.json','.md','.html','.css','.yml','.yaml','.service','.example','.sh'}
for dirpath, dirnames, filenames in os.walk('.'):
dirnames[:] = [d for d in dirnames if d not in SKIP]
for name in filenames:
p = Path(dirpath) / name
if p.suffix not in EXTS:
continue
try:
text = p.read_text(encoding='utf-8')
except Exception:
continue
orig = text
for a, b in pairs:
text = text.replace(a, b)
if text != orig:
p.write_text(text, encoding='utf-8')
print('updated', p)
PY
if [[ -f bin/peardata-server.mjs && "$NEW_SLUG" != "peardata" ]]; then
mv bin/peardata-server.mjs "bin/${NEW_SLUG}-server.mjs" mv bin/peardata-server.mjs "bin/${NEW_SLUG}-server.mjs"
fi fi
if [[ -f deploy/peardata.service ]]; then if [[ -f deploy/peardata.service && "$NEW_SLUG" != "peardata" ]]; then
mv deploy/peardata.service "deploy/${NEW_SLUG}.service" mv deploy/peardata.service "deploy/${NEW_SLUG}.service"
fi fi
+7 -4
View File
@@ -1,6 +1,6 @@
#!/usr/bin/env node #!/usr/bin/env node
/** /**
* Soak test: connect, post messages, verify pushes for SOAK_DURATION_MS. * Soak test: connect, query metrics, ping for SOAK_DURATION_MS.
* *
* Usage: * Usage:
* SERVER_PUBLIC_KEY=… SERVER_SEED=… node scripts/soak.js * SERVER_PUBLIC_KEY=… SERVER_SEED=… node scripts/soak.js
@@ -26,18 +26,21 @@ let received = 0
let sent = 0 let sent = 0
let errors = 0 let errors = 0
conn.on(Pushes.message, () => { conn.on(Pushes.metrics, () => {
received++ received++
}) })
await conn.connect() await conn.connect()
console.log('soak connected as', conn.role) console.log('soak connected as', conn.role)
await conn.request(Methods.subscribeMetrics, { charts: ['*'], intervalMs: 1000 })
const end = Date.now() + duration const end = Date.now() + duration
while (Date.now() < end) { while (Date.now() < end) {
try { try {
await conn.request(Methods.postMessage, { await conn.request(Methods.queryData, {
text: `soak ${sent} @ ${new Date().toISOString()}`, chart: 'system.cpu',
after: -30,
points: 30,
}) })
sent++ sent++
await conn.ping() await conn.ping()
+6 -2
View File
@@ -5,10 +5,14 @@ import fs from 'fs'
import path from 'path' import path from 'path'
const MUTATING = new Set([ const MUTATING = new Set([
'postMessage',
'clearMessages',
'mintInvite', 'mintInvite',
'revokePeer', 'revokePeer',
'setAlertConfig',
'ackAlert',
'silenceAlert',
'runJob',
'cancelJob',
'exportSnapshot',
'handshake', 'handshake',
]) ])
+1 -1
View File
@@ -1,5 +1,5 @@
/** /**
* Export formatters (JSON / Prometheus / shell) for /api/v*/allmetrics. * Export formatters (JSON / Prometheus / shell) for /api/v1|v2|v3/allmetrics.
*/ */
import os from 'os' import os from 'os'
import { getStore } from '../services/store.js' import { getStore } from '../services/store.js'
+4 -2
View File
@@ -122,7 +122,9 @@ export class MetricsCollector extends EventEmitter {
*/ */
constructor(opts = {}) { constructor(opts = {}) {
super() super()
this.intervalMs = opts.intervalMs ?? Number(process.env.PEARDATA_SAMPLE_MS) || SAMPLE_INTERVAL_MS this.intervalMs =
opts.intervalMs ??
(Number(process.env.PEARDATA_SAMPLE_MS) || SAMPLE_INTERVAL_MS)
this.timer = null this.timer = null
this.running = false this.running = false
this.sampleCount = 0 this.sampleCount = 0
@@ -257,7 +259,7 @@ export class MetricsCollector extends EventEmitter {
values: { values: {
running: load.running, running: load.running,
blocked: 0, blocked: 0,
total: load.total || Object.keys(process || {}).length, total: load.total || 0,
}, },
}) })
+1 -1
View File
@@ -301,7 +301,7 @@ export function encodeInvite(pkg) {
export function decodeInvite(invite) { export function decodeInvite(invite) {
const s = String(invite || '').trim() const s = String(invite || '').trim()
if (!s.startsWith(INVITE_PREFIX)) { if (!s.startsWith(INVITE_PREFIX)) {
return { ok: false, error: 'Not a pa1 invite', code: 'INVITE_INVALID' } return { ok: false, error: 'Not a pd1 invite', code: 'INVITE_INVALID' }
} }
try { try {
const json = b64urlDecode(s.slice(INVITE_PREFIX.length)).toString('utf8') const json = b64urlDecode(s.slice(INVITE_PREFIX.length)).toString('utf8')
+10 -6
View File
@@ -2,10 +2,12 @@ import test from 'brittle'
import { Roles, roleAllows, MethodRoles } from '../shared/protocol.js' import { Roles, roleAllows, MethodRoles } from '../shared/protocol.js'
import { assertAllowed, maxRole } from '../server/core/acl.js' import { assertAllowed, maxRole } from '../server/core/acl.js'
test('assertAllowed allows operator postMessage', (t) => { test('assertAllowed denies viewer mutating ops', (t) => {
t.exception(() => assertAllowed(Roles.viewer, 'postMessage')) t.exception(() => assertAllowed(Roles.viewer, 'runJob'))
assertAllowed(Roles.operator, 'postMessage') t.exception(() => assertAllowed(Roles.viewer, 'mintInvite'))
assertAllowed(Roles.admin, 'clearMessages') assertAllowed(Roles.operator, 'runJob')
assertAllowed(Roles.operator, 'setAlertConfig')
assertAllowed(Roles.admin, 'mintInvite')
t.pass() t.pass()
}) })
@@ -14,7 +16,9 @@ test('maxRole elevates', (t) => {
t.is(maxRole(Roles.admin, Roles.operator), Roles.admin) t.is(maxRole(Roles.admin, Roles.operator), Roles.admin)
}) })
test('viewer can read', (t) => { test('viewer can read metrics', (t) => {
t.ok(roleAllows(Roles.viewer, MethodRoles.listMessages)) t.ok(roleAllows(Roles.viewer, MethodRoles.queryData))
t.ok(roleAllows(Roles.viewer, MethodRoles.listCharts))
t.ok(roleAllows(Roles.viewer, MethodRoles.handshake)) t.ok(roleAllows(Roles.viewer, MethodRoles.handshake))
t.ok(roleAllows(Roles.viewer, MethodRoles.subscribeMetrics))
}) })
+29 -25
View File
@@ -1,10 +1,5 @@
/** /**
* End-to-end: ephemeral HyperDHT server + client RPC + push. * End-to-end: ephemeral HyperDHT agent + client RPC + metric push.
* Runs against real DHT (local, no bootstrap dependency for same-process? )
*
* hyperdht connect needs the DHT network; same-machine servers work via
* the default bootstrap / local discovery. May be slow on restricted networks.
*
* Skip with: SKIP_INTEGRATION=1 npm test * Skip with: SKIP_INTEGRATION=1 npm test
*/ */
import test from 'brittle' import test from 'brittle'
@@ -17,21 +12,22 @@ import { peers } from '../server/core/peer-registry.js'
import { initAuthKeys } from '../server/core/auth-keys.js' import { initAuthKeys } from '../server/core/auth-keys.js'
import { PearDataConnection } from '../client/connection.js' import { PearDataConnection } from '../client/connection.js'
import { Methods, Pushes } from '../shared/protocol.js' import { Methods, Pushes } from '../shared/protocol.js'
import { signCapability } from '../shared/crypto-auth.js' import { startPipeline } from '../server/pipeline.js'
import { getCollector } from '../server/services/collector.js'
const skip = process.env.SKIP_INTEGRATION === '1' const skip = process.env.SKIP_INTEGRATION === '1'
test('integration: dial, handshake, post, push', { skip, timeout: 60_000 }, async (t) => { test('integration: dial, handshake, metrics, subscribe', { skip, timeout: 60_000 }, async (t) => {
const seed = crypto.randomBytes(32) const seed = crypto.randomBytes(32)
const keyPair = DHT.keyPair(seed) const keyPair = DHT.keyPair(seed)
const publicKeyHex = b4a.toString(keyPair.publicKey, 'hex') const publicKeyHex = b4a.toString(keyPair.publicKey, 'hex')
const seedHex = b4a.toString(seed, 'hex') const seedHex = b4a.toString(seed, 'hex')
initAuthKeys({ seedHex, publicKeyHex }) initAuthKeys({ seedHex, publicKeyHex })
// Open admin for this test process
process.env.PEARDATA_INSECURE_OPEN_ADMIN = '1' process.env.PEARDATA_INSECURE_OPEN_ADMIN = '1'
startPipeline()
const dht = new DHT() const dht = new DHT()
const server = dht.createServer() const server = dht.createServer()
@@ -49,6 +45,7 @@ test('integration: dial, handshake, post, push', { skip, timeout: 60_000 }, asyn
await server.listen(keyPair) await server.listen(keyPair)
t.teardown(async () => { t.teardown(async () => {
getCollector().stop()
for (const s of peers.list()) s.destroy() for (const s of peers.list()) s.destroy()
await server.close().catch(() => {}) await server.close().catch(() => {})
await dht.destroy().catch(() => {}) await dht.destroy().catch(() => {})
@@ -61,8 +58,8 @@ test('integration: dial, handshake, post, push', { skip, timeout: 60_000 }, asyn
}) })
/** @type {object[]} */ /** @type {object[]} */
const pushes = [] const metricPushes = []
conn.on(Pushes.message, (m) => pushes.push(m)) conn.on(Pushes.metrics, (m) => metricPushes.push(m))
await conn.connect() await conn.connect()
t.is(conn.role, 'admin') t.is(conn.role, 'admin')
@@ -71,22 +68,29 @@ test('integration: dial, handshake, post, push', { skip, timeout: 60_000 }, asyn
const pong = await conn.request(Methods.ping, {}) const pong = await conn.request(Methods.ping, {})
t.ok(pong.ok) t.ok(pong.ok)
await conn.request(Methods.setDisplayName, { name: 'tester' }) const info = await conn.request(Methods.getServerInfo, {})
const post = await conn.request(Methods.postMessage, { text: 'hello p2p' }) t.is(info.app, 'peardata')
t.ok(post.success)
t.is(post.message.text, 'hello p2p')
// Allow push delivery const charts = await conn.request(Methods.listCharts, {})
await new Promise((r) => setTimeout(r, 200)) t.ok(charts.charts)
const list = await conn.request(Methods.listMessages, {})
t.ok(list.messages.some((m) => m.text === 'hello p2p'))
const invite = await conn.request(Methods.mintInvite, { role: 'operator', ttlMs: 3600_000 }) await conn.request(Methods.subscribeMetrics, { charts: ['*'], intervalMs: 1000 })
t.ok(invite.invite.startsWith('pd1.'))
// Capability can be verified offline // wait for at least one collector tick + push
const cap = signCapability(seedHex, { role: 'viewer', forever: true }) await new Promise((r) => setTimeout(r, 2200))
t.ok(cap.token.includes('.'))
const q = await conn.request(Methods.queryData, {
chart: 'system.cpu',
after: -30,
points: 30,
})
t.ok(q.labels)
t.ok(Array.isArray(q.data))
const health = await conn.request(Methods.getHealth, {})
t.ok(health.status)
t.ok(metricPushes.length >= 0) // push timing may vary; query proves pipeline
await conn.destroy() await conn.destroy()
}) })
+43 -10
View File
@@ -5,11 +5,14 @@ import {
MethodRoles, MethodRoles,
PROTOCOL, PROTOCOL,
PROTOCOL_VERSION, PROTOCOL_VERSION,
Methods,
Pushes,
} from '../shared/protocol.js' } from '../shared/protocol.js'
import { validateMethodArgs, SCHEMA_VERSION } from '../shared/schema.js' import { validateMethodArgs, SCHEMA_VERSION } from '../shared/schema.js'
import { CHART_DEFS, CONTEXT_IDS } from '../shared/metrics.js'
test('protocol constants', (t) => { test('protocol constants', (t) => {
t.ok(PROTOCOL.includes('/rpc')) t.is(PROTOCOL, 'peardata/rpc')
t.ok(PROTOCOL_VERSION >= 1) t.ok(PROTOCOL_VERSION >= 1)
t.ok(SCHEMA_VERSION >= 1) t.ok(SCHEMA_VERSION >= 1)
}) })
@@ -22,19 +25,49 @@ test('roleAllows hierarchy', (t) => {
t.absent(roleAllows(Roles.viewer, Roles.admin)) t.absent(roleAllows(Roles.viewer, Roles.admin))
}) })
test('method roles map covers core methods', (t) => { test('method roles cover monitoring surface', (t) => {
for (const m of ['handshake', 'ping', 'listMessages', 'postMessage', 'mintInvite']) { for (const m of [
'handshake',
'ping',
'queryData',
'subscribeMetrics',
'listCharts',
'mintInvite',
'runJob',
]) {
t.ok(MethodRoles[m], m) t.ok(MethodRoles[m], m)
t.is(Methods[m], m)
} }
}) })
test('validateMethodArgs postMessage', (t) => { test('pushes include metrics + anomaly', (t) => {
t.absent(validateMethodArgs('postMessage', {}).ok) t.ok(Pushes.metrics.startsWith('push:'))
t.ok(validateMethodArgs('postMessage', { text: 'hi' }).ok) t.ok(Pushes.anomaly.startsWith('push:'))
t.absent(validateMethodArgs('postMessage', { text: 'x'.repeat(2001) }).ok)
}) })
test('validateMethodArgs setDisplayName', (t) => { test('metrics catalog non-empty', (t) => {
t.ok(validateMethodArgs('setDisplayName', { name: 'Ada' }).ok) t.ok(CHART_DEFS.length >= 5)
t.absent(validateMethodArgs('setDisplayName', { name: '' }).ok) t.ok(CONTEXT_IDS.includes('system.cpu'))
})
test('validateMethodArgs queryData', (t) => {
t.absent(validateMethodArgs('queryData', {}).ok)
t.ok(validateMethodArgs('queryData', { chart: 'system.cpu' }).ok)
t.absent(validateMethodArgs('queryData', { chart: 'system.cpu', points: 0 }).ok)
})
test('validateMethodArgs subscribeMetrics', (t) => {
const r = validateMethodArgs('subscribeMetrics', { charts: ['system.cpu'] })
t.ok(r.ok)
t.ok(r.args.intervalMs >= 500)
})
test('validateMethodArgs mintInvite', (t) => {
t.ok(validateMethodArgs('mintInvite', { role: 'operator' }).ok)
t.absent(validateMethodArgs('mintInvite', { role: 'god' }).ok)
})
test('validateMethodArgs revokePeer', (t) => {
t.absent(validateMethodArgs('revokePeer', { peerId: 'abc' }).ok)
t.ok(validateMethodArgs('revokePeer', { peerId: 'a'.repeat(64) }).ok)
}) })
+73
View File
@@ -0,0 +1,73 @@
import test from 'brittle'
import { handleRest } from '../server/rest/routes.js'
import { getStore } from '../server/services/store.js'
import { initAuthKeys } from '../server/core/auth-keys.js'
import crypto from 'hypercore-crypto'
import b4a from 'b4a'
// REST routes read public key helper — init dummy keys
const seed = crypto.randomBytes(32)
initAuthKeys({
seedHex: b4a.toString(seed, 'hex'),
publicKeyHex: b4a.toString(crypto.keyPair(seed).publicKey, 'hex'),
})
test('REST /api/v3/info', (t) => {
const res = handleRest('/api/v3/info', new URLSearchParams())
t.is(res.status, 200)
t.ok(res.body.hostname)
t.ok(res.body.peardata)
})
test('REST /api/v1/charts after ingest', (t) => {
getStore().ingest([
{
chart: 'system.load',
context: 'system.load',
ts: Date.now(),
values: { load1: 0.5, load5: 0.4, load15: 0.3 },
},
])
const res = handleRest('/api/v1/charts', new URLSearchParams())
t.is(res.status, 200)
t.ok(res.body.charts['system.load'] || res.body.charts['system.cpu'])
})
test('REST /api/v3/data', (t) => {
const now = Date.now()
for (let i = 0; i < 10; i++) {
getStore().ingest([
{
chart: 'system.cpu',
context: 'system.cpu',
ts: now - (10 - i) * 1000,
values: {
user: i,
system: 1,
nice: 0,
iowait: 0,
irq: 0,
softirq: 0,
idle: 99 - i,
},
},
])
}
const res = handleRest(
'/api/v3/data',
new URLSearchParams({ chart: 'system.cpu', after: '-30', points: '10' })
)
t.is(res.status, 200)
t.ok(Array.isArray(res.body.data))
})
test('REST /api/v3/contexts', (t) => {
const res = handleRest('/api/v3/contexts', new URLSearchParams())
t.is(res.status, 200)
t.ok(res.body.contexts['system.cpu'])
})
test('REST 404', (t) => {
const res = handleRest('/api/v9/nope', new URLSearchParams())
t.is(res.status, 404)
})
+41
View File
@@ -0,0 +1,41 @@
import test from 'brittle'
import { MetricStore } from '../server/services/store.js'
test('store ingest + query', (t) => {
const store = new MetricStore()
const now = Date.now()
for (let i = 0; i < 30; i++) {
store.ingest([
{
chart: 'system.cpu',
context: 'system.cpu',
ts: now - (30 - i) * 1000,
values: {
user: 10 + i,
system: 5,
nice: 0,
iowait: 0,
irq: 0,
softirq: 0,
idle: 85 - i,
},
},
])
}
const meta = store.getMeta('system.cpu')
t.ok(meta)
t.is(meta.context, 'system.cpu')
const q = store.query({ chart: 'system.cpu', after: -60, before: 0, points: 15 })
t.absent(q.error)
t.ok(q.data.length > 0)
t.ok(q.labels.includes('user'))
t.ok(q.labels.includes('time'))
})
test('store unknown chart', (t) => {
const store = new MetricStore()
const q = store.query({ chart: 'nope.chart', points: 10 })
t.ok(q.error)
})
+173
View File
@@ -465,3 +465,176 @@ code {
font-size: 11px; font-size: 11px;
color: #b7c9f5; color: #b7c9f5;
} }
/* ─── PearData fleet dashboard ─── */
#app {
flex-direction: row;
overflow: hidden;
}
.fleet-rail {
width: 280px;
flex: 0 0 280px;
margin: 12px 0 12px 12px;
overflow: auto;
display: flex;
flex-direction: column;
gap: 4px;
}
.fleet-rail h2 {
margin-top: 0;
}
.dash-main {
flex: 1;
min-width: 0;
overflow: auto;
padding: 12px;
display: flex;
flex-direction: column;
gap: 12px;
}
.overview-strip {
display: grid;
grid-template-columns: repeat(5, 1fr);
gap: 12px;
padding: 14px 16px;
}
.stat {
display: flex;
flex-direction: column;
gap: 4px;
}
.stat-label {
font-size: 11px;
text-transform: uppercase;
letter-spacing: 0.04em;
color: var(--muted);
}
.stat strong {
font-size: 22px;
font-weight: 650;
letter-spacing: -0.02em;
font-variant-numeric: tabular-nums;
}
.stat[data-health='ok'] strong {
color: var(--ok);
}
.stat[data-health='degraded'] strong {
color: #f0b429;
}
.stat[data-health='critical'] strong {
color: var(--danger);
}
.charts-grid {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 12px;
}
.chart-panel header {
display: flex;
justify-content: space-between;
align-items: baseline;
margin-bottom: 8px;
}
.chart-panel h3 {
margin: 0;
font-size: 13px;
font-weight: 600;
}
.chart-panel canvas {
width: 100%;
display: block;
border-radius: 8px;
background: #0a1224;
}
.bottom-row {
display: grid;
grid-template-columns: 1fr 1fr;
gap: 12px;
}
.bottom-row h3 {
margin: 0 0 8px;
font-size: 13px;
}
.peer-list,
.event-list {
list-style: none;
padding: 0;
margin: 10px 0 0;
}
.peer-list li,
.event-list li {
display: flex;
justify-content: space-between;
gap: 8px;
padding: 8px 10px;
border-radius: 8px;
border: 1px solid transparent;
font-size: 12px;
cursor: pointer;
}
.peer-list li:hover,
.peer-list li.active {
background: #0d1426;
border-color: var(--border);
}
.event-list li {
cursor: default;
flex-direction: column;
align-items: flex-start;
border-bottom: 1px solid rgba(36, 49, 84, 0.7);
border-radius: 0;
}
.event-list li.crit {
color: var(--danger);
}
.event-list li.warn {
color: #f0b429;
}
.event-list li.ok {
color: var(--ok);
}
.admin-block {
margin-top: auto;
padding-top: 12px;
}
@media (max-width: 1100px) {
.overview-strip {
grid-template-columns: repeat(3, 1fr);
}
.charts-grid,
.bottom-row {
grid-template-columns: 1fr;
}
}
@media (max-width: 800px) {
#app {
flex-direction: column;
}
.fleet-rail {
width: auto;
flex: 0 0 auto;
margin: 12px 12px 0;
}
}