snxraven/peardata
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Updates

Browse Source
This commit is contained in:
Raven Scott
2026-07-18 16:22:56 -04:00
parent 015d92a257
commit f747ffbd25
36 changed files with 1975 additions and 873 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+123 -151
View File
@@ -1,193 +1,165 @@
# Architecture
PearData is a **decentralized, P2P clone of the Netdata real-time monitoring experience**, built on the same HyperDHT + protomux-rpc patterns as PearDock-class apps (via the pear-app template).
## Design goals
1. **No central control plane** — peers dial a public key, not a SaaS tenant.
2. **Cryptographic identity** — HyperDHT Noise streams authenticate both ends.
3. **Clear AuthZ** — roles, rate limits, audit, optional allowlist / revoke.
4. **Shared wire contract** — `shared/*` is the single source of truth for client + server.
5. **Replaceable domain** — demo room is a thin layer over the session stack.
6. **Pear-native desktop** — `pear-electron` shell with `<pear-ctrl>` window chrome.
1. **No central control plane** — dial an agent by Ed25519 public key.
2. **Instant live truth** — ~1s metric push to connected desktops.
3. **Dual API** — P2P RPC for the Pear client; Netdata-style REST for scripts/Grafana.
4. **Clear AuthZ** — viewer (pubkey) vs operator/admin (`pd1.` invite / seed proof).
5. **Low agent overhead** — Node/`os` + `/proc` collectors, ring buffers, hot-path RPCs.
6. **Ecosystem-ready** — shared wire contract in `shared/` for PearDock / PearVirt adapters later.
## Mapping to PearDock / template components
| PearDock-class concept | PearData component |
|------------------------|--------------------|
| HyperDHT secret stream | `server/server.js` + `client/connection.js` |
| protomux-rpc methods | `shared/protocol.js` + `server/handlers/monitor.js` |
| Capability invites (`pd1.`) | `shared/crypto-auth.js` |
| Roles viewer/operator/admin | `shared/protocol.js` `MethodRoles` + `server/core/acl.js` |
| Peer policy / revoke | `server/core/peer-policy.js` |
| Connection manager / multi-peer | `client/manager.js` |
| Job tray | `server/services/jobs.js` + desktop actions |
| Domain service | **Metrics pipeline** (`collector` → `store` → `anomaly` → pushes) |
| Optional HTTP surface | `server/rest/*` (Netdata v1/v2/v3) |
## System context
```mermaid
flowchart LR
UI[Pear desktop / scripts] -->|HyperDHT Noise| SRV[Node server]
SRV --> STATE[Room / your domain]
UI -.->|bootstrap / punch| NET[HyperDHT network]
SRV -.-> NET
```
## Layered stack
```mermaid
flowchart TB
subgraph Presentation
HTML[index.html + app.js + ui/styles.css]
PEAR[index.js pear-electron + bridge]
end
subgraph ClientCore
MGR[client/manager.js]
CON[client/connection.js]
ID[client/identity.js]
end
subgraph Wire
PROT[shared/protocol.js]
ENC[shared/encodings.js]
AUTH[shared/crypto-auth.js]
SCH[shared/schema.js]
end
subgraph ServerCore
BOOT[server/server.js]
SESS[server/rpc/session.js]
ACL[server/core/acl.js]
HAND[server/handlers/*]
SVC[server/services/*]
end
PEAR --> HTML
HTML --> MGR --> CON
CON --> PROT
CON --> AUTH
SESS --> PROT
SESS --> ACL
SESS --> SCH
HAND --> SVC
BOOT --> SESS
CON <-->|secret stream| SESS
UI[PearData desktop] -->|HyperDHT Noise + RPC| AG[PearMonitor agent]
SCRIPTS[curl / Grafana / Prometheus scrapers] -->|HTTP REST :19999| AG
AG --> COL[Collector 1s]
COL --> STORE[Tiered ring buffers]
COL --> ANO[Anomaly engine]
AG -.->|bootstrap / punch| DHT[HyperDHT]
UI -.-> DHT
```
## Process model
| Process | Entry | Responsibility |
|---------|-------|----------------|
| **Server** | `server/server.js` or `bin/peardata-server.mjs` | HyperDHT listen, RPC, domain state |
| **Desktop** | `index.js` → Pear Runtime | Window + HTML UI; dials servers as a client |
| **Scripts** | `scripts/*` | mint-invite, healthcheck, soak (use client stack) |
| **Agent** | `server/server.js` / `bin/peardata-server.mjs` | Collect, store, P2P RPC, optional REST |
| **Desktop** | `index.js` → Pear Runtime | Fleet UI, multi-peer dial, live charts |
| **Scripts** | `scripts/*` | mint-invite, healthcheck, soak |
Server and desktop are **independent**. You can run many clients against one server, or headless scripts with no UI.
Many desktops may dial one agent; one desktop may dial many agents.
## Server boot
## Planes
1. `loadOrCreateKeyPair()` → persist `SERVER_SEED` / `SERVER_PUBLIC_KEY` in `.env`
2. `initAuthKeys()` → HKDF MAC key for capabilities
3. `loadPeerPolicy()` → roles / revocations / spent JTIs from `PEARDATA_DATA_DIR`
4. `dht.createServer().listen(keyPair)`
5. On connection → revoke check → `PeerSession` → `registerAllHandlers` → peer registry
6. Banner logs public key + secure/insecure mode
7. `graceful-goodbye` / SIGINT / SIGTERM drain peers and destroy DHT
### 1. Control / metadata (RPC)
## Session middleware (every non-hot RPC)
Handshake, node info, chart/context catalog, alert config, invites, jobs, ACL.
### 2. High-frequency metrics
- **Ingest:** collector emits sample batches every `PEARDATA_SAMPLE_MS` (default 1000).
- **Store:** tier0 (1s) + tier1 (downsampled averages).
- **Push:** `push:metrics` to subscribed peers (protomux-rpc events).
- **Pull:** `queryData` / REST `/api/v3/data` for history windows.
### 3. Anomaly / health
Threshold engine evaluates each batch; transitions emit `push:anomaly` / `push:alert`; health snapshot on an interval via `push:health`.
### 4. Local REST (optional)
Netdata-compatible HTTP on `127.0.0.1:19999` by default — for local tooling without P2P. Disable with `PEARDATA_REST=0`.
## Layered stack
```mermaid
flowchart TD
IN[method + args] --> RL{Rate limit}
RL -->|deny| E1[RATE_LIMIT_EXCEEDED]
RL -->|ok| ACL{roleAllows MethodRoles}
ACL -->|deny| E2[PERMISSION_DENIED + audit]
ACL -->|ok| VAL{validateMethodArgs}
VAL -->|fail| E3[INVALID_ARGS]
VAL -->|ok| H[Handler]
H --> OK[Result + optional audit]
flowchart TB
subgraph Presentation
HTML[index.html + app.js]
PEAR[index.js pear-electron]
end
subgraph ClientCore
MGR[client/manager.js]
CON[client/connection.js]
end
subgraph Wire
PROT[shared/protocol.js]
MET[shared/metrics.js]
SCH[shared/schema.js]
AUTH[shared/crypto-auth.js]
end
subgraph Agent
BOOT[server/server.js]
PIPE[server/pipeline.js]
COL[collector]
STORE[store]
ANO[anomaly]
HAND[handlers/monitor.js]
REST[rest/http-server.js]
end
PEAR --> HTML --> MGR --> CON
CON <-->|Noise + protomux-rpc| HAND
BOOT --> PIPE --> COL --> STORE
COL --> ANO
REST --> STORE
HAND --> STORE
```
**Hot path** (`session.respond(method, handler, { hot: true })` or stream method names):
## Agent boot sequence
- Still rate-limited and ACL-checked
- Skips full schema validation / success audit (for high-frequency streams)
1. Load/create `SERVER_SEED` / public key → `initAuthKeys`
2. Load peer policy from `PEARDATA_DATA_DIR`
3. `startPipeline()` — collector + store + anomaly fan-out
4. `startRestServer()` — unless disabled
5. HyperDHT `createServer().listen(keyPair)`
6. On connection → revoke check → `PeerSession` → register monitor handlers
7. Banner prints pubkey + REST URL
## Client connection states
## Session middleware
```
idle → dialing → handshaking → ready
↘ closed → (manager reconnect timer)
```
| State | Meaning |
|-------|---------|
| `idle` | Constructed, not dialing |
| `dialing` | `dht.connect(serverPk)` in flight |
| `handshaking` | Stream open; `handshake` RPC |
| `ready` | Authenticated; RPCs allowed |
| `closed` | Torn down |
`ConnectionManager` tracks multiple peers, active selection, and reconnect (max `PEARDATA_MAX_RECONNECT`).
Same as the template: rate limit → ACL (`MethodRoles`) → schema validate → handler.
Hot methods (`queryData`, `subscribeMetrics`, `ping`, …) skip heavy audit.
## Identity planes
| Plane | Storage | Purpose |
|-------|---------|---------|
| **Server keypair** | `.env` (`SERVER_SEED`) | DHT listen address + HMAC root |
| **Client keypair** | `~/.config/peardata/identity.json` | Stable peerId for AuthZ / revoke |
| **Capabilities** | Issued as tokens / `pd1.` invites | Role grants with optional expiry & peer bind |
| **Peer policy** | `data/peer-policy.json` | Registered roles, revocations, spent JTIs |
| **Audit** | `data/audit.log` | Mutating RPC trail |
| Agent keypair | `.env` `SERVER_SEED` | DHT address + HMAC root |
| Client keypair | `~/.config/peardata/identity.json` | Stable peerId |
| Capabilities | `pd1.` invites / raw tokens | Role grants |
| Peer policy | `data/peer-policy.json` | Roles, revokes, spent JTIs |
| Audit | `data/audit.log` | Mutating RPC trail |
## Parent peer (future)
A heavier agent may subscribe to child agents over P2P, downsample into its own store, and expose fleet REST — still no central SaaS. Design leaves room via `stream_path` REST stub and multi-peer desktop manager.
## Module map
### `shared/`
### Keep from template
| File | Role |
|------|------|
| `protocol.js` | `PROTOCOL`, roles, `MethodRoles`, `Methods`, `Pushes` |
| `encodings.js` | compact-encoding JSON for protomux-rpc |
| `crypto-auth.js` | MAC key, capabilities, admin proof, invites |
| `schema.js` | Lightweight request validation |
`server/core/*`, `server/rpc/session.js`, `client/*`, `shared/crypto-auth.js`, `shared/encodings.js`, Pear titlebar patterns.
### `server/`
### PearData domain
| Path | Role |
|------|------|
| `server.js` | Boot + DHT accept loop |
| `core/keys.js` | Seed load / generate |
| `core/auth-keys.js` | Process-wide MAC key |
| `core/acl.js` | Role resolution + assert |
| `core/peer-policy.js` | File-backed policy |
| `core/peer-registry.js` | Live sessions |
| `core/audit.js` | Audit log writer |
| `rpc/session.js` | ProtomuxRPC + middleware |
| `rpc/register.js` | Wire handlers per session |
| `handlers/demo.js` | **Replace** — domain RPCs |
| `services/room.js` | **Replace** — domain state |
| `utils/logger.js` | Structured / pretty logs |
| `utils/rateLimiter.js` | Per-peer RPM |
### `client/`
| File | Role |
|------|------|
| `identity.js` | Persistent client seed |
| `connection.js` | Single peer RPC client |
| `manager.js` | Multi-peer + reconnect |
| `errors.js` | Error normalization |
| `index.js` | Public re-exports |
### Desktop shell
| File | Role |
|------|------|
| `index.js` | Pear Runtime + Bridge |
| `index.html` | Titlebar (`pear-ctrl`) + layout |
| `app.js` | UI → manager |
| `ui/styles.css` | Drag regions + theme |
See [DESKTOP.md](./DESKTOP.md).
## What to keep vs replace
| Keep | Replace when productizing |
|------|---------------------------|
| `shared/*` wire + crypto | Method names / schema for your domain |
| `server/rpc/session.js` | Rarely — middleware is generic |
| `server/core/*` | Peer policy storage backend if needed |
| `client/connection.js` + `manager.js` | UI-specific multi-peer UX |
| Titlebar / `pear-ctrl` patterns | Visual design only — keep drag + controls |
| `server/handlers/demo.js` + `services/room.js` | **Your product** |
| `shared/metrics.js` | Chart/context catalog |
| `shared/data-model.js` | Typed shapes |
| `server/services/collector.js` | System sampling |
| `server/services/store.js` | Tiered buffers + query |
| `server/services/anomaly.js` | Thresholds |
| `server/services/alerts.js` | Alert CRUD helpers |
| `server/services/subscriptions.js` | Push fan-out |
| `server/services/jobs.js` | On-demand jobs |
| `server/handlers/monitor.js` | RPC surface |
| `server/rest/*` | Netdata HTTP API |
| `server/pipeline.js` | Wire collector→store→push |
## Related docs
- [PROTOCOL.md](./PROTOCOL.md)
- [SECURITY.md](./SECURITY.md)
- [DESKTOP.md](./DESKTOP.md)
- [CONFIGURATION.md](./CONFIGURATION.md)
- [EXTENDING.md](./EXTENDING.md)
- [PROTOCOL.md](./PROTOCOL.md) — RPC methods & pushes
- [DATA-MODEL.md](./DATA-MODEL.md) — metrics / anomalies / health
- [REST-API.md](./REST-API.md) — `/api/v1|v2|v3`
- [TECH-CHOICES.md](./TECH-CHOICES.md) — collector & charts
- [ROADMAP.md](./ROADMAP.md) — phases
- [SECURITY.md](./SECURITY.md) — threat model