peardata/docs/ARCHITECTURE.md

Architecture

PearData is a decentralized, P2P real-time host monitoring stack, built on the same HyperDHT + protomux-rpc patterns as PearDock-class apps (via the pear-app template).

Design goals

1. No central control plane — dial an agent by Ed25519 public key.
2. Instant live truth — ~1s metric push to connected desktops.
3. Dual API — P2P RPC for the Pear client; agent-style REST for scripts/Grafana.
4. Clear AuthZ — viewer (pubkey) vs operator/admin (pd1. invite / seed proof).
5. Low agent overhead — os/bare-os + /proc collectors, ring buffers, hot-path RPCs.
6. Bare/Pear-ready — package.json import maps route builtins → bare-*; no Node-only core deps in the Pear path.
7. Ecosystem-ready — shared wire contract in shared/ for PearDock / PearVirt adapters later.

Mapping to PearDock / template components

PearDock-class concept	PearData component
HyperDHT secret stream	server/server.js + client/connection.js
protomux-rpc methods	shared/protocol.js + server/handlers/monitor.js
Capability invites (pd1.)	shared/crypto-auth.js
Roles viewer/operator/admin	shared/protocol.js MethodRoles + server/core/acl.js
Peer policy / revoke	server/core/peer-policy.js
Connection manager / multi-peer	client/manager.js
Job tray	server/services/jobs.js + desktop actions
Domain service	Metrics pipeline (collector → store → anomaly → pushes)
Optional HTTP surface	server/rest/* (REST v1/v2/v3)

System context

flowchart LR
  UI[PearData desktop] -->|HyperDHT Noise + RPC| AG[PearMonitor agent]
  SCRIPTS[curl / Grafana / Prometheus scrapers] -->|HTTP REST :18888| AG
  AG --> COL[Collector 1s]
  COL --> STORE[Tiered ring buffers]
  COL --> ANO[Anomaly engine]
  AG -.->|bootstrap / punch| DHT[HyperDHT]
  UI -.-> DHT

Process model

Process	Entry	Responsibility
Agent	server/server.js / bin/peardata-server.mjs	Collect, store, P2P RPC, optional REST
Desktop	index.js → Pear Runtime	Fleet UI, multi-peer dial, live charts
Scripts	scripts/*	mint-invite, healthcheck, soak

Many desktops may dial one agent; one desktop may dial many agents.

Planes

1. Control / metadata (RPC)

Handshake, node info, chart/context catalog, alert config, invites, jobs, ACL.

2. High-frequency metrics

• Ingest: collector emits sample batches every PEARDATA_SAMPLE_MS (default 1000).
• Store: tier0 (1s) + tier1 (downsampled averages) + HyperDB warm flush.
• Retention: server/services/retention.js — retention.json, auto-prune (default warm age 3 months, 1 GiB budget, clearUnlinked+compact GC), Data Manager RPC.
• Push: push:metrics to subscribed peers (protomux-rpc events).
• Pull: queryData / REST /api/v3/data for history windows.

3. Anomaly / health

Threshold engine evaluates each batch; transitions emit push:anomaly / push:alert; health snapshot on an interval via push:health.

4. Local REST (optional)

agent-compatible HTTP on 127.0.0.1:18888 by default — for local tooling without P2P. Disable with PEARDATA_REST=0.

Layered stack

flowchart TB
  subgraph Presentation
    HTML[index.html + app.js]
    PEAR[index.js pear-electron]
  end
  subgraph ClientCore
    MGR[client/manager.js]
    CON[client/connection.js]
  end
  subgraph Wire
    PROT[shared/protocol.js]
    MET[shared/metrics.js]
    SCH[shared/schema.js]
    AUTH[shared/crypto-auth.js]
  end
  subgraph Agent
    BOOT[server/server.js]
    PIPE[server/pipeline.js]
    COL[collector]
    STORE[store]
    ANO[anomaly]
    HAND[handlers/monitor.js]
    REST[rest/http-server.js]
  end
  PEAR --> HTML --> MGR --> CON
  CON <-->|Noise + protomux-rpc| HAND
  BOOT --> PIPE --> COL --> STORE
  COL --> ANO
  REST --> STORE
  HAND --> STORE

Agent boot sequence

1. Load/create SERVER_SEED / public key → initAuthKeys
2. Load peer policy from PEARDATA_DATA_DIR
3. startPipeline() — collector + store + anomaly fan-out
4. startRestServer() — unless disabled
5. HyperDHT createServer().listen(keyPair)
6. On connection → revoke check → PeerSession → register monitor handlers
7. Banner prints pubkey + REST URL

Session middleware

Same as the template: rate limit → ACL (MethodRoles) → schema validate → handler.
Hot methods (queryData, subscribeMetrics, ping, …) skip heavy audit.

Identity planes

Plane	Storage	Purpose
Agent keypair	.env SERVER_SEED	DHT address + HMAC root
Client keypair	~/.config/peardata/identity.json	Stable peerId
Capabilities	pd1. invites / raw tokens	Role grants
Peer policy	data/peer-policy.json	Roles, revokes, spent JTIs
Audit	data/audit.log	Mutating RPC trail

Parent peer (future)

A heavier agent may subscribe to child agents over P2P, downsample into its own store, and expose fleet REST — still no central SaaS. Design leaves room via stream_path REST stub and multi-peer desktop manager.

Module map

Keep from template

server/core/*, server/rpc/session.js, client/*, shared/crypto-auth.js, shared/encodings.js, Pear titlebar patterns.

PearData domain

Path	Role
shared/metrics.js	Chart/context catalog
shared/data-model.js	Typed shapes
server/services/collector.js	System sampling
server/services/store.js	Tiered buffers + query + live retention trim
server/services/retention.js	Data Manager policy, auto-prune, storage usage
server/services/anomaly.js	Thresholds
server/services/weights.js	Metric Correlations scoring (volume / ks2 / …)
server/services/logs.js	System log query (anomaly / audit / journalctl)
server/services/alerts.js	Alert CRUD helpers
server/services/subscriptions.js	Push fan-out
server/services/jobs.js	On-demand jobs (gcBuffers → prune)
server/services/collectors/docker.js	Opt-in Docker metrics + socket name enrichment
server/handlers/monitor.js	RPC surface
server/rest/*	Agent HTTP API
server/pipeline.js	Wire collector→store→push
server/db/*	HyperDB model, Corestore, swarm replicate
spec/	Generated Hyperschema + HyperDB defs
scripts/build-db.js	Schema codegen

HyperDB design: STORAGE-HYPERDB.md.

Related docs

• User guide — desktop workflows
• PROTOCOL.md — RPC methods & pushes
• DATA-MODEL.md — metrics / anomalies / health / weights
• REST-API.md — /api/v1|v2|v3
• TECH-CHOICES.md — collector & charts
• ROADMAP.md — phases
• SECURITY.md — threat model