peardata/docs/ARCHITECTURE.md

# Architecture

PearData is a **decentralized, P2P real-time host monitoring stack**, built on the same HyperDHT + protomux-rpc patterns as PearDock-class apps (via the pear-app template).

## Design goals

1. **No central control plane** — dial an agent by Ed25519 public key.
2. **Instant live truth** — ~1s metric push to connected desktops.
3. **Dual API** — P2P RPC for the Pear client; agent-style REST for scripts/Grafana.
4. **Clear AuthZ** — viewer (pubkey) vs operator/admin (`pd1.` invite / seed proof).
5. **Low agent overhead** — `os`/`bare-os` + `/proc` collectors, ring buffers, hot-path RPCs.
6. **Bare/Pear-ready** — `package.json` import maps route builtins → `bare-*`; no Node-only core deps in the Pear path.
7. **Ecosystem-ready** — shared wire contract in `shared/` for PearDock / PearVirt adapters later.

## Mapping to PearDock / template components

| PearDock-class concept | PearData component |
|------------------------|--------------------|
| HyperDHT secret stream | `server/server.js` + `client/connection.js` |
| protomux-rpc methods | `shared/protocol.js` + `server/handlers/monitor.js` |
| Capability invites (`pd1.`) | `shared/crypto-auth.js` |
| Roles viewer/operator/admin | `shared/protocol.js` `MethodRoles` + `server/core/acl.js` |
| Peer policy / revoke | `server/core/peer-policy.js` |
| Connection manager / multi-peer | `client/manager.js` |
| Job tray | `server/services/jobs.js` + desktop actions |
| Domain service | **Metrics pipeline** (`collector` → `store` → `anomaly` → pushes) |
| Optional HTTP surface | `server/rest/*` (REST v1/v2/v3) |

## System context

```mermaid
flowchart LR
  UI[PearData desktop] -->|HyperDHT Noise + RPC| AG[PearMonitor agent]
  SCRIPTS[curl / Grafana / Prometheus scrapers] -->|HTTP REST :18888| AG
  AG --> COL[Collector 1s]
  COL --> STORE[Tiered ring buffers]
  COL --> ANO[Anomaly engine]
  AG -.->|bootstrap / punch| DHT[HyperDHT]
  UI -.-> DHT
```

## Process model

| Process | Entry | Responsibility |
|---------|-------|----------------|
| **Agent** | `server/server.js` / `bin/peardata-server.mjs` | Collect, store, P2P RPC, optional REST |
| **Desktop** | `index.js` → Pear Runtime | Fleet UI, multi-peer dial, live charts |
| **Scripts** | `scripts/*` | mint-invite, healthcheck, soak |

Many desktops may dial one agent; one desktop may dial many agents.

## Planes

### 1. Control / metadata (RPC)

Handshake, node info, chart/context catalog, alert config, invites, jobs, ACL.

### 2. High-frequency metrics

- **Ingest:** collector emits sample batches every `PEARDATA_SAMPLE_MS` (default 1000).
- **Store:** tier0 (1s) + tier1 (downsampled averages) + HyperDB warm flush.
- **Retention:** `server/services/retention.js` — `retention.json`, auto-prune (default warm age **3 months**, **1 GiB** budget, clearUnlinked+compact GC), Data Manager RPC.
- **Push:** `push:metrics` to subscribed peers (protomux-rpc events).
- **Pull:** `queryData` / REST `/api/v3/data` for history windows.

### 3. Anomaly / health

Threshold engine evaluates each batch; transitions emit `push:anomaly` / `push:alert`; health snapshot on an interval via `push:health`.

### 4. Local REST (optional)

agent-compatible HTTP on `127.0.0.1:18888` by default — for local tooling without P2P. Disable with `PEARDATA_REST=0`.

## Layered stack

```mermaid
flowchart TB
  subgraph Presentation
    HTML[index.html + app.js]
    PEAR[index.js pear-electron]
  end
  subgraph ClientCore
    MGR[client/manager.js]
    CON[client/connection.js]
  end
  subgraph Wire
    PROT[shared/protocol.js]
    MET[shared/metrics.js]
    SCH[shared/schema.js]
    AUTH[shared/crypto-auth.js]
  end
  subgraph Agent
    BOOT[server/server.js]
    PIPE[server/pipeline.js]
    COL[collector]
    STORE[store]
    ANO[anomaly]
    HAND[handlers/monitor.js]
    REST[rest/http-server.js]
  end
  PEAR --> HTML --> MGR --> CON
  CON <-->|Noise + protomux-rpc| HAND
  BOOT --> PIPE --> COL --> STORE
  COL --> ANO
  REST --> STORE
  HAND --> STORE
```

## Agent boot sequence

1. Load/create `SERVER_SEED` / public key → `initAuthKeys`
2. Load peer policy from `PEARDATA_DATA_DIR`
3. `startPipeline()` — collector + store + anomaly fan-out
4. `startRestServer()` — unless disabled
5. HyperDHT `createServer().listen(keyPair)`
6. On connection → revoke check → `PeerSession` → register monitor handlers
7. Banner prints pubkey + REST URL

## Session middleware

Same as the template: rate limit → ACL (`MethodRoles`) → schema validate → handler.  
Hot methods (`queryData`, `subscribeMetrics`, `ping`, …) skip heavy audit.

## Identity planes

| Plane | Storage | Purpose |
|-------|---------|---------|
| Agent keypair | `.env` `SERVER_SEED` | DHT address + HMAC root |
| Client keypair | `~/.config/peardata/identity.json` | Stable peerId |
| Capabilities | `pd1.` invites / raw tokens | Role grants |
| Peer policy | `data/peer-policy.json` | Roles, revokes, spent JTIs |
| Audit | `data/audit.log` | Mutating RPC trail |

## Parent peer (future)

A heavier agent may subscribe to child agents over P2P, downsample into its own store, and expose fleet REST — still no central SaaS. Design leaves room via `stream_path` REST stub and multi-peer desktop manager.

## Module map

### Keep from template

`server/core/*`, `server/rpc/session.js`, `client/*`, `shared/crypto-auth.js`, `shared/encodings.js`, Pear titlebar patterns.

### PearData domain

| Path | Role |
|------|------|
| `shared/metrics.js` | Chart/context catalog |
| `shared/data-model.js` | Typed shapes |
| `server/services/collector.js` | System sampling |
| `server/services/store.js` | Tiered buffers + query + live retention trim |
| `server/services/retention.js` | Data Manager policy, auto-prune, storage usage |
| `server/services/anomaly.js` | Thresholds |
| `server/services/weights.js` | Metric Correlations scoring (`volume` / `ks2` / …) |
| `server/services/logs.js` | System log query (anomaly / audit / journalctl) |
| `server/services/alerts.js` | Alert CRUD helpers |
| `server/services/subscriptions.js` | Push fan-out |
| `server/services/jobs.js` | On-demand jobs (`gcBuffers` → prune) |
| `server/services/collectors/docker.js` | Opt-in Docker metrics + socket name enrichment |
| `server/handlers/monitor.js` | RPC surface |
| `server/rest/*` | Agent HTTP API |
| `server/pipeline.js` | Wire collector→store→push |
| `server/db/*` | HyperDB model, Corestore, swarm replicate |
| `spec/` | Generated Hyperschema + HyperDB defs |
| `scripts/build-db.js` | Schema codegen |

HyperDB design: [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md).

## Related docs

- [User guide](../user-guide/README.md) — desktop workflows
- [PROTOCOL.md](./PROTOCOL.md) — RPC methods & pushes
- [DATA-MODEL.md](./DATA-MODEL.md) — metrics / anomalies / health / weights
- [REST-API.md](./REST-API.md) — `/api/v1|v2|v3`
- [TECH-CHOICES.md](./TECH-CHOICES.md) — collector & charts
- [ROADMAP.md](./ROADMAP.md) — phases
- [SECURITY.md](./SECURITY.md) — threat model