snxraven/peardata
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Docs
CI / test (push) Successful in 1m3s
Details
Release rolling / release (push) Successful in 6m51s
Details

Browse Source
This commit is contained in:
Raven Scott
2026-07-18 22:46:24 -04:00
parent 93a239e8ef
commit 7815037737
24 changed files with 576 additions and 16 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/ARCHITECTURE.md
+3 -1
View File
@@ -149,6 +149,7 @@ A heavier agent may subscribe to child agents over P2P, downsample into its own
| `server/services/collector.js` | System sampling |
| `server/services/store.js` | Tiered buffers + query |
| `server/services/anomaly.js` | Thresholds |
| `server/services/weights.js` | Metric Correlations scoring (`volume` / `ks2` / …) |
| `server/services/alerts.js` | Alert CRUD helpers |
| `server/services/subscriptions.js` | Push fan-out |
| `server/services/jobs.js` | On-demand jobs |
@@ -163,8 +164,9 @@ HyperDB design: [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md).
## Related docs
- [User guide](../user-guide/README.md) — desktop workflows
- [PROTOCOL.md](./PROTOCOL.md) — RPC methods & pushes
- [DATA-MODEL.md](./DATA-MODEL.md) — metrics / anomalies / health
- [DATA-MODEL.md](./DATA-MODEL.md) — metrics / anomalies / health / weights
- [REST-API.md](./REST-API.md) — `/api/v1|v2|v3`
- [TECH-CHOICES.md](./TECH-CHOICES.md) — collector & charts
- [ROADMAP.md](./ROADMAP.md) — phases
docs/DASHBOARD.md
+3 -1
View File
@@ -3,7 +3,7 @@
Product plan for turning the Charts tab into a full host/fleet metrics wall:
sectioned TOC, every collected chart, synchronized time, and investigation controls.
Related: [ROADMAP.md](./ROADMAP.md) Phase 6 · [DESKTOP.md](./DESKTOP.md) · [DATA-MODEL.md](./DATA-MODEL.md)
Related: [ROADMAP.md](./ROADMAP.md) Phase 6 · [DESKTOP.md](./DESKTOP.md) · [DATA-MODEL.md](./DATA-MODEL.md) · **[User guide](../user-guide/README.md)** (operator walkthroughs)
---
@@ -154,3 +154,5 @@ Do not name third-party products in code, commits, or user-facing copy.
| Goal | Find metrics that **changed** vs a baseline around a highlight | Find charts **related** to a seed chart |
| Engine | Server `weights.js` (`volume` / `ks2` / …) | Client taxonomy + Pearson |
| UX | Toolbar Correlate → brush ≥15s → Find Correlations → filtered wall | Per-card ⇢ / dblclick → related panel |
Full operator docs: [metric-correlations.md](../user-guide/metric-correlations.md) · [related-metrics.md](../user-guide/related-metrics.md) · [weights-api.md](../user-guide/weights-api.md).
docs/DATA-MODEL.md
+33 -1
View File
@@ -146,13 +146,45 @@ Aggregate: any CRITICAL → `critical`; else any WARNING → `degraded`; else `o
}
```
## Weights / Metric Correlations
Engine: `server/services/weights.js`. Not Pearson pairwise correlation — highlight-vs-baseline (or single-window) **scoring**.
| Method | Windows | Score meaning |
|--------|---------|---------------|
| `volume` | highlight + baseline | Average change × time above/below baseline |
| `ks2` | highlight + baseline | Two-sample KS on pairwise diffs |
| `anomaly-rate` | highlight | Anomaly / alert activity in window |
| `value` | highlight | Aggregated value (default `time_group=cv`) |
| `alerts` | none | Legacy active-alert ranking (Related panel boost) |
Default baseline when omitted: ends at highlight start, duration ≈ highlight × 4 (power-of-two snap). Reject if fewer than ~15 usable points.
Response shape:
```json
{
"method": "volume",
"view": {
"highlight": { "after": 1710000000, "before": 1710000060 },
"baseline": { "after": 1710000000, "before": 1710000000 }
},
"results": [
{ "id": "system.cpu", "weight": 0.91, "context": "system.cpu", "family": "cpu" }
],
"stats": { "scored": 100, "elapsed_ms": 35 }
}
```
Desktop UX: [user-guide/metric-correlations.md](../user-guide/metric-correlations.md). Related (Pearson/taxonomy) is separate: `shared/related-metrics.js`.
## REST ↔ RPC parity
| Concept | RPC | REST |
|---------|-----|------|
| Charts | `listCharts` | `GET /api/v1/charts` |
| Data | `queryData` | `GET /api/v3/data` |
| Weights | `getWeights` | `GET /api/v3/weights` |
| Weights | `getWeights` | `GET /api/v1\|v2\|v3/weights` |
| Contexts | `listContexts` | `GET /api/v3/contexts` |
| Nodes | `getNodeInfo` | `GET /api/v3/nodes` |
| Alerts | `listAlerts` | `GET /api/v3/alerts` |
docs/DESKTOP.md
+12
View File
@@ -14,6 +14,7 @@ The client shell is a **Pear desktop application** built with `pear-electron` +
| `ui/styles.css` | Layout, theme, **titlebar drag regions** |
| `client/*` | HyperDHT connection stack used by the UI |
| `docs/DASHBOARD.md` | Master Charts / metrics wall plan |
| `user-guide/` | End-user workflows (Correlate, Related, Fleet, …) |
```bash
npm start # pear run -d .
@@ -188,8 +189,19 @@ Override root with `PEARDATA_HOME`. Writes are atomic (`tmp` → rename, mode `0
This template ships the **Pear run** path only. For Electron-forge / multi-arch standalone binaries, adapt packaging from a fuller product (e.g. peardock’s forge + bare-standalone scripts) once the app stabilizes.
## Charts investigation (summary)
| Control | Doc |
|---------|-----|
| Metrics wall, time, board, gestures | [user-guide/charts.md](../user-guide/charts.md) · [DASHBOARD.md](./DASHBOARD.md) |
| **Correlate** (highlight → Find Correlations) | [user-guide/metric-correlations.md](../user-guide/metric-correlations.md) |
| **Related (⇢)** (taxonomy + Pearson) | [user-guide/related-metrics.md](../user-guide/related-metrics.md) |
Implementation: `ui/dashboard.js`, `server/services/weights.js`, `shared/related-metrics.js`.
## Related
- [User guide](../user-guide/README.md)
- [GETTING-STARTED.md](./GETTING-STARTED.md)
- [ARCHITECTURE.md](./ARCHITECTURE.md)
- [CONFIGURATION.md](./CONFIGURATION.md)
docs/GETTING-STARTED.md
+1
View File
@@ -144,6 +144,7 @@ From-source Node agents can still use `npm run start:server` under a custom unit
## Next reading
- **[User guide](../user-guide/README.md)** — Charts, Metric Correlations, Fleet, Alerts
- [REST-API.md](./REST-API.md)
- [ARCHITECTURE.md](./ARCHITECTURE.md)
- [ROADMAP.md](./ROADMAP.md)
docs/PROTOCOL.md
+14
View File
@@ -51,10 +51,24 @@ Auth modes at handshake: public key (viewer), capability token / `pd1.` invite,
| `listCharts` | viewer | — | agent-style chart map |
| `getChart` | viewer | `{ id }` | Chart summary |
| `queryData` | viewer | `{ chart, after, before, points, group, tier }` | Time series |
| `getWeights` | viewer | see below | Metric Correlations / alert weights |
| `getAllMetrics` | viewer | `{ format: json\|prometheus\|shell }` | Latest export |
`after` / `before`: absolute unix seconds, or relative (negative = relative to `before`/`now`), agent-style.
#### `getWeights`
Scores charts for [Metric Correlations](../user-guide/metric-correlations.md) (or legacy alert weights when no highlight window).
| Arg | Notes |
|-----|--------|
| `method` | `volume` \| `ks2` \| `anomaly-rate` \| `value` \| `alerts` |
| `after` / `before` | Highlight window |
| `baseline_after` / `baseline_before` | Optional baseline (default 4× preceding highlight) |
| `points`, `time_group`, `contexts`, `charts`, `dimensions`, `limit`, `timeout` | Same as REST |
REST parity: `GET /api/v*/weights`. Engine: `server/services/weights.js`.
### Live subscriptions
| Method | Role | Args |
docs/README.md
+13 -3
View File
@@ -1,13 +1,22 @@
# Documentation index
## End users
| Doc | Contents |
|-----|----------|
| **[User guide](../user-guide/README.md)** | Desktop workflows: Connect, Overview, Charts, Metric Correlations, Related, Alerts, Fleet, Settings |
| [GETTING-STARTED.md](./GETTING-STARTED.md) | Install agent / desktop, first dial, systemd |
## Engineers & operators
| Doc | Audience | Contents |
|-----|----------|----------|
| [GETTING-STARTED.md](./GETTING-STARTED.md) | Operators | Install, run agent, REST, desktop, systemd |
| [ROADMAP.md](./ROADMAP.md) | Everyone | Phased MVP → advanced real-time features |
| [ARCHITECTURE.md](./ARCHITECTURE.md) | Engineers | Planes, PearDock mapping, module map |
| [PROTOCOL.md](./PROTOCOL.md) | Protocol owners | RPC methods, pushes, versioning |
| [DATA-MODEL.md](./DATA-MODEL.md) | Engineers | Metrics, anomalies, alerts, jobs |
| [DATA-MODEL.md](./DATA-MODEL.md) | Engineers | Metrics, anomalies, alerts, jobs, weights |
| [REST-API.md](./REST-API.md) | Integrators | agent-style `/api/v1\|v2\|v3` |
| [DASHBOARD.md](./DASHBOARD.md) | UI developers | Charts wall / Correlate / Related product plan |
| [TECH-CHOICES.md](./TECH-CHOICES.md) | Engineers | Collector, charts, libraries |
| [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | Engineers | HyperDB / Corestore / Autobase sync design |
| [DESKTOP.md](./DESKTOP.md) | UI developers | Pear shell, titlebar, identity |
@@ -17,5 +26,6 @@
| [CI.md](./CI.md) | Maintainers | GitHub/Gitea pipelines |
| [RELEASE.md](./RELEASE.md) | Maintainers | Version, tag, tarball |
| [EXTENDING.md](./EXTENDING.md) | Product builders | New charts, collectors, parents |
| [SWAGGER_PLAN.md](./SWAGGER_PLAN.md) | Integrators | Endpoint parity tracker |
Start here: **[GETTING-STARTED.md](./GETTING-STARTED.md)** · Product overview: **[../README.md](../README.md)**.
Start here (ops): **[GETTING-STARTED.md](./GETTING-STARTED.md)** · Day-to-day UI: **[User guide](../user-guide/README.md)** · Product overview: **[../README.md](../README.md)**.
docs/REST-API.md
+5 -1
View File
@@ -103,7 +103,7 @@ Single-agent returns one node. With `PEARDATA_PARENT=1`, `/nodes` and `/fleet` i
| Method | Path | Notes |
|--------|------|-------|
| GET | `/api/v3/q?q=` | Full-text over chart ids/titles |
| GET | `/api/v3/weights` | Per-chart anomaly weights (`?chart=&limit=`) |
| GET | `/api/v1\|v2\|v3/weights` | Metric Correlations + alert weights — see [Weights](#weights--metric-correlations); user guide: [weights-api](../user-guide/weights-api.md) |
### Alerts
@@ -141,6 +141,8 @@ Single-agent returns one node. With `PEARDATA_PARENT=1`, `/nodes` and `/fleet` i
`GET /api/v1/weights` · `/api/v2/weights` · `/api/v3/weights` (same engine; also RPC `getWeights`)
Desktop walkthrough: [user-guide/metric-correlations.md](../user-guide/metric-correlations.md) · Parameter cheat sheet: [user-guide/weights-api.md](../user-guide/weights-api.md)
| Param | Notes |
|-------|--------|
| `method` | `volume` (default with window), `ks2`, `anomaly-rate`, `value`, `alerts` |
@@ -155,6 +157,8 @@ Without a highlight window, `method` defaults to **alerts** (active anomaly/aler
Response includes `method`, `view.highlight` / `view.baseline`, and `results[{ id, weight, context, family, info }]`. Higher `weight` = more changed / interesting.
Engine: `server/services/weights.js`. Unit tests: `test/weights.test.js`.
## CORS
`Access-Control-Allow-Origin` defaults to `*` (override with `PEARDATA_REST_CORS`).
docs/ROADMAP.md
+1
View File
@@ -194,6 +194,7 @@ time + play/pause; multi-dim series; Overview stays the compact home grid.
| Doc | Role |
|-----|------|
| [User guide](../user-guide/README.md) | End-user desktop workflows (including Metric Correlations) |
| [DASHBOARD.md](./DASHBOARD.md) | Master Charts / metrics wall plan |
| [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | Warm store + swarm + Autobase |
| [CI.md](./CI.md) / [RELEASE.md](./RELEASE.md) | Binary matrix + installer |
docs/SWAGGER_PLAN.md
+7 -7
View File
@@ -171,10 +171,10 @@ Legend:
| Op | Path | PearData | Notes |
|----|------|----------|-------|
| GET | `/api/v1/weights` | **alias-miss** | Use v2/v3 |
| GET | `/api/v1/weights` | **done** | Same engine as v2/v3 |
| GET | `/api/v1/metric_correlations` | **skip** | Deprecated upstream |
| GET | `/api/v2/weights` | **partial** | Health-derived scores, not full MC/Anomaly Weight API |
| GET | `/api/v3/weights` | **partial** | Same |
| GET | `/api/v2/weights` | **done** | MC: `volume` / `ks2` / `anomaly-rate` / `value`; no window → `alerts` |
| GET | `/api/v3/weights` | **done** | Same; see [REST-API](./REST-API.md) + [user-guide/weights-api](../user-guide/weights-api.md). Multi-node `scope_nodes` / fleet aggregation still out of scope |
### B4. Nodes / info / health
@@ -265,7 +265,7 @@ Legend:
| Prometheus scrape (`allmetrics`) | yes | **done** |
| Badge SVG | yes | **partial** |
| Context search (`/q`) | yes | **partial** |
| Weights / anomaly scoring API | yes | **partial** (health MVP) |
| Weights / Metric Correlations API | yes | **done** (single-host; no multi-node scope) |
| v2/v3 `group_by` across nodes | yes (v2+) | **todo** |
| HTTP function execute | yes | **todo** (P2P today) |
| Full DynCFG | yes | **skip** (env + jobs) |
@@ -282,14 +282,14 @@ Legend:
### P1 — close swagger gaps that matter for tooling
1. Add **v1 aliases**: `/api/v1/contexts`, `/api/v1/context`, `/api/v1/weights`, `/api/v1/variable` → existing handlers
1. Add **v1 aliases**: `/api/v1/contexts`, `/api/v1/context`, `/api/v1/variable` → existing handlers (`/api/v1/weights` already done)
2. Add **v2 aliases**: `alert_transitions`, `alert_config`, `node_instances`, `versions`
3. Extend `allmetrics`: accept `format=prometheus_all_hosts` as alias of `prometheus`; honor `filter`, `prefix`
4. Document response-field parity for Grafana/Prometheus dashboards in `REST-API.md`
### P2 — deepen metrics APIs
1. Richer `/api/v3/weights` (per-chart anomaly scores already in engine)
1. ~~Richer `/api/v3/weights`~~ ✅ single-host MC engine (`volume` / `ks2` / `anomaly-rate` / `value`)
2. `/api/v1/alarm_count`, `/api/v1/alarms_values`, `/api/v1/alarm_log` thin wrappers
3. Optional PUT `/api/v3/settings` (localhost-only)
4. HTTP `function` bridge to `runJob` with strict localhost/admin gate
@@ -362,4 +362,4 @@ curl -s 'http://127.0.0.1:19999/api/v3/allmetrics?format=prometheus' | head
| Full swagger operation parity (72 ops) | **NO** (~38 matched; rest alias/skip/todo) |
| Cloud / WebRTC / DynCFG | **N/A by design** |
**Bottom line:** Every host metric chart PearData defines is collected. Every swagger endpoint required to *discover and scrape those metrics* is implemented on v1/v2/v3 as appropriate. Remaining work is API alias completeness, deeper alert/weights/function semantics, and optional extra collectors — not missing core system metrics.
**Bottom line:** Every host metric chart PearData defines is collected. Every swagger endpoint required to *discover and scrape those metrics* is implemented on v1/v2/v3 as appropriate. Weights / Metric Correlations are **done** for single-host (`volume` / `ks2` / `anomaly-rate` / `value`). Remaining work is API alias completeness, deeper alert/function semantics, multi-node weight scope, and optional extra collectors — not missing core system metrics.
docs/TESTING.md
+3
View File
@@ -16,6 +16,7 @@
| `test/crypto-auth.test.js` | MAC key, capabilities, admin proof, invites, classify input |
| `test/protocol.test.js` | Constants, monitoring `MethodRoles`, schema validators |
| `test/store.test.js` | Metric ring buffer ingest + query |
| `test/weights.test.js` | Metric Correlations engine (`volume` / `ks2` / windows / errors) |
| `test/hyperdb.test.js` | HyperDB model (node, links, warm points, alerts) |
| `test/rest.test.js` | agent-style `/api/v3` routes |
| `test/integration.test.js` | Live HyperDHT agent + handshake + metrics query |
@@ -84,6 +85,8 @@ When adding an RPC method:
| Viewer denial | Public key only → `runJob` / `mintInvite` fail with permission error |
| Health | With server up: `SERVER_PUBLIC_KEY=… npm run healthcheck` |
| Soak | `SERVER_PUBLIC_KEY=… SERVER_SEED=… npm run soak` |
| Metric Correlations | Charts → Correlate → brush ≥15s → Find Correlations → wall filters; Clear restores; Related ⇢ still works |
| Weights REST | `curl -sG 'http://127.0.0.1:19999/api/v3/weights' --data-urlencode 'method=volume' --data-urlencode 'after=-60' --data-urlencode 'before=0'` |
## Soak test