350 lines
17 KiB
Markdown
350 lines
17 KiB
Markdown
# Swagger / metrics compatibility plan
|
||
|
||
Plan and verification for PearData’s agent-style REST surface (`/api/v1|v2|v3`):
|
||
|
||
- Spec shape: OpenAPI-style path inventory — **68 paths / 72 operations** used as a historical checklist (GET/POST/PUT)
|
||
- PearData router: `server/rest/routes.js`
|
||
- Chart catalog: `shared/metrics.js` + collectors under `server/services/`
|
||
|
||
This document answers two questions:
|
||
|
||
1. Which **REST operations** from that checklist does PearData implement?
|
||
2. Which **metric charts / dimensions** (the data those APIs serve) are collected?
|
||
|
||
Verified: **2026-07-18** (static analysis of routes + collector emit sites; unit tests `73/73`).
|
||
|
||
---
|
||
|
||
## Summary
|
||
|
||
| Surface | Upstream | PearData | Status |
|
||
|---------|----------|----------|--------|
|
||
| OpenAPI operations | 72 | **38 path matches** + PearData-only extras | Core metrics path covered; cloud/auth/WebRTC out of scope |
|
||
| Host static charts | (collector-defined; not enumerated in swagger) | **35 / 35** defined & emitted | Complete |
|
||
| Instance charts (cpu/disk/net/mount) | same | Patterns implemented in collector | Complete on Linux |
|
||
| Opt-in collectors | app plugins | docker / processes / nginx / redis / postgres / fleet / peardock | Complete when env-enabled |
|
||
| Data query (`/api/v*/data`) | v1–v3 | v1–v3 | Implemented (MVP grouping) |
|
||
| allmetrics export | shell / prometheus / json | shell / prometheus / json | Implemented (`prometheus_all_hosts` not separate) |
|
||
| Contexts | v1–v3 | **v2 + v3** (v1 missing) | Partial |
|
||
| Weights / search / logs | v1–v3 | v1–v3 weights + logs | Done (single-host) |
|
||
| Alerts | rich v1–v3 | core read paths | Partial |
|
||
| Cloud / claim / ACLK / bearer / WebRTC | many | none | Intentionally out of scope |
|
||
|
||
**Verdict:** All **core host metrics** required for agent-style dashboards and scrapers are implemented. REST **metric read** endpoints (`charts`, `data`, `allmetrics`, `contexts` v2/v3, badges, health) are implemented. Gaps are mostly **legacy v1 aliases**, **deep alert/config APIs**, **function execution over HTTP**, and **cloud/auth** endpoints that PearData replaces with P2P + HyperDHT.
|
||
|
||
---
|
||
|
||
## Part A — Metric charts (what scrapers/UI need)
|
||
|
||
Swagger describes HTTP shapes; the metric *inventory* lives in collectors. PearData’s target catalog (agent-compatible context IDs) is below.
|
||
|
||
### A1. Host-wide static charts — verified implemented
|
||
|
||
All IDs are in `STATIC_CHART_DEFS` and referenced by `server/services/collector.js` emit paths.
|
||
|
||
| Chart ID | Dimensions (key) | Units | Status |
|
||
|----------|------------------|-------|--------|
|
||
| `system.cpu` | guest_nice, guest, steal, softirq, irq, user, system, nice, iowait, idle | percentage | **done** |
|
||
| `system.intr` | interrupts | interrupts/s | **done** |
|
||
| `system.ctxt` | switches | context switches/s | **done** |
|
||
| `system.forks` | started | processes/s | **done** |
|
||
| `system.processes` | running, blocked | processes | **done** |
|
||
| `system.active_processes` | active | processes | **done** |
|
||
| `system.load` | load1, load5, load15 | load | **done** |
|
||
| `system.uptime` | uptime | seconds | **done** |
|
||
| `system.entropy` | entropy | entropy | **done** |
|
||
| `system.ram` | free, used, cached, buffers | MiB | **done** |
|
||
| `mem.available` | avail | MiB | **done** |
|
||
| `mem.swap` | free, used | MiB | **done** |
|
||
| `mem.swap_cached` | cached | MiB | **done** |
|
||
| `mem.kernel` | slab, kernel_stack, page_tables, vmalloc_used | MiB | **done** |
|
||
| `mem.slab` | reclaimable, unreclaimable | MiB | **done** |
|
||
| `mem.writeback` | dirty, writeback | MiB | **done** |
|
||
| `mem.committed` | Committed_AS | MiB | **done** |
|
||
| `mem.swapio` | in, out | KiB/s | **done** |
|
||
| `system.pgpgio` | in, out | KiB/s | **done** |
|
||
| `system.pgfaults` | minor, major | faults/s | **done** |
|
||
| `system.io` | in, out | KiB/s | **done** |
|
||
| `system.net` | received, sent | kilobits/s | **done** |
|
||
| `system.ip` | received, sent | kilobits/s | **done** |
|
||
| `system.ipv6` | received, sent | kilobits/s | **done** |
|
||
| `ip.tcppackets` | received, sent | packets/s | **done** |
|
||
| `ip.tcperrors` | InErrs, InCsumErrors, RetransSegs | packets/s | **done** |
|
||
| `ip.tcpopens` | active, passive | connections/s | **done** |
|
||
| `ip.tcpsock` | connections | connections | **done** |
|
||
| `ipv4.packets` | received, sent, forwarded, delivered | packets/s | **done** |
|
||
| `ipv4.errors` | InDiscards, OutDiscards, InHdrErrors, OutNoRoutes | packets/s | **done** |
|
||
| `ipv4.udppackets` | received, sent | packets/s | **done** |
|
||
| `ipv4.udperrors` | RcvbufErrors, SndbufErrors, InErrors, NoPorts | packets/s | **done** |
|
||
| `system.cpu_some_pressure` | some10, some60, some300 | percentage | **done** (Linux PSI) |
|
||
| `system.memory_some_pressure` | some10, some60, some300 | percentage | **done** (Linux PSI) |
|
||
| `system.io_some_pressure` | some10, some60, some300 | percentage | **done** (Linux PSI) |
|
||
|
||
**Score: 35 / 35 static host charts implemented.**
|
||
|
||
### A2. Instance charts — verified implemented (runtime)
|
||
|
||
| Pattern | Context | Dimensions | Status |
|
||
|---------|---------|------------|--------|
|
||
| `cpu.cpu{N}` | `cpu.cpu` | same as `system.cpu` | **done** |
|
||
| `disk_io.{dev}` | `disk.io` | reads, writes | **done** |
|
||
| `disk_ops.{dev}` | `disk.ops` | reads, writes | **done** |
|
||
| `disk_util.{dev}` | `disk.util` | utilization | **done** |
|
||
| `net.{iface}` | `net.net` | received, sent | **done** |
|
||
| `net_packets.{iface}` | `net.packets` | received, sent, multicast | **done** |
|
||
| `net_errors.{iface}` | `net.errors` | inbound, outbound | **done** |
|
||
| `net_drops.{iface}` | `net.drops` | inbound, outbound | **done** |
|
||
| `disk_space.{mount}` | `disk.space` | avail, used, reserved_for_root | **done** |
|
||
| `disk_inodes.{mount}` | `disk.inodes` | avail, used, reserved_for_root | **done** |
|
||
|
||
### A3. Opt-in / plugin charts — verified implemented
|
||
|
||
| Env gate | Charts | Status |
|
||
|----------|--------|--------|
|
||
| `PEARDATA_DOCKER=1` | `docker.containers`, `docker.cpu.*`, `docker.mem.*` (human titles via socket; installer auto-enables) | **done** |
|
||
| `PEARDATA_PROCESSES=1` | `processes.top_cpu`, `processes.top_rss` | **done** |
|
||
| `PEARDATA_NGINX=1` | `nginx.connections`, `nginx.requests` | **done** |
|
||
| `PEARDATA_REDIS=1` | `redis.memory`, `redis.clients`, `redis.stats` | **done** |
|
||
| `PEARDATA_POSTGRES=1` | `postgres.up`, `postgres.stats` | **done** |
|
||
| `PEARDATA_PARENT=1` | `fleet.cpu`, `fleet.ram`, `fleet.children` | **done** |
|
||
| `PEARDATA_PEARDOCK=1` | `peardock.*` (remapped) | **done** |
|
||
|
||
### A4. Metric gaps (not required by swagger paths, but common agent charts)
|
||
|
||
These appear in many agent installs but are **not** in PearData yet. Track as optional collector work:
|
||
|
||
| Family | Examples | Priority |
|
||
|--------|----------|----------|
|
||
| cgroup / systemd services | `cgroup.*`, `systemd.service.*` | P2 |
|
||
| apps.plugin | `apps.cpu`, `apps.mem`, `apps.vmem`, … | P2 |
|
||
| disk SMART / mdstat | `disk_await.*`, `md.health` | P3 |
|
||
| wireless / wifi | `wireless.*` | P3 |
|
||
| sensors / hwmon | `sensors.*` | P3 |
|
||
| eBPF / softnet | `softnet.*`, `ebpf.*` | P3 |
|
||
| Windows deep collectors | beyond `os` fallback CPU/RAM/load | P2 (roadmap) |
|
||
|
||
---
|
||
|
||
## Part B — OpenAPI endpoint matrix
|
||
|
||
Legend:
|
||
|
||
| Code | Meaning |
|
||
|------|---------|
|
||
| **done** | Path handled in `routes.js` with useful behavior |
|
||
| **partial** | Path exists but MVP / stub vs full upstream semantics |
|
||
| **alias-miss** | Upstream has path; PearData covers same capability on another version |
|
||
| **skip** | Out of scope (cloud, proprietary auth, deprecated) |
|
||
| **todo** | In scope for PearData; not implemented |
|
||
|
||
### B1. Charts & contexts (metrics catalog)
|
||
|
||
| Op | Path | PearData | Notes |
|
||
|----|------|----------|-------|
|
||
| GET | `/api/v1/charts` | **done** | Full chart map |
|
||
| GET | `/api/v1/chart` | **done** | Single chart |
|
||
| GET | `/api/v1/contexts` | **alias-miss** | Use `/api/v2\|v3/contexts` |
|
||
| GET | `/api/v1/context` | **alias-miss** | Use `/api/v2\|v3/context` |
|
||
| GET | `/api/v2/contexts` | **done** | |
|
||
| GET | `/api/v3/contexts` | **done** | Preferred |
|
||
| GET | `/api/v3/context` | **done** | |
|
||
| GET | `/api/v2/q` | **partial** | Chart id/title/family search |
|
||
| GET | `/api/v3/q` | **partial** | Same |
|
||
|
||
### B2. Data & export (core metrics APIs)
|
||
|
||
| Op | Path | PearData | Notes |
|
||
|----|------|----------|-------|
|
||
| GET | `/api/v1/data` | **done** | chart/context, after/before/points/group/tier/format |
|
||
| GET | `/api/v2/data` | **partial** | No full `group_by` / multi-node aggregation |
|
||
| GET | `/api/v3/data` | **partial** | Preferred; HyperDB warm fallback supported |
|
||
| GET | `/api/v1/allmetrics` | **done** | `json` \| `prometheus` \| `shell` |
|
||
| GET | `/api/v2/allmetrics` | **done** | |
|
||
| GET | `/api/v3/allmetrics` | **done** | Preferred; `prometheus_all_hosts` → treat as prometheus |
|
||
| GET | `/api/v1/badge.svg` | **partial** | Basic SVG; limited styling params |
|
||
| GET | `/api/v3/badge.svg` | **partial** | Same |
|
||
| GET | `/api/v1/variable` | **alias-miss** | Use `/api/v3/variable` |
|
||
| GET | `/api/v3/variable` | **partial** | Alert/variable lookup MVP |
|
||
|
||
### B3. Weights / scoring
|
||
|
||
| Op | Path | PearData | Notes |
|
||
|----|------|----------|-------|
|
||
| GET | `/api/v1/weights` | **done** | Same engine as v2/v3 |
|
||
| GET | `/api/v1/metric_correlations` | **skip** | Deprecated upstream |
|
||
| GET | `/api/v2/weights` | **done** | MC: `volume` / `ks2` / `anomaly-rate` / `value`; no window → `alerts` |
|
||
| GET | `/api/v3/weights` | **done** | Same; see [REST-API](./REST-API.md) + [user-guide/weights-api](../user-guide/weights-api.md). Multi-node `scope_nodes` / fleet aggregation still out of scope |
|
||
| GET | `/api/v1\|v2\|v3/logs` | **done** | PearData extension — anomaly / audit / journal (`queryLogs`); see [REST-API](./REST-API.md) + [user-guide/logs](../user-guide/logs.md) |
|
||
|
||
### B4. Nodes / info / health
|
||
|
||
| Op | Path | PearData | Notes |
|
||
|----|------|----------|-------|
|
||
| GET | `/api/v1/info` | **done** | |
|
||
| GET | `/api/v2/info` | **done** | |
|
||
| GET | `/api/v3/info` | **done** | Includes HyperDB metadata |
|
||
| GET | `/api/v2/nodes` | **done** | Single node; parent expands children |
|
||
| GET | `/api/v3/nodes` | **done** | |
|
||
| GET | `/api/v2/node_instances` | **alias-miss** | Use `/api/v3/node_instances` |
|
||
| GET | `/api/v3/node_instances` | **done** | |
|
||
| GET | `/api/v2/versions` | **alias-miss** | Use `/api/v3/versions` |
|
||
| GET | `/api/v3/versions` | **done** | |
|
||
| GET | `/api/v3/stream_path` | **partial** | Local + parent children hops |
|
||
| GET | `/api/v3/stream_info` | **todo** | Streaming stats (map to parent/P2P later) |
|
||
| GET | `/health`, `/api/v1/health`, `/api/v3/health` | **done** | PearData extension (not all in swagger) |
|
||
|
||
### B5. Alerts
|
||
|
||
| Op | Path | PearData | Notes |
|
||
|----|------|----------|-------|
|
||
| GET | `/api/v1/alarms` | **done** | |
|
||
| GET | `/api/v1/alarms_values` | **todo** | |
|
||
| GET | `/api/v1/alarm_log` | **todo** | Transitions ≈ `/api/v3/alert_transitions` |
|
||
| GET | `/api/v1/alarm_count` | **todo** | Derivable from alerts list |
|
||
| GET | `/api/v1/alarm_variables` | **partial** | |
|
||
| GET | `/api/v2/alerts` | **done** | |
|
||
| GET | `/api/v3/alerts` | **done** | Preferred |
|
||
| GET | `/api/v2/alert_transitions` | **alias-miss** | Use v3 |
|
||
| GET | `/api/v3/alert_transitions` | **partial** | Recent anomaly events |
|
||
| GET | `/api/v2/alert_config` | **alias-miss** | Use v3 |
|
||
| GET | `/api/v3/alert_config` | **partial** | List configs |
|
||
|
||
### B6. Functions / config / settings
|
||
|
||
| Op | Path | PearData | Notes |
|
||
|----|------|----------|-------|
|
||
| GET | `/api/v1/functions` | **alias-miss** | Use v2/v3 |
|
||
| GET | `/api/v1/function` | **skip** / **todo** | Execute via P2P `runJob` |
|
||
| GET | `/api/v2/functions` | **partial** | List only |
|
||
| GET | `/api/v3/functions` | **partial** | List only |
|
||
| GET/POST | `/api/v3/function` | **todo** | Prefer authenticated P2P |
|
||
| GET | `/api/v2/progress` | **skip** | |
|
||
| GET | `/api/v3/progress` | **todo** | Only if HTTP functions land |
|
||
| GET | `/api/v3/settings` | **partial** | Read-only runtime knobs |
|
||
| PUT | `/api/v3/settings` | **todo** | GET returns retention + storage; writes use P2P admin RPC (`setRetentionConfig` / `pruneHistory`) |
|
||
| GET/POST | `/api/v1/config` | **skip** | DynCFG; use env + P2P |
|
||
| GET/POST | `/api/v3/config` | **partial** | Aliases settings GET; no DynCFG tree |
|
||
|
||
### B7. Auth / cloud / realtime — out of scope
|
||
|
||
| Op | Path | PearData | Notes |
|
||
|----|------|----------|-------|
|
||
| GET | `/api/v3/me` | **partial** | Anonymous REST identity note |
|
||
| GET | `/api/v*/bearer_*` | **skip** | Future optional bearer |
|
||
| GET | `/api/v*/claim` | **skip** | Cloud claiming N/A |
|
||
| GET | `/api/v1/aclk` | **skip** | Cloud link N/A |
|
||
| GET | `/api/v1/registry` | **skip** | Deprecated |
|
||
| GET | `/api/v1/manage/health` | **skip** | Use P2P alert ops |
|
||
| GET | `/api/v1/dbengine_stats` | **skip** | Use `/api/v3/db` (HyperDB) |
|
||
| GET | `/api/v1/ml_info` | **skip** | Deprecated; anomaly via engine |
|
||
| POST | `/api/v*/rtc_offer` | **skip** | WebRTC N/A; use HyperDHT / tunnel |
|
||
|
||
### B8. PearData-only REST (not in swagger)
|
||
|
||
| Op | Path | Purpose |
|
||
|----|------|---------|
|
||
| GET | `/api/v3/fleet` | Parent fleet health |
|
||
| GET | `/api/v3/export` | Snapshot / prometheus write |
|
||
| GET | `/api/v3/tunnel` | HyperDHT REST tunnel status |
|
||
| GET | `/api/v3/db` | HyperDB keys + collections |
|
||
| GET | `/`, `/api` | Service index |
|
||
|
||
---
|
||
|
||
## Part C — Implementation depth checklist (metrics path)
|
||
|
||
| Capability | Required by swagger consumers | PearData |
|
||
|------------|-------------------------------|----------|
|
||
| List charts with dimensions | yes | **done** |
|
||
| Query time-series by chart | yes | **done** |
|
||
| Query by context | yes | **done** |
|
||
| Relative `after=-60` | yes | **done** |
|
||
| `points` + downsample | yes | **done** |
|
||
| `group=average\|min\|max\|sum` | yes | **done** |
|
||
| Multi-format data (`json`, csv, array) | yes | **done** |
|
||
| Prometheus scrape (`allmetrics`) | yes | **done** |
|
||
| Badge SVG | yes | **partial** |
|
||
| Context search (`/q`) | yes | **partial** |
|
||
| Weights / Metric Correlations API | yes | **done** (single-host; no multi-node scope) |
|
||
| v2/v3 `group_by` across nodes | yes (v2+) | **todo** |
|
||
| HTTP function execute | yes | **todo** (P2P today) |
|
||
| Full DynCFG | yes | **skip** (env + jobs) |
|
||
|
||
---
|
||
|
||
## Part D — Work plan (priority)
|
||
|
||
### P0 — already satisfied for metrics consumers
|
||
|
||
- Host chart catalog + Linux collectors
|
||
- `/api/v1/charts`, `/api/v*/data`, `/api/v*/allmetrics`
|
||
- `/api/v3/contexts`, health, nodes, badges (basic)
|
||
|
||
### P1 — close swagger gaps that matter for tooling
|
||
|
||
1. Add **v1 aliases**: `/api/v1/contexts`, `/api/v1/context`, `/api/v1/variable` → existing handlers (`/api/v1/weights` already done)
|
||
2. Add **v2 aliases**: `alert_transitions`, `alert_config`, `node_instances`, `versions`
|
||
3. Extend `allmetrics`: accept `format=prometheus_all_hosts` as alias of `prometheus`; honor `filter`, `prefix`
|
||
4. Document response-field parity for Grafana/Prometheus dashboards in `REST-API.md`
|
||
|
||
### P2 — deepen metrics APIs
|
||
|
||
1. ~~Richer `/api/v3/weights`~~ ✅ single-host MC engine (`volume` / `ks2` / `anomaly-rate` / `value`)
|
||
2. `/api/v1/alarm_count`, `/api/v1/alarms_values`, `/api/v1/alarm_log` thin wrappers
|
||
3. Optional PUT `/api/v3/settings` (localhost-only) — today prefer P2P Data Manager RPC
|
||
4. HTTP `function` bridge to `runJob` with strict localhost/admin gate
|
||
5. `/api/v3/stream_info` from parent collector stats
|
||
|
||
### P3 — optional collectors (not swagger-listed)
|
||
|
||
- cgroup / apps.plugin-style process trees
|
||
- sensors, SMART, md
|
||
- Windows collector depth
|
||
|
||
### Explicit non-goals
|
||
|
||
- Cloud claim / ACLK / registry
|
||
- WebRTC offers
|
||
- Full DynCFG tree parity
|
||
- Bearer auth (until someone binds REST publicly)
|
||
|
||
---
|
||
|
||
## Part E — How to re-verify
|
||
|
||
```bash
|
||
# 1) Diff documented REST paths against the router
|
||
rg -o '/api/v[123]/[a-zA-Z0-9_./{}-]+' docs/REST-API.md docs/SWAGGER_PLAN.md | sort -u > /tmp/doc-paths.txt
|
||
rg -o '/api/v[123]/[a-zA-Z0-9_./-]+' server/rest/routes.js | sort -u > /tmp/route-paths.txt
|
||
comm -23 /tmp/doc-paths.txt /tmp/route-paths.txt || true
|
||
|
||
# 2) Confirm every static chart is emitted
|
||
node -e "
|
||
import { readFileSync } from 'fs'
|
||
import { STATIC_CHART_DEFS } from './shared/metrics.js'
|
||
const c = readFileSync('server/services/collector.js','utf8')
|
||
const missing = STATIC_CHART_DEFS.filter(d => !c.includes(d.id)).map(d => d.id)
|
||
console.log(missing.length ? missing : 'all static charts referenced')
|
||
"
|
||
|
||
# 3) Live smoke (agent running on :18888)
|
||
curl -s http://127.0.0.1:18888/api/v1/charts | jq '.charts | keys | length'
|
||
curl -s 'http://127.0.0.1:18888/api/v3/data?chart=system.cpu&after=-60&points=30' | jq '.labels,.points'
|
||
curl -s 'http://127.0.0.1:18888/api/v3/allmetrics?format=prometheus' | head
|
||
```
|
||
|
||
---
|
||
|
||
## Part F — Quick status board
|
||
|
||
| Area | Implemented? |
|
||
|------|----------------|
|
||
| All 35 static host metric charts | **YES** |
|
||
| Per-CPU / disk / net / mount instance charts | **YES** |
|
||
| Opt-in docker/nginx/redis/postgres/processes/fleet | **YES** (env-gated) |
|
||
| Swagger core metrics REST (`charts`/`data`/`allmetrics`/`contexts` v3) | **YES** |
|
||
| Full swagger operation parity (72 ops) | **NO** (~38 matched; rest alias/skip/todo) |
|
||
| Cloud / WebRTC / DynCFG | **N/A by design** |
|
||
|
||
**Bottom line:** Every host metric chart PearData defines is collected. Every swagger endpoint required to *discover and scrape those metrics* is implemented on v1/v2/v3 as appropriate. Weights / Metric Correlations and Logs (`/api/v*/logs`) are **done** for single-host. Default REST port is **18888**. Remaining work is API alias completeness, deeper alert/function semantics, multi-node weight scope, and optional extra collectors — not missing core system metrics.
|