peardata/docs/ROADMAP.md

PearData Roadmap

Phased plan from MVP agent → real-time fleet observability on pure P2P.

Guiding principles

1. Instant value — connect a pubkey, see live charts in seconds.
2. Agent efficiency — stay in a tight ballpark for CPU/RAM overhead.
3. PearDock patterns — HyperDHT identity, protomux-rpc, roles, pd1. invites.
4. Dual access — P2P desktop + agent-style REST (/api/v1|v2|v3).
5. Bare/Pear-ready — no Node builtins at runtime; import maps → bare-*.
6. Ecosystem glue — ready for PearDock / PearVirt / HoneyPeer / BareOS later.

---

Phase 0 — Foundation ✅

Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MVP, docs.

---

Phase 1 — Hardening, packaging & UX ✅

Item	Status
Bare import maps + bare-node-runtime bootstrap	Done
Full system chart catalog + instance collectors	Done
One-line installer (scripts/install.sh)	Done
Bare Linux server + Electron clients (all arches) + CI	Done
brittle tests (protocol, crypto, store, REST, collector, HyperDB)	Done
Persistent peer bookmarks + aliases	Done
Multi-peer CPU compare overlay	Done
Instance / explore chart picker (listCharts)	Done
Alert silence TTL auto-reenable	Done
Reconnection (manager auto-reconnect)	Done

Phase 1 exit — all criteria met (install path, dual archives, bookmarks, compare).

---

Phase 2 — Storage & history depth ✅ / 🔄

Item	Status
HyperDB bee + Corestore (local warm)	Done
Hot path stays memory (no 1s → HyperDB)	Done
RPC peer-link + getDbInfo / REST /api/v3/db	Done
Query fallback: memory miss / long window → HyperDB warm	Done
Hyperswarm mesh (PEARDATA_SWARM=1)	Done (opt-in)
Configurable retention knobs (tier sizes)	Done (PEARDATA_TIER* + Data Manager RPC/UI)
Data Manager (usage, age rotate, auto-prune)	Done (retention.json, Settings → Data)
Documented warm history across restart (operator M4)	Done (store-hyperdb-fallback + STORAGE doc)
Export snapshot job → JSON / Prometheus push	Done (exportSnapshot, prometheusPush)
Autobase multi-writer parents	Later (Phase 2c)
Rocks engine for local-only desktop cache	Optional

Phase 2 exit criteria

• queryData / REST /data can return source: "hyperdb-warm"
• Soak test: empty memory + reopen Corestore → hyperdb-warm
• Linked peer pulls warm points (source: "hyperdb-remote", test/swarm-pull.test.js)

---

Phase 3 — Discovery & app collectors ← in progress

Item	Status
Docker / container discovery	Done (PEARDATA_DOCKER=1; installer enables + docker group when detected; human names)
Offline banner (last-known samples)	Done
Process top-N	Done (opt-in PEARDATA_PROCESSES=1)
Service plugins	nginx + redis + postgres done
PearDock bridge	Done spike (PEARDATA_PEARDOCK=1, remap docker→peardock)
PearVirt / BareOS adapters	Thin translators into shared contexts
Holesail optional REST expose	Tunneled agent API
Local AI (QVAC)	Phase 0–1: tab + onboarding + tool RPCs + REST /api/v3/ai/* + keyword RAG + settings (profile/RAG/idle unload) + expanded tools (fleet/logs/alerts/silence); optional @qvac/sdk — see QVAC.md

Phase 3 exit criteria

• One container host can emit per-container CPU/mem charts (docker.cpu.* / docker.mem.*)
• Plugin docs + Docker collector in EXTENDING.md
• Process top-N opt-in collector (processes.top_cpu / processes.top_rss)
• Plugin base + nginx / redis / postgres collectors

---

Phase 4 — Smarter anomalies & fleet ← started

Item	Status
Parent peer aggregation	Done spike (PEARDATA_PARENT=1)
Fleet REST / RPC	Done (/api/v3/fleet, getFleetHealth)
Desktop fleet strip	Done (getFleetHealth + multi-dial peers)
Anomaly scoring + UI highlight	Done (score + panel tint + threshold line)
Notifications	Done (desktop + PEARDATA_WEBHOOK_URL)
Streaming z-score / retrain job	Done (PEARDATA_ANOMALY_MODE, job retrainAnomaly)
/api/v3/weights / Metric Correlations	Done — highlight-vs-baseline scoring (ks2/volume/anomaly-rate/value) + Charts Correlate UI
/api/v3/logs / System Log tab	Done — journal (UI + omitted-source default) + anomaly + audit; Charts Filters menu

---

Phase 5 — Product polish

Item	Notes
Desktop persistence + multi-agent	Done — PearDock-style cache/settings.json + cache/peers.json, boot restore, last-active switch (docs/DESKTOP.md)
Signed / notarized macOS clients	Beyond ad-hoc rcodesign
Role templates	viewer / SRE / admin presets
Plugin SDK	Collector + chart registration API
Grafana cookbook	REST + Prometheus recipes
HoneyPeer directory	Opt-in agent presence
Windows collector depth	Close /proc-only gaps

---

Phase 6 — Master metrics dashboard (Charts tab) ← done + polish

Turn Charts from a single-chart catalog into a full metrics wall (taxonomy TOC, shared time, multi-dim cards, investigation). Detail checklist: DASHBOARD.md.

Item	Status
Taxonomy sections + group catalog by context/family	Done (shared/taxonomy.js)
Long-scroll metrics wall + TOC jump links	Done (ui/dashboard.js)
Global time bar (play/pause, presets)	Done
Multi-dimension live + history cards	Done
Lazy visible-card query / paint	Done
Dim show/hide chips + basic synced hover	Done
Pan/zoom/reset + shared window from gestures	Done (P1)
Section KPIs, chart type, dim sort, resize	Done (P1)
Alert → scroll-to-chart + pause near event	Done
Persist metrics layout prefs + pins	Done
Group aggregation + force-play + related	Done
Board mode (pinned-only wall)	Done
Drag-reorder pins + keyboard shortcuts	Done
Taxonomy coverage test + EXTENDING note	Done
Desktop polish (layout fill, plot domain, legacy detail removed, Fleet chips)	Done
Metric Correlations (weights engine + Correlate brush UI)	Done
Charts Filters menu (collapsible TOC / search / group / sort)	Done
System Log tab (queryLogs — anomaly / audit / journal)	Done
Multi-named custom boards	Later

Phase 6 exit (P0) — Charts tab lists every agent chart in sections; shared time + play/pause; multi-dim series; Overview stays the compact home grid.

---

Suggested near-term sequence

1. HyperDB soak ✅
2. Docker collector spike ✅ (PEARDATA_DOCKER=1)
3. Offline banner ✅
4. Process top-N ✅ (PEARDATA_PROCESSES=1)
5. Parent peer prototype ✅ (PEARDATA_PARENT=1)
6. Swarm / linked warm pull ✅ (hyperdb-remote)
7. Postgres / Redis plugins ✅
8. Desktop fleet strip ✅
9. Export snapshot / Prometheus push ✅
10. Anomaly scoring UI ✅
11. Notifications ✅ (desktop + webhook)
12. PearDock bridge ✅ (PEARDATA_PEARDOCK=1)
13. Webhook HMAC signing ✅ (X-PearData-Signature)
14. z-score / retrain job ✅
15. REST HyperDHT tunnel ✅ (PEARDATA_REST_TUNNEL=1)
16. Master metrics dashboard (Charts) ✅ Phase 6 core (wall, pins, related, weights)
17. Multi-named custom boards / Autobase parents — later
18. Autobase multi-writer parents — HA fleet history
19. Windows collector depth — close /proc-only gaps
20. Plugin SDK polish — public collector registration API

---

Non-goals (for now)

• Replacing centralized multi-tenant SaaS monitoring products
• Third-party proprietary on-disk metric DB formats
• Browser-only public dashboard without auth (REST stays localhost unless bound)
• Byte-identical third-party plugin surface (system stats first; apps as plugins)

---

Milestone checklist

Milestone	You can…	Status
M0	Agent + REST info	✅
M1	Desktop live charts from pubkey	✅
M2	Mint pd1. invite; revoke peer	✅
M3	curl | bash install; rolling server+client	✅
M4	Historical scrub across restart (HyperDB warm)	✅
M5	Container charts from Docker hosts	✅ opt-in spike
M6	Parent peer rolling up a fleet	✅ opt-in spike
M7	Charts tab = sectioned master metrics wall	✅ Phase 6 core

---

Related docs

Doc	Role
User guide	End-user desktop workflows (Charts, Correlate, Logs, Fleet, …)
DASHBOARD.md	Master Charts / metrics wall plan
STORAGE-HYPERDB.md	Warm store + swarm + Autobase
CI.md / RELEASE.md	Binary matrix + installer
TECH-CHOICES.md	Bare maps, collectors
EXTENDING.md	New charts / RPCs / plugins
DESKTOP.md	Pear / Electron shell