peardata/docs/ROADMAP.md

# PearData Roadmap

Phased plan from MVP agent → real-time fleet observability on pure P2P.

## Guiding principles

1. **Instant value** — connect a pubkey, see live charts in seconds.
2. **Agent efficiency** — stay in a tight ballpark for CPU/RAM overhead.
3. **PearDock patterns** — HyperDHT identity, protomux-rpc, roles, `pd1.` invites.
4. **Dual access** — P2P desktop + agent-style REST (`/api/v1|v2|v3`).
5. **Bare/Pear-ready** — no Node builtins at runtime; import maps → `bare-*`.
6. **Ecosystem glue** — ready for PearDock / PearVirt / HoneyPeer / BareOS later.

---

## Phase 0 — Foundation ✅

Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MVP, docs.

---

## Phase 1 — Hardening, packaging & UX ✅

| Item | Status |
|------|--------|
| Bare import maps + `bare-node-runtime` bootstrap | Done |
| Full system chart catalog + instance collectors | Done |
| One-line installer (`scripts/install.sh`) | Done |
| Bare Linux server + Electron clients (all arches) + CI | Done |
| brittle tests (protocol, crypto, store, REST, collector, HyperDB) | Done |
| Persistent peer bookmarks + aliases | Done |
| Multi-peer CPU compare overlay | Done |
| Instance / explore chart picker (`listCharts`) | Done |
| Alert silence TTL auto-reenable | Done |
| Reconnection (manager auto-reconnect) | Done |

**Phase 1 exit** — all criteria met (install path, dual archives, bookmarks, compare).

---

## Phase 2 — Storage & history depth ✅ / 🔄

| Item | Status |
|------|--------|
| HyperDB bee + Corestore (local warm) | Done |
| Hot path stays memory (no 1s → HyperDB) | Done |
| RPC peer-link + `getDbInfo` / REST `/api/v3/db` | Done |
| Query fallback: memory miss / long window → HyperDB warm | Done |
| Hyperswarm mesh (`PEARDATA_SWARM=1`) | Done (opt-in) |
| Configurable retention knobs (tier sizes) | Done (`PEARDATA_TIER*` + Data Manager RPC/UI) |
| Data Manager (usage, age rotate, auto-prune) | Done (`retention.json`, Settings → Data) |
| Documented warm history across restart (operator M4) | Done (`store-hyperdb-fallback` + STORAGE doc) |
| Export snapshot job → JSON / Prometheus push | Done (`exportSnapshot`, `prometheusPush`) |
| Autobase multi-writer parents | Later (Phase 2c) |
| Rocks engine for local-only desktop cache | Optional |

**Phase 2 exit criteria**

- [x] `queryData` / REST `/data` can return `source: "hyperdb-warm"`
- [x] Soak test: empty memory + reopen Corestore → `hyperdb-warm`
- [x] Linked peer pulls warm points (`source: "hyperdb-remote"`, `test/swarm-pull.test.js`)

---

## Phase 3 — Discovery & app collectors  ← **in progress**

| Item | Status |
|------|--------|
| Docker / container discovery | Done (`PEARDATA_DOCKER=1`; installer enables + `docker` group when detected; human names) |
| Offline banner (last-known samples) | Done |
| Process top-N | Done (opt-in `PEARDATA_PROCESSES=1`) |
| Service plugins | nginx + redis + postgres done |
| PearDock bridge | Done spike (`PEARDATA_PEARDOCK=1`, remap docker→peardock) |
| PearVirt / BareOS adapters | Thin translators into shared contexts |
| Holesail optional REST expose | Tunneled agent API |
| **Local AI (QVAC)** | Phase 0–1: tab + onboarding + tool RPCs + REST `/api/v3/ai/*` + keyword RAG + settings (profile/RAG/idle unload) + expanded tools (fleet/logs/alerts/silence); optional `@qvac/sdk` — see [QVAC.md](./QVAC.md) |

**Phase 3 exit criteria**

- [x] One container host can emit per-container CPU/mem charts (`docker.cpu.*` / `docker.mem.*`)
- [x] Plugin docs + Docker collector in [EXTENDING.md](./EXTENDING.md)
- [x] Process top-N opt-in collector (`processes.top_cpu` / `processes.top_rss`)
- [x] Plugin base + nginx / redis / postgres collectors

---

## Phase 4 — Smarter anomalies & fleet  ← **started**

| Item | Status |
|------|--------|
| Parent peer aggregation | Done spike (`PEARDATA_PARENT=1`) |
| Fleet REST / RPC | Done (`/api/v3/fleet`, `getFleetHealth`) |
| Desktop fleet strip | Done (`getFleetHealth` + multi-dial peers) |
| Anomaly scoring + UI highlight | Done (score + panel tint + threshold line) |
| Notifications | Done (desktop + `PEARDATA_WEBHOOK_URL`) |
| Streaming z-score / retrain job | Done (`PEARDATA_ANOMALY_MODE`, job `retrainAnomaly`) |
| `/api/v3/weights` / Metric Correlations | Done — highlight-vs-baseline scoring (`ks2`/`volume`/`anomaly-rate`/`value`) + Charts Correlate UI |
| `/api/v3/logs` / System Log tab | Done — journal (UI + omitted-source default) + anomaly + audit; Charts Filters menu |

---

## Phase 5 — Product polish

| Item | Notes |
|------|--------|
| Desktop persistence + multi-agent | Done — PearDock-style `cache/settings.json` + `cache/peers.json`, boot restore, last-active switch (`docs/DESKTOP.md`) |
| Signed / notarized macOS clients | Beyond ad-hoc `rcodesign` |
| Role templates | viewer / SRE / admin presets |
| Plugin SDK | Collector + chart registration API |
| Grafana cookbook | REST + Prometheus recipes |
| HoneyPeer directory | Opt-in agent presence |
| Windows collector depth | Close `/proc`-only gaps |

---

## Phase 6 — Master metrics dashboard (Charts tab)  ← **done + polish**

Turn Charts from a single-chart catalog into a full metrics wall (taxonomy TOC,
shared time, multi-dim cards, investigation). Detail checklist:
[DASHBOARD.md](./DASHBOARD.md).

| Item | Status |
|------|--------|
| Taxonomy sections + group catalog by context/family | Done (`shared/taxonomy.js`) |
| Long-scroll metrics wall + TOC jump links | Done (`ui/dashboard.js`) |
| Global time bar (play/pause, presets) | Done |
| Multi-dimension live + history cards | Done |
| Lazy visible-card query / paint | Done |
| Dim show/hide chips + basic synced hover | Done |
| Pan/zoom/reset + shared window from gestures | Done (P1) |
| Section KPIs, chart type, dim sort, resize | Done (P1) |
| Alert → scroll-to-chart + pause near event | Done |
| Persist metrics layout prefs + pins | Done |
| Group aggregation + force-play + related | Done |
| Board mode (pinned-only wall) | Done |
| Drag-reorder pins + keyboard shortcuts | Done |
| Taxonomy coverage test + EXTENDING note | Done |
| Desktop polish (layout fill, plot domain, legacy detail removed, Fleet chips) | Done |
| Metric Correlations (weights engine + Correlate brush UI) | Done |
| Charts Filters menu (collapsible TOC / search / group / sort) | Done |
| System Log tab (`queryLogs` — anomaly / audit / journal) | Done |
| Multi-named custom boards | Later |

**Phase 6 exit (P0)** — Charts tab lists every agent chart in sections; shared
time + play/pause; multi-dim series; Overview stays the compact home grid.

---

## Suggested near-term sequence

1. ~~HyperDB soak~~ ✅  
2. ~~Docker collector spike~~ ✅ (`PEARDATA_DOCKER=1`)  
3. ~~Offline banner~~ ✅  
4. ~~Process top-N~~ ✅ (`PEARDATA_PROCESSES=1`)  
5. ~~Parent peer prototype~~ ✅ (`PEARDATA_PARENT=1`)  
6. ~~Swarm / linked warm pull~~ ✅ (`hyperdb-remote`)  
7. ~~Postgres / Redis plugins~~ ✅  
8. ~~Desktop fleet strip~~ ✅  
9. ~~Export snapshot / Prometheus push~~ ✅  
10. ~~Anomaly scoring UI~~ ✅  
11. ~~Notifications~~ ✅ (desktop + webhook)  
12. ~~PearDock bridge~~ ✅ (`PEARDATA_PEARDOCK=1`)  
13. ~~Webhook HMAC signing~~ ✅ (`X-PearData-Signature`)  
14. ~~z-score / retrain job~~ ✅  
15. ~~REST HyperDHT tunnel~~ ✅ (`PEARDATA_REST_TUNNEL=1`)  
16. ~~Master metrics dashboard (Charts)~~ ✅ Phase 6 core (wall, pins, related, weights)  
17. Multi-named custom boards / Autobase parents — later  
18. Autobase multi-writer parents — HA fleet history  
19. Windows collector depth — close `/proc`-only gaps  
20. Plugin SDK polish — public collector registration API

---

## Non-goals (for now)

- Replacing centralized multi-tenant SaaS monitoring products  
- Third-party proprietary on-disk metric DB formats  
- Browser-only public dashboard without auth (REST stays localhost unless bound)  
- Byte-identical third-party plugin surface (system stats first; apps as plugins)

---

## Milestone checklist

| Milestone | You can… | Status |
|-----------|----------|--------|
| M0 | Agent + REST info | ✅ |
| M1 | Desktop live charts from pubkey | ✅ |
| M2 | Mint `pd1.` invite; revoke peer | ✅ |
| M3 | `curl \| bash` install; rolling server+client | ✅ |
| M4 | Historical scrub across restart (HyperDB warm) | ✅ |
| M5 | Container charts from Docker hosts | ✅ opt-in spike |
| M6 | Parent peer rolling up a fleet | ✅ opt-in spike |
| M7 | Charts tab = sectioned master metrics wall | ✅ Phase 6 core |

---

## Related docs

| Doc | Role |
|-----|------|
| [User guide](../user-guide/README.md) | End-user desktop workflows (Charts, Correlate, Logs, Fleet, …) |
| [DASHBOARD.md](./DASHBOARD.md) | Master Charts / metrics wall plan |
| [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | Warm store + swarm + Autobase |
| [CI.md](./CI.md) / [RELEASE.md](./RELEASE.md) | Binary matrix + installer |
| [TECH-CHOICES.md](./TECH-CHOICES.md) | Bare maps, collectors |
| [EXTENDING.md](./EXTENDING.md) | New charts / RPCs / plugins |
| [DESKTOP.md](./DESKTOP.md) | Pear / Electron shell |