snxraven/peardata
0
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

Update Docs
CI / test (push) Successful in 1m1s
Details
Release rolling / release (push) Has been cancelled
Details

Browse Source
This commit is contained in:
Raven Scott
2026-07-19 13:35:19 -04:00
parent 584e1c9c42
commit 5521f2952a
25 changed files with 191 additions and 64 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+9 -8
View File
@@ -27,6 +27,7 @@ curl -fsSL https://git.ssh.surf/snxraven/peardata/raw/branch/main/scripts/instal
```bash ```bash
# Server only (Linux Bare + systemd → /opt/peardata) # Server only (Linux Bare + systemd → /opt/peardata)
# Enables journal Logs + Docker metrics/names when Docker is detected
curl -fsSL https://git.ssh.surf/snxraven/peardata/raw/branch/main/scripts/install.sh | bash -s -- --server --yes curl -fsSL https://git.ssh.surf/snxraven/peardata/raw/branch/main/scripts/install.sh | bash -s -- --server --yes
# Desktop client only # Desktop client only
@@ -77,13 +78,13 @@ peardata/
│ ├── pipeline.js # collector → store → anomaly → push │ ├── pipeline.js # collector → store → anomaly → push
│ ├── db/ # HyperDB model + Corestore + swarm replicate │ ├── db/ # HyperDB model + Corestore + swarm replicate
│ ├── handlers/monitor.js # Domain RPCs │ ├── handlers/monitor.js # Domain RPCs
│ ├── services/ # collector, store, warm-flush, anomaly, jobs │ ├── services/ # collector, store, retention, warm-flush, anomaly, jobs
│ ├── rest/ # agent-compatible HTTP API │ ├── rest/ # agent-compatible HTTP API
│ └── core/ rpc/ utils/ # Auth, ACL, sessions (template) │ └── core/ rpc/ utils/ # Auth, ACL, sessions (template)
├── client/ # Multi-peer connection manager ├── client/ # Multi-peer connection manager
├── user-guide/ # End-user desktop workflows ├── user-guide/ # End-user desktop workflows
├── docs/ # Engineer / operator documentation ├── docs/ # Engineer / operator documentation
├── deploy/peardata.service # systemd ├── deploy/peardata.service # systemd (installer also writes journal/docker groups)
└── test/ └── test/
``` ```
@@ -122,17 +123,17 @@ peardata/
| Doc | Contents | | Doc | Contents |
|-----|----------| |-----|----------|
| **[User guide](./user-guide/README.md)** | Desktop workflows — Charts, Metric Correlations, Logs, Fleet, Alerts | | **[User guide](./user-guide/README.md)** | Desktop — Charts, Correlations, Logs, Containers, Fleet, Settings / Data Manager |
| [Getting started](./docs/GETTING-STARTED.md) | Install, agent, REST, desktop, systemd | | [Getting started](./docs/GETTING-STARTED.md) | Install, agent, REST, desktop, systemd (journal + Docker) |
| [Docs index](./docs/README.md) | Full engineer / operator doc set | | [Docs index](./docs/README.md) | Full engineer / operator doc set |
| [Roadmap](./docs/ROADMAP.md) | MVP → advanced phases | | [Roadmap](./docs/ROADMAP.md) | MVP → advanced phases |
| [Architecture](./docs/ARCHITECTURE.md) | PearDock mapping, planes, modules | | [Architecture](./docs/ARCHITECTURE.md) | PearDock mapping, planes, modules |
| [Protocol](./docs/PROTOCOL.md) | RPC methods & pushes | | [Protocol](./docs/PROTOCOL.md) | RPC methods & pushes (incl. Data Manager) |
| [Data model](./docs/DATA-MODEL.md) | Metrics, anomalies, health, weights, logs | | [Data model](./docs/DATA-MODEL.md) | Metrics, anomalies, health, weights, logs, retention tiers |
| [REST API](./docs/REST-API.md) | `/api/v1\|v2\|v3` | | [REST API](./docs/REST-API.md) | `/api/v1\|v2\|v3` |
| [Dashboard](./docs/DASHBOARD.md) | Charts wall / Correlate product notes | | [Dashboard](./docs/DASHBOARD.md) | Charts wall / Correlate / Containers naming |
| [Tech choices](./docs/TECH-CHOICES.md) | Collector, charts, Bare maps | | [Tech choices](./docs/TECH-CHOICES.md) | Collector, charts, Bare maps |
| [HyperDB storage](./docs/STORAGE-HYPERDB.md) | Warm history, peer links, swarm sync | | [HyperDB storage](./docs/STORAGE-HYPERDB.md) | Warm history, peer links, swarm sync, prune |
| [Release](./docs/RELEASE.md) | Binary host matrix + rolling CI | | [Release](./docs/RELEASE.md) | Binary host matrix + rolling CI |
| [CI](./docs/CI.md) | Gitea / GitHub pipelines | | [CI](./docs/CI.md) | Gitea / GitHub pipelines |
| [Security](./docs/SECURITY.md) | Threat model & hardening | | [Security](./docs/SECURITY.md) | Threat model & hardening |
docs/ARCHITECTURE.md
+6 -3
View File
@@ -58,7 +58,8 @@ Handshake, node info, chart/context catalog, alert config, invites, jobs, ACL.
### 2. High-frequency metrics ### 2. High-frequency metrics
- **Ingest:** collector emits sample batches every `PEARDATA_SAMPLE_MS` (default 1000). - **Ingest:** collector emits sample batches every `PEARDATA_SAMPLE_MS` (default 1000).
- **Store:** tier0 (1s) + tier1 (downsampled averages). - **Store:** tier0 (1s) + tier1 (downsampled averages) + HyperDB warm flush.
- **Retention:** `server/services/retention.js` — `retention.json`, auto-prune (default warm age **1 year**), Data Manager RPC.
- **Push:** `push:metrics` to subscribed peers (protomux-rpc events). - **Push:** `push:metrics` to subscribed peers (protomux-rpc events).
- **Pull:** `queryData` / REST `/api/v3/data` for history windows. - **Pull:** `queryData` / REST `/api/v3/data` for history windows.
@@ -147,13 +148,15 @@ A heavier agent may subscribe to child agents over P2P, downsample into its own
| `shared/metrics.js` | Chart/context catalog | | `shared/metrics.js` | Chart/context catalog |
| `shared/data-model.js` | Typed shapes | | `shared/data-model.js` | Typed shapes |
| `server/services/collector.js` | System sampling | | `server/services/collector.js` | System sampling |
| `server/services/store.js` | Tiered buffers + query | | `server/services/store.js` | Tiered buffers + query + live retention trim |
| `server/services/retention.js` | Data Manager policy, auto-prune, storage usage |
| `server/services/anomaly.js` | Thresholds | | `server/services/anomaly.js` | Thresholds |
| `server/services/weights.js` | Metric Correlations scoring (`volume` / `ks2` / …) | | `server/services/weights.js` | Metric Correlations scoring (`volume` / `ks2` / …) |
| `server/services/logs.js` | System log query (anomaly / audit / journalctl) | | `server/services/logs.js` | System log query (anomaly / audit / journalctl) |
| `server/services/alerts.js` | Alert CRUD helpers | | `server/services/alerts.js` | Alert CRUD helpers |
| `server/services/subscriptions.js` | Push fan-out | | `server/services/subscriptions.js` | Push fan-out |
| `server/services/jobs.js` | On-demand jobs | | `server/services/jobs.js` | On-demand jobs (`gcBuffers` → prune) |
| `server/services/collectors/docker.js` | Opt-in Docker metrics + socket name enrichment |
| `server/handlers/monitor.js` | RPC surface | | `server/handlers/monitor.js` | RPC surface |
| `server/rest/*` | Agent HTTP API | | `server/rest/*` | Agent HTTP API |
| `server/pipeline.js` | Wire collector→store→push | | `server/pipeline.js` | Wire collector→store→push |
docs/CI.md
+1 -1
View File
@@ -61,7 +61,7 @@ DRY_RUN=1 bash scripts/gitea-rolling-release.sh
| Script | Role | | Script | Role |
|--------|------| |--------|------|
| `scripts/install.sh` | End-user installer (server systemd + desktop client) | | `scripts/install.sh` | End-user installer (server systemd + desktop client; journal + Docker socket when detected) |
| `scripts/hosts.cjs` | Host lists (`SERVER_LINUX`, `ALL_64`) | | `scripts/hosts.cjs` | Host lists (`SERVER_LINUX`, `ALL_64`) |
| `scripts/make.cjs` | Orchestrate server/client builds | | `scripts/make.cjs` | Orchestrate server/client builds |
| `scripts/bare-standalone.cjs` | Pack Bare `peardata-server` | | `scripts/bare-standalone.cjs` | Pack Bare `peardata-server` |
docs/CONFIGURATION.md
+7 -5
View File
@@ -45,7 +45,9 @@ Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators.
``` ```
data/ data/
├── peer-policy.json ├── peer-policy.json
└── audit.log ├── retention.json # Data Manager policy (after first boot / Save)
├── audit.log
└── corestore/ # HyperDB warm history
``` ```
Permissions: directory `0700`. Do **not** commit `data/` or `.env`. Permissions: directory `0700`. Do **not** commit `data/` or `.env`.
@@ -60,12 +62,12 @@ Permissions: directory `0700`. Do **not** commit `data/` or `.env`.
| `PEARDATA_TIER0_POINTS` | `3600` | High-res ring size (~1h @ 1s); seeds Data Manager | | `PEARDATA_TIER0_POINTS` | `3600` | High-res ring size (~1h @ 1s); seeds Data Manager |
| `PEARDATA_TIER1_POINTS` | `1440` | Downsampled ring size; seeds Data Manager | | `PEARDATA_TIER1_POINTS` | `1440` | Downsampled ring size; seeds Data Manager |
| `PEARDATA_TIER1_EVERY` | `60` | Samples per tier1 average (also HyperDB warm flush) | | `PEARDATA_TIER1_EVERY` | `60` | Samples per tier1 average (also HyperDB warm flush) |
| `PEARDATA_WARM_RETENTION_MS` | `31536000000` (1y) | Soft age limit for HyperDB warm points (seed) | | `PEARDATA_WARM_RETENTION_MS` | `31536000000` (1y) | Soft age limit for HyperDB warm points (seed; default preset **1 year**) |
| `PEARDATA_WARM_MAX_BYTES` | `0` | Soft corestore budget in bytes (`0` = unlimited) | | `PEARDATA_WARM_MAX_BYTES` | `0` | Soft corestore budget in bytes (`0` = unlimited) |
| `PEARDATA_WARM_MAX_POINTS` | `0` | Cap on warm points (`0` = unlimited) | | `PEARDATA_WARM_MAX_POINTS` | `0` | Cap on warm points (`0` = unlimited) |
| `PEARDATA_AUTO_PRUNE` | on | `0` disables scheduled prune on boot | | `PEARDATA_AUTO_PRUNE` | on | `0` disables scheduled prune on boot |
Live retention is edited in the desktop **Settings → Data** tab (admin) and persisted to `$PEARDATA_DATA_DIR/retention.json`. Env values only seed the file on first boot. Live retention is edited in the desktop **Settings → Data** tab (admin) and persisted to `$PEARDATA_DATA_DIR/retention.json`. Env values only seed the file on first boot. UI walkthrough: [user-guide/settings.md](../user-guide/settings.md).
--- ---
@@ -75,8 +77,8 @@ Live retention is edited in the desktop **Settings → Data** tab (admin) and pe
|----------|---------|-------------| |----------|---------|-------------|
| `PEARDATA_HYPERDB` | on | `0` / `off` disables HyperDB | | `PEARDATA_HYPERDB` | on | `0` / `off` disables HyperDB |
| `PEARDATA_SWARM` | off | `1` enables Hyperswarm Corestore replication | | `PEARDATA_SWARM` | off | `1` enables Hyperswarm Corestore replication |
| `PEARDATA_DOCKER` | off* | `1` enables Docker Engine collector (socket + per-container cpu/mem). *Installer sets `1` when Docker is detected | | `PEARDATA_DOCKER` | off* | `1` enables Docker Engine collector (socket + per-container cpu/mem + **human names**). *Installer sets `1` when Docker is detected and adds `peardata` to the `docker` group |
| `PEARDATA_DOCKER_SOCKET` | auto | Socket path for names + API (`/var/run/docker.sock`, `/run/docker.sock`, Podman). Installer writes the detected path | | `PEARDATA_DOCKER_SOCKET` | auto | Socket path for names + API (`/var/run/docker.sock`, `/run/docker.sock`, Podman). Installer writes the detected path. See [user-guide/containers.md](../user-guide/containers.md) |
| `PEARDATA_CGROUPS` | on (Linux) | `0` disables cgroup v2 CPU/mem/IO charts; `1` forces on | | `PEARDATA_CGROUPS` | on (Linux) | `0` disables cgroup v2 CPU/mem/IO charts; `1` forces on |
| `PEARDATA_CGROUPS_MAX` | `32` | Max cgroup instances to chart | | `PEARDATA_CGROUPS_MAX` | `32` | Max cgroup instances to chart |
| `PEARDATA_SENSORS` | on (Linux) | `0` disables hwmon/thermal charts | | `PEARDATA_SENSORS` | on (Linux) | `0` disables hwmon/thermal charts |
docs/DASHBOARD.md
+12 -5
View File
@@ -11,9 +11,10 @@ Related: [ROADMAP.md](./ROADMAP.md) Phase 6 · [DESKTOP.md](./DESKTOP.md) · [DA
| Surface | Today | | Surface | Today |
|---------|--------| |---------|--------|
| Overview | Compact 6-panel live grid + KPIs; spark depth follows Charts timeframe when available | | Overview | Compact 6-panel live grid + KPIs (auto-scaled RAM/net units); sparks + KPIs sync from history seed |
| Charts | Sectioned metrics wall (TOC + cards), shared time bar, multi-dim series | | Charts | Sectioned metrics wall (TOC + cards), shared time bar, multi-dim series |
| Time | Charts presets (1m–6h) + play/pause; retention is advisory (`thin-history`) | | Containers | Docker/cgroup cards use **human names** (e.g. `dozzle · CPU`); ids stay `docker.cpu.<shortId>` |
| Time | Charts presets (1m–6h) + play/pause; UI thin-history advisory; agent retention via Settings → Data |
| Dimensions | Legend chips show/hide; chart type cycle (line/area/stacked/bar/pie) | | Dimensions | Legend chips show/hide; chart type cycle (line/area/stacked/bar/pie) |
| Alerts → chart | Charts wall scroll-to-card + pause near event | | Alerts → chart | Charts wall scroll-to-card + pause near event |
@@ -105,7 +106,7 @@ Do not name third-party products in code, commits, or user-facing copy.
| Section | Match (chart id / context / plugin prefixes) | | Section | Match (chart id / context / plugin prefixes) |
|---------|-----------------------------------------------| |---------|-----------------------------------------------|
| System | `system.*`, `cpu.*`, `mem.*`, `disk_*`, `net.*`, `ip.*`, `ipv*`, `system.*pressure*` | | System | `system.*`, `cpu.*`, `mem.*`, `disk_*`, `net.*`, `ip.*`, `ipv*`, `system.*pressure*` |
| Containers | `docker.*`, `cgroup.*`, `peardock.*` | | Containers | `docker.*`, `cgroup.*`, `peardock.*` — titles/families from Docker socket names ([user-guide/containers.md](../user-guide/containers.md)) |
| Storage | `zfs.*`, `bcache.*`, `dmcache.*`, `md.*`, `smart.*`, `fs.*` | | Storage | `zfs.*`, `bcache.*`, `dmcache.*`, `md.*`, `smart.*`, `fs.*` |
| Hardware | `sensors.*`, `gpu.*`, `ipmi.*`, `power*` | | Hardware | `sensors.*`, `gpu.*`, `ipmi.*`, `power*` |
| Applications | `nginx.*`, `apache.*`, `redis.*`, `mysql.*`, `postgres.*`, `memcached.*`, `mongodb.*`, `rabbitmq.*`, `kafka.*`, `nats.*`, `unbound.*` | | Applications | `nginx.*`, `apache.*`, `redis.*`, `mysql.*`, `postgres.*`, `memcached.*`, `mongodb.*`, `rabbitmq.*`, `kafka.*`, `nats.*`, `unbound.*` |
@@ -125,7 +126,11 @@ Do not name third-party products in code, commits, or user-facing copy.
| `server/services/weights.js` | Metric Correlations scoring (ks2 / volume / AR / value) | | `server/services/weights.js` | Metric Correlations scoring (ks2 / volume / AR / value) |
| `ui/dashboard.js` | Metrics wall controller + Correlate mode | | `ui/dashboard.js` | Metrics wall controller + Correlate mode |
| `ui/charts.js` | Canvas paint + hover helpers | | `ui/charts.js` | Canvas paint + hover helpers |
| `index.html` | Charts view shell (TOC + wall + time bar) | | `ui/data-manager.js` | Settings → Data (retention / prune / usage) |
| `ui/logs.js` | System Log tab (journal default) |
| `shared/container-names.js` | Docker/cgroup display names + card subtitles |
| `shared/format.js` | KPI unit scaling (MiB, kb/s, …) |
| `index.html` | Charts / Logs / Settings shells |
| `ui/styles.css` | Metrics shell layout | | `ui/styles.css` | Metrics shell layout |
| `app.js` | Wire catalog / samples / view switch | | `app.js` | Wire catalog / samples / view switch |
@@ -147,7 +152,9 @@ Do not name third-party products in code, commits, or user-facing copy.
- Hover pad shared with Y-axis; hero-aware resize; hover paint batched via rAF - Hover pad shared with Y-axis; hero-aware resize; hover paint batched via rAF
- Fleet cards show CPU/RAM chips when available; empty → Connect CTA - Fleet cards show CPU/RAM chips when available; empty → Connect CTA
- Charts **Filters** menu (TOC / search / group / sort collapsed by default) - Charts **Filters** menu (TOC / search / group / sort collapsed by default)
- **Logs** tab: journal/anomaly/audit stream, Follow, search highlight - **Logs** tab: journal (default) / anomaly / audit stream, Follow, search highlight
- **Settings → Data**: agent Data Manager (1y warm default, auto-prune, usage)
- Container cards: human names via Docker socket; subtitle hides bare hashes
## Metric Correlations vs Related ## Metric Correlations vs Related
docs/DATA-MODEL.md
+8 -5
View File
@@ -90,10 +90,13 @@ Chart summary objects use fields: `id`, `context`, `units`, `dimensions`, `updat
## Storage tiers ## Storage tiers
| Tier | Resolution | Default retention | Env | | Tier | Resolution | Default retention | Env / control |
|------|------------|-------------------|-----| |------|------------|-------------------|---------------|
| 0 | 1s | 3600 points (~1h) | `PEARDATA_TIER0_POINTS` | | 0 (hot RAM) | 1s | 3600 points (~1h) | `PEARDATA_TIER0_POINTS` → seeds Data Manager |
| 1 | avg every N samples (default 60 → ~1m) | 1440 points (~24h) | `PEARDATA_TIER1_POINTS`, `PEARDATA_TIER1_EVERY` | | 1 (warm RAM) | avg every N samples (default 60 → ~1m) | 1440 points (~24h in RAM) | `PEARDATA_TIER1_POINTS`, `PEARDATA_TIER1_EVERY` |
| HyperDB warm | tier1 flush | **1 year** age rotate (default) | `PEARDATA_WARM_RETENTION_MS` seed; live policy in `retention.json` |
Live retention, soft disk budgets, and auto-prune are managed via desktop **Settings → Data** (RPC `getStorageInfo` / `setRetentionConfig` / `pruneHistory`). See [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) and [user-guide/settings.md](../user-guide/settings.md).
## AnomalyEvent ## AnomalyEvent
@@ -194,7 +197,7 @@ Normalized shape from `queryLogs` / `GET /api/v*/logs`:
} }
``` ```
Sources: `anomaly` | `audit` | `journal`. See [user-guide/logs.md](../user-guide/logs.md). Sources: `anomaly` | `audit` | `journal`. Desktop UI and omitted `source` default to **journal** (admin); viewers fall back to anomalies. See [user-guide/logs.md](../user-guide/logs.md).
## REST ↔ RPC parity ## REST ↔ RPC parity
docs/DESKTOP.md
+7 -3
View File
@@ -8,14 +8,18 @@ The client shell is a **Pear desktop application** built with `pear-electron` +
|------|------| |------|------|
| `index.js` | Pear process entry — starts `pear-electron` Runtime + `pear-bridge` | | `index.js` | Pear process entry — starts `pear-electron` Runtime + `pear-bridge` |
| `index.html` | GUI main (`pear.gui.main`) — titlebar + panels | | `index.html` | GUI main (`pear.gui.main`) — titlebar + panels |
| `app.js` | UI logic (connect, fleet, live charts, invites) | | `app.js` | UI logic (connect, fleet, live charts, invites, Data Manager wiring) |
| `ui/dashboard.js` | Charts tab master metrics wall | | `ui/dashboard.js` | Charts tab master metrics wall |
| `ui/data-manager.js` | Settings → Data (retention / prune / storage usage) |
| `ui/logs.js` | System Log tab (journal default for admins) |
| `shared/taxonomy.js` | Section grouping for the metrics wall | | `shared/taxonomy.js` | Section grouping for the metrics wall |
| `shared/container-names.js` | Human Docker/cgroup labels for the wall |
| `shared/format.js` | Overview KPI unit scaling |
| `shared/retention.js` | Retention presets shared with the agent |
| `ui/styles.css` | Layout, theme, **titlebar drag regions** | | `ui/styles.css` | Layout, theme, **titlebar drag regions** |
| `client/*` | HyperDHT connection stack used by the UI | | `client/*` | HyperDHT connection stack used by the UI |
| `docs/DASHBOARD.md` | Master Charts / metrics wall plan | | `docs/DASHBOARD.md` | Master Charts / metrics wall plan |
| `user-guide/` | End-user workflows (Correlate, Related, Logs, Fleet, …) | | `user-guide/` | End-user workflows (Correlate, Related, Logs, Containers, Settings, …) |
| `ui/logs.js` | System Log tab controller |
```bash ```bash
npm start # pear run -d . npm start # pear run -d .
docs/EXTENDING.md
+7 -5
View File
@@ -59,7 +59,7 @@ Operators run via `runJob` (`retrainAnomaly` is registered).
``` ```
server/services/collectors/ server/services/collectors/
docker.js # PEARDATA_DOCKER=1 — cgroup + optional Docker socket docker.js # PEARDATA_DOCKER=1 — cgroup + Docker socket names (installer auto-enables)
peardock.js # bridge (planned) peardock.js # bridge (planned)
``` ```
@@ -67,10 +67,12 @@ Have `pipeline.js` start each enabled collector; all emit `samples` batches into
### Docker collector (shipped spike) ### Docker collector (shipped spike)
1. Set `PEARDATA_DOCKER=1` on the agent. 1. Set `PEARDATA_DOCKER=1` on the agent (or re-run the installer when Docker is present).
2. Charts: `docker.containers`, `docker.cpu.<id>`, `docker.mem.<id>` (registered at runtime). 2. Ensure `peardata` can read the socket (`usermod -aG docker peardata` + unit `SupplementaryGroups=… docker`).
3. Discovery: cgroup v2 `docker-*.scope` / `libpod-*.scope`; names from the Docker socket (`PEARDATA_DOCKER_SOCKET`). Installer adds `peardata` to the `docker` group and enables `PEARDATA_DOCKER=1` when Docker is detected. 3. Charts: `docker.containers`, `docker.cpu.<shortId>`, `docker.mem.<shortId>` (stable ids); **titles/families** use human names (`dozzle · CPU`).
4. Pick them in the desktop Explore chart dropdown (`docker.*`). 4. Discovery: cgroup v2 `docker-*.scope` / `libpod-*.scope`; names from `fetchDockerNames` / Compose labels (`shared/container-names.js`).
5. Cgroups collector reuses the same socket map so `cgroup.*` cards are not bare hashes.
6. UI: [user-guide/containers.md](../user-guide/containers.md).
### Process top-N collector ### Process top-N collector
docs/PROTOCOL.md
+4 -2
View File
@@ -76,7 +76,7 @@ REST parity: `GET /api/v*/weights`. Engine: `server/services/weights.js`.
| Arg | Notes | | Arg | Notes |
|-----|--------| |-----|--------|
| `source` | `anomaly` (RPC/REST default) \| `audit` \| `journal` (desktop UI default for admins) | | `source` | `journal` (default) \| `anomaly` \| `audit` — desktop prefers journal for admins |
| `q` | Case-insensitive substring | | `q` | Case-insensitive substring |
| `since` / `until` | Absolute ms/sec or relative (`-1h`) | | `since` / `until` | Absolute ms/sec or relative (`-1h`) |
| `priority` / `unit` | Journal filters | | `priority` / `unit` | Journal filters |
@@ -116,10 +116,12 @@ Journal is **enabled by default** on Linux (`PEARDATA_JOURNAL=0` to disable). In
| Method | Role | Notes | | Method | Role | Notes |
|--------|------|-------| |--------|------|-------|
| `getStorageInfo` | viewer | Disk usage, memory rings, warm sample stats, effective retention | | `getStorageInfo` | viewer | Disk usage, memory rings, warm sample stats, effective retention |
| `getRetentionConfig` | viewer | Current policy (`retention.json`) | | `getRetentionConfig` | viewer | Current policy (`retention.json`; default warm age **1 year**) |
| `setRetentionConfig` | admin | Update hot/warm rings, age rotate, disk budget, auto-prune | | `setRetentionConfig` | admin | Update hot/warm rings, age rotate, disk budget, auto-prune |
| `pruneHistory` | admin | `{ dryRun?, beforeMs? }` — delete warm points past retention | | `pruneHistory` | admin | `{ dryRun?, beforeMs? }` — delete warm points past retention |
UI: [user-guide/settings.md](../user-guide/settings.md) → **Data**. Job alias: `runJob({ name: 'gcBuffers' })` runs the same prune path.
### HyperDB / peer links ### HyperDB / peer links
| Method | Role | Notes | | Method | Role | Notes |
docs/README.md
+6 -6
View File
@@ -4,8 +4,8 @@
| Doc | Contents | | Doc | Contents |
|-----|----------| |-----|----------|
| **[User guide](../user-guide/README.md)** | Desktop workflows: Connect, Overview, Charts, Metric Correlations, Logs, Related, Alerts, Fleet, Settings | | **[User guide](../user-guide/README.md)** | Desktop workflows: Connect, Overview, Charts, Metric Correlations, Logs, Containers, Related, Alerts, Fleet, Settings (Data Manager) |
| [GETTING-STARTED.md](./GETTING-STARTED.md) | Install agent / desktop, first dial, systemd | | [GETTING-STARTED.md](./GETTING-STARTED.md) | Install agent / desktop, first dial, systemd (journal + Docker socket when detected) |
## Engineers & operators ## Engineers & operators
@@ -18,10 +18,10 @@
| [REST-API.md](./REST-API.md) | Integrators | agent-style `/api/v1\|v2\|v3` | | [REST-API.md](./REST-API.md) | Integrators | agent-style `/api/v1\|v2\|v3` |
| [DASHBOARD.md](./DASHBOARD.md) | UI developers | Charts wall / Correlate / Related product plan | | [DASHBOARD.md](./DASHBOARD.md) | UI developers | Charts wall / Correlate / Related product plan |
| [TECH-CHOICES.md](./TECH-CHOICES.md) | Engineers | Collector, charts, libraries | | [TECH-CHOICES.md](./TECH-CHOICES.md) | Engineers | Collector, charts, libraries |
| [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | Engineers | HyperDB / Corestore / Autobase sync design | | [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | Engineers | HyperDB / Corestore / Autobase + Data Manager retention |
| [DESKTOP.md](./DESKTOP.md) | UI developers | Pear shell, titlebar, identity | | [DESKTOP.md](./DESKTOP.md) | UI developers | Pear shell, titlebar, identity, Data Manager / Logs |
| [SECURITY.md](./SECURITY.md) | Operators / security | Trust model, secrets, hardening | | [SECURITY.md](./SECURITY.md) | Operators / security | Trust model, secrets, journal / Docker socket hardening |
| [CONFIGURATION.md](./CONFIGURATION.md) | Operators | Full env vars + npm scripts | | [CONFIGURATION.md](./CONFIGURATION.md) | Operators | Full env vars + npm scripts (retention seeds, Docker) |
| [TESTING.md](./TESTING.md) | Contributors | brittle, soak, manual QA | | [TESTING.md](./TESTING.md) | Contributors | brittle, soak, manual QA |
| [CI.md](./CI.md) | Maintainers | GitHub/Gitea pipelines | | [CI.md](./CI.md) | Maintainers | GitHub/Gitea pipelines |
| [RELEASE.md](./RELEASE.md) | Maintainers | Version, tag, tarball | | [RELEASE.md](./RELEASE.md) | Maintainers | Version, tag, tarball |
docs/RELEASE.md
+2
View File
@@ -17,6 +17,8 @@ curl -fsSL https://git.ssh.surf/snxraven/peardata/raw/branch/main/scripts/instal
Script: [`scripts/install.sh`](../scripts/install.sh) · unit: [`deploy/peardata.service`](../deploy/peardata.service) Script: [`scripts/install.sh`](../scripts/install.sh) · unit: [`deploy/peardata.service`](../deploy/peardata.service)
Server install (`--server`) writes systemd under `/opt/peardata`, enables **journal** Logs (`peardata` ∈ `systemd-journal`), and when **Docker** is detected enables container metrics/names (`peardata` ∈ `docker`, `PEARDATA_DOCKER=1`). See [GETTING-STARTED.md](./GETTING-STARTED.md) and [user-guide/containers.md](../user-guide/containers.md).
## Local builds ## Local builds
```bash ```bash
docs/REST-API.md
+9 -4
View File
@@ -45,7 +45,7 @@ curl -s http://127.0.0.1:18888/api/v3/health | jq
| GET | `/api/v3/db` | HyperDB public/discovery keys + collections | | GET | `/api/v3/db` | HyperDB public/discovery keys + collections |
| GET | `/api/v3/versions` | Agent / protocol / API versions | | GET | `/api/v3/versions` | Agent / protocol / API versions |
| GET | `/api/v3/me` | Anonymous REST identity note | | GET | `/api/v3/me` | Anonymous REST identity note |
| GET | `/api/v3/settings` | Runtime knobs + retention + storage usage summary | | GET | `/api/v3/settings` | Runtime knobs + **retention** + **storage usage** summary (Data Manager read model) |
| GET | `/api/v3/config` | alias of settings | | GET | `/api/v3/config` | alias of settings |
| GET | `/health`, `/api/v1/health`, `/api/v3/health` | Aggregate health | | GET | `/health`, `/api/v1/health`, `/api/v3/health` | Aggregate health |
@@ -168,7 +168,7 @@ Desktop walkthrough: [user-guide/logs.md](../user-guide/logs.md)
| Param | Notes | | Param | Notes |
|-------|--------| |-------|--------|
| `source` | `anomaly` (API default) \| `audit` \| `journal` — desktop UI defaults to **journal** for admins | | `source` | `journal` (default) \| `anomaly` \| `audit` — desktop prefers **journal** for admins |
| `q` | Case-insensitive substring over message / unit / severity | | `q` | Case-insensitive substring over message / unit / severity |
| `since` / `until` | Absolute ms/sec, ISO, or relative (`-1h`) | | `since` / `until` | Absolute ms/sec, ISO, or relative (`-1h`) |
| `priority` | Journal max priority (`journalctl -p`) | | `priority` | Journal max priority (`journalctl -p`) |
@@ -177,10 +177,14 @@ Desktop walkthrough: [user-guide/logs.md](../user-guide/logs.md)
**Auth:** REST is localhost-open (like other GETs). Over P2P, `audit` and `journal` require **admin**; `anomaly` allows viewer. **Auth:** REST is localhost-open (like other GETs). Over P2P, `audit` and `journal` require **admin**; `anomaly` allows viewer.
**Journal:** enabled by default (`PEARDATA_JOURNAL=0` to disable). Linux only; installer adds `peardata` to `systemd-journal`. **Journal:** enabled by default (`PEARDATA_JOURNAL=0` to disable). Linux only; installer adds `peardata` to `systemd-journal`. Omitted `source` defaults to **journal**.
Engine: `server/services/logs.js`. Unit tests: `test/logs.test.js`. Engine: `server/services/logs.js`. Unit tests: `test/logs.test.js`.
### Data Manager (read via settings)
`GET /api/v3/settings` includes `retention` and a `storage` summary (data dir / corestore / memory rings). Mutating retention / prune is **P2P admin RPC** (`setRetentionConfig`, `pruneHistory`) — see [PROTOCOL.md](./PROTOCOL.md) and [user-guide/settings.md](../user-guide/settings.md).
## CORS ## CORS
`Access-Control-Allow-Origin` defaults to `*` (override with `PEARDATA_REST_CORS`). `Access-Control-Allow-Origin` defaults to `*` (override with `PEARDATA_REST_CORS`).
@@ -190,7 +194,8 @@ Engine: `server/services/logs.js`. Unit tests: `test/logs.test.js`.
| Area | PearData | | Area | PearData |
|------|----------| |------|----------|
| Weights / Metric Correlations | `GET /api/v*/weights` — highlight-vs-baseline scoring (`method=volume\|ks2\|anomaly-rate\|value`) or alert weights when no window; see above | | Weights / Metric Correlations | `GET /api/v*/weights` — highlight-vs-baseline scoring (`method=volume\|ks2\|anomaly-rate\|value`) or alert weights when no window; see above |
| Logs | `GET /api/v*/logs` — anomaly / audit / journal; journal on by default; see above | | Logs | `GET /api/v*/logs` — journal (default) / anomaly / audit; see above |
| Retention / storage | `GET /api/v*/settings` includes retention + usage; writes via P2P admin RPC |
| Multi-node parent streaming | Single node; parent planned | | Multi-node parent streaming | Single node; parent planned |
| Cloud POST spaces APIs | Not implemented (agent GET style only) | | Cloud POST spaces APIs | Not implemented (agent GET style only) |
| App/plugin charts (nginx, DB, …) | System/OS charts; plugins later | | App/plugin charts (nginx, DB, …) | System/OS charts; plugins later |
docs/ROADMAP.md
+2 -2
View File
@@ -66,7 +66,7 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
| Item | Status | | Item | Status |
|------|--------| |------|--------|
| Docker / container discovery | Done (opt-in `PEARDATA_DOCKER=1`) | | Docker / container discovery | Done (`PEARDATA_DOCKER=1`; installer enables + `docker` group when detected; human names) |
| Offline banner (last-known samples) | Done | | Offline banner (last-known samples) | Done |
| Process top-N | Done (opt-in `PEARDATA_PROCESSES=1`) | | Process top-N | Done (opt-in `PEARDATA_PROCESSES=1`) |
| Service plugins | nginx + redis + postgres done | | Service plugins | nginx + redis + postgres done |
@@ -94,7 +94,7 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
| Notifications | Done (desktop + `PEARDATA_WEBHOOK_URL`) | | Notifications | Done (desktop + `PEARDATA_WEBHOOK_URL`) |
| Streaming z-score / retrain job | Done (`PEARDATA_ANOMALY_MODE`, job `retrainAnomaly`) | | Streaming z-score / retrain job | Done (`PEARDATA_ANOMALY_MODE`, job `retrainAnomaly`) |
| `/api/v3/weights` / Metric Correlations | Done — highlight-vs-baseline scoring (`ks2`/`volume`/`anomaly-rate`/`value`) + Charts Correlate UI | | `/api/v3/weights` / Metric Correlations | Done — highlight-vs-baseline scoring (`ks2`/`volume`/`anomaly-rate`/`value`) + Charts Correlate UI |
| `/api/v3/logs` / System Log tab | Done — journal (default on) + anomaly + audit (`queryLogs`); Charts Filters menu | | `/api/v3/logs` / System Log tab | Done — journal (UI + omitted-source default) + anomaly + audit; Charts Filters menu |
--- ---
docs/SECURITY.md
+3
View File
@@ -48,6 +48,7 @@
- [ ] Do not embed seed in frontend builds, CI logs, or crash reports - [ ] Do not embed seed in frontend builds, CI logs, or crash reports
- [ ] Review `LOG_LEVEL=debug` before production (avoid verbose auth noise) - [ ] Review `LOG_LEVEL=debug` before production (avoid verbose auth noise)
- [ ] Journal is **on by default** (admin-only over P2P); it can expose secrets from other units — set `PEARDATA_JOURNAL=0` if you do not want host log access - [ ] Journal is **on by default** (admin-only over P2P); it can expose secrets from other units — set `PEARDATA_JOURNAL=0` if you do not want host log access
- [ ] If Docker is enabled, `peardata` ∈ **`docker`** can inspect all containers via the socket — treat that as host-equivalent visibility; disable with `PEARDATA_DOCKER=0` and remove the group if unwanted
- [ ] Keep Pear / dependency updates current (`npm outdated`) - [ ] Keep Pear / dependency updates current (`npm outdated`)
## Threat notes ## Threat notes
@@ -87,6 +88,8 @@ Implementation: `shared/crypto-auth.js`.
| `data/peer-policy.json` | High | Contains roles & JTIs | | `data/peer-policy.json` | High | Contains roles & JTIs |
| `data/audit.log` | Medium | Peer activity metadata; readable via Logs/audit (admin) | | `data/audit.log` | Medium | Peer activity metadata; readable via Logs/audit (admin) |
| Host journal (`PEARDATA_JOURNAL`) | High | May include secrets from other units; admin-only; **on by default** (disable with `=0`) | | Host journal (`PEARDATA_JOURNAL`) | High | May include secrets from other units; admin-only; **on by default** (disable with `=0`) |
| Docker socket (`PEARDATA_DOCKER`) | High | Container inventory/names/stats; installer adds `peardata` to `docker` when detected |
| `data/retention.json` | Medium | Retention / prune policy (admin-writable over P2P) |
| `identity.json` | High for that user | Per-machine client secret | | `identity.json` | High for that user | Per-machine client secret |
| Release tarballs | Low | Source only; no secrets | | Release tarballs | Low | Source only; no secrets |
docs/STORAGE-HYPERDB.md
+4 -1
View File
@@ -148,6 +148,9 @@ Banner fields on agent start:
| `linkPeer` | admin | Upsert link; optionally `joinRemoteTopic` if swarm on | | `linkPeer` | admin | Upsert link; optionally `joinRemoteTopic` if swarm on |
| `unlinkPeer` | admin | Remove link | | `unlinkPeer` | admin | Remove link |
| `queryData` | viewer | Memory first; HyperDB warm if miss / `tier≥1` | | `queryData` | viewer | Memory first; HyperDB warm if miss / `tier≥1` |
| `getStorageInfo` | viewer | Disk + memory usage for Data Manager |
| `getRetentionConfig` / `setRetentionConfig` | viewer / admin | Warm age rotate (default **1y**), budgets, auto-prune |
| `pruneHistory` | admin | Delete warm points past retention (`gcBuffers` job alias) |
`linkPeer` args: `linkPeer` args:
@@ -305,7 +308,7 @@ curl -s 'http://127.0.0.1:18888/api/v3/data?chart=system.cpu&after=-7200&tier=1&
- [ ] Back up **keys** (`SERVER_SEED`, swarm keypair in corestore) — not by zipping live corestore while writing - [ ] Back up **keys** (`SERVER_SEED`, swarm keypair in corestore) — not by zipping live corestore while writing
- [ ] Keep REST on localhost; swarm only when linking trusted peers - [ ] Keep REST on localhost; swarm only when linking trusted peers
- [ ] Prefer `pd1.` invites for RPC admin; share db discovery keys only with linked nodes - [ ] Prefer `pd1.` invites for RPC admin; share db discovery keys only with linked nodes
- [ ] Monitor disk under `data/corestore` as warm retention grows - [ ] Monitor disk under `data/corestore` (or use **Settings → Data** usage cards + auto-prune)
--- ---
docs/SWAGGER_PLAN.md
+3 -3
View File
@@ -101,7 +101,7 @@ All IDs are in `STATIC_CHART_DEFS` and referenced by `server/services/collector.
| Env gate | Charts | Status | | Env gate | Charts | Status |
|----------|--------|--------| |----------|--------|--------|
| `PEARDATA_DOCKER=1` | `docker.containers`, `docker.cpu.*`, `docker.mem.*` | **done** | | `PEARDATA_DOCKER=1` | `docker.containers`, `docker.cpu.*`, `docker.mem.*` (human titles via socket; installer auto-enables) | **done** |
| `PEARDATA_PROCESSES=1` | `processes.top_cpu`, `processes.top_rss` | **done** | | `PEARDATA_PROCESSES=1` | `processes.top_cpu`, `processes.top_rss` | **done** |
| `PEARDATA_NGINX=1` | `nginx.connections`, `nginx.requests` | **done** | | `PEARDATA_NGINX=1` | `nginx.connections`, `nginx.requests` | **done** |
| `PEARDATA_REDIS=1` | `redis.memory`, `redis.clients`, `redis.stats` | **done** | | `PEARDATA_REDIS=1` | `redis.memory`, `redis.clients`, `redis.stats` | **done** |
@@ -221,7 +221,7 @@ Legend:
| GET | `/api/v2/progress` | **skip** | | | GET | `/api/v2/progress` | **skip** | |
| GET | `/api/v3/progress` | **todo** | Only if HTTP functions land | | GET | `/api/v3/progress` | **todo** | Only if HTTP functions land |
| GET | `/api/v3/settings` | **partial** | Read-only runtime knobs | | GET | `/api/v3/settings` | **partial** | Read-only runtime knobs |
| PUT | `/api/v3/settings` | **todo** | Not wired (GET-only today) | | PUT | `/api/v3/settings` | **todo** | GET returns retention + storage; writes use P2P admin RPC (`setRetentionConfig` / `pruneHistory`) |
| GET/POST | `/api/v1/config` | **skip** | DynCFG; use env + P2P | | GET/POST | `/api/v1/config` | **skip** | DynCFG; use env + P2P |
| GET/POST | `/api/v3/config` | **partial** | Aliases settings GET; no DynCFG tree | | GET/POST | `/api/v3/config` | **partial** | Aliases settings GET; no DynCFG tree |
@@ -291,7 +291,7 @@ Legend:
1. ~~Richer `/api/v3/weights`~~ ✅ single-host MC engine (`volume` / `ks2` / `anomaly-rate` / `value`) 1. ~~Richer `/api/v3/weights`~~ ✅ single-host MC engine (`volume` / `ks2` / `anomaly-rate` / `value`)
2. `/api/v1/alarm_count`, `/api/v1/alarms_values`, `/api/v1/alarm_log` thin wrappers 2. `/api/v1/alarm_count`, `/api/v1/alarms_values`, `/api/v1/alarm_log` thin wrappers
3. Optional PUT `/api/v3/settings` (localhost-only) 3. Optional PUT `/api/v3/settings` (localhost-only) — today prefer P2P Data Manager RPC
4. HTTP `function` bridge to `runJob` with strict localhost/admin gate 4. HTTP `function` bridge to `runJob` with strict localhost/admin gate
5. `/api/v3/stream_info` from parent collector stats 5. `/api/v3/stream_info` from parent collector stats
docs/TECH-CHOICES.md
+4 -3
View File
@@ -57,8 +57,9 @@ Target overhead: single timer, no child processes per tick, ring buffers only.
| Tier | Implementation | | Tier | Implementation |
|------|----------------| |------|----------------|
| Now | In-memory ring (`server/services/store.js`) | | Hot | In-memory ring (`server/services/store.js`) |
| Next | **HyperDB** + Corestore (+ Hyperswarm replicate) — see [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) | | Warm | **HyperDB** + Corestore (+ optional Hyperswarm) — see [STORAGE-HYPERDB.md](./STORAGE-HYPERDB.md) |
| Policy | Data Manager (`retention.json`, default **1y** warm age, auto-prune) — `server/services/retention.js` |
| HA / multi-writer parents | Autobase with HyperDB view (`extension: false`) | | HA / multi-writer parents | Autobase with HyperDB view (`extension: false`) |
| Avoid | SQLite as primary P2P store; copying corestore folders as “backup” | | Avoid | SQLite as primary P2P store; copying corestore folders as “backup” |
@@ -89,7 +90,7 @@ Target overhead: single timer, no child processes per tick, ring buffers only.
| Agent | Node 20+ or Bare (`bin/peardata-server.mjs`), systemd unit `deploy/peardata.service` | | Agent | Node 20+ or Bare (`bin/peardata-server.mjs`), systemd unit `deploy/peardata.service` |
| Desktop | Pear (`pear-electron` + `pear-bridge`) with Bare-ready imports | | Desktop | Pear (`pear-electron` + `pear-bridge`) with Bare-ready imports |
| Invites | `pd1.` tokens (PearDock-style) | | Invites | `pd1.` tokens (PearDock-style) |
| Installer | `scripts/install.sh` (rolling binaries; see [RELEASE.md](./RELEASE.md)) | | Installer | `scripts/install.sh` (rolling binaries; journal + Docker socket when detected; see [RELEASE.md](./RELEASE.md)) |
## Reusable library extraction (recommended later) ## Reusable library extraction (recommended later)
docs/TESTING.md
+7
View File
@@ -18,6 +18,10 @@
| `test/store.test.js` | Metric ring buffer ingest + query | | `test/store.test.js` | Metric ring buffer ingest + query |
| `test/weights.test.js` | Metric Correlations engine (`volume` / `ks2` / windows / errors) | | `test/weights.test.js` | Metric Correlations engine (`volume` / `ks2` / windows / errors) |
| `test/logs.test.js` | Log query engine (roles, filters, journal argv, audit file) | | `test/logs.test.js` | Log query engine (roles, filters, journal argv, audit file) |
| `test/retention.test.js` | Retention presets, store trim, warm prune delete, Data Manager RPC roles |
| `test/format.test.js` | KPI unit scaling (`formatMib`, `formatKilobitsPerSec`, `formatBytes`) |
| `test/container-names.test.js` | Docker/Compose display names, cgroup resolve, card subtitles |
| `test/docker-collector.test.js` | Docker chart helpers + socket resolve |
| `test/hyperdb.test.js` | HyperDB model (node, links, warm points, alerts) | | `test/hyperdb.test.js` | HyperDB model (node, links, warm points, alerts) |
| `test/rest.test.js` | agent-style `/api/v3` routes | | `test/rest.test.js` | agent-style `/api/v3` routes |
| `test/integration.test.js` | Live HyperDHT agent + handshake + metrics query | | `test/integration.test.js` | Live HyperDHT agent + handshake + metrics query |
@@ -91,6 +95,9 @@ When adding an RPC method:
| Weights REST | `curl -sG 'http://127.0.0.1:18888/api/v3/weights' --data-urlencode 'method=volume' --data-urlencode 'after=-60' --data-urlencode 'before=0'` | | Weights REST | `curl -sG 'http://127.0.0.1:18888/api/v3/weights' --data-urlencode 'method=volume' --data-urlencode 'after=-60' --data-urlencode 'before=0'` |
| Logs | Journal default (admin); Follow / Refresh / search highlight; Anomalies for viewers; Audit admin-only | | Logs | Journal default (admin); Follow / Refresh / search highlight; Anomalies for viewers; Audit admin-only |
| Logs REST | `curl -sG 'http://127.0.0.1:18888/api/v3/logs' --data-urlencode 'source=journal' --data-urlencode 'limit=20'` | | Logs REST | `curl -sG 'http://127.0.0.1:18888/api/v3/logs' --data-urlencode 'source=journal' --data-urlencode 'limit=20'` |
| Data Manager | Settings → Data (admin): usage cards, 1y warm default, Save, dry-run prune |
| Settings REST | `curl -s http://127.0.0.1:18888/api/v3/settings \| jq '.retention,.storage'` |
| Docker names | With `PEARDATA_DOCKER=1` + socket access: Charts → Containers shows `dozzle · CPU` (not bare hashes) |
## Soak test ## Soak test
user-guide/README.md
+2 -1
View File
@@ -19,8 +19,9 @@ For engineers (protocol, REST deep-dive, architecture), see [docs/](../docs/READ
| [Related metrics](./related-metrics.md) | Per-chart ⇢ related (taxonomy + Pearson) — different from Correlate | | [Related metrics](./related-metrics.md) | Per-chart ⇢ related (taxonomy + Pearson) — different from Correlate |
| [Alerts](./alerts.md) | Anomaly list, Show / Correlate actions | | [Alerts](./alerts.md) | Anomaly list, Show / Correlate actions |
| [Logs](./logs.md) | System log stream — journal (default), anomalies, audit; Follow / search | | [Logs](./logs.md) | System log stream — journal (default), anomalies, audit; Follow / search |
| [Containers](./containers.md) | Docker metrics with human names (installer socket access) |
| [Fleet](./fleet.md) | Multi-host roster, set active, reconnect, forget | | [Fleet](./fleet.md) | Multi-host roster, set active, reconnect, forget |
| [Settings](./settings.md) | Theme, motion, spark depth, reconnect | | [Settings](./settings.md) | Appearance, Data Manager (retention / prune), connections |
| [Keyboard shortcuts](./keyboard.md) | Charts and shell shortcuts | | [Keyboard shortcuts](./keyboard.md) | Charts and shell shortcuts |
| [Weights API](./weights-api.md) | REST / RPC for Metric Correlations from scripts | | [Weights API](./weights-api.md) | REST / RPC for Metric Correlations from scripts |
user-guide/charts.md
+5 -1
View File
@@ -32,7 +32,11 @@ The Charts tab is the **metrics wall**: every chart the agent catalogs, grouped
| **Group** | Downsample aggregation (inside Filters): average / min / max / sum | | **Group** | Downsample aggregation (inside Filters): average / min / max / sum |
| **thin-history** (preset style) | Advisory — buffered history may be shorter than the window | | **thin-history** (preset style) | Advisory — buffered history may be shorter than the window |
Retention never hard-locks presets once an agent is connected; longer windows may look sparse until samples accumulate. Retention never hard-locks presets once an agent is connected; longer windows may look sparse until samples accumulate. Agent-side history length and disk prune live under **Settings → Data** (default warm keep **1 year**) — see [Settings](./settings.md).
## Containers section
Docker / cgroup cards group under **Containers** with human titles (`dozzle · CPU`) when the agent has Docker socket access. Details: [Containers](./containers.md).
## Gestures ## Gestures
user-guide/containers.md
+63
View File
@@ -0,0 +1,63 @@
# Containers (Docker)
PearData can chart per-container CPU and memory and show **human names** (e.g. `dozzle`) instead of short hashes.
## Requirements
On the **agent** host:
1. Docker (or Podman with a compatible socket) installed
2. Agent user can read the engine socket
3. `PEARDATA_DOCKER=1` in the agent `.env`
The [one-line installer](../docs/GETTING-STARTED.md) does this automatically when Docker is detected:
- Adds `peardata` to the **`docker`** group
- Sets unit `SupplementaryGroups=… docker`
- Writes `PEARDATA_DOCKER=1` and `PEARDATA_DOCKER_SOCKET=…`
Then restart: `sudo systemctl restart peardata`.
## Already installed without Docker setup?
```bash
sudo usermod -aG docker peardata
# /opt/peardata/.env — add or set:
# PEARDATA_DOCKER=1
# PEARDATA_DOCKER_SOCKET=/var/run/docker.sock
# Unit must include docker in SupplementaryGroups (re-run installer --server, or edit peardata.service)
sudo systemctl daemon-reload
sudo systemctl restart peardata
```
Confirm names loaded:
```bash
sudo journalctl -u peardata -n 50 --no-pager | grep -i docker
```
You should see a line like `Docker container names loaded` with a count.
## What you see in the UI
| Surface | Behavior |
|---------|----------|
| **Charts → Containers** | Cards titled `dozzle · CPU`, `dozzle · memory`; family groups by container name |
| Card subtitle | Units only (chart id stays in the tooltip for debugging) |
| Explore dropdown | Human titles, not `docker.cpu.<hash>` |
| Cgroup charts | Same name enrichment when the socket is readable |
Chart **ids** remain `docker.cpu.<shortId>` / `docker.mem.<shortId>` so history stays stable when a container is recreated with the same id prefix.
## Name resolution order
1. Compose / Swarm / Kubernetes service labels (`web`, `web · 2`)
2. Docker `Names` (e.g. `/dozzle` → `dozzle`)
3. Image short name
4. Fallback `container <12-char id>` if the socket is unavailable
## Related
- Env: [CONFIGURATION.md](../docs/CONFIGURATION.md) (`PEARDATA_DOCKER*`)
- Collector: [EXTENDING.md](../docs/EXTENDING.md)
- Taxonomy section **Containers**: [DASHBOARD.md](../docs/DASHBOARD.md)
user-guide/logs.md
+1 -1
View File
@@ -20,7 +20,7 @@ The **Logs** tab is a searchable stream from the active agent: host **journal**
| **Anomalies** | Viewer+ | Threshold / z-score events | | **Anomalies** | Viewer+ | Threshold / z-score events |
| **Audit** | Admin | Mutating RPC trail (`data/audit.log`) | | **Audit** | Admin | Mutating RPC trail (`data/audit.log`) |
Non-admin sessions see Anomalies only; Journal/Audit tabs stay disabled. The tab **prefers Journal** whenever your role allows it (including after you connect as admin). Non-admin sessions fall back to Anomalies; Journal/Audit tabs stay disabled until you have admin.
## Search & follow ## Search & follow
user-guide/overview-tab.md
+13 -1
View File
@@ -4,7 +4,19 @@ Compact home dashboard for the **active** agent.
## KPIs ## KPIs
Top strip shows live scalars (CPU, RAM, load, net, health). Health tint follows anomaly engine status (ok / degraded / critical). Top strip shows live scalars (CPU, RAM, load, net, health), filled from history seed and live samples.
| KPI | Notes |
|-----|--------|
| **CPU** | Percent used |
| **RAM used** | Auto-scales B → KiB → MiB → GiB → TiB |
| **Load 1m** | Plain load average |
| **Net RX** | Auto-scales b/s → kb/s → Mb/s → Gb/s |
| **Health** | Anomaly engine status (ok / degraded / critical) with tint |
## Sparks vs KPIs
Overview charts load from `queryData` history immediately; KPI labels sync from the same series so they are not stuck on `—` while graphs already have data.
## Spark panels ## Spark panels
user-guide/overview.md
+3 -1
View File
@@ -19,7 +19,9 @@ You dial agents by **public key** (viewer) or **`pd1.` invite** / admin seed for
| **Overview** | Compact home: KPIs + six spark panels + optional fleet strip | | **Overview** | Compact home: KPIs + six spark panels + optional fleet strip |
| **Charts** | Full metrics wall — every catalog chart, shared time, investigation | | **Charts** | Full metrics wall — every catalog chart, shared time, investigation |
| **Alerts** | Recent anomaly / threshold events | | **Alerts** | Recent anomaly / threshold events |
| **Logs** | Host journal (default), anomalies, and audit trail with search / Follow | | **Logs** | Host journal (default for admins), anomalies, and audit trail with search / Follow |
| **Settings → Data** | Agent Data Manager — retention, auto-prune, disk usage (admin) |
| **Containers** | Docker charts with human names when the agent has socket access |
| **Fleet** | Saved + live agents; set active, reconnect, open Charts | | **Fleet** | Saved + live agents; set active, reconnect, open Charts |
| **Connect** | Dial a new key or invite | | **Connect** | Dial a new key or invite |
| **Settings** | Theme, notifications, history depth, reconnect | | **Settings** | Theme, notifications, history depth, reconnect |
user-guide/settings.md
+3 -3
View File
@@ -25,14 +25,14 @@ Controls **agent-side** history so disks and RAM stay bounded. Requires an **adm
| **Usage cards** | Total data dir size, Corestore size, in-memory hot/warm points, warm HyperDB sample | | **Usage cards** | Total data dir size, Corestore size, in-memory hot/warm points, warm HyperDB sample |
| **Hot memory** | 1s ring depth (15m → 24h presets or custom points) | | **Hot memory** | 1s ring depth (15m → 24h presets or custom points) |
| **Warm memory** | Downsampled in-RAM ring + samples-per-bucket | | **Warm memory** | Downsampled in-RAM ring + samples-per-bucket |
| **Disk history** | Auto-rotate by age (1 day → 1 year, forever, or custom days) | | **Disk history** | Auto-rotate by age (default **1 year**; also 1 day → forever, or custom days) |
| **Soft disk budget** | Optional Corestore size target that triggers extra prune | | **Soft disk budget** | Optional Corestore size target that triggers extra prune |
| **Max warm points** | Hard cap on stored warm samples (`0` = unlimited) | | **Max warm points** | Hard cap on stored warm samples (`0` = unlimited) |
| **Auto prune** | Scheduled prune interval (hours) | | **Auto prune** | Scheduled prune interval (hours; default on) |
| **Dry-run / Prune now** | Preview or immediately delete expired warm points | | **Dry-run / Prune now** | Preview or immediately delete expired warm points |
| **Save to agent** | Applies live + writes `retention.json` on the agent | | **Save to agent** | Applies live + writes `retention.json` on the agent |
Env seeds (`PEARDATA_TIER*`, `PEARDATA_WARM_*`) only apply when no `retention.json` exists yet. See [CONFIGURATION.md](../docs/CONFIGURATION.md) and [STORAGE-HYPERDB.md](../docs/STORAGE-HYPERDB.md). Default warm retention preset is **1 year**. Env seeds (`PEARDATA_TIER*`, `PEARDATA_WARM_*`) only apply when no `retention.json` exists yet. See [CONFIGURATION.md](../docs/CONFIGURATION.md) and [STORAGE-HYPERDB.md](../docs/STORAGE-HYPERDB.md).
## Connections ## Connections