System Log

2026-07-18 23:05:44 -04:00
parent 7815037737
commit 6ef3afe821
33 changed files with 1360 additions and 32 deletions
@@ -150,6 +150,7 @@ A heavier agent may subscribe to child agents over P2P, downsample into its own
 | `server/services/store.js` | Tiered buffers + query |
 | `server/services/anomaly.js` | Thresholds |
 | `server/services/weights.js` | Metric Correlations scoring (`volume` / `ks2` / …) |
+| `server/services/logs.js` | System log query (anomaly / audit / journalctl) |
 | `server/services/alerts.js` | Alert CRUD helpers |
 | `server/services/subscriptions.js` | Push fan-out |
 | `server/services/jobs.js` | On-demand jobs |
@@ -40,6 +40,7 @@ Treat `SERVER_SEED` like a root password. Prefer `pd1.` invites for operators.
 | `PEARDATA_STORAGE` | — | Electron/Pear storage dir override (bookmarks prefer `Pear.config.storage`) |
 | `PEARDATA_RATE_LIMIT_RPM` | `120` | Per-peer RPC requests per minute |
 | `PEARDATA_MAX_RECONNECT` | `20` | Client manager reconnect attempts per peer |
+| `PEARDATA_JOURNAL` | off | `1` enables host `journalctl` for Logs / `queryLogs` (Linux; needs journal ACL) |

 ```
 data/
@@ -155,4 +155,4 @@ Do not name third-party products in code, commits, or user-facing copy.
 | Engine | Server `weights.js` (`volume` / `ks2` / …) | Client taxonomy + Pearson |
 | UX | Toolbar Correlate → brush ≥15s → Find Correlations → filtered wall | Per-card ⇢ / dblclick → related panel |

-Full operator docs: [metric-correlations.md](../user-guide/metric-correlations.md) · [related-metrics.md](../user-guide/related-metrics.md) · [weights-api.md](../user-guide/weights-api.md).
+Full operator docs: [metric-correlations.md](../user-guide/metric-correlations.md) · [related-metrics.md](../user-guide/related-metrics.md) · [weights-api.md](../user-guide/weights-api.md) · [logs.md](../user-guide/logs.md) · [charts.md](../user-guide/charts.md) (Filters menu).
@@ -178,6 +178,24 @@ Response shape:

 Desktop UX: [user-guide/metric-correlations.md](../user-guide/metric-correlations.md). Related (Pearson/taxonomy) is separate: `shared/related-metrics.js`.

+## Log entries
+
+Normalized shape from `queryLogs` / `GET /api/v*/logs`:
+
+```json
+{
+  "id": "anomaly:1710000000000:system.cpu",
+  "ts": 1710000000000,
+  "source": "anomaly",
+  "severity": "warning",
+  "unit": "system.cpu",
+  "message": "…",
+  "fields": {}
+}
+```
+
+Sources: `anomaly` | `audit` | `journal`. See [user-guide/logs.md](../user-guide/logs.md).
+
 ## REST ↔ RPC parity

 | Concept | RPC | REST |
@@ -185,6 +203,7 @@ Desktop UX: [user-guide/metric-correlations.md](../user-guide/metric-correlation
 | Charts | `listCharts` | `GET /api/v1/charts` |
 | Data | `queryData` | `GET /api/v3/data` |
 | Weights | `getWeights` | `GET /api/v1\|v2\|v3/weights` |
+| Logs | `queryLogs` | `GET /api/v1\|v2\|v3/logs` |
 | Contexts | `listContexts` | `GET /api/v3/contexts` |
 | Nodes | `getNodeInfo` | `GET /api/v3/nodes` |
 | Alerts | `listAlerts` | `GET /api/v3/alerts` |
@@ -14,7 +14,8 @@ The client shell is a **Pear desktop application** built with `pear-electron` +
 | `ui/styles.css` | Layout, theme, **titlebar drag regions** |
 | `client/*` | HyperDHT connection stack used by the UI |
 | `docs/DASHBOARD.md` | Master Charts / metrics wall plan |
-| `user-guide/` | End-user workflows (Correlate, Related, Fleet, …) |
+| `user-guide/` | End-user workflows (Correlate, Related, Logs, Fleet, …) |
+| `ui/logs.js` | System Log tab controller |

 ```bash
 npm start          # pear run -d .
@@ -196,6 +197,8 @@ This template ships the **Pear run** path only. For Electron-forge / multi-arch
 | Metrics wall, time, board, gestures | [user-guide/charts.md](../user-guide/charts.md) · [DASHBOARD.md](./DASHBOARD.md) |
 | **Correlate** (highlight → Find Correlations) | [user-guide/metric-correlations.md](../user-guide/metric-correlations.md) |
 | **Related (⇢)** (taxonomy + Pearson) | [user-guide/related-metrics.md](../user-guide/related-metrics.md) |
+| **Filters** menu (search / TOC / group / sort) | [user-guide/charts.md](../user-guide/charts.md) |
+| **Logs** tab | [user-guide/logs.md](../user-guide/logs.md) |

 Implementation: `ui/dashboard.js`, `server/services/weights.js`, `shared/related-metrics.js`.

@@ -52,8 +52,11 @@ Auth modes at handshake: public key (viewer), capability token / `pd1.` invite,
 | `getChart` | viewer | `{ id }` | Chart summary |
 | `queryData` | viewer | `{ chart, after, before, points, group, tier }` | Time series |
 | `getWeights` | viewer | see below | Metric Correlations / alert weights |
+| `queryLogs` | viewer* | see below | Anomaly / audit / journal lines |
 | `getAllMetrics` | viewer | `{ format: json\|prometheus\|shell }` | Latest export |

+\* `queryLogs` method role is viewer so anomaly search works for all dialers; **audit** and **journal** sources require **admin** inside the handler.
+
 `after` / `before`: absolute unix seconds, or relative (negative = relative to `before`/`now`), agent-style.

 #### `getWeights`
@@ -69,6 +72,18 @@ Scores charts for [Metric Correlations](../user-guide/metric-correlations.md) (o

 REST parity: `GET /api/v*/weights`. Engine: `server/services/weights.js`.

+#### `queryLogs`
+
+| Arg | Notes |
+|-----|--------|
+| `source` | `anomaly` (default) \| `audit` \| `journal` |
+| `q` | Case-insensitive substring |
+| `since` / `until` | Absolute ms/sec or relative (`-1h`) |
+| `priority` / `unit` | Journal filters |
+| `limit` / `cursor` | Cap (≤2000) + pagination offset |
+
+Journal requires `PEARDATA_JOURNAL=1` on Linux. REST: `GET /api/v*/logs`. Engine: `server/services/logs.js`. UI: [user-guide/logs.md](../user-guide/logs.md).
+
 ### Live subscriptions

 | Method | Role | Args |
@@ -4,7 +4,7 @@

 | Doc | Contents |
 |-----|----------|
-| **[User guide](../user-guide/README.md)** | Desktop workflows: Connect, Overview, Charts, Metric Correlations, Related, Alerts, Fleet, Settings |
+| **[User guide](../user-guide/README.md)** | Desktop workflows: Connect, Overview, Charts, Metric Correlations, Logs, Related, Alerts, Fleet, Settings |
 | [GETTING-STARTED.md](./GETTING-STARTED.md) | Install agent / desktop, first dial, systemd |

 ## Engineers & operators
@@ -104,6 +104,7 @@ Single-agent returns one node. With `PEARDATA_PARENT=1`, `/nodes` and `/fleet` i
 |--------|------|-------|
 | GET | `/api/v3/q?q=` | Full-text over chart ids/titles |
 | GET | `/api/v1\|v2\|v3/weights` | Metric Correlations + alert weights — see [Weights](#weights--metric-correlations); user guide: [weights-api](../user-guide/weights-api.md) |
+| GET | `/api/v1\|v2\|v3/logs` | Log search — `source=anomaly\|audit\|journal`, `q`, `since`, `until`, `priority`, `unit`, `limit` — [user-guide/logs](../user-guide/logs.md) |

 ### Alerts

@@ -93,6 +93,7 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
 | Notifications | Done (desktop + `PEARDATA_WEBHOOK_URL`) |
 | Streaming z-score / retrain job | Done (`PEARDATA_ANOMALY_MODE`, job `retrainAnomaly`) |
 | `/api/v3/weights` / Metric Correlations | Done — highlight-vs-baseline scoring (`ks2`/`volume`/`anomaly-rate`/`value`) + Charts Correlate UI |
+| `/api/v3/logs` / System Log tab | Done — anomaly + audit + opt-in journal (`queryLogs`); Charts Filters menu |

 ---

@@ -134,6 +135,8 @@ shared time, multi-dim cards, investigation). Detail checklist:
 | Taxonomy coverage test + EXTENDING note | Done |
 | Desktop polish (layout fill, plot domain, legacy detail removed, Fleet chips) | Done |
 | Metric Correlations (weights engine + Correlate brush UI) | Done |
+| Charts Filters menu (collapsible TOC / search / group / sort) | Done |
+| System Log tab (`queryLogs` — anomaly / audit / journal) | Done |
 | Multi-named custom boards | Later |

 **Phase 6 exit (P0)** — Charts tab lists every agent chart in sections; shared
@@ -47,6 +47,7 @@
 - [ ] Run under systemd with `ProtectSystem` / `NoNewPrivileges` (see `deploy/`)
 - [ ] Do not embed seed in frontend builds, CI logs, or crash reports
 - [ ] Review `LOG_LEVEL=debug` before production (avoid verbose auth noise)
+- [ ] Keep `PEARDATA_JOURNAL` off unless operators need host logs; journal access is **admin-only** over P2P and can expose secrets from other units — add `SupplementaryGroups=systemd-journal` only when enabling
 - [ ] Keep Pear / dependency updates current (`npm outdated`)

 ## Threat notes
@@ -84,7 +85,8 @@ Implementation: `shared/crypto-auth.js`.
 |----------|-------------|----------|
 | `.env` | Critical | Never commit; backup offline |
 | `data/peer-policy.json` | High | Contains roles & JTIs |
-| `data/audit.log` | Medium | Peer activity metadata |
+| `data/audit.log` | Medium | Peer activity metadata; readable via Logs/audit (admin) |
+| Host journal (`PEARDATA_JOURNAL`) | High | May include secrets from other units; admin-only; off by default |
 | `identity.json` | High for that user | Per-machine client secret |
 | Release tarballs | Low | Source only; no secrets |

@@ -17,6 +17,7 @@
 | `test/protocol.test.js` | Constants, monitoring `MethodRoles`, schema validators |
 | `test/store.test.js` | Metric ring buffer ingest + query |
 | `test/weights.test.js` | Metric Correlations engine (`volume` / `ks2` / windows / errors) |
+| `test/logs.test.js` | Log query engine (roles, filters, journal argv, audit file) |
 | `test/hyperdb.test.js` | HyperDB model (node, links, warm points, alerts) |
 | `test/rest.test.js` | agent-style `/api/v3` routes |
 | `test/integration.test.js` | Live HyperDHT agent + handshake + metrics query |
@@ -86,7 +87,10 @@ When adding an RPC method:
 | Health | With server up: `SERVER_PUBLIC_KEY=… npm run healthcheck` |
 | Soak | `SERVER_PUBLIC_KEY=… SERVER_SEED=… npm run soak` |
 | Metric Correlations | Charts → Correlate → brush ≥15s → Find Correlations → wall filters; Clear restores; Related ⇢ still works |
+| Charts Filters | Filters closed → full-width wall; `/` opens search; chip when filter active; Esc closes panel |
 | Weights REST | `curl -sG 'http://127.0.0.1:19999/api/v3/weights' --data-urlencode 'method=volume' --data-urlencode 'after=-60' --data-urlencode 'before=0'` |
+| Logs | Logs tab → Anomalies search; admin → Audit; with `PEARDATA_JOURNAL=1` → Journal |
+| Logs REST | `curl -sG 'http://127.0.0.1:19999/api/v3/logs' --data-urlencode 'source=anomaly' --data-urlencode 'limit=20'` |

 ## Soak test