Usage — total data dir, Corestore, memory rings, warm HyperDB sample
CI / test (push) Successful in 1m2s
Release rolling / release (push) Has been cancelled

Hot memory — 15m → 24h (or custom points)
Warm memory — downsampled ring + samples-per-bucket
Disk history — auto-rotate by age (1 day → 1 year, forever, or custom days)
Soft disk budget + max warm points
Auto prune on a configurable hour interval
Dry-run prune / Prune now / Save to agent (admin role)
This commit is contained in:
Raven Scott
2026-07-19 12:55:13 -04:00
parent b5b2907496
commit c3b4240e47
26 changed files with 1610 additions and 17 deletions
+8 -2
View File
@@ -57,9 +57,15 @@ Permissions: directory `0700`. Do **not** commit `data/` or `.env`.
| Variable | Default | Description |
|----------|---------|-------------|
| `PEARDATA_SAMPLE_MS` | `1000` | Collector interval |
| `PEARDATA_TIER0_POINTS` | `3600` | High-res ring size (~1h @ 1s) |
| `PEARDATA_TIER1_POINTS` | `1440` | Downsampled ring size |
| `PEARDATA_TIER0_POINTS` | `3600` | High-res ring size (~1h @ 1s); seeds Data Manager |
| `PEARDATA_TIER1_POINTS` | `1440` | Downsampled ring size; seeds Data Manager |
| `PEARDATA_TIER1_EVERY` | `60` | Samples per tier1 average (also HyperDB warm flush) |
| `PEARDATA_WARM_RETENTION_MS` | `604800000` (7d) | Soft age limit for HyperDB warm points (seed) |
| `PEARDATA_WARM_MAX_BYTES` | `0` | Soft corestore budget in bytes (`0` = unlimited) |
| `PEARDATA_WARM_MAX_POINTS` | `0` | Cap on warm points (`0` = unlimited) |
| `PEARDATA_AUTO_PRUNE` | on | `0` disables scheduled prune on boot |
Live retention is edited in the desktop **Settings → Data** tab (admin) and persisted to `$PEARDATA_DATA_DIR/retention.json`. Env values only seed the file on first boot.
---
+10 -1
View File
@@ -108,9 +108,18 @@ Journal is **enabled by default** on Linux (`PEARDATA_JOURNAL=0` to disable). In
| Method | Role | Known jobs |
|--------|------|------------|
| `listJobs` | viewer | — |
| `runJob` | operator | `collectOnce`, `snapshot`, `gcBuffers` |
| `runJob` | operator | `collectOnce`, `snapshot`, `gcBuffers` (prune) |
| `cancelJob` | operator | By job id |
### Data Manager (retention / storage)
| Method | Role | Notes |
|--------|------|-------|
| `getStorageInfo` | viewer | Disk usage, memory rings, warm sample stats, effective retention |
| `getRetentionConfig` | viewer | Current policy (`retention.json`) |
| `setRetentionConfig` | admin | Update hot/warm rings, age rotate, disk budget, auto-prune |
| `pruneHistory` | admin | `{ dryRun?, beforeMs? }` — delete warm points past retention |
### HyperDB / peer links
| Method | Role | Notes |
+1 -1
View File
@@ -45,7 +45,7 @@ curl -s http://127.0.0.1:18888/api/v3/health | jq
| GET | `/api/v3/db` | HyperDB public/discovery keys + collections |
| GET | `/api/v3/versions` | Agent / protocol / API versions |
| GET | `/api/v3/me` | Anonymous REST identity note |
| GET | `/api/v3/settings` | Runtime knobs |
| GET | `/api/v3/settings` | Runtime knobs + retention + storage usage summary |
| GET | `/api/v3/config` | alias of settings |
| GET | `/health`, `/api/v1/health`, `/api/v3/health` | Aggregate health |
+2 -1
View File
@@ -47,7 +47,8 @@ Template rebrand, protocol, collector, memory store, anomalies, REST, desktop MV
| RPC peer-link + `getDbInfo` / REST `/api/v3/db` | Done |
| Query fallback: memory miss / long window → HyperDB warm | Done |
| Hyperswarm mesh (`PEARDATA_SWARM=1`) | Done (opt-in) |
| Configurable retention knobs (tier sizes) | Done (`PEARDATA_TIER*`) |
| Configurable retention knobs (tier sizes) | Done (`PEARDATA_TIER*` + Data Manager RPC/UI) |
| Data Manager (usage, age rotate, auto-prune) | Done (`retention.json`, Settings → Data) |
| Documented warm history across restart (operator M4) | Done (`store-hyperdb-fallback` + STORAGE doc) |
| Export snapshot job → JSON / Prometheus push | Done (`exportSnapshot`, `prometheusPush`) |
| Autobase multi-writer parents | Later (Phase 2c) |
+5
View File
@@ -129,6 +129,11 @@ Banner fields on agent start:
| `PEARDATA_SWARM` | off | `1` enables Hyperswarm `store.replicate` |
| `PEARDATA_TIER1_EVERY` | `60` | Samples per warm bucket (~60s @ 1Hz) |
| `PEARDATA_TIER1_POINTS` | `1440` | In-memory tier1 ring (HyperDB keeps longer) |
| `PEARDATA_WARM_RETENTION_MS` | `7d` | Seeds Data Manager age rotate |
| `PEARDATA_WARM_MAX_BYTES` / `_POINTS` | `0` | Soft caps; `0` = unlimited |
| `PEARDATA_AUTO_PRUNE` | on | Scheduled prune after boot |
**Data Manager:** desktop **Settings → Data** (admin) calls `setRetentionConfig` / `pruneHistory`. Policy file: `$PEARDATA_DATA_DIR/retention.json`. Auto-prune runs on an interval and deletes HyperDB `@peardata/metric-point` rows older than the configured window (and enforces optional point/disk caps). Note: Hypercore deletes are logical; reclaiming physical disk may require compaction over time.
---