Updates
CI / test (push) Successful in 1m15s
Release rolling / release (push) Successful in 7m6s

This commit is contained in:
Raven Scott
2026-07-18 19:49:57 -04:00
parent 2e1a3e9b06
commit a639b3c953
19 changed files with 1705 additions and 293 deletions
+12 -2
View File
@@ -44,8 +44,18 @@ PEARDATA_DEFAULT_ROLE=viewer
# PEARDATA_SMART=0
# PEARDATA_NVIDIA=0
# PEARDATA_IPMI=0
# PEARDATA_EBPF=0
# PEARDATA_EBPF_PATH=/run/peardata/ebpf.ndjson
# PEARDATA_EBPF=1
# PEARDATA_EBPF_PATH=
# PEARDATA_HELPER_DIR=
# PEARDATA_IOPING=0
# PEARDATA_ZFS=1
# PEARDATA_BCACHE=1
# PEARDATA_DMCACHE=1
# PEARDATA_FS_STATS=1
# PEARDATA_KAFKA=0
# PEARDATA_NATS=0
# PEARDATA_UNBOUND=0
# PEARDATA_SOCKETS=1
# PEARDATA_PARENT=0
# PEARDATA_PARENT_PEERS=
# PEARDATA_PARENT_PEERS_FILE=
+14 -2
View File
@@ -95,8 +95,20 @@ Permissions: directory `0700`. Do **not** commit `data/` or `.env`.
| `PEARDATA_SMART` | off | `smartctl` disk SMART |
| `PEARDATA_NVIDIA` | off | `nvidia-smi` GPU metrics |
| `PEARDATA_IPMI` | off | `ipmitool sensor` |
| `PEARDATA_EBPF` | off | Ingest eBPF helper NDJSON |
| `PEARDATA_EBPF_PATH` | `/run/peardata/ebpf.ndjson` | Helper output path |
| `PEARDATA_EBPF` | on (Linux) | Embedded eBPF-family helper / JS fallback; `0` disables |
| `PEARDATA_EBPF_PATH` | — | Optional external NDJSON bridge path |
| `PEARDATA_HELPER_DIR` | tmp/helpers | Where embedded helpers are extracted |
| `PEARDATA_IOPING` | off | Active disk latency probe |
| `PEARDATA_ZFS` | auto | ZFS ARC/pool when kstat present |
| `PEARDATA_BCACHE` | auto | BCache when `/sys/fs/bcache` exists |
| `PEARDATA_DMCACHE` | auto | dm-cache when device-mapper cache present; `0` disables |
| `PEARDATA_FS_STATS` | on (Linux) | ext4/xfs lifetime IO stats; `0` disables |
| `PEARDATA_KAFKA` | off | Kafka exporter scrape |
| `PEARDATA_KAFKA_URL` | `http://127.0.0.1:9308/metrics` | |
| `PEARDATA_NATS` | off | NATS `/varz` |
| `PEARDATA_NATS_URL` | `http://127.0.0.1:8222` | |
| `PEARDATA_UNBOUND` | off | Unbound stats |
| `PEARDATA_SOCKETS` | on (Linux) | TCP/UDP socket state charts |
| `PEARDATA_PARENT` | off | `1` enables parent peer fleet aggregator |
| `PEARDATA_PARENT_PEERS` | empty | Comma-separated child agent public keys (64 hex) |
| `PEARDATA_PARENT_PEERS_FILE` | — | Optional file with one child pubkey per line |
-88
View File
@@ -1,88 +0,0 @@
# Missing / deferred metrics
Everything from the advanced host-metrics inventory that PearData can collect in-process (or via optional tools / bridges) is implemented. This file tracks **remaining gaps** that need native code, external products, or are non-metric product features.
Last reviewed: 2026-07-18.
---
## Implemented (complete for agent scope)
| Area | Implementation |
|------|----------------|
| CPU / scheduler | `system.cpu`, per-core, cpufreq, throttle, **cpuidle C-states**, softirqs, per-IRQ |
| Memory | Full meminfo/vmstat pack including zswap, THP matrix, reclaiming, KSM (+savings/ratios), NUMA nodes, pagetype, CMA/directmap, EDAC; debugfs zswap/extfrag when readable |
| Disk | io/ops/util/await/avgsz/qops/busy/iotime/discard/**svctm/merged/flush**; optional partitions (`PEARDATA_DISK_PARTITIONS=1`); MD RAID (`mdstat`); SMART (`PEARDATA_SMART=1` + smartctl) |
| Network | iface + speed/duplex/MTU/queue, softnet, ICMP/ICMPmsg, IPv6, TCP advanced, **conntrack** |
| PSI | some + full |
| Processes | top CPU/RSS/IO/threads (`PEARDATA_PROCESSES=1`) |
| cgroups | CPU/mem/IO + mem detail + throttle + pressure (default on Linux) |
| Sensors | hwmon temp/fan/voltage/power + thermal zones; IPMI via ipmitool; NVIDIA via nvidia-smi; RAPL powercap |
| Apps | nginx, redis, postgres, mysql, apache, memcached, rabbitmq, mongodb, prometheus scrape, statsd ingest |
| Self | `peardata.cpu` / `mem` / `charts` |
| eBPF | **Bridge** (`PEARDATA_EBPF=1` + helper NDJSON file) — see below |
Catalog: `shared/metrics.js` + `metrics-deep.js` + `metrics-more.js`.
---
## Still deferred
### Native eBPF programs (in-process)
Loading CO-RE eBPF from the Node/Bare agent is **not** supported. Use the bridge:
```bash
PEARDATA_EBPF=1
PEARDATA_EBPF_PATH=/run/peardata/ebpf.ndjson
```
A privileged helper should append lines like:
```json
{"chart":"ebpf.cachestat","values":{"hits":12,"misses":3},"ts":1710000000000}
```
Until a first-party helper ships, charts such as cachestat / FD leaks / exec-trace appear only when an external helper feeds the bridge.
### Active probes / specialized FS
| Item | Notes |
|------|-------|
| ioping latency probes | Active I/O; opt-in probe not shipped |
| ext4/xfs internal debugfs deep dive | Partial via debugfs when permitted |
| BCache / DM-Cache / ZFS pool collectors | Use `PEARDATA_PROMETHEUS_URLS` or custom plugin |
| LVM detailed | Not dedicated; mdstat covers MD |
| Kafka / NATS / CoreDNS / Unbound / PowerDNS | Use Prometheus scrape (`PEARDATA_PROMETHEUS=1`) |
| libvirt / Xen / KVM guest balloon beyond vmstat | Partial via `mem.balloon` |
| Rich K8s label enrichment | cgroup discovery only; scrape kube-state-metrics via Prometheus |
| Live socket connection viewer | Interactive function, not a timeseries chart |
| Windows Perflib / macOS deep sysctl | Linux-first; OS fallback CPU/RAM/load |
| Full DynCFG / 400+ alert templates / auto-correlation UI | Product features, not collectors |
| ML on every metric | Threshold + optional z-score |
### Privileges
| Need | Metrics affected |
|------|------------------|
| root / `CAP_DAC_READ_SEARCH` | debugfs zswap pool, extfrag |
| `smartctl`, `nvidia-smi`, `ipmitool` | SMART / GPU / IPMI collectors |
| ptrace/DAC for other users `/proc/PID/io` | process I/O may be incomplete |
---
## Env quick reference (new collectors)
| Variable | Default | Purpose |
|----------|---------|---------|
| `PEARDATA_CGROUPS` | on (Linux) | cgroup v2 charts |
| `PEARDATA_SENSORS` | on (Linux) | hwmon/thermal |
| `PEARDATA_DISK_PARTITIONS` | off | Include partition devices |
| `PEARDATA_IRQ_MAX` | `24` | Per-IRQ chart limit |
| `PEARDATA_MYSQL` / `APACHE` / `MEMCACHED` / `MONGODB` / `RABBITMQ` | off | App collectors |
| `PEARDATA_PROMETHEUS` + `PEARDATA_PROMETHEUS_URLS` | off | Generic scrape |
| `PEARDATA_STATSD` | off | UDP StatsD ingest |
| `PEARDATA_SMART` / `NVIDIA` / `IPMI` | off | Tool-backed hardware |
| `PEARDATA_EBPF` + `PEARDATA_EBPF_PATH` | off | eBPF helper bridge |
See [CONFIGURATION.md](./CONFIGURATION.md).
+268
View File
@@ -0,0 +1,268 @@
/**
* peardata-ebpf — embedded host helper for eBPF-family charts.
*
* Emits NDJSON lines on stdout every interval (default 1000ms):
* {"chart":"ebpf.cachestat","ts":...,"values":{...}}
*
* Uses /proc + /sys (works without CAP_BPF). When compiled with
* -DPEARDATA_HAVE_LIBBPF and libbpf is linked, additional BPF-backed
* series can be enabled later without changing the agent protocol.
*
* Build:
* cc -O2 -o peardata-ebpf main.c
*/
#define _GNU_SOURCE
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <time.h>
#include <stdint.h>
#include <errno.h>
#include <dirent.h>
#include <sys/stat.h>
static long long now_ms(void) {
struct timespec ts;
clock_gettime(CLOCK_REALTIME, &ts);
return (long long)ts.tv_sec * 1000LL + ts.tv_nsec / 1000000LL;
}
static int read_file(const char *path, char *buf, size_t n) {
FILE *f = fopen(path, "r");
if (!f) return -1;
size_t r = fread(buf, 1, n - 1, f);
fclose(f);
buf[r] = 0;
return (int)r;
}
static long long read_ll(const char *path) {
char buf[64];
if (read_file(path, buf, sizeof(buf)) < 0) return -1;
return atoll(buf);
}
static long long vmstat(const char *key) {
FILE *f = fopen("/proc/vmstat", "r");
if (!f) return 0;
char line[256];
char k[64];
long long v = 0;
while (fgets(line, sizeof(line), f)) {
if (sscanf(line, "%63s %lld", k, &v) == 2 && strcmp(k, key) == 0) {
fclose(f);
return v;
}
}
fclose(f);
return 0;
}
static void emit(const char *chart, const char *values_json) {
printf("{\"chart\":\"%s\",\"ts\":%lld,\"values\":%s}\n", chart, now_ms(), values_json);
fflush(stdout);
}
static void collect_cachestat(
long long *prev_hit,
long long *prev_miss,
long long *prev_in,
long long *prev_out,
double dt
) {
/* Page-cache efficiency proxy from vmstat (no CAP_BPF required). */
long long pgfault = vmstat("pgfault");
long long pgmajfault = vmstat("pgmajfault");
long long pgpgin = vmstat("pgpgin");
long long pgpgout = vmstat("pgpgout");
long long hit = pgfault > pgmajfault ? pgfault - pgmajfault : 0;
long long miss = pgmajfault;
double hits = 0, misses = 0, ratio = 0, rin = 0, rout = 0;
if (*prev_hit >= 0 && dt > 0) {
hits = (hit - *prev_hit) / dt;
misses = (miss - *prev_miss) / dt;
rin = (pgpgin - *prev_in) / dt;
rout = (pgpgout - *prev_out) / dt;
double den = hits + misses;
ratio = den > 0 ? (hits / den) * 100.0 : 0;
}
*prev_hit = hit;
*prev_miss = miss;
*prev_in = pgpgin;
*prev_out = pgpgout;
char buf[256];
snprintf(
buf,
sizeof(buf),
"{\"hits\":%.3f,\"misses\":%.3f,\"ratio\":%.3f,\"pgpgin\":%.3f,\"pgpgout\":%.3f}",
hits,
misses,
ratio,
rin,
rout
);
emit("ebpf.cachestat", buf);
}
static void collect_fd(void) {
char buf[128];
long long alloc = 0, max = 0;
if (read_file("/proc/sys/fs/file-nr", buf, sizeof(buf)) > 0) {
sscanf(buf, "%lld %*s %lld", &alloc, &max);
}
char out[128];
snprintf(out, sizeof(out), "{\"open\":%lld,\"max\":%lld}", alloc, max);
emit("ebpf.fd", out);
}
static void collect_oom(long long *prev, double dt) {
long long kills = vmstat("oom_kill");
double rate = 0;
if (*prev >= 0 && dt > 0) rate = (kills - *prev) / dt;
*prev = kills;
char out[64];
snprintf(out, sizeof(out), "{\"kills\":%.3f,\"total\":%lld}", rate, kills);
emit("ebpf.oom", out);
}
static void collect_process(long long *prev_forks, long long *prev_ctxt, double dt) {
FILE *f = fopen("/proc/stat", "r");
long long forks = 0, ctxt = 0;
if (f) {
char line[256];
while (fgets(line, sizeof(line), f)) {
if (strncmp(line, "processes ", 10) == 0) forks = atoll(line + 10);
else if (strncmp(line, "ctxt ", 5) == 0) ctxt = atoll(line + 5);
}
fclose(f);
}
double fr = 0, cr = 0;
if (*prev_forks >= 0 && dt > 0) {
fr = (forks - *prev_forks) / dt;
cr = (ctxt - *prev_ctxt) / dt;
}
*prev_forks = forks;
*prev_ctxt = ctxt;
char out[128];
snprintf(out, sizeof(out), "{\"forks\":%.3f,\"ctxt\":%.3f}", fr, cr);
emit("ebpf.process", out);
}
static void collect_shm(void) {
FILE *f = fopen("/proc/sysvipc/shm", "r");
long long segs = 0, bytes = 0;
if (f) {
char line[512];
int first = 1;
while (fgets(line, sizeof(line), f)) {
if (first) {
first = 0;
continue;
}
long long size = 0;
/* key shmid perms size ... */
if (sscanf(line, "%*s %*s %*s %lld", &size) >= 1) {
segs++;
bytes += size;
}
}
fclose(f);
}
char out[128];
snprintf(out, sizeof(out), "{\"segments\":%lld,\"bytes\":%lld}", segs, bytes);
emit("ebpf.shm", out);
}
static void collect_swap(long long *prev_in, long long *prev_out, double dt) {
long long in = vmstat("pswpin");
long long out = vmstat("pswpout");
double ri = 0, ro = 0;
if (*prev_in >= 0 && dt > 0) {
ri = (in - *prev_in) / dt;
ro = (out - *prev_out) / dt;
}
*prev_in = in;
*prev_out = out;
char buf[128];
snprintf(buf, sizeof(buf), "{\"in\":%.3f,\"out\":%.3f}", ri, ro);
emit("ebpf.swap", buf);
}
static void collect_vfs(long long *prev_read, long long *prev_write, double dt) {
long long r = vmstat("pgpgout"); /* proxy */
long long w = vmstat("pgpgin");
/* Prefer ext4-ish: use /proc/vmstat nr_dirty_threshold unrelated — keep simple */
double rr = 0, ww = 0;
if (*prev_read >= 0 && dt > 0) {
rr = (r - *prev_read) / dt;
ww = (w - *prev_write) / dt;
}
*prev_read = r;
*prev_write = w;
char buf[128];
snprintf(buf, sizeof(buf), "{\"read_pages\":%.3f,\"write_pages\":%.3f}", rr, ww);
emit("ebpf.vfs", buf);
}
static void collect_net(long long *prev_tcp, double dt) {
FILE *f = fopen("/proc/net/sockstat", "r");
long long tcp = 0;
if (f) {
char line[256];
while (fgets(line, sizeof(line), f)) {
if (strncmp(line, "TCP:", 4) == 0) {
sscanf(line, "TCP: inuse %lld", &tcp);
}
}
fclose(f);
}
double rate = 0;
if (*prev_tcp >= 0 && dt > 0) rate = (tcp - *prev_tcp) / dt;
*prev_tcp = tcp;
char buf[128];
snprintf(buf, sizeof(buf), "{\"tcp_inuse\":%lld,\"tcp_delta\":%.3f}", tcp, rate);
emit("ebpf.socket", buf);
}
int main(int argc, char **argv) {
int interval_ms = 1000;
for (int i = 1; i < argc; i++) {
if (strcmp(argv[i], "--interval") == 0 && i + 1 < argc) {
interval_ms = atoi(argv[++i]);
if (interval_ms < 200) interval_ms = 200;
}
}
long long prev_hit = -1, prev_miss = -1, prev_cin = -1, prev_cout = -1;
long long prev_oom = -1;
long long prev_forks = -1, prev_ctxt = -1;
long long prev_swap_in = -1, prev_swap_out = -1;
long long prev_vfs_r = -1, prev_vfs_w = -1;
long long prev_tcp = -1;
long long last = now_ms();
fprintf(stderr, "peardata-ebpf ready interval=%dms\n", interval_ms);
for (;;) {
long long t0 = now_ms();
double dt = (t0 - last) / 1000.0;
if (dt <= 0) dt = interval_ms / 1000.0;
last = t0;
collect_cachestat(&prev_hit, &prev_miss, &prev_cin, &prev_cout, dt);
collect_fd();
collect_oom(&prev_oom, dt);
collect_process(&prev_forks, &prev_ctxt, dt);
collect_shm();
collect_swap(&prev_swap_in, &prev_swap_out, dt);
collect_vfs(&prev_vfs_r, &prev_vfs_w, dt);
collect_net(&prev_tcp, dt);
long long elapsed = now_ms() - t0;
long long sleep_ms = interval_ms - elapsed;
if (sleep_ms < 50) sleep_ms = 50;
usleep((useconds_t)(sleep_ms * 1000));
}
return 0;
}
+1
View File
@@ -45,6 +45,7 @@
"soak": "node scripts/soak.js",
"mint-invite": "node scripts/mint-invite.js",
"build:db": "node scripts/build-db.js",
"build:native": "node scripts/build-native-helpers.cjs",
"rename": "bash scripts/rename-template.sh",
"release:notes": "node -e \"console.log('See docs/RELEASE.md')\"",
"build:client-bundle": "node scripts/build-client-bundle.cjs",
+26
View File
@@ -91,6 +91,18 @@ async function buildOne(host, outRoot) {
)
}
// Ensure native helpers are compiled + base64-embedded into the JS bundle
console.log('[bare-standalone] building/embedding native helpers…')
const { spawnSync } = require('child_process')
const nh = spawnSync(process.execPath, [path.join(root, 'scripts', 'build-native-helpers.cjs')], {
cwd: root,
stdio: 'inherit',
env: process.env,
})
if (nh.status !== 0) {
console.warn('[bare-standalone] native helper build failed (JS fallback still works)')
}
const name = 'peardata-server'
const outDir = path.join(outRoot, `${name}-${host}`)
fs.rmSync(outDir, { recursive: true, force: true })
@@ -160,6 +172,20 @@ async function buildOne(host, outRoot) {
fs.copyFileSync(binary, flat)
binary = flat
}
// Also ship raw helper next to binary for ops that prefer an external file
const pre = path.join(root, 'native', 'prebuilds', `peardata-ebpf-${host}`)
const preAlt = path.join(root, 'native', 'prebuilds', 'peardata-ebpf')
const helperSrc = fs.existsSync(pre) ? pre : fs.existsSync(preAlt) ? preAlt : null
if (helperSrc) {
const helperDest = path.join(outDir, 'peardata-ebpf')
fs.copyFileSync(helperSrc, helperDest)
try {
fs.chmodSync(helperDest, 0o755)
} catch {
// ignore
}
console.log(`[bare-standalone] shipped helper ${helperDest}`)
}
console.log(`[bare-standalone] wrote ${binary}`)
} else {
console.warn(`[bare-standalone] WARN: expected binary not found under ${outDir}`)
+93
View File
@@ -0,0 +1,93 @@
#!/usr/bin/env node
/**
* Compile native helpers and embed them as base64 into
* server/native/embedded-helpers.js so bare-pack includes them
* inside peardata-server.
*
* node scripts/build-native-helpers.cjs
*/
'use strict'
const fs = require('fs')
const path = require('path')
const { spawnSync } = require('child_process')
const root = path.resolve(__dirname, '..')
const src = path.join(root, 'native', 'peardata-ebpf', 'main.c')
const outDir = path.join(root, 'native', 'prebuilds')
const embedOut = path.join(root, 'server', 'native', 'embedded-helpers.js')
function ensureDir(d) {
fs.mkdirSync(d, { recursive: true })
}
function compile(hostTag) {
ensureDir(outDir)
const outBin = path.join(outDir, `peardata-ebpf-${hostTag}`)
const cc = process.env.CC || 'cc'
const args = ['-O2', '-Wall', '-o', outBin, src]
console.log(`[native] ${cc} ${args.join(' ')}`)
const res = spawnSync(cc, args, { cwd: root, stdio: 'inherit' })
if (res.status !== 0) {
console.warn('[native] compile failed — embedding placeholder stub')
return null
}
try {
fs.chmodSync(outBin, 0o755)
} catch {
// ignore
}
return outBin
}
function hostTag() {
const p = process.platform
const a = process.arch
return `${p}-${a}`
}
function main() {
const tag = hostTag()
let bin = null
if (process.platform === 'linux') {
bin = compile(tag)
} else {
console.warn(`[native] skip compile on ${process.platform} (Linux helper only)`)
}
/** @type {Record<string, string>} */
const helpers = {}
if (bin && fs.existsSync(bin)) {
helpers[`peardata-ebpf-${tag}`] = fs.readFileSync(bin).toString('base64')
// also copy generic name for current host
helpers['peardata-ebpf'] = helpers[`peardata-ebpf-${tag}`]
}
// Keep any previously embedded linux builds if present
try {
const prev = fs.readFileSync(embedOut, 'utf8')
const m = prev.match(/export const EMBEDDED_HELPERS = (\{[\s\S]*?\})\n/)
if (m) {
const old = Function(`return (${m[1]})`)()
for (const [k, v] of Object.entries(old || {})) {
if (!helpers[k] && typeof v === 'string' && v.length > 32) helpers[k] = v
}
}
} catch {
// first run
}
ensureDir(path.dirname(embedOut))
const body = `/* AUTO-GENERATED by scripts/build-native-helpers.cjs — do not edit */
export const EMBEDDED_HELPERS = ${JSON.stringify(helpers, null, 2)}
export const EMBEDDED_HELPER_NAMES = ${JSON.stringify(Object.keys(helpers))}
`
fs.writeFileSync(embedOut, body)
console.log(
`[native] wrote ${embedOut} (${Object.keys(helpers).length} helper(s), ` +
`${Math.round(JSON.stringify(helpers).length / 1024)} KiB json)`
)
}
main()
+4
View File
@@ -0,0 +1,4 @@
/* AUTO-GENERATED by scripts/build-native-helpers.cjs — do not edit */
export const EMBEDDED_HELPERS = {}
export const EMBEDDED_HELPER_NAMES = []
+74
View File
@@ -0,0 +1,74 @@
/**
* Extract embedded native helpers next to the agent storage dir
* (or PEARDATA_HELPER_DIR) so they can be exec'd from Bare/Node.
*/
import fs from 'fs'
import path from 'path'
import os from 'os'
import { EMBEDDED_HELPERS } from './embedded-helpers.js'
function helperDir() {
if (process.env.PEARDATA_HELPER_DIR) return process.env.PEARDATA_HELPER_DIR
if (process.env.PEARDATA_HOME) return path.join(process.env.PEARDATA_HOME, 'helpers')
if (process.env.PEARDATA_DATA_DIR) return path.join(process.env.PEARDATA_DATA_DIR, 'helpers')
return path.join(os.tmpdir(), 'peardata-helpers')
}
function hostKey() {
return `${process.platform}-${process.arch}`
}
/**
* @param {string} name e.g. peardata-ebpf
* @returns {string|null} absolute path to executable
*/
export function extractHelper(name) {
const keyed = `${name}-${hostKey()}`
const b64 = EMBEDDED_HELPERS[keyed] || EMBEDDED_HELPERS[name]
if (!b64) {
// Fall back to sibling of process.execPath / PATH
const siblings = [
path.join(path.dirname(process.execPath || ''), name),
path.join(process.cwd(), 'native', 'prebuilds', `${name}-${hostKey()}`),
path.join(process.cwd(), 'native', 'prebuilds', name),
`/usr/local/lib/peardata/${name}`,
]
for (const p of siblings) {
try {
if (fs.existsSync(p)) {
fs.accessSync(p, fs.constants.X_OK)
return p
}
} catch {
// continue
}
}
return null
}
const dir = helperDir()
fs.mkdirSync(dir, { recursive: true })
const dest = path.join(dir, name)
const buf = Buffer.from(b64, 'base64')
let needWrite = true
try {
const st = fs.statSync(dest)
if (st.size === buf.length) needWrite = false
} catch {
needWrite = true
}
if (needWrite) {
fs.writeFileSync(dest, buf, { mode: 0o755 })
try {
fs.chmodSync(dest, 0o755)
} catch {
// ignore
}
}
return dest
}
export function hasEmbeddedHelper(name) {
const keyed = `${name}-${hostKey()}`
return Boolean(EMBEDDED_HELPERS[keyed] || EMBEDDED_HELPERS[name])
}
+130 -5
View File
@@ -90,9 +90,45 @@ import {
isMongodbEnabled,
} from './services/collectors/mongodb.js'
import {
getEbpfBridgeCollector,
isEbpfBridgeEnabled,
} from './services/collectors/ebpf-bridge.js'
getEbpfCollector,
isEbpfEnabled,
} from './services/collectors/ebpf.js'
import {
getIopingCollector,
isIopingEnabled,
} from './services/collectors/ioping.js'
import {
getZfsCollector,
isZfsEnabled,
} from './services/collectors/zfs.js'
import {
getBcacheCollector,
isBcacheEnabled,
} from './services/collectors/bcache.js'
import {
getKafkaCollector,
isKafkaEnabled,
} from './services/collectors/kafka.js'
import {
getNatsCollector,
isNatsEnabled,
} from './services/collectors/nats.js'
import {
getUnboundCollector,
isUnboundEnabled,
} from './services/collectors/unbound.js'
import {
getSocketsCollector,
isSocketsEnabled,
} from './services/collectors/sockets.js'
import {
getDmcacheCollector,
isDmcacheEnabled,
} from './services/collectors/dmcache.js'
import {
getFsStatsCollector,
isFsStatsEnabled,
} from './services/collectors/fs-stats.js'
import { notifyAnomaly } from './services/notify.js'
import { getStore } from './services/store.js'
import { getAnomalyEngine } from './services/anomaly.js'
@@ -320,12 +356,83 @@ export function startPipeline() {
}
let ebpf = null
if (isEbpfBridgeEnabled()) {
ebpf = getEbpfBridgeCollector()
if (isEbpfEnabled()) {
ebpf = getEbpfCollector()
ebpf.on('samples', (batch) => ingestBatch(store, anomalies, batch))
ebpf.start()
}
let ioping = null
if (isIopingEnabled()) {
ioping = getIopingCollector()
ioping.on('samples', (batch) => ingestBatch(store, anomalies, batch))
ioping.on('error', (err) => log.warn('ioping error', { error: err.message }))
ioping.start()
}
let zfs = null
if (isZfsEnabled()) {
zfs = getZfsCollector()
zfs.on('samples', (batch) => ingestBatch(store, anomalies, batch))
zfs.on('error', (err) => log.warn('ZFS error', { error: err.message }))
zfs.start()
}
let bcache = null
if (isBcacheEnabled()) {
bcache = getBcacheCollector()
bcache.on('samples', (batch) => ingestBatch(store, anomalies, batch))
bcache.on('error', (err) => log.warn('BCache error', { error: err.message }))
bcache.start()
}
let kafka = null
if (isKafkaEnabled()) {
kafka = getKafkaCollector()
kafka.on('samples', (batch) => ingestBatch(store, anomalies, batch))
kafka.on('error', (err) => log.warn('Kafka error', { error: err.message }))
kafka.start()
}
let nats = null
if (isNatsEnabled()) {
nats = getNatsCollector()
nats.on('samples', (batch) => ingestBatch(store, anomalies, batch))
nats.on('error', (err) => log.warn('NATS error', { error: err.message }))
nats.start()
}
let unbound = null
if (isUnboundEnabled()) {
unbound = getUnboundCollector()
unbound.on('samples', (batch) => ingestBatch(store, anomalies, batch))
unbound.on('error', (err) => log.warn('Unbound error', { error: err.message }))
unbound.start()
}
let sockets = null
if (isSocketsEnabled()) {
sockets = getSocketsCollector()
sockets.on('samples', (batch) => ingestBatch(store, anomalies, batch))
sockets.start()
}
let dmcache = null
if (isDmcacheEnabled()) {
dmcache = getDmcacheCollector()
dmcache.on('samples', (batch) => ingestBatch(store, anomalies, batch))
dmcache.on('error', (err) => log.warn('dm-cache error', { error: err.message }))
dmcache.start()
}
let fsStats = null
if (isFsStatsEnabled()) {
fsStats = getFsStatsCollector()
fsStats.on('samples', (batch) => ingestBatch(store, anomalies, batch))
fsStats.on('error', (err) => log.warn('fs-stats error', { error: err.message }))
fsStats.start()
}
log.info('Metrics pipeline started', {
hyperdb: Boolean(getDb()),
docker: Boolean(docker),
@@ -350,6 +457,15 @@ export function startPipeline() {
rabbitmq: Boolean(rabbitmq),
mongodb: Boolean(mongodb),
ebpf: Boolean(ebpf),
ioping: Boolean(ioping),
zfs: Boolean(zfs),
bcache: Boolean(bcache),
kafka: Boolean(kafka),
nats: Boolean(nats),
unbound: Boolean(unbound),
sockets: Boolean(sockets),
dmcache: Boolean(dmcache),
fsStats: Boolean(fsStats),
})
return {
collector,
@@ -377,5 +493,14 @@ export function startPipeline() {
rabbitmq,
mongodb,
ebpf,
ioping,
zfs,
bcache,
kafka,
nats,
unbound,
sockets,
dmcache,
fsStats,
}
}
+49 -101
View File
@@ -1,44 +1,11 @@
/**
* bcache sysfs stats collector.
*
* Enable: PEARDATA_BCACHE=1 or auto when /sys/fs/bcache exists
* Disable: PEARDATA_BCACHE=0
*
* Charts: bcache.hit_ratio, bcache.dirty
* BCache collector via /sys/fs/bcache and per-block bcache sysfs.
* Enable: PEARDATA_BCACHE=1 (auto when sysfs present)
*/
import fs from 'fs'
import path from 'path'
import { CollectorPlugin } from './plugin.js'
import { registerChart } from '../../../shared/metrics.js'
import logger from '../../utils/logger.js'
const log = logger.child('bcache')
const CHART_HIT = {
id: 'bcache.hit_ratio',
name: 'bcache.hit_ratio',
context: 'bcache.hit_ratio',
title: 'bcache hit ratio',
units: 'percent',
family: 'bcache',
chartType: 'line',
priority: 4320,
plugin: 'bcache',
dimensions: [{ id: 'hit_ratio', name: 'hit_ratio', algorithm: 'absolute' }],
}
const CHART_DIRTY = {
id: 'bcache.dirty',
name: 'bcache.dirty',
context: 'bcache.dirty',
title: 'bcache dirty data',
units: 'bytes',
family: 'bcache',
chartType: 'line',
priority: 4330,
plugin: 'bcache',
dimensions: [{ id: 'dirty', name: 'dirty', algorithm: 'absolute' }],
}
export function isBcacheEnabled() {
const v = process.env.PEARDATA_BCACHE
@@ -49,94 +16,75 @@ export function isBcacheEnabled() {
function readNum(p) {
try {
const n = Number(fs.readFileSync(p, 'utf8').trim())
return Number.isFinite(n) ? n : null
return Number(fs.readFileSync(p, 'utf8').trim()) || 0
} catch {
return null
}
}
/**
* @returns {{ hit_ratio: number|null, dirty: number|null }}
*/
export function collectBcacheStats() {
const root = '/sys/fs/bcache'
if (!fs.existsSync(root)) return { hit_ratio: null, dirty: null }
let hitSum = 0
let hitCount = 0
let dirtySum = 0
let dirtyCount = 0
for (const uuid of fs.readdirSync(root)) {
const base = path.join(root, uuid)
const hit =
readNum(path.join(base, 'stats_total', 'cache_hit_ratio')) ??
readNum(path.join(base, 'cache_hit_ratio'))
if (hit != null) {
hitSum += hit
hitCount++
}
const dirty =
readNum(path.join(base, 'dirty_data')) ?? readNum(path.join(base, 'stats_total', 'dirty'))
if (dirty != null) {
dirtySum += dirty
dirtyCount++
}
}
return {
hit_ratio: hitCount ? hitSum / hitCount : null,
dirty: dirtyCount ? dirtySum : null,
return 0
}
}
export class BcacheCollector extends CollectorPlugin {
constructor(opts = {}) {
super({ name: 'bcache', intervalMs: opts.intervalMs })
constructor() {
super({ name: 'bcache' })
this.prev = new Map()
}
isEnabled() {
return isBcacheEnabled()
}
start() {
if (!this.isEnabled()) return
registerChart(CHART_HIT)
registerChart(CHART_DIRTY)
log.info('bcache collector started')
super.start()
}
async collect() {
const stats = collectBcacheStats()
if (stats.hit_ratio == null && stats.dirty == null) return []
const ts = Date.now()
/** @type {Array<{ chart: string, context: string, ts: number, values: Record<string, number|null> }>} */
/** @type {Array<{ chart: string, context: string, ts: number, values: object }>} */
const batch = []
if (stats.hit_ratio != null) {
batch.push({
chart: 'bcache.hit_ratio',
context: 'bcache.hit_ratio',
ts,
values: { hit_ratio: stats.hit_ratio },
})
let dirs = []
try {
dirs = fs.readdirSync('/sys/fs/bcache')
} catch {
return batch
}
if (stats.dirty != null) {
batch.push({
chart: 'bcache.dirty',
context: 'bcache.dirty',
ts,
values: { dirty: stats.dirty },
for (const id of dirs) {
const base = path.join('/sys/fs/bcache', id)
if (!fs.existsSync(path.join(base, 'stats_total'))) continue
const hits = readNum(path.join(base, 'stats_total', 'cache_hits'))
const misses = readNum(path.join(base, 'stats_total', 'cache_misses'))
const bypass = readNum(path.join(base, 'stats_total', 'cache_bypass_hits'))
const safe = id.slice(0, 12)
const chart = `bcache.${safe}`
registerChart({
id: chart,
name: chart,
context: 'bcache.cache',
title: `BCache ${safe}`,
units: 'events/s',
family: safe,
chartType: 'line',
priority: 4700,
plugin: 'bcache',
dimensions: [
{ id: 'hits', name: 'hits', algorithm: 'incremental' },
{ id: 'misses', name: 'misses', algorithm: 'incremental' },
{ id: 'bypass', name: 'bypass', algorithm: 'incremental' },
],
})
const prev = this.prev.get(id)
const dt = this.intervalMs / 1000
batch.push({
chart,
context: 'bcache.cache',
ts,
values: {
hits: prev ? Math.max(0, hits - prev.hits) / dt : 0,
misses: prev ? Math.max(0, misses - prev.misses) / dt : 0,
bypass: prev ? Math.max(0, bypass - prev.bypass) / dt : 0,
},
})
this.prev.set(id, { hits, misses, bypass })
}
return batch
}
}
/** @type {BcacheCollector|null} */
let singleton = null
export function getBcacheCollector() {
if (!singleton) singleton = new BcacheCollector()
return singleton
+329
View File
@@ -0,0 +1,329 @@
/**
* Embedded eBPF-family collector.
*
* Prefer the bundled `peardata-ebpf` helper (extracted from the server
* binary). Falls back to the NDJSON file bridge, then to an in-process
* JS implementation of the same charts.
*
* Enable: PEARDATA_EBPF=1 (default on when helper is available)
* Disable: PEARDATA_EBPF=0
*/
import { spawn } from 'child_process'
import { EventEmitter } from 'events'
import fs from 'fs'
import os from 'os'
import {
SAMPLE_INTERVAL_MS,
registerChart,
} from '../../../shared/metrics.js'
import { extractHelper, hasEmbeddedHelper } from '../../native/extract-helper.js'
import logger from '../../utils/logger.js'
const log = logger.child('ebpf')
export function isEbpfEnabled() {
const v = process.env.PEARDATA_EBPF
if (v === '0' || v === 'off' || v === 'false') return false
if (v === '1' || v === 'on' || v === 'true') return true
// Auto: on Linux when helper exists or can be extracted
return os.platform() === 'linux'
}
function chartDef(id, title, units, dims) {
return {
id,
name: id,
context: id,
title,
units,
family: 'ebpf',
chartType: 'line',
priority: 9500,
plugin: 'ebpf',
dimensions: dims.map((d) => ({ id: d, name: d, algorithm: 'absolute' })),
}
}
function ensureCharts() {
registerChart(
chartDef('ebpf.cachestat', 'Page cache efficiency', 'events/s', [
'hits',
'misses',
'ratio',
'pgpgin',
'pgpgout',
])
)
registerChart(chartDef('ebpf.fd', 'Open file descriptors', 'files', ['open', 'max']))
registerChart(chartDef('ebpf.oom', 'OOM kills', 'kills/s', ['kills', 'total']))
registerChart(
chartDef('ebpf.process', 'Process lifecycle', 'events/s', ['forks', 'ctxt'])
)
registerChart(chartDef('ebpf.shm', 'System V shared memory', 'bytes', ['segments', 'bytes']))
registerChart(chartDef('ebpf.swap', 'Swap I/O (eBPF family)', 'pages/s', ['in', 'out']))
registerChart(
chartDef('ebpf.vfs', 'VFS page I/O', 'pages/s', ['read_pages', 'write_pages'])
)
registerChart(
chartDef('ebpf.socket', 'Socket tracking', 'sockets', ['tcp_inuse', 'tcp_delta'])
)
}
export class EbpfCollector extends EventEmitter {
constructor(opts = {}) {
super()
this.intervalMs =
opts.intervalMs ?? (Number(process.env.PEARDATA_SAMPLE_MS) || SAMPLE_INTERVAL_MS)
this.child = null
this.timer = null
this.running = false
this.buf = ''
this.mode = 'none'
/** JS fallback state */
this.prev = {}
this.lastTs = 0
this.bridgeOffset = 0
}
start() {
if (this.running) return
this.running = true
ensureCharts()
const helper = extractHelper('peardata-ebpf')
if (helper) {
this.mode = 'helper'
this._startHelper(helper)
log.info('eBPF helper started', { helper, embedded: hasEmbeddedHelper('peardata-ebpf') })
return
}
const bridge = process.env.PEARDATA_EBPF_PATH
if (bridge) {
this.mode = 'bridge'
this.timer = setInterval(() => this._tickBridge(bridge), this.intervalMs)
if (this.timer.unref) this.timer.unref()
log.info('eBPF bridge mode', { path: bridge })
return
}
this.mode = 'js'
this.timer = setInterval(() => this._tickJs(), this.intervalMs)
if (this.timer.unref) this.timer.unref()
this._tickJs()
log.info('eBPF JS fallback collector started')
}
stop() {
this.running = false
if (this.timer) clearInterval(this.timer)
this.timer = null
if (this.child) {
try {
this.child.kill('SIGTERM')
} catch {
// ignore
}
this.child = null
}
}
_startHelper(bin) {
this.child = spawn(bin, ['--interval', String(this.intervalMs)], {
stdio: ['ignore', 'pipe', 'pipe'],
})
this.child.stdout.setEncoding('utf8')
this.child.stdout.on('data', (chunk) => this._onStdout(chunk))
this.child.stderr.on('data', (chunk) => {
const s = String(chunk).trim()
if (s) log.info('helper', { msg: s })
})
this.child.on('exit', (code) => {
log.warn('eBPF helper exited', { code })
this.child = null
if (this.running) {
this.mode = 'js'
this.timer = setInterval(() => this._tickJs(), this.intervalMs)
if (this.timer.unref) this.timer.unref()
}
})
}
_onStdout(chunk) {
this.buf += chunk
const parts = this.buf.split('\n')
this.buf = parts.pop() || ''
const ts = Date.now()
/** @type {Array<{ chart: string, context: string, ts: number, values: object }>} */
const batch = []
for (const line of parts) {
if (!line.trim()) continue
try {
const row = JSON.parse(line)
if (!row.chart || !row.values) continue
const chart = String(row.chart).startsWith('ebpf.')
? String(row.chart)
: `ebpf.${row.chart}`
batch.push({
chart,
context: chart,
ts: row.ts || ts,
values: row.values,
})
} catch {
// ignore bad lines
}
}
if (batch.length) this.emit('samples', batch)
}
_tickBridge(p) {
let st
try {
st = fs.statSync(p)
} catch {
return
}
if (st.size < this.bridgeOffset) this.bridgeOffset = 0
if (st.size === this.bridgeOffset) return
try {
const fd = fs.openSync(p, 'r')
const len = Math.min(st.size - this.bridgeOffset, 256 * 1024)
const buf = Buffer.alloc(len)
const n = fs.readSync(fd, buf, 0, len, this.bridgeOffset)
fs.closeSync(fd)
this.bridgeOffset += n
this._onStdout(buf.slice(0, n).toString('utf8') + '\n')
} catch {
// ignore
}
}
_readVm() {
/** @type {Record<string, number>} */
const out = {}
try {
for (const line of fs.readFileSync('/proc/vmstat', 'utf8').split('\n')) {
const [k, v] = line.trim().split(/\s+/)
if (k) out[k] = Number(v) || 0
}
} catch {
// ignore
}
return out
}
_tickJs() {
const ts = Date.now()
const dt = this.lastTs ? (ts - this.lastTs) / 1000 : this.intervalMs / 1000
this.lastTs = ts
const vm = this._readVm()
const prev = this.prev
const rate = (k) => (prev[k] != null ? Math.max(0, (vm[k] || 0) - prev[k]) / dt : 0)
const hit = Math.max(0, (vm.pgfault || 0) - (vm.pgmajfault || 0))
const miss = vm.pgmajfault || 0
const dHit = prev.hit != null ? Math.max(0, hit - prev.hit) / dt : 0
const dMiss = prev.miss != null ? Math.max(0, miss - prev.miss) / dt : 0
const den = dHit + dMiss
let fileNr = [0, 0]
try {
const parts = fs.readFileSync('/proc/sys/fs/file-nr', 'utf8').trim().split(/\s+/)
fileNr = [Number(parts[0]) || 0, Number(parts[2]) || 0]
} catch {
// ignore
}
let forks = 0
let ctxt = 0
try {
for (const line of fs.readFileSync('/proc/stat', 'utf8').split('\n')) {
if (line.startsWith('processes ')) forks = Number(line.slice(10)) || 0
if (line.startsWith('ctxt ')) ctxt = Number(line.slice(5)) || 0
}
} catch {
// ignore
}
let tcp = 0
try {
const m = fs.readFileSync('/proc/net/sockstat', 'utf8').match(/TCP:\s+inuse\s+(\d+)/)
if (m) tcp = Number(m[1]) || 0
} catch {
// ignore
}
const batch = [
{
chart: 'ebpf.cachestat',
context: 'ebpf.cachestat',
ts,
values: {
hits: dHit,
misses: dMiss,
ratio: den > 0 ? (dHit / den) * 100 : 0,
pgpgin: rate('pgpgin'),
pgpgout: rate('pgpgout'),
},
},
{
chart: 'ebpf.fd',
context: 'ebpf.fd',
ts,
values: { open: fileNr[0], max: fileNr[1] },
},
{
chart: 'ebpf.oom',
context: 'ebpf.oom',
ts,
values: { kills: rate('oom_kill'), total: vm.oom_kill || 0 },
},
{
chart: 'ebpf.process',
context: 'ebpf.process',
ts,
values: {
forks: prev.forks != null ? Math.max(0, forks - prev.forks) / dt : 0,
ctxt: prev.ctxt != null ? Math.max(0, ctxt - prev.ctxt) / dt : 0,
},
},
{
chart: 'ebpf.swap',
context: 'ebpf.swap',
ts,
values: { in: rate('pswpin'), out: rate('pswpout') },
},
{
chart: 'ebpf.vfs',
context: 'ebpf.vfs',
ts,
values: { read_pages: rate('pgpgout'), write_pages: rate('pgpgin') },
},
{
chart: 'ebpf.socket',
context: 'ebpf.socket',
ts,
values: {
tcp_inuse: tcp,
tcp_delta: prev.tcp != null ? (tcp - prev.tcp) / dt : 0,
},
},
{
chart: 'ebpf.shm',
context: 'ebpf.shm',
ts,
values: { segments: 0, bytes: 0 },
},
]
this.prev = { ...vm, hit, miss, forks, ctxt, tcp }
this.emit('samples', batch)
}
}
let singleton = null
export function getEbpfCollector() {
if (!singleton) singleton = new EbpfCollector()
return singleton
}
+41 -95
View File
@@ -1,132 +1,78 @@
/**
* Disk I/O latency probe via ioping or fsync fallback.
*
* Active disk latency probe (embedded, no external ioping binary).
* Enable: PEARDATA_IOPING=1
* Path: PEARDATA_IOPING_PATH=/ (default)
*
* Charts: ioping.latency
* Path: PEARDATA_IOPING_PATH=/tmp/.peardata-ioping
*/
import fs from 'fs'
import path from 'path'
import os from 'os'
import { execFile } from 'child_process'
import { promisify } from 'util'
import { CollectorPlugin } from './plugin.js'
import { registerChart } from '../../../shared/metrics.js'
import logger from '../../utils/logger.js'
const log = logger.child('ioping')
const execFileAsync = promisify(execFile)
const CHART_LATENCY = {
id: 'ioping.latency',
name: 'ioping.latency',
context: 'ioping.latency',
title: 'I/O ping latency',
units: 'ms',
family: 'ioping',
chartType: 'line',
priority: 4200,
plugin: 'ioping',
dimensions: [{ id: 'latency_ms', name: 'latency_ms', algorithm: 'absolute' }],
}
export function isIopingEnabled() {
const v = process.env.PEARDATA_IOPING
return v === '1' || v === 'on' || v === 'true'
}
function targetPath() {
return process.env.PEARDATA_IOPING_PATH || '/'
}
/**
* @param {string} output
* @returns {number|null}
*/
export function parseIopingOutput(output) {
const m = String(output || '').match(/([\d.]+)\s*ms/i)
if (m) return Number(m[1])
const n = Number(String(output || '').trim().split(/\s+/).pop())
return Number.isFinite(n) ? n : null
}
/**
* @param {string} target
* @returns {Promise<number|null>}
*/
export async function probeIoping(target) {
try {
const { stdout } = await execFileAsync('ioping', ['-c', '1', '-q', target], {
timeout: 10000,
})
return parseIopingOutput(stdout)
} catch {
return null
}
}
/**
* @param {string} target
* @returns {number|null}
*/
export function probeFsyncFallback(target) {
const dir = fs.existsSync(target) && fs.statSync(target).isDirectory() ? target : os.tmpdir()
const file = path.join(dir, `.peardata-ioping-${process.pid}`)
const started = Date.now()
try {
fs.writeFileSync(file, `${Date.now()}\n`)
const fd = fs.openSync(file, 'r+')
fs.fsyncSync(fd)
fs.closeSync(fd)
return Date.now() - started
} catch {
return null
} finally {
try {
fs.unlinkSync(file)
} catch {
// ignore
}
}
function probePath() {
return (
process.env.PEARDATA_IOPING_PATH ||
path.join(os.tmpdir(), '.peardata-ioping')
)
}
export class IopingCollector extends CollectorPlugin {
constructor(opts = {}) {
super({ name: 'ioping', intervalMs: opts.intervalMs })
this.target = opts.path || targetPath()
constructor() {
super({ name: 'ioping' })
}
isEnabled() {
return isIopingEnabled()
}
start() {
if (!this.isEnabled()) return
registerChart(CHART_LATENCY)
log.info('Ioping collector started', { path: this.target })
super.start()
}
async collect() {
const ts = Date.now()
let latency = await probeIoping(this.target)
if (latency == null) latency = probeFsyncFallback(this.target)
if (latency == null) return []
const p = probePath()
const buf = Buffer.alloc(4096, 0x5a)
let latency = 0
try {
const t0 = process.hrtime.bigint()
fs.writeFileSync(p, buf)
fs.readFileSync(p)
const t1 = process.hrtime.bigint()
latency = Number(t1 - t0) / 1e6
try {
fs.unlinkSync(p)
} catch {
// ignore
}
} catch {
latency = -1
}
registerChart({
id: 'disk.ioping',
name: 'disk.ioping',
context: 'disk.ioping',
title: 'Disk probe latency',
units: 'milliseconds',
family: 'ioping',
chartType: 'line',
priority: 2100,
plugin: 'ioping',
dimensions: [{ id: 'latency', name: 'latency', algorithm: 'absolute' }],
})
return [
{
chart: 'ioping.latency',
context: 'ioping.latency',
chart: 'disk.ioping',
context: 'disk.ioping',
ts,
values: { latency_ms: latency },
values: { latency: latency < 0 ? 0 : latency },
},
]
}
}
/** @type {IopingCollector|null} */
let singleton = null
export function getIopingCollector() {
if (!singleton) singleton = new IopingCollector()
return singleton
+88
View File
@@ -0,0 +1,88 @@
/**
* Kafka JMX/exporter HTTP probe (Burrow-style or prometheus jmx exporter).
* Enable: PEARDATA_KAFKA=1
* URL: PEARDATA_KAFKA_URL=http://127.0.0.1:9308/metrics (prometheus) OR
* PEARDATA_KAFKA_URL=http://127.0.0.1:8080 (plain JSON under /stats)
*/
import { CollectorPlugin } from './plugin.js'
import { registerChart } from '../../../shared/metrics.js'
export function isKafkaEnabled() {
const v = process.env.PEARDATA_KAFKA
return v === '1' || v === 'on' || v === 'true'
}
function url() {
return (process.env.PEARDATA_KAFKA_URL || 'http://127.0.0.1:9308/metrics').replace(/\/$/, '')
}
export class KafkaCollector extends CollectorPlugin {
constructor() {
super({ name: 'kafka' })
}
isEnabled() {
return isKafkaEnabled()
}
async collect() {
const ts = Date.now()
registerChart({
id: 'kafka.up',
name: 'kafka.up',
context: 'kafka.up',
title: 'Kafka exporter up',
units: 'boolean',
family: 'kafka',
chartType: 'line',
priority: 8800,
plugin: 'kafka',
dimensions: [{ id: 'up', name: 'up', algorithm: 'absolute' }],
})
try {
const res = await fetch(url(), { signal: AbortSignal.timeout(2500) })
if (!res.ok) throw new Error(String(res.status))
const text = await res.text()
/** @type {Record<string, number>} */
const vals = { brokers: 0, under_replicated: 0, offline_count: 0 }
for (const line of text.split('\n')) {
if (line.startsWith('#') || !line.trim()) continue
const m = line.match(/^([a-zA-Z0-9_:]+)(?:\{[^}]*\})?\s+([0-9.eE+-]+)/)
if (!m) continue
const name = m[1]
const v = Number(m[2])
if (name.includes('broker') && name.includes('info')) vals.brokers += 1
if (name.includes('under_replicated')) vals.under_replicated += v
if (name.includes('topic') && name.includes('partitions')) vals.partition_count += v
}
registerChart({
id: 'kafka.cluster',
name: 'kafka.cluster',
context: 'kafka.cluster',
title: 'Kafka cluster',
units: 'count',
family: 'kafka',
chartType: 'line',
priority: 8810,
plugin: 'kafka',
dimensions: Object.keys(vals).map((id) => ({
id,
name: id,
algorithm: 'absolute',
})),
})
return [
{ chart: 'kafka.up', context: 'kafka.up', ts, values: { up: 1 } },
{ chart: 'kafka.cluster', context: 'kafka.cluster', ts, values: vals },
]
} catch {
return [{ chart: 'kafka.up', context: 'kafka.up', ts, values: { up: 0 } }]
}
}
}
let singleton = null
export function getKafkaCollector() {
if (!singleton) singleton = new KafkaCollector()
return singleton
}
+105
View File
@@ -0,0 +1,105 @@
/**
* NATS monitoring endpoint collector.
* Enable: PEARDATA_NATS=1
* URL: PEARDATA_NATS_URL=http://127.0.0.1:8222
*/
import { CollectorPlugin } from './plugin.js'
import { registerChart } from '../../../shared/metrics.js'
export function isNatsEnabled() {
const v = process.env.PEARDATA_NATS
return v === '1' || v === 'on' || v === 'true'
}
function base() {
return (process.env.PEARDATA_NATS_URL || 'http://127.0.0.1:8222').replace(/\/$/, '')
}
export class NatsCollector extends CollectorPlugin {
constructor() {
super({ name: 'nats' })
}
isEnabled() {
return isNatsEnabled()
}
async collect() {
const ts = Date.now()
registerChart({
id: 'nats.up',
name: 'nats.up',
context: 'nats.up',
title: 'NATS up',
units: 'boolean',
family: 'nats',
chartType: 'line',
priority: 8850,
plugin: 'nats',
dimensions: [{ id: 'up', name: 'up', algorithm: 'absolute' }],
})
try {
const res = await fetch(`${base()}/varz`, { signal: AbortSignal.timeout(2000) })
if (!res.ok) throw new Error(String(res.status))
const j = await res.json()
registerChart({
id: 'nats.connections',
name: 'nats.connections',
context: 'nats.connections',
title: 'NATS connections',
units: 'connections',
family: 'nats',
chartType: 'line',
priority: 8860,
plugin: 'nats',
dimensions: [
{ id: 'connections', name: 'connections', algorithm: 'absolute' },
{ id: 'routes', name: 'routes', algorithm: 'absolute' },
{ id: 'remotes', name: 'remotes', algorithm: 'absolute' },
],
})
registerChart({
id: 'nats.messages',
name: 'nats.messages',
context: 'nats.messages',
title: 'NATS messages',
units: 'messages',
family: 'nats',
chartType: 'line',
priority: 8870,
plugin: 'nats',
dimensions: [
{ id: 'in_msgs', name: 'in_msgs', algorithm: 'absolute' },
{ id: 'out_msgs', name: 'out_msgs', algorithm: 'absolute' },
],
})
return [
{ chart: 'nats.up', context: 'nats.up', ts, values: { up: 1 } },
{
chart: 'nats.connections',
context: 'nats.connections',
ts,
values: {
connections: j.connections || 0,
routes: j.routes || 0,
remotes: j.remotes || 0,
},
},
{
chart: 'nats.messages',
context: 'nats.messages',
ts,
values: { in_msgs: j.in_msgs || 0, out_msgs: j.out_msgs || 0 },
},
]
} catch {
return [{ chart: 'nats.up', context: 'nats.up', ts, values: { up: 0 } }]
}
}
}
let singleton = null
export function getNatsCollector() {
if (!singleton) singleton = new NatsCollector()
return singleton
}
+155
View File
@@ -0,0 +1,155 @@
/**
* TCP/UDP socket state timeseries from /proc/net/tcp{,6} and udp.
* Always-on on Linux (cheap enough); disable with PEARDATA_SOCKETS=0.
*/
import fs from 'fs'
import os from 'os'
import { EventEmitter } from 'events'
import { SAMPLE_INTERVAL_MS, registerChart } from '../../../shared/metrics.js'
import logger from '../../utils/logger.js'
const log = logger.child('sockets')
const TCP_STATES = {
'01': 'established',
'02': 'syn_sent',
'03': 'syn_recv',
'04': 'fin_wait1',
'05': 'fin_wait2',
'06': 'time_wait',
'07': 'close',
'08': 'close_wait',
'09': 'last_ack',
'0A': 'listen',
'0B': 'closing',
}
export function isSocketsEnabled() {
const v = process.env.PEARDATA_SOCKETS
if (v === '0' || v === 'off' || v === 'false') return false
if (v === '1' || v === 'on' || v === 'true') return true
return os.platform() === 'linux'
}
function countTcp(file) {
/** @type {Record<string, number>} */
const counts = {
established: 0,
listen: 0,
time_wait: 0,
close_wait: 0,
syn_sent: 0,
syn_recv: 0,
other: 0,
}
let raw
try {
raw = fs.readFileSync(file, 'utf8')
} catch {
return counts
}
for (const line of raw.split('\n').slice(1)) {
const parts = line.trim().split(/\s+/)
if (parts.length < 4) continue
const st = parts[3]
const name = TCP_STATES[st]
if (name && name in counts) counts[name]++
else counts.other++
}
return counts
}
function countUdp(file) {
try {
return Math.max(0, fs.readFileSync(file, 'utf8').trim().split('\n').length - 1)
} catch {
return 0
}
}
export class SocketsCollector extends EventEmitter {
constructor(opts = {}) {
super()
this.intervalMs =
opts.intervalMs ?? (Number(process.env.PEARDATA_SAMPLE_MS) || SAMPLE_INTERVAL_MS)
this.timer = null
this.running = false
}
start() {
if (this.running) return
this.running = true
registerChart({
id: 'net.socket_states',
name: 'net.socket_states',
context: 'net.socket_states',
title: 'TCP socket states',
units: 'sockets',
family: 'sockets',
chartType: 'stacked',
priority: 775,
plugin: 'sockets',
dimensions: [
'established',
'listen',
'time_wait',
'close_wait',
'syn_sent',
'syn_recv',
'other',
].map((id) => ({ id, name: id, algorithm: 'absolute' })),
})
registerChart({
id: 'net.udp_sockets',
name: 'net.udp_sockets',
context: 'net.udp_sockets',
title: 'UDP sockets',
units: 'sockets',
family: 'sockets',
chartType: 'line',
priority: 776,
plugin: 'sockets',
dimensions: [
{ id: 'udp', name: 'udp', algorithm: 'absolute' },
{ id: 'udp6', name: 'udp6', algorithm: 'absolute' },
],
})
this._tick()
this.timer = setInterval(() => this._tick(), this.intervalMs)
if (this.timer.unref) this.timer.unref()
log.info('Sockets collector started')
}
stop() {
this.running = false
if (this.timer) clearInterval(this.timer)
this.timer = null
}
_tick() {
const ts = Date.now()
const v4 = countTcp('/proc/net/tcp')
const v6 = countTcp('/proc/net/tcp6')
/** @type {Record<string, number>} */
const merged = {}
for (const k of Object.keys(v4)) merged[k] = (v4[k] || 0) + (v6[k] || 0)
this.emit('samples', [
{ chart: 'net.socket_states', context: 'net.socket_states', ts, values: merged },
{
chart: 'net.udp_sockets',
context: 'net.udp_sockets',
ts,
values: {
udp: countUdp('/proc/net/udp'),
udp6: countUdp('/proc/net/udp6'),
},
},
])
}
}
let singleton = null
export function getSocketsCollector() {
if (!singleton) singleton = new SocketsCollector()
return singleton
}
+111
View File
@@ -0,0 +1,111 @@
/**
* Unbound DNS collector via unbound-control or remote HTTP stats.
* Enable: PEARDATA_UNBOUND=1
*/
import { spawnSync } from 'child_process'
import { CollectorPlugin } from './plugin.js'
import { registerChart } from '../../../shared/metrics.js'
export function isUnboundEnabled() {
const v = process.env.PEARDATA_UNBOUND
return v === '1' || v === 'on' || v === 'true'
}
export class UnboundCollector extends CollectorPlugin {
constructor() {
super({ name: 'unbound' })
this.prev = null
this.prevTs = 0
}
isEnabled() {
return isUnboundEnabled()
}
async collect() {
const ts = Date.now()
/** @type {Record<string, number>} */
const stats = {}
const http = process.env.PEARDATA_UNBOUND_URL
if (http) {
try {
const res = await fetch(http, { signal: AbortSignal.timeout(2000) })
const text = await res.text()
for (const line of text.split('\n')) {
const [k, v] = line.split('=')
if (k && v != null && Number.isFinite(Number(v))) stats[k.trim()] = Number(v)
}
} catch {
// fall through
}
}
if (!Object.keys(stats).length) {
const res = spawnSync('unbound-control', ['stats_noreset'], {
encoding: 'utf8',
timeout: 2000,
})
if (res.status === 0 && res.stdout) {
for (const line of res.stdout.split('\n')) {
const [k, v] = line.split('=')
if (k && v != null && Number.isFinite(Number(v))) stats[k.trim()] = Number(v)
}
}
}
registerChart({
id: 'unbound.up',
name: 'unbound.up',
context: 'unbound.up',
title: 'Unbound up',
units: 'boolean',
family: 'unbound',
chartType: 'line',
priority: 8900,
plugin: 'unbound',
dimensions: [{ id: 'up', name: 'up', algorithm: 'absolute' }],
})
if (!Object.keys(stats).length) {
return [{ chart: 'unbound.up', context: 'unbound.up', ts, values: { up: 0 } }]
}
const dt = this.prevTs ? (ts - this.prevTs) / 1000 : 0
const prev = this.prev
const num = (k) => stats[k] || 0
const rate = (k) => (prev && dt > 0 ? Math.max(0, num(k) - (prev[k] || 0)) / dt : 0)
registerChart({
id: 'unbound.queries',
name: 'unbound.queries',
context: 'unbound.queries',
title: 'Unbound queries',
units: 'queries/s',
family: 'unbound',
chartType: 'line',
priority: 8910,
plugin: 'unbound',
dimensions: [
{ id: 'total', name: 'total', algorithm: 'incremental' },
{ id: 'cache_hits', name: 'cache_hits', algorithm: 'incremental' },
{ id: 'cache_misses', name: 'cache_misses', algorithm: 'incremental' },
],
})
this.prev = stats
this.prevTs = ts
return [
{ chart: 'unbound.up', context: 'unbound.up', ts, values: { up: 1 } },
{
chart: 'unbound.queries',
context: 'unbound.queries',
ts,
values: {
total: rate('total.num.queries') || rate('num.queries'),
cache_hits: rate('total.num.cachehits') || rate('num.cachehits'),
cache_misses: rate('total.num.cachemiss') || rate('num.cachemiss'),
},
},
]
}
}
let singleton = null
export function getUnboundCollector() {
if (!singleton) singleton = new UnboundCollector()
return singleton
}
+171
View File
@@ -0,0 +1,171 @@
/**
* ZFS / ZFS-on-Linux collector via /proc/spl/kstat/zfs and zpool list.
* Enable: PEARDATA_ZFS=1 (auto when kstat present and unset)
*/
import fs from 'fs'
import path from 'path'
import { spawnSync } from 'child_process'
import { CollectorPlugin } from './plugin.js'
import { registerChart } from '../../../shared/metrics.js'
export function isZfsEnabled() {
const v = process.env.PEARDATA_ZFS
if (v === '0' || v === 'off' || v === 'false') return false
if (v === '1' || v === 'on' || v === 'true') return true
return fs.existsSync('/proc/spl/kstat/zfs')
}
function readArcstats() {
const p = '/proc/spl/kstat/zfs/arcstats'
if (!fs.existsSync(p)) return null
/** @type {Record<string, number>} */
const out = {}
for (const line of fs.readFileSync(p, 'utf8').split('\n')) {
const parts = line.trim().split(/\s+/)
if (parts.length >= 3 && /^\d+$/.test(parts[2])) out[parts[0]] = Number(parts[2])
}
return out
}
export class ZfsCollector extends CollectorPlugin {
constructor() {
super({ name: 'zfs' })
this.prev = null
this.prevTs = 0
}
isEnabled() {
return isZfsEnabled()
}
async collect() {
const ts = Date.now()
const arc = readArcstats()
/** @type {Array<{ chart: string, context: string, ts: number, values: object }>} */
const batch = []
if (arc) {
registerChart({
id: 'zfs.arc',
name: 'zfs.arc',
context: 'zfs.arc',
title: 'ZFS ARC size',
units: 'MiB',
family: 'zfs',
chartType: 'stacked',
priority: 4600,
plugin: 'zfs',
dimensions: [
{ id: 'size', name: 'size', algorithm: 'absolute' },
{ id: 'target', name: 'target', algorithm: 'absolute' },
{ id: 'mru', name: 'mru', algorithm: 'absolute' },
{ id: 'mfu', name: 'mfu', algorithm: 'absolute' },
],
})
registerChart({
id: 'zfs.arc_hits',
name: 'zfs.arc_hits',
context: 'zfs.arc_hits',
title: 'ZFS ARC hits/misses',
units: 'events/s',
family: 'zfs',
chartType: 'line',
priority: 4610,
plugin: 'zfs',
dimensions: [
{ id: 'hits', name: 'hits', algorithm: 'incremental' },
{ id: 'misses', name: 'misses', algorithm: 'incremental' },
],
})
const dt = this.prevTs ? (ts - this.prevTs) / 1000 : 0
const prev = this.prev
const mib = (n) => (n || 0) / (1024 * 1024)
batch.push({
chart: 'zfs.arc',
context: 'zfs.arc',
ts,
values: {
size: mib(arc.size),
target: mib(arc.c),
mru: mib(arc.mru_size),
mfu: mib(arc.mfu_size),
},
})
batch.push({
chart: 'zfs.arc_hits',
context: 'zfs.arc_hits',
ts,
values: {
hits: prev && dt > 0 ? Math.max(0, (arc.hits || 0) - (prev.hits || 0)) / dt : 0,
misses:
prev && dt > 0 ? Math.max(0, (arc.misses || 0) - (prev.misses || 0)) / dt : 0,
},
})
this.prev = arc
this.prevTs = ts
}
// zpool list -Hp
try {
const res = spawnSync('zpool', ['list', '-Hp', '-o', 'name,size,alloc,free,health'], {
encoding: 'utf8',
timeout: 2000,
})
if (res.status === 0 && res.stdout) {
for (const line of res.stdout.trim().split('\n')) {
const [name, size, alloc, free, health] = line.split('\t')
if (!name) continue
const id = `zfs.pool.${name.replace(/[^\w.-]/g, '_')}`
registerChart({
id,
name: id,
context: 'zfs.pool',
title: `ZFS pool ${name}`,
units: 'GiB',
family: name,
chartType: 'stacked',
priority: 4620,
plugin: 'zfs',
dimensions: [
{ id: 'alloc', name: 'alloc', algorithm: 'absolute' },
{ id: 'free', name: 'free', algorithm: 'absolute' },
{ id: 'online', name: 'online', algorithm: 'absolute' },
],
})
batch.push({
chart: id,
context: 'zfs.pool',
ts,
values: {
alloc: Number(alloc) / (1024 ** 3),
free: Number(free) / (1024 ** 3),
online: String(health).toUpperCase() === 'ONLINE' ? 1 : 0,
},
})
void size
}
}
} catch {
// zpool not installed
}
// Also scan /proc/spl/kstat/zfs/*/io
try {
const root = '/proc/spl/kstat/zfs'
for (const ent of fs.readdirSync(root)) {
const ioPath = path.join(root, ent, 'io')
if (!fs.existsSync(ioPath)) continue
// pool-level io kstat — skip if not file
}
} catch {
// ignore
}
return batch
}
}
let singleton = null
export function getZfsCollector() {
if (!singleton) singleton = new ZfsCollector()
return singleton
}
+34
View File
@@ -0,0 +1,34 @@
import test from 'brittle'
import { EbpfCollector, isEbpfEnabled } from '../server/services/collectors/ebpf.js'
import { hasEmbeddedHelper } from '../server/native/extract-helper.js'
import { EMBEDDED_HELPERS } from '../server/native/embedded-helpers.js'
test('ebpf enablement is boolean', (t) => {
t.ok(typeof isEbpfEnabled() === 'boolean')
t.ok(typeof EMBEDDED_HELPERS === 'object')
t.ok(typeof hasEmbeddedHelper('peardata-ebpf') === 'boolean')
})
test('ebpf JS fallback emits family charts', async (t) => {
const prev = process.env.PEARDATA_EBPF
process.env.PEARDATA_EBPF = '1'
const c = new EbpfCollector({ intervalMs: 40 })
/** @type {any[]} */
let batch = []
c.on('samples', (b) => {
batch = b
})
// Force JS path
c.mode = 'js'
c.running = true
c._tickJs()
await new Promise((r) => setTimeout(r, 60))
c._tickJs()
c.stop()
if (prev === undefined) delete process.env.PEARDATA_EBPF
else process.env.PEARDATA_EBPF = prev
t.ok(batch.length >= 1)
const charts = new Set(batch.map((s) => s.chart))
t.ok(charts.has('ebpf.cachestat') || charts.has('ebpf.fd'))
})