Alert System Changes: Fixes applied (feature kept, load bounded)
Release rolling / release (push) Successful in 8m4s
Release rolling / release (push) Successful in 8m4s
1. Removed docker.df() entirely 2. Poll only what rules need • docker_daemon → cheap ping • container_health / stack_health → listContainers • resource → host statfs only (no Docker) • Event-only rules → no poll timer 3. Poll floor 60s, default 120s (was 15s min / 60s default) 4. No overlapping polls (pollInFlight) 5. Event path: drop noisy actions (exec_*, attach, …), match only relevant rules, serialize work (bounded queue) 6. UI status shows whether poll is active
This commit is contained in:
@@ -4,15 +4,16 @@
|
|||||||
|
|
||||||
export const CONFIG = {
|
export const CONFIG = {
|
||||||
// Stats collection (server/services/stats.js)
|
// Stats collection (server/services/stats.js)
|
||||||
// Prefer one-shot polls over perpetual stream:true attachments — each live
|
// Round-robin one-shot samples — not full-fleet stream:true (Dozzle-style streams
|
||||||
// stream forces dockerd to sample that container ~1Hz and is a host CPU hog.
|
// keep dockerd busy ~1Hz × N containers). See Settings → Performance.
|
||||||
// Override at runtime: PEARDOCK_STATS_INTERVAL_MS, PEARDOCK_STATS_CONCURRENCY.
|
|
||||||
STATS: {
|
STATS: {
|
||||||
INTERVAL_MS: 2000, // Min gap between allStats broadcasts
|
INTERVAL_MS: 5000, // Min gap between allStats broadcasts
|
||||||
ACTIVE_INTERVAL_MS: 2000, // Collection tick while peers are connected
|
ACTIVE_INTERVAL_MS: 5000, // Idle gap after each sample batch completes
|
||||||
IDLE_INTERVAL_MS: 5000, // Reserved (peer-idle uses full pause)
|
IDLE_INTERVAL_MS: 5000, // Reserved (peer-idle uses full pause)
|
||||||
CACHE_TTL_MS: 1000, // Per-container broadcast cache TTL
|
CACHE_TTL_MS: 2000, // Per-container broadcast cache TTL
|
||||||
CONCURRENCY: 6, // Max parallel one-shot stats to dockerd
|
CONCURRENCY: 2, // Max parallel one-shot stats to dockerd
|
||||||
|
SAMPLES_PER_TICK: 4, // Max containers sampled per tick (round-robin)
|
||||||
|
LIST_INTERVAL_MS: 15000, // How often to re-list containers
|
||||||
SMOOTHING_FACTOR: 0.2, // Client-side smoothing (if used)
|
SMOOTHING_FACTOR: 0.2, // Client-side smoothing (if used)
|
||||||
},
|
},
|
||||||
|
|
||||||
|
|||||||
+12
-6
@@ -105,16 +105,22 @@ Subscribes to Engine events; fans out `push:dockerEvent` and may trigger list re
|
|||||||
|
|
||||||
Periodic container stats → `push:allStats`; history buffer for charts.
|
Periodic container stats → `push:allStats`; history buffer for charts.
|
||||||
|
|
||||||
**Important:** collection uses **one-shot** `stats({ stream: false })` with bounded concurrency (default 6), not perpetual per-container streams. Live `stream: true` attachments force dockerd to sample every running container ~1 Hz and routinely pin host load average. Stats only run while `peers.size > 0`. Operators can tune intervals and concurrency from the client under **Settings → Performance** (admin) without restarting the server.
|
**Important:** peardock does **not** open perpetual per-container `stream: true` stats (Dozzle does, and [documents elevated dockerd CPU](https://dozzle.dev/guide/faq) as expected). Instead:
|
||||||
|
|
||||||
|
1. **Round-robin one-shot** samples (`stream: false`) — at most `samplesPerTick` containers per batch
|
||||||
|
2. **Post-batch idle gap** — next batch waits `collectIntervalMs` after the previous finishes (no overlapping sweeps)
|
||||||
|
3. **Roster throttle** — `listContainers` only every `listIntervalMs`
|
||||||
|
4. **Peer-idle pause** — no sampling when `peers.size === 0`
|
||||||
|
|
||||||
| Env / UI | Default | Meaning |
|
| Env / UI | Default | Meaning |
|
||||||
|----------|---------|---------|
|
|----------|---------|---------|
|
||||||
| `PEARDOCK_STATS_INTERVAL_MS` / Settings → Performance | 2000 | Collection tick while peers connected |
|
| Collect gap / `PEARDOCK_STATS_INTERVAL_MS` | 5000 | Idle after each sample batch |
|
||||||
| `PEARDOCK_STATS_BROADCAST_MS` | 2000 | Min gap between `push:allStats` |
|
| Samples/tick / `PEARDOCK_STATS_SAMPLES_PER_TICK` | 4 | Max containers sampled per batch (main load knob) |
|
||||||
| `PEARDOCK_STATS_CONCURRENCY` | 6 | Max parallel one-shot stats requests |
|
| Concurrency / `PEARDOCK_STATS_CONCURRENCY` | 2 | Parallel one-shots within a batch |
|
||||||
| `PEARDOCK_STATS_CACHE_TTL_MS` | 1000 | Idle-container broadcast cache TTL |
|
| List interval / `PEARDOCK_STATS_LIST_INTERVAL_MS` | 15000 | Roster refresh |
|
||||||
|
| Broadcast / `PEARDOCK_STATS_BROADCAST_MS` | 5000 | Min gap between `push:allStats` |
|
||||||
|
|
||||||
Runtime values persist in `peardock-stats.json` (cwd or `PEARDOCK_STATS_PATH`). RPC: `getStatsConfig` (viewer), `updateStatsConfig` (admin). The Settings → **Performance** tab applies changes live with no server restart.
|
Runtime values persist in `peardock-stats.json` (cwd or `PEARDOCK_STATS_PATH`). RPC: `getStatsConfig` (viewer), `updateStatsConfig` (admin). Settings → **Performance** applies live. On hot hosts: raise collect gap, keep samples/tick at 2–4 and concurrency at 1–2.
|
||||||
|
|
||||||
### Metrics (`services/metrics.js`)
|
### Metrics (`services/metrics.js`)
|
||||||
|
|
||||||
|
|||||||
+25
-14
@@ -2906,33 +2906,43 @@ services:
|
|||||||
<p class="small text-muted mb-2">Changes apply live on the server — no restart and no Save button.</p>
|
<p class="small text-muted mb-2">Changes apply live on the server — no restart and no Save button.</p>
|
||||||
<div class="row g-3 mt-1">
|
<div class="row g-3 mt-1">
|
||||||
<div class="col-md-3">
|
<div class="col-md-3">
|
||||||
<label class="form-label" for="stats-collect-sec">Collect interval (sec)</label>
|
<label class="form-label" for="stats-collect-sec">Idle gap after batch (sec)</label>
|
||||||
<input type="number" id="stats-collect-sec" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="1" max="60" step="1" value="2">
|
<input type="number" id="stats-collect-sec" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="1" max="60" step="1" value="5">
|
||||||
<div class="form-text">How often the server starts a stats sweep while a client is connected. Higher = less dockerd CPU.</div>
|
<div class="form-text">Wait this long <em>after</em> a sample batch finishes before the next. Higher = less dockerd CPU.</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="col-md-3">
|
<div class="col-md-3">
|
||||||
<label class="form-label" for="stats-broadcast-sec">Broadcast interval (sec)</label>
|
<label class="form-label" for="stats-samples-per-tick">Samples per tick</label>
|
||||||
<input type="number" id="stats-broadcast-sec" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="1" max="60" step="1" value="2">
|
<input type="number" id="stats-samples-per-tick" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="1" max="100" step="1" value="4">
|
||||||
<div class="form-text">Min gap between <code>push:allStats</code> updates to clients.</div>
|
<div class="form-text">Containers sampled each tick (round-robin). <strong>Main load control</strong> — do not set to fleet size.</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="col-md-3">
|
<div class="col-md-3">
|
||||||
<label class="form-label" for="stats-concurrency">Sample concurrency</label>
|
<label class="form-label" for="stats-concurrency">Sample concurrency</label>
|
||||||
<input type="number" id="stats-concurrency" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="1" max="32" step="1" value="6">
|
<input type="number" id="stats-concurrency" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="1" max="32" step="1" value="2">
|
||||||
<div class="form-text">Max parallel one-shot stats requests to dockerd. Keep modest on busy hosts.</div>
|
<div class="form-text">Max parallel one-shot stats to dockerd within a tick. Keep at 1–2 on busy hosts.</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="col-md-3">
|
<div class="col-md-3">
|
||||||
<label class="form-label" for="stats-cache-ttl-ms">Idle cache TTL (ms)</label>
|
<label class="form-label" for="stats-broadcast-sec">Broadcast interval (sec)</label>
|
||||||
<input type="number" id="stats-cache-ttl-ms" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="0" max="30000" step="100" value="1000">
|
<input type="number" id="stats-broadcast-sec" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="1" max="60" step="1" value="5">
|
||||||
<div class="form-text">Reuse last sample for idle containers when broadcasting (0 = always fresh).</div>
|
<div class="form-text">Min gap between <code>push:allStats</code> to clients (can reuse last samples).</div>
|
||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="row g-3 mt-1">
|
<div class="row g-3 mt-1">
|
||||||
|
<div class="col-md-3">
|
||||||
|
<label class="form-label" for="stats-list-sec">List containers every (sec)</label>
|
||||||
|
<input type="number" id="stats-list-sec" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="2" max="120" step="1" value="15">
|
||||||
|
<div class="form-text">Roster refresh rate (start/stop detection for the sampler).</div>
|
||||||
|
</div>
|
||||||
|
<div class="col-md-3">
|
||||||
|
<label class="form-label" for="stats-cache-ttl-ms">Idle cache TTL (ms)</label>
|
||||||
|
<input type="number" id="stats-cache-ttl-ms" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="0" max="30000" step="100" value="2000">
|
||||||
|
<div class="form-text">Reuse last sample for idle containers when broadcasting.</div>
|
||||||
|
</div>
|
||||||
<div class="col-md-3">
|
<div class="col-md-3">
|
||||||
<label class="form-label" for="stats-warn-containers">Large-fleet warn threshold</label>
|
<label class="form-label" for="stats-warn-containers">Large-fleet warn threshold</label>
|
||||||
<input type="number" id="stats-warn-containers" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="10" max="10000" step="1" value="80">
|
<input type="number" id="stats-warn-containers" class="form-control bg-dark text-white stats-live-field" data-min-role="admin" min="10" max="10000" step="1" value="80">
|
||||||
<div class="form-text">Log a one-time warning when this many containers are running.</div>
|
<div class="form-text">One-time server log when this many containers are running.</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="col-md-9 d-flex align-items-end flex-wrap gap-2">
|
<div class="col-md-3 d-flex align-items-end flex-wrap gap-2">
|
||||||
<button type="button" class="btn btn-outline-secondary btn-sm" id="stats-reset-defaults" data-min-role="admin" title="Reset to server defaults">
|
<button type="button" class="btn btn-outline-secondary btn-sm" id="stats-reset-defaults" data-min-role="admin" title="Reset to server defaults">
|
||||||
<i class="fas fa-undo me-1"></i>Reset to defaults
|
<i class="fas fa-undo me-1"></i>Reset to defaults
|
||||||
</button>
|
</button>
|
||||||
@@ -3098,7 +3108,8 @@ services:
|
|||||||
</div>
|
</div>
|
||||||
<div class="col-md-2">
|
<div class="col-md-2">
|
||||||
<label class="form-label" for="alerts-poll-ms">Health poll (sec)</label>
|
<label class="form-label" for="alerts-poll-ms">Health poll (sec)</label>
|
||||||
<input type="number" id="alerts-poll-ms" class="form-control bg-dark text-white alerts-live-field" data-min-role="admin" min="15" max="3600" value="60">
|
<input type="number" id="alerts-poll-ms" class="form-control bg-dark text-white alerts-live-field" data-min-role="admin" min="60" max="3600" value="120" title="Minimum 60s — lower values thrash Docker">
|
||||||
|
<div class="form-text">Min 60s. Only runs for daemon / health / stack / disk rules (not pure event rules).</div>
|
||||||
</div>
|
</div>
|
||||||
<div class="col-md-2">
|
<div class="col-md-2">
|
||||||
<label class="form-label" for="alerts-rate-limit">Max / minute</label>
|
<label class="form-label" for="alerts-rate-limit">Max / minute</label>
|
||||||
|
|||||||
+296
-123
@@ -3,7 +3,14 @@
|
|||||||
*
|
*
|
||||||
* Fully customizable rules + webhook channels (Discord, Slack, Teams,
|
* Fully customizable rules + webhook channels (Discord, Slack, Teams,
|
||||||
* generic HTTP, ntfy, Gotify, Telegram). Evaluates Docker events and
|
* generic HTTP, ntfy, Gotify, Telegram). Evaluates Docker events and
|
||||||
* periodic health polls even when no client is connected.
|
* light periodic health polls even when no client is connected.
|
||||||
|
*
|
||||||
|
* Host-load rules (keep this cheap on dockerd):
|
||||||
|
* - Never call docker.df() (expensive, unused).
|
||||||
|
* - Poll only kinds that have enabled rules (skip listContainers when unused).
|
||||||
|
* - Min poll interval 60s (default 120s); never overlap polls.
|
||||||
|
* - Docker event path filters noise (exec_*, attach, …) and serializes work.
|
||||||
|
* - Event-only rules need no Docker polling.
|
||||||
*
|
*
|
||||||
* Config (mode 0600), first match wins:
|
* Config (mode 0600), first match wins:
|
||||||
* PEARDOCK_ALERTS_PATH
|
* PEARDOCK_ALERTS_PATH
|
||||||
@@ -45,10 +52,36 @@ const FILE = resolveAlertsFilePath()
|
|||||||
const LEGACY_CWD_FILE = path.join(process.cwd(), 'peardock-alerts.json')
|
const LEGACY_CWD_FILE = path.join(process.cwd(), 'peardock-alerts.json')
|
||||||
|
|
||||||
const HISTORY_MAX = 200
|
const HISTORY_MAX = 200
|
||||||
const DEFAULT_POLL_MS = 60_000
|
/** Default health poll — 2 minutes (was 60s). */
|
||||||
|
const DEFAULT_POLL_MS = 120_000
|
||||||
|
/** Hard floor — sub-minute listContainers/ping loops thrash dockerd. */
|
||||||
|
const MIN_POLL_MS = 60_000
|
||||||
|
const MAX_POLL_MS = 3_600_000
|
||||||
|
/** Drop oldest under event flood rather than unbounded memory. */
|
||||||
|
const EVENT_QUEUE_MAX = 200
|
||||||
const SEVERITIES = ['info', 'warning', 'critical']
|
const SEVERITIES = ['info', 'warning', 'critical']
|
||||||
const SEV_RANK = { info: 1, warning: 2, critical: 3 }
|
const SEV_RANK = { info: 1, warning: 2, critical: 3 }
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Docker event actions that never match default peardock alert rules and are
|
||||||
|
* extremely chatty (terminals, attach, top). Skip before evaluateRules.
|
||||||
|
*/
|
||||||
|
const NOISY_DOCKER_ACTIONS = new Set([
|
||||||
|
'exec_create',
|
||||||
|
'exec_start',
|
||||||
|
'exec_die',
|
||||||
|
'exec_detach',
|
||||||
|
'attach',
|
||||||
|
'detach',
|
||||||
|
'resize',
|
||||||
|
'top',
|
||||||
|
'export',
|
||||||
|
'commit',
|
||||||
|
'copy',
|
||||||
|
'archive-path',
|
||||||
|
'extract-to-dir',
|
||||||
|
])
|
||||||
|
|
||||||
const CHANNEL_TYPES = new Set([
|
const CHANNEL_TYPES = new Set([
|
||||||
'discord',
|
'discord',
|
||||||
'slack',
|
'slack',
|
||||||
@@ -85,6 +118,11 @@ let deliveriesThisMinute = 0
|
|||||||
let deliveriesMinuteStart = Date.now()
|
let deliveriesMinuteStart = Date.now()
|
||||||
let lastDaemonOk = true
|
let lastDaemonOk = true
|
||||||
let started = false
|
let started = false
|
||||||
|
/** Prevent overlapping pollHealth (setInterval + slow listContainers). */
|
||||||
|
let pollInFlight = false
|
||||||
|
/** Serialize docker-event alert evaluation under noisy fleets. */
|
||||||
|
let eventQueue = []
|
||||||
|
let eventDraining = false
|
||||||
|
|
||||||
function newId(prefix) {
|
function newId(prefix) {
|
||||||
return `${prefix}_${Date.now().toString(36)}_${randomBytes(3).toString('hex')}`
|
return `${prefix}_${Date.now().toString(36)}_${randomBytes(3).toString('hex')}`
|
||||||
@@ -316,8 +354,8 @@ function normalizeConfig(raw) {
|
|||||||
version: 1,
|
version: 1,
|
||||||
enabled: r.enabled !== false,
|
enabled: r.enabled !== false,
|
||||||
pollIntervalMs: Math.max(
|
pollIntervalMs: Math.max(
|
||||||
15_000,
|
MIN_POLL_MS,
|
||||||
Math.min(3_600_000, Number(r.pollIntervalMs) || DEFAULT_POLL_MS)
|
Math.min(MAX_POLL_MS, Number(r.pollIntervalMs) || DEFAULT_POLL_MS)
|
||||||
),
|
),
|
||||||
minSeverity: normalizeSeverity(r.minSeverity, 'info'),
|
minSeverity: normalizeSeverity(r.minSeverity, 'info'),
|
||||||
rateLimitPerMinute: Math.max(1, Math.min(300, Number(r.rateLimitPerMinute) || 40)),
|
rateLimitPerMinute: Math.max(1, Math.min(300, Number(r.rateLimitPerMinute) || 40)),
|
||||||
@@ -1031,141 +1069,260 @@ async function evaluateRules(kind, ctx) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Called from Docker event stream.
|
* @param {string} kind
|
||||||
* @param {object} event
|
* @returns {boolean}
|
||||||
*/
|
*/
|
||||||
export async function onDockerEvent(event) {
|
function hasEnabledRuleKind(kind) {
|
||||||
if (!event || !config.enabled) return
|
return config.rules.some(
|
||||||
try {
|
(r) =>
|
||||||
await evaluateRules('docker_event', event)
|
r.enabled &&
|
||||||
// Health status often arrives as container health_status event
|
r.kind === kind &&
|
||||||
const action = String(event.Action || event.status || '')
|
severityAtLeast(r.severity, config.minSeverity)
|
||||||
if (event.Type === 'container' && action.startsWith('health_status')) {
|
)
|
||||||
const health = action.includes(':')
|
}
|
||||||
? action.split(':').slice(1).join(':').trim()
|
|
||||||
: event.status
|
/**
|
||||||
await evaluateRules('container_health', {
|
* Kinds that require a periodic Docker / host poll (not pure events).
|
||||||
id: event.id || event.Actor?.ID,
|
*/
|
||||||
name: event.Actor?.Attributes?.name,
|
function pollNeeds() {
|
||||||
labels: event.Actor?.Attributes,
|
return {
|
||||||
health: health || event.Actor?.Attributes?.health_status,
|
daemon: hasEnabledRuleKind('docker_daemon'),
|
||||||
})
|
list: hasEnabledRuleKind('container_health') || hasEnabledRuleKind('stack_health'),
|
||||||
}
|
resource: hasEnabledRuleKind('resource'),
|
||||||
} catch (err) {
|
|
||||||
logger.warn('alerts: onDockerEvent failed', { error: err.message })
|
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether this Docker event could fire any enabled alert rule.
|
||||||
|
* Drops terminal/exec noise that was previously evaluated on every event.
|
||||||
|
* @param {object} event
|
||||||
|
*/
|
||||||
|
function isAlertRelevantEvent(event) {
|
||||||
|
if (!event) return false
|
||||||
|
const type = String(event.Type || event.type || '')
|
||||||
|
const actionRaw = String(event.Action || event.status || '')
|
||||||
|
const action = actionRaw.split(':')[0]
|
||||||
|
|
||||||
|
if (NOISY_DOCKER_ACTIONS.has(action)) return false
|
||||||
|
|
||||||
|
// Healthcheck transitions → container_health rules
|
||||||
|
if (
|
||||||
|
type === 'container' &&
|
||||||
|
actionRaw.startsWith('health_status') &&
|
||||||
|
hasEnabledRuleKind('container_health')
|
||||||
|
) {
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!hasEnabledRuleKind('docker_event')) return false
|
||||||
|
|
||||||
|
for (const rule of config.rules) {
|
||||||
|
if (!rule.enabled || rule.kind !== 'docker_event') continue
|
||||||
|
if (!severityAtLeast(rule.severity, config.minSeverity)) continue
|
||||||
|
const match = rule.match || {}
|
||||||
|
if (Array.isArray(match.types) && match.types.length && !match.types.includes(type)) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
Array.isArray(match.actions) &&
|
||||||
|
match.actions.length &&
|
||||||
|
!match.actions.includes(action)
|
||||||
|
) {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Process one Docker event for alert rules (awaited).
|
||||||
|
* @param {object} event
|
||||||
|
*/
|
||||||
|
async function processDockerEvent(event) {
|
||||||
|
await evaluateRules('docker_event', event)
|
||||||
|
const action = String(event.Action || event.status || '')
|
||||||
|
if (event.Type === 'container' && action.startsWith('health_status')) {
|
||||||
|
const health = action.includes(':')
|
||||||
|
? action.split(':').slice(1).join(':').trim()
|
||||||
|
: event.status
|
||||||
|
await evaluateRules('container_health', {
|
||||||
|
id: event.id || event.Actor?.ID,
|
||||||
|
name: event.Actor?.Attributes?.name,
|
||||||
|
labels: event.Actor?.Attributes,
|
||||||
|
health: health || event.Actor?.Attributes?.health_status,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function drainEventQueue() {
|
||||||
|
if (eventDraining) return
|
||||||
|
eventDraining = true
|
||||||
|
try {
|
||||||
|
while (eventQueue.length) {
|
||||||
|
const ev = eventQueue.shift()
|
||||||
|
if (!ev || !config.enabled) continue
|
||||||
|
try {
|
||||||
|
await processDockerEvent(ev)
|
||||||
|
} catch (err) {
|
||||||
|
logger.warn('alerts: onDockerEvent failed', { error: err.message })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
eventDraining = false
|
||||||
|
// Work may have been enqueued while draining
|
||||||
|
if (eventQueue.length) {
|
||||||
|
drainEventQueue().catch(() => {})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Called from Docker event stream (fire-and-forget from events.js).
|
||||||
|
* Cheap sync filter + bounded queue; never piles concurrent evaluate/fire work.
|
||||||
|
* @param {object} event
|
||||||
|
*/
|
||||||
|
export function onDockerEvent(event) {
|
||||||
|
if (!event || !config.enabled) return
|
||||||
|
if (!isAlertRelevantEvent(event)) return
|
||||||
|
if (eventQueue.length >= EVENT_QUEUE_MAX) {
|
||||||
|
eventQueue.shift()
|
||||||
|
}
|
||||||
|
eventQueue.push(event)
|
||||||
|
drainEventQueue().catch((err) => {
|
||||||
|
logger.debug('alerts: event drain error', { error: err?.message })
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Periodic health poll. Skips Docker work for disabled rule kinds.
|
||||||
|
* Never calls docker.df (expensive and unused).
|
||||||
|
*/
|
||||||
async function pollHealth() {
|
async function pollHealth() {
|
||||||
if (!config.enabled) return
|
if (!config.enabled) return
|
||||||
|
if (pollInFlight) {
|
||||||
// Docker daemon
|
logger.debug('alerts: poll skipped (previous still running)')
|
||||||
try {
|
return
|
||||||
await docker.ping()
|
}
|
||||||
lastDaemonOk = true
|
const needs = pollNeeds()
|
||||||
await evaluateRules('docker_daemon', { ok: true })
|
if (!needs.daemon && !needs.list && !needs.resource) {
|
||||||
} catch (err) {
|
// Only docker_event / peardock rules — pure event path, no polling work
|
||||||
lastDaemonOk = false
|
return
|
||||||
await evaluateRules('docker_daemon', {
|
|
||||||
ok: false,
|
|
||||||
error: err.message || String(err),
|
|
||||||
})
|
|
||||||
return // skip further polls if docker is down
|
|
||||||
}
|
}
|
||||||
|
|
||||||
// One listContainers for both container_health and stack_health (half the socket work)
|
pollInFlight = true
|
||||||
/** @type {object[]} */
|
|
||||||
let containers = []
|
|
||||||
try {
|
try {
|
||||||
containers = (await docker.listContainers({ all: true })) || []
|
if (needs.daemon) {
|
||||||
} catch (err) {
|
try {
|
||||||
logger.debug('alerts: container list poll failed', { error: err.message })
|
await docker.ping()
|
||||||
}
|
lastDaemonOk = true
|
||||||
|
await evaluateRules('docker_daemon', { ok: true })
|
||||||
// Container health
|
} catch (err) {
|
||||||
try {
|
lastDaemonOk = false
|
||||||
for (const c of containers) {
|
await evaluateRules('docker_daemon', {
|
||||||
const health = c.Status?.match(/\((healthy|unhealthy|health: starting)\)/i)?.[1]
|
ok: false,
|
||||||
|| c.State // running/exited — Health may be in inspect only
|
error: err.message || String(err),
|
||||||
// Prefer Health from inspect-lite if present
|
})
|
||||||
const h =
|
return // skip list/resource if docker is down
|
||||||
c.Health?.Status ||
|
|
||||||
(typeof health === 'string' ? health.toLowerCase().replace('health: ', '') : null)
|
|
||||||
if (!h || h === 'none') continue
|
|
||||||
const name = (c.Names?.[0] || '').replace(/^\//, '') || c.Id?.slice(0, 12)
|
|
||||||
await evaluateRules('container_health', {
|
|
||||||
id: c.Id,
|
|
||||||
name,
|
|
||||||
labels: c.Labels,
|
|
||||||
health: h === 'starting' ? 'starting' : h,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
} catch (err) {
|
|
||||||
logger.debug('alerts: container health poll failed', { error: err.message })
|
|
||||||
}
|
|
||||||
|
|
||||||
// Stack / compose project health (group by com.docker.compose.project)
|
|
||||||
try {
|
|
||||||
/** @type {Map<string, { name: string, running: number, total: number, unhealthy: number }>} */
|
|
||||||
const stacks = new Map()
|
|
||||||
for (const c of containers) {
|
|
||||||
const project =
|
|
||||||
c.Labels?.['com.docker.compose.project'] ||
|
|
||||||
c.Labels?.['com.docker.stack.namespace']
|
|
||||||
if (!project) continue
|
|
||||||
let s = stacks.get(project)
|
|
||||||
if (!s) {
|
|
||||||
s = { name: project, running: 0, total: 0, unhealthy: 0 }
|
|
||||||
stacks.set(project, s)
|
|
||||||
}
|
}
|
||||||
s.total += 1
|
|
||||||
if (c.State === 'running') s.running += 1
|
|
||||||
const st = String(c.Status || '')
|
|
||||||
if (/\(unhealthy\)/i.test(st)) s.unhealthy += 1
|
|
||||||
}
|
}
|
||||||
for (const s of stacks.values()) {
|
|
||||||
const degraded = s.running < s.total || s.unhealthy > 0
|
|
||||||
await evaluateRules('stack_health', {
|
|
||||||
name: s.name,
|
|
||||||
status: degraded ? 'degraded' : 'ok',
|
|
||||||
running: s.running,
|
|
||||||
desired: s.total,
|
|
||||||
message: degraded
|
|
||||||
? `${s.running}/${s.total} running${s.unhealthy ? `, ${s.unhealthy} unhealthy` : ''}`
|
|
||||||
: 'all services running',
|
|
||||||
})
|
|
||||||
}
|
|
||||||
} catch (err) {
|
|
||||||
logger.debug('alerts: stack poll failed', { error: err.message })
|
|
||||||
}
|
|
||||||
|
|
||||||
// Disk via system df
|
/** @type {object[]} */
|
||||||
try {
|
let containers = []
|
||||||
if (typeof docker.df === 'function') {
|
if (needs.list) {
|
||||||
const df = await docker.df()
|
try {
|
||||||
// Docker df doesn't give host disk %; approximate via layers size vs nothing.
|
containers = (await docker.listContainers({ all: true })) || []
|
||||||
// Prefer host free space from / if available via system info
|
} catch (err) {
|
||||||
|
logger.debug('alerts: container list poll failed', { error: err.message })
|
||||||
|
}
|
||||||
}
|
}
|
||||||
// Host disk from Node (server filesystem)
|
|
||||||
try {
|
if (needs.list && hasEnabledRuleKind('container_health')) {
|
||||||
const { statfsSync } = await import('fs')
|
try {
|
||||||
if (typeof statfsSync === 'function') {
|
for (const c of containers) {
|
||||||
const st = statfsSync('/')
|
const health =
|
||||||
const total = Number(st.blocks) * Number(st.bsize)
|
c.Status?.match(/\((healthy|unhealthy|health: starting)\)/i)?.[1] || c.State
|
||||||
const free = Number(st.bfree) * Number(st.bsize)
|
const h =
|
||||||
if (total > 0) {
|
c.Health?.Status ||
|
||||||
const usedPct = ((total - free) / total) * 100
|
(typeof health === 'string'
|
||||||
await evaluateRules('resource', {
|
? health.toLowerCase().replace('health: ', '')
|
||||||
diskPercent: usedPct,
|
: null)
|
||||||
mount: '/',
|
if (!h || h === 'none') continue
|
||||||
|
// Skip pure state strings that aren't healthcheck results
|
||||||
|
if (h === 'running' || h === 'exited' || h === 'created' || h === 'paused') {
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
const name = (c.Names?.[0] || '').replace(/^\//, '') || c.Id?.slice(0, 12)
|
||||||
|
await evaluateRules('container_health', {
|
||||||
|
id: c.Id,
|
||||||
|
name,
|
||||||
|
labels: c.Labels,
|
||||||
|
health: h === 'starting' ? 'starting' : h,
|
||||||
})
|
})
|
||||||
}
|
}
|
||||||
|
} catch (err) {
|
||||||
|
logger.debug('alerts: container health poll failed', { error: err.message })
|
||||||
}
|
}
|
||||||
} catch {
|
|
||||||
// statfs not available on all platforms
|
|
||||||
}
|
}
|
||||||
} catch (err) {
|
|
||||||
logger.debug('alerts: resource poll failed', { error: err.message })
|
if (needs.list && hasEnabledRuleKind('stack_health')) {
|
||||||
|
try {
|
||||||
|
/** @type {Map<string, { name: string, running: number, total: number, unhealthy: number }>} */
|
||||||
|
const stacks = new Map()
|
||||||
|
for (const c of containers) {
|
||||||
|
const project =
|
||||||
|
c.Labels?.['com.docker.compose.project'] ||
|
||||||
|
c.Labels?.['com.docker.stack.namespace']
|
||||||
|
if (!project) continue
|
||||||
|
let s = stacks.get(project)
|
||||||
|
if (!s) {
|
||||||
|
s = { name: project, running: 0, total: 0, unhealthy: 0 }
|
||||||
|
stacks.set(project, s)
|
||||||
|
}
|
||||||
|
s.total += 1
|
||||||
|
if (c.State === 'running') s.running += 1
|
||||||
|
const st = String(c.Status || '')
|
||||||
|
if (/\(unhealthy\)/i.test(st)) s.unhealthy += 1
|
||||||
|
}
|
||||||
|
for (const s of stacks.values()) {
|
||||||
|
const degraded = s.running < s.total || s.unhealthy > 0
|
||||||
|
await evaluateRules('stack_health', {
|
||||||
|
name: s.name,
|
||||||
|
status: degraded ? 'degraded' : 'ok',
|
||||||
|
running: s.running,
|
||||||
|
desired: s.total,
|
||||||
|
message: degraded
|
||||||
|
? `${s.running}/${s.total} running${s.unhealthy ? `, ${s.unhealthy} unhealthy` : ''}`
|
||||||
|
: 'all services running',
|
||||||
|
})
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
logger.debug('alerts: stack poll failed', { error: err.message })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// Host disk only — never docker.df() (expensive Engine inventory scan).
|
||||||
|
if (needs.resource) {
|
||||||
|
try {
|
||||||
|
const { statfsSync } = await import('fs')
|
||||||
|
if (typeof statfsSync === 'function') {
|
||||||
|
const st = statfsSync('/')
|
||||||
|
const total = Number(st.blocks) * Number(st.bsize)
|
||||||
|
const free = Number(st.bfree) * Number(st.bsize)
|
||||||
|
if (total > 0) {
|
||||||
|
const usedPct = ((total - free) / total) * 100
|
||||||
|
await evaluateRules('resource', {
|
||||||
|
diskPercent: usedPct,
|
||||||
|
mount: '/',
|
||||||
|
})
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} catch (err) {
|
||||||
|
logger.debug('alerts: resource poll failed', { error: err?.message })
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
pollInFlight = false
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1175,7 +1332,12 @@ function armPoll() {
|
|||||||
pollTimer = null
|
pollTimer = null
|
||||||
}
|
}
|
||||||
if (!config.enabled) return
|
if (!config.enabled) return
|
||||||
const ms = config.pollIntervalMs || DEFAULT_POLL_MS
|
const needs = pollNeeds()
|
||||||
|
if (!needs.daemon && !needs.list && !needs.resource) {
|
||||||
|
logger.debug('alerts: poll timer off (no poll-backed rules enabled)')
|
||||||
|
return
|
||||||
|
}
|
||||||
|
const ms = Math.max(MIN_POLL_MS, config.pollIntervalMs || DEFAULT_POLL_MS)
|
||||||
pollTimer = setInterval(() => {
|
pollTimer = setInterval(() => {
|
||||||
pollHealth().catch((err) => {
|
pollHealth().catch((err) => {
|
||||||
logger.debug('alerts: poll error', { error: err.message })
|
logger.debug('alerts: poll error', { error: err.message })
|
||||||
@@ -1228,6 +1390,9 @@ export function stopAlerts() {
|
|||||||
clearInterval(pollTimer)
|
clearInterval(pollTimer)
|
||||||
pollTimer = null
|
pollTimer = null
|
||||||
}
|
}
|
||||||
|
eventQueue = []
|
||||||
|
eventDraining = false
|
||||||
|
pollInFlight = false
|
||||||
if (saveTimer) {
|
if (saveTimer) {
|
||||||
clearTimeout(saveTimer)
|
clearTimeout(saveTimer)
|
||||||
saveTimer = null
|
saveTimer = null
|
||||||
@@ -1333,6 +1498,8 @@ export function upsertAlertRule(input) {
|
|||||||
if (idx >= 0) config.rules[idx] = rule
|
if (idx >= 0) config.rules[idx] = rule
|
||||||
else config.rules.push(rule)
|
else config.rules.push(rule)
|
||||||
flushSave()
|
flushSave()
|
||||||
|
// Rule kind/enable changes which Docker polls we need
|
||||||
|
armPoll()
|
||||||
logger.debug('alerts: rule upserted live', {
|
logger.debug('alerts: rule upserted live', {
|
||||||
id: rule.id,
|
id: rule.id,
|
||||||
enabled: rule.enabled,
|
enabled: rule.enabled,
|
||||||
@@ -1345,6 +1512,7 @@ export function deleteAlertRule(id) {
|
|||||||
const before = config.rules.length
|
const before = config.rules.length
|
||||||
config.rules = config.rules.filter((r) => r.id !== id)
|
config.rules = config.rules.filter((r) => r.id !== id)
|
||||||
flushSave()
|
flushSave()
|
||||||
|
armPoll()
|
||||||
return before !== config.rules.length
|
return before !== config.rules.length
|
||||||
}
|
}
|
||||||
|
|
||||||
@@ -1354,6 +1522,7 @@ export function listAlertHistory(limit = 50) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
export function getAlertsStatus() {
|
export function getAlertsStatus() {
|
||||||
|
const needs = pollNeeds()
|
||||||
return {
|
return {
|
||||||
enabled: config.enabled,
|
enabled: config.enabled,
|
||||||
channelCount: config.channels.filter((c) => c.enabled).length,
|
channelCount: config.channels.filter((c) => c.enabled).length,
|
||||||
@@ -1361,6 +1530,10 @@ export function getAlertsStatus() {
|
|||||||
lastDaemonOk,
|
lastDaemonOk,
|
||||||
historyCount: history.length,
|
historyCount: history.length,
|
||||||
pollIntervalMs: config.pollIntervalMs,
|
pollIntervalMs: config.pollIntervalMs,
|
||||||
|
pollActive: Boolean(pollTimer),
|
||||||
|
pollInFlight,
|
||||||
|
pollNeeds: needs,
|
||||||
|
eventQueueDepth: eventQueue.length,
|
||||||
quietHoursActive: inQuietHours(),
|
quietHoursActive: inQuietHours(),
|
||||||
rateLimitPerMinute: config.rateLimitPerMinute,
|
rateLimitPerMinute: config.rateLimitPerMinute,
|
||||||
}
|
}
|
||||||
|
|||||||
+181
-66
@@ -1,15 +1,18 @@
|
|||||||
/**
|
/**
|
||||||
* Container stats collection and broadcast.
|
* Container stats collection and broadcast.
|
||||||
*
|
*
|
||||||
* Uses **one-shot** Docker stats (stream:false) with bounded concurrency —
|
* Strategy (tuned against dockerd load, informed by Dozzle's approach):
|
||||||
* not perpetual `stats({ stream: true })` attachments.
|
|
||||||
*
|
*
|
||||||
* Why: each live stats stream forces dockerd to sample cgroups ~1 Hz for that
|
* - **One-shot** `stats({ stream: false })` — no perpetual stream:true attachments.
|
||||||
* container for as long as the stream is open. With N running containers that
|
* Dozzle uses long-lived streams for all running containers and documents
|
||||||
* is N concurrent streams and is a well-known host CPU / load-average killer.
|
* elevated dockerd CPU as expected; we intentionally avoid that.
|
||||||
* One-shot polls only work when we ask, and we can throttle concurrency.
|
* - **Round-robin batches** — each tick samples at most `samplesPerTick`
|
||||||
*
|
* containers (not the entire fleet). Full-fleet one-shot sweeps are often
|
||||||
* Collection only runs while peers.size > 0 (paused when idle).
|
* *worse* than streams when sweep time > interval (dockerd never idles).
|
||||||
|
* - **Post-sweep spacing** — next tick is scheduled only after the previous
|
||||||
|
* finishes + collectIntervalMs (not a fixed setInterval that overlaps work).
|
||||||
|
* - **Roster cache** — listContainers is not called every tick.
|
||||||
|
* - **Peer-idle pause** — no collection when peers.size === 0.
|
||||||
*/
|
*/
|
||||||
import fs from 'fs'
|
import fs from 'fs'
|
||||||
import path from 'path'
|
import path from 'path'
|
||||||
@@ -28,6 +31,8 @@ export const STATS_LIMITS = Object.freeze({
|
|||||||
collectIntervalMs: { min: 1000, max: 60_000 },
|
collectIntervalMs: { min: 1000, max: 60_000 },
|
||||||
broadcastIntervalMs: { min: 1000, max: 60_000 },
|
broadcastIntervalMs: { min: 1000, max: 60_000 },
|
||||||
concurrency: { min: 1, max: 32 },
|
concurrency: { min: 1, max: 32 },
|
||||||
|
samplesPerTick: { min: 1, max: 100 },
|
||||||
|
listIntervalMs: { min: 2000, max: 120_000 },
|
||||||
cacheTtlMs: { min: 0, max: 30_000 },
|
cacheTtlMs: { min: 0, max: 30_000 },
|
||||||
warnContainers: { min: 10, max: 10_000 },
|
warnContainers: { min: 10, max: 10_000 },
|
||||||
})
|
})
|
||||||
@@ -45,10 +50,13 @@ function clampInt(value, min, max, fallback) {
|
|||||||
|
|
||||||
/**
|
/**
|
||||||
* Built-in defaults, then env overrides (bootstrap before disk / UI).
|
* Built-in defaults, then env overrides (bootstrap before disk / UI).
|
||||||
|
* Conservative defaults keep dockerd idle between batches on busy hosts.
|
||||||
* @returns {{
|
* @returns {{
|
||||||
* collectIntervalMs: number,
|
* collectIntervalMs: number,
|
||||||
* broadcastIntervalMs: number,
|
* broadcastIntervalMs: number,
|
||||||
* concurrency: number,
|
* concurrency: number,
|
||||||
|
* samplesPerTick: number,
|
||||||
|
* listIntervalMs: number,
|
||||||
* cacheTtlMs: number,
|
* cacheTtlMs: number,
|
||||||
* warnContainers: number,
|
* warnContainers: number,
|
||||||
* }}
|
* }}
|
||||||
@@ -57,29 +65,41 @@ export function defaultStatsConfig() {
|
|||||||
const collectIntervalMs = clampInt(
|
const collectIntervalMs = clampInt(
|
||||||
envInt(
|
envInt(
|
||||||
'PEARDOCK_STATS_INTERVAL_MS',
|
'PEARDOCK_STATS_INTERVAL_MS',
|
||||||
CONFIG.STATS?.ACTIVE_INTERVAL_MS ?? CONFIG.STATS?.INTERVAL_MS ?? 2000
|
CONFIG.STATS?.ACTIVE_INTERVAL_MS ?? CONFIG.STATS?.INTERVAL_MS ?? 5000
|
||||||
),
|
),
|
||||||
STATS_LIMITS.collectIntervalMs.min,
|
STATS_LIMITS.collectIntervalMs.min,
|
||||||
STATS_LIMITS.collectIntervalMs.max,
|
STATS_LIMITS.collectIntervalMs.max,
|
||||||
2000
|
5000
|
||||||
)
|
)
|
||||||
const broadcastIntervalMs = clampInt(
|
const broadcastIntervalMs = clampInt(
|
||||||
envInt('PEARDOCK_STATS_BROADCAST_MS', CONFIG.STATS?.INTERVAL_MS ?? 2000),
|
envInt('PEARDOCK_STATS_BROADCAST_MS', CONFIG.STATS?.INTERVAL_MS ?? 5000),
|
||||||
STATS_LIMITS.broadcastIntervalMs.min,
|
STATS_LIMITS.broadcastIntervalMs.min,
|
||||||
STATS_LIMITS.broadcastIntervalMs.max,
|
STATS_LIMITS.broadcastIntervalMs.max,
|
||||||
2000
|
5000
|
||||||
)
|
)
|
||||||
const concurrency = clampInt(
|
const concurrency = clampInt(
|
||||||
envInt('PEARDOCK_STATS_CONCURRENCY', CONFIG.STATS?.CONCURRENCY ?? 6),
|
envInt('PEARDOCK_STATS_CONCURRENCY', CONFIG.STATS?.CONCURRENCY ?? 2),
|
||||||
STATS_LIMITS.concurrency.min,
|
STATS_LIMITS.concurrency.min,
|
||||||
STATS_LIMITS.concurrency.max,
|
STATS_LIMITS.concurrency.max,
|
||||||
6
|
2
|
||||||
|
)
|
||||||
|
const samplesPerTick = clampInt(
|
||||||
|
envInt('PEARDOCK_STATS_SAMPLES_PER_TICK', CONFIG.STATS?.SAMPLES_PER_TICK ?? 4),
|
||||||
|
STATS_LIMITS.samplesPerTick.min,
|
||||||
|
STATS_LIMITS.samplesPerTick.max,
|
||||||
|
4
|
||||||
|
)
|
||||||
|
const listIntervalMs = clampInt(
|
||||||
|
envInt('PEARDOCK_STATS_LIST_INTERVAL_MS', CONFIG.STATS?.LIST_INTERVAL_MS ?? 15_000),
|
||||||
|
STATS_LIMITS.listIntervalMs.min,
|
||||||
|
STATS_LIMITS.listIntervalMs.max,
|
||||||
|
15_000
|
||||||
)
|
)
|
||||||
const cacheTtlMs = clampInt(
|
const cacheTtlMs = clampInt(
|
||||||
envInt('PEARDOCK_STATS_CACHE_TTL_MS', CONFIG.STATS?.CACHE_TTL_MS ?? 1000),
|
envInt('PEARDOCK_STATS_CACHE_TTL_MS', CONFIG.STATS?.CACHE_TTL_MS ?? 2000),
|
||||||
STATS_LIMITS.cacheTtlMs.min,
|
STATS_LIMITS.cacheTtlMs.min,
|
||||||
STATS_LIMITS.cacheTtlMs.max,
|
STATS_LIMITS.cacheTtlMs.max,
|
||||||
1000
|
2000
|
||||||
)
|
)
|
||||||
const warnContainers = clampInt(
|
const warnContainers = clampInt(
|
||||||
envInt('PEARDOCK_STATS_WARN_CONTAINERS', 80),
|
envInt('PEARDOCK_STATS_WARN_CONTAINERS', 80),
|
||||||
@@ -91,6 +111,8 @@ export function defaultStatsConfig() {
|
|||||||
collectIntervalMs,
|
collectIntervalMs,
|
||||||
broadcastIntervalMs,
|
broadcastIntervalMs,
|
||||||
concurrency,
|
concurrency,
|
||||||
|
samplesPerTick,
|
||||||
|
listIntervalMs,
|
||||||
cacheTtlMs,
|
cacheTtlMs,
|
||||||
warnContainers,
|
warnContainers,
|
||||||
}
|
}
|
||||||
@@ -120,6 +142,8 @@ function saveStatsDisk() {
|
|||||||
collectIntervalMs: runtime.collectIntervalMs,
|
collectIntervalMs: runtime.collectIntervalMs,
|
||||||
broadcastIntervalMs: runtime.broadcastIntervalMs,
|
broadcastIntervalMs: runtime.broadcastIntervalMs,
|
||||||
concurrency: runtime.concurrency,
|
concurrency: runtime.concurrency,
|
||||||
|
samplesPerTick: runtime.samplesPerTick,
|
||||||
|
listIntervalMs: runtime.listIntervalMs,
|
||||||
cacheTtlMs: runtime.cacheTtlMs,
|
cacheTtlMs: runtime.cacheTtlMs,
|
||||||
warnContainers: runtime.warnContainers,
|
warnContainers: runtime.warnContainers,
|
||||||
}
|
}
|
||||||
@@ -138,7 +162,7 @@ function saveStatsDisk() {
|
|||||||
*/
|
*/
|
||||||
export function normalizeStatsConfig(partial = {}, base = defaultStatsConfig()) {
|
export function normalizeStatsConfig(partial = {}, base = defaultStatsConfig()) {
|
||||||
const src = partial && typeof partial === 'object' ? partial : {}
|
const src = partial && typeof partial === 'object' ? partial : {}
|
||||||
// Accept seconds from UI as collectIntervalSec / broadcastIntervalSec
|
// Accept seconds from UI as collectIntervalSec / broadcastIntervalSec / listIntervalSec
|
||||||
let collectMs = src.collectIntervalMs
|
let collectMs = src.collectIntervalMs
|
||||||
if (collectMs == null && src.collectIntervalSec != null) {
|
if (collectMs == null && src.collectIntervalSec != null) {
|
||||||
collectMs = Number(src.collectIntervalSec) * 1000
|
collectMs = Number(src.collectIntervalSec) * 1000
|
||||||
@@ -147,6 +171,10 @@ export function normalizeStatsConfig(partial = {}, base = defaultStatsConfig())
|
|||||||
if (broadcastMs == null && src.broadcastIntervalSec != null) {
|
if (broadcastMs == null && src.broadcastIntervalSec != null) {
|
||||||
broadcastMs = Number(src.broadcastIntervalSec) * 1000
|
broadcastMs = Number(src.broadcastIntervalSec) * 1000
|
||||||
}
|
}
|
||||||
|
let listMs = src.listIntervalMs
|
||||||
|
if (listMs == null && src.listIntervalSec != null) {
|
||||||
|
listMs = Number(src.listIntervalSec) * 1000
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
collectIntervalMs: clampInt(
|
collectIntervalMs: clampInt(
|
||||||
collectMs ?? base.collectIntervalMs,
|
collectMs ?? base.collectIntervalMs,
|
||||||
@@ -166,6 +194,18 @@ export function normalizeStatsConfig(partial = {}, base = defaultStatsConfig())
|
|||||||
STATS_LIMITS.concurrency.max,
|
STATS_LIMITS.concurrency.max,
|
||||||
base.concurrency
|
base.concurrency
|
||||||
),
|
),
|
||||||
|
samplesPerTick: clampInt(
|
||||||
|
src.samplesPerTick ?? base.samplesPerTick,
|
||||||
|
STATS_LIMITS.samplesPerTick.min,
|
||||||
|
STATS_LIMITS.samplesPerTick.max,
|
||||||
|
base.samplesPerTick
|
||||||
|
),
|
||||||
|
listIntervalMs: clampInt(
|
||||||
|
listMs ?? base.listIntervalMs,
|
||||||
|
STATS_LIMITS.listIntervalMs.min,
|
||||||
|
STATS_LIMITS.listIntervalMs.max,
|
||||||
|
base.listIntervalMs
|
||||||
|
),
|
||||||
cacheTtlMs: clampInt(
|
cacheTtlMs: clampInt(
|
||||||
src.cacheTtlMs ?? base.cacheTtlMs,
|
src.cacheTtlMs ?? base.cacheTtlMs,
|
||||||
STATS_LIMITS.cacheTtlMs.min,
|
STATS_LIMITS.cacheTtlMs.min,
|
||||||
@@ -210,13 +250,17 @@ export function getStatsConfig() {
|
|||||||
collectIntervalMs: runtime.collectIntervalMs,
|
collectIntervalMs: runtime.collectIntervalMs,
|
||||||
broadcastIntervalMs: runtime.broadcastIntervalMs,
|
broadcastIntervalMs: runtime.broadcastIntervalMs,
|
||||||
concurrency: runtime.concurrency,
|
concurrency: runtime.concurrency,
|
||||||
|
samplesPerTick: runtime.samplesPerTick,
|
||||||
|
listIntervalMs: runtime.listIntervalMs,
|
||||||
cacheTtlMs: runtime.cacheTtlMs,
|
cacheTtlMs: runtime.cacheTtlMs,
|
||||||
warnContainers: runtime.warnContainers,
|
warnContainers: runtime.warnContainers,
|
||||||
// Convenience for UI
|
// Convenience for UI
|
||||||
collectIntervalSec: Math.round(runtime.collectIntervalMs / 1000),
|
collectIntervalSec: Math.round(runtime.collectIntervalMs / 1000),
|
||||||
broadcastIntervalSec: Math.round(runtime.broadcastIntervalMs / 1000),
|
broadcastIntervalSec: Math.round(runtime.broadcastIntervalMs / 1000),
|
||||||
active: Boolean(intervalHandle),
|
listIntervalSec: Math.round(runtime.listIntervalMs / 1000),
|
||||||
|
active: collectionLoopActive,
|
||||||
peers: peers.size,
|
peers: peers.size,
|
||||||
|
runningTracked: rosterRunning.length,
|
||||||
limits: { ...STATS_LIMITS },
|
limits: { ...STATS_LIMITS },
|
||||||
path: STATS_FILE,
|
path: STATS_FILE,
|
||||||
}
|
}
|
||||||
@@ -232,35 +276,21 @@ export function updateStatsConfig(partial = {}, opts = {}) {
|
|||||||
ensureStatsConfigLoaded()
|
ensureStatsConfigLoaded()
|
||||||
const base = opts.replace ? defaultStatsConfig() : { ...runtime }
|
const base = opts.replace ? defaultStatsConfig() : { ...runtime }
|
||||||
const next = normalizeStatsConfig(partial, base)
|
const next = normalizeStatsConfig(partial, base)
|
||||||
const intervalChanged = next.collectIntervalMs !== runtime.collectIntervalMs
|
|
||||||
runtime = next
|
runtime = next
|
||||||
if (opts.persist !== false) saveStatsDisk()
|
if (opts.persist !== false) saveStatsDisk()
|
||||||
if (intervalChanged && intervalHandle) {
|
// Interval is applied on the next schedule after a tick completes
|
||||||
rearmCollectInterval()
|
|
||||||
}
|
|
||||||
logger.info('Stats config updated', {
|
logger.info('Stats config updated', {
|
||||||
collectIntervalMs: runtime.collectIntervalMs,
|
collectIntervalMs: runtime.collectIntervalMs,
|
||||||
broadcastIntervalMs: runtime.broadcastIntervalMs,
|
broadcastIntervalMs: runtime.broadcastIntervalMs,
|
||||||
concurrency: runtime.concurrency,
|
concurrency: runtime.concurrency,
|
||||||
|
samplesPerTick: runtime.samplesPerTick,
|
||||||
|
listIntervalMs: runtime.listIntervalMs,
|
||||||
cacheTtlMs: runtime.cacheTtlMs,
|
cacheTtlMs: runtime.cacheTtlMs,
|
||||||
persist: opts.persist !== false,
|
persist: opts.persist !== false,
|
||||||
})
|
})
|
||||||
return getStatsConfig()
|
return getStatsConfig()
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Rearm setInterval after collect interval change (while peers connected). */
|
|
||||||
function rearmCollectInterval() {
|
|
||||||
if (!intervalHandle) return
|
|
||||||
clearInterval(intervalHandle)
|
|
||||||
intervalHandle = setInterval(() => {
|
|
||||||
statsTick().catch(() => {})
|
|
||||||
}, runtime.collectIntervalMs)
|
|
||||||
if (typeof intervalHandle.unref === 'function') intervalHandle.unref()
|
|
||||||
logger.debug('Stats collect interval rearmed', {
|
|
||||||
intervalMs: runtime.collectIntervalMs,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
|
|
||||||
export function getStatsFilePath() {
|
export function getStatsFilePath() {
|
||||||
return STATS_FILE
|
return STATS_FILE
|
||||||
}
|
}
|
||||||
@@ -270,7 +300,10 @@ const containerStats = {}
|
|||||||
const statsCache = new Map()
|
const statsCache = new Map()
|
||||||
const containerActivity = new Map()
|
const containerActivity = new Map()
|
||||||
|
|
||||||
let intervalHandle = null
|
/** @type {ReturnType<typeof setTimeout>|null} */
|
||||||
|
let loopTimer = null
|
||||||
|
/** True while peer-connected collection loop is armed */
|
||||||
|
let collectionLoopActive = false
|
||||||
/** Peer-registry unsubscribe; set while stats service is armed */
|
/** Peer-registry unsubscribe; set while stats service is armed */
|
||||||
let unsubPeers = null
|
let unsubPeers = null
|
||||||
/** True after startStatsBroadcast(); false after stopStatsBroadcast() */
|
/** True after startStatsBroadcast(); false after stopStatsBroadcast() */
|
||||||
@@ -282,6 +315,14 @@ let dockerBackoffUntil = 0
|
|||||||
let tickInFlight = false
|
let tickInFlight = false
|
||||||
let largeFleetWarned = false
|
let largeFleetWarned = false
|
||||||
|
|
||||||
|
/** Cached running containers from last listContainers */
|
||||||
|
let rosterRunning = []
|
||||||
|
/** @type {Set<string>} */
|
||||||
|
let rosterAllIds = new Set()
|
||||||
|
let rosterFetchedAt = 0
|
||||||
|
/** Round-robin cursor into rosterRunning */
|
||||||
|
let rrIndex = 0
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Docker Engine CPU % (same idea as `docker stats`).
|
* Docker Engine CPU % (same idea as `docker stats`).
|
||||||
* One-shot stats populate precpu_stats after ~1s wait inside the engine.
|
* One-shot stats populate precpu_stats after ~1s wait inside the engine.
|
||||||
@@ -580,30 +621,41 @@ export function destroyStatsForContainer(id) {
|
|||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* List once, sample running containers with bounded one-shot stats.
|
* Refresh container roster from Docker (throttled).
|
||||||
* No perpetual streams — dockerd only does work while we request samples.
|
* @param {boolean} [force]
|
||||||
*/
|
*/
|
||||||
async function collectContainerStats() {
|
async function refreshRoster(force = false) {
|
||||||
// Single list (all) — derive running from State; was 2× listContainers/tick
|
const now = Date.now()
|
||||||
|
if (!force && rosterFetchedAt && now - rosterFetchedAt < runtime.listIntervalMs) {
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
const all = await docker.listContainers({ all: true })
|
const all = await docker.listContainers({ all: true })
|
||||||
const running = all.filter((c) => String(c.State || '').toLowerCase() === 'running')
|
const running = all
|
||||||
|
.filter((c) => String(c.State || '').toLowerCase() === 'running')
|
||||||
|
.sort((a, b) => String(a.Id).localeCompare(String(b.Id)))
|
||||||
const allIds = new Set(all.map((c) => c.Id))
|
const allIds = new Set(all.map((c) => c.Id))
|
||||||
const runningIds = new Set(running.map((c) => c.Id))
|
const runningIds = new Set(running.map((c) => c.Id))
|
||||||
|
|
||||||
|
rosterRunning = running
|
||||||
|
rosterAllIds = allIds
|
||||||
|
rosterFetchedAt = now
|
||||||
|
if (rrIndex >= running.length) rrIndex = 0
|
||||||
|
|
||||||
if (running.length >= runtime.warnContainers && !largeFleetWarned) {
|
if (running.length >= runtime.warnContainers && !largeFleetWarned) {
|
||||||
largeFleetWarned = true
|
largeFleetWarned = true
|
||||||
logger.warn(
|
logger.warn(
|
||||||
'Large running fleet for stats collection — dockerd load scales with container count',
|
'Large running fleet for stats collection — dockerd load scales with samples/tick',
|
||||||
{
|
{
|
||||||
running: running.length,
|
running: running.length,
|
||||||
|
samplesPerTick: runtime.samplesPerTick,
|
||||||
concurrency: runtime.concurrency,
|
concurrency: runtime.concurrency,
|
||||||
intervalMs: runtime.collectIntervalMs,
|
intervalMs: runtime.collectIntervalMs,
|
||||||
hint: 'Settings → Performance, or PEARDOCK_STATS_INTERVAL_MS / PEARDOCK_STATS_CONCURRENCY',
|
hint: 'Settings → Performance: raise interval, lower samples/tick & concurrency',
|
||||||
}
|
}
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
|
||||||
// Ensure entries + drop gone / zero-out stopped
|
|
||||||
for (const containerInfo of running) {
|
for (const containerInfo of running) {
|
||||||
ensureStatsEntry(containerInfo)
|
ensureStatsEntry(containerInfo)
|
||||||
}
|
}
|
||||||
@@ -625,9 +677,35 @@ async function collectContainerStats() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
|
|
||||||
// Bounded parallel one-shot samples (only running)
|
/**
|
||||||
await mapPool(running, runtime.concurrency, async (containerInfo) => {
|
* Pick next round-robin batch of running containers (at most samplesPerTick).
|
||||||
|
* @param {object[]} running
|
||||||
|
* @returns {object[]}
|
||||||
|
*/
|
||||||
|
function nextSampleBatch(running) {
|
||||||
|
if (!running.length) return []
|
||||||
|
const n = Math.min(runtime.samplesPerTick, running.length)
|
||||||
|
const batch = []
|
||||||
|
for (let i = 0; i < n; i++) {
|
||||||
|
batch.push(running[(rrIndex + i) % running.length])
|
||||||
|
}
|
||||||
|
rrIndex = (rrIndex + n) % running.length
|
||||||
|
return batch
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Throttled roster + round-robin one-shot samples.
|
||||||
|
* Dockerd cost ≈ samplesPerTick × ~1s / concurrency per tick, then idle for interval.
|
||||||
|
*/
|
||||||
|
async function collectContainerStats() {
|
||||||
|
await refreshRoster(false)
|
||||||
|
|
||||||
|
const batch = nextSampleBatch(rosterRunning)
|
||||||
|
if (!batch.length) return
|
||||||
|
|
||||||
|
await mapPool(batch, runtime.concurrency, async (containerInfo) => {
|
||||||
if (peers.size === 0) return
|
if (peers.size === 0) return
|
||||||
const statsData = containerStats[containerInfo.Id]
|
const statsData = containerStats[containerInfo.Id]
|
||||||
if (!statsData) return
|
if (!statsData) return
|
||||||
@@ -654,17 +732,26 @@ function destroyAllStatsEntries() {
|
|||||||
statsCache.clear()
|
statsCache.clear()
|
||||||
containerActivity.clear()
|
containerActivity.clear()
|
||||||
largeFleetWarned = false
|
largeFleetWarned = false
|
||||||
|
rosterRunning = []
|
||||||
|
rosterAllIds = new Set()
|
||||||
|
rosterFetchedAt = 0
|
||||||
|
rrIndex = 0
|
||||||
|
}
|
||||||
|
|
||||||
|
function clearLoopTimer() {
|
||||||
|
if (loopTimer) {
|
||||||
|
clearTimeout(loopTimer)
|
||||||
|
loopTimer = null
|
||||||
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Pause collection: stop interval and drop in-memory state.
|
* Pause collection: stop loop and drop in-memory state.
|
||||||
* Safe to call when already paused.
|
* Safe to call when already paused.
|
||||||
*/
|
*/
|
||||||
export function pauseStatsCollection() {
|
export function pauseStatsCollection() {
|
||||||
if (intervalHandle) {
|
collectionLoopActive = false
|
||||||
clearInterval(intervalHandle)
|
clearLoopTimer()
|
||||||
intervalHandle = null
|
|
||||||
}
|
|
||||||
destroyAllStatsEntries()
|
destroyAllStatsEntries()
|
||||||
tickInFlight = false
|
tickInFlight = false
|
||||||
logger.debug('Stats collection paused (no peers)')
|
logger.debug('Stats collection paused (no peers)')
|
||||||
@@ -769,6 +856,33 @@ async function statsTick() {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Run one stats tick, then schedule the next after collectIntervalMs.
|
||||||
|
* Spacing is measured from tick *completion* so long sweeps cannot overlap.
|
||||||
|
*/
|
||||||
|
function runLoopTick() {
|
||||||
|
if (!collectionLoopActive || !serviceArmed || peers.size === 0) {
|
||||||
|
collectionLoopActive = false
|
||||||
|
clearLoopTimer()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
statsTick()
|
||||||
|
.catch(() => {})
|
||||||
|
.finally(() => {
|
||||||
|
if (!collectionLoopActive || !serviceArmed || peers.size === 0) {
|
||||||
|
collectionLoopActive = false
|
||||||
|
clearLoopTimer()
|
||||||
|
return
|
||||||
|
}
|
||||||
|
clearLoopTimer()
|
||||||
|
loopTimer = setTimeout(() => {
|
||||||
|
loopTimer = null
|
||||||
|
runLoopTick()
|
||||||
|
}, runtime.collectIntervalMs)
|
||||||
|
if (typeof loopTimer.unref === 'function') loopTimer.unref()
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
/**
|
/**
|
||||||
* Resume collection when at least one peer is connected.
|
* Resume collection when at least one peer is connected.
|
||||||
* Idempotent; kicks an immediate tick so first client is not waiting a full interval.
|
* Idempotent; kicks an immediate tick so first client is not waiting a full interval.
|
||||||
@@ -776,20 +890,19 @@ async function statsTick() {
|
|||||||
export function resumeStatsCollection() {
|
export function resumeStatsCollection() {
|
||||||
if (!serviceArmed) return
|
if (!serviceArmed) return
|
||||||
if (peers.size === 0) return
|
if (peers.size === 0) return
|
||||||
if (!intervalHandle) {
|
if (collectionLoopActive) return
|
||||||
intervalHandle = setInterval(() => {
|
collectionLoopActive = true
|
||||||
statsTick().catch(() => {})
|
|
||||||
}, runtime.collectIntervalMs)
|
|
||||||
if (typeof intervalHandle.unref === 'function') intervalHandle.unref()
|
|
||||||
logger.debug('Stats collection resumed', {
|
|
||||||
peers: peers.size,
|
|
||||||
intervalMs: runtime.collectIntervalMs,
|
|
||||||
concurrency: runtime.concurrency,
|
|
||||||
})
|
|
||||||
}
|
|
||||||
// Immediate kick so first connected client gets samples ASAP
|
|
||||||
lastBroadcast = 0
|
lastBroadcast = 0
|
||||||
statsTick().catch(() => {})
|
logger.debug('Stats collection resumed', {
|
||||||
|
peers: peers.size,
|
||||||
|
intervalMs: runtime.collectIntervalMs,
|
||||||
|
concurrency: runtime.concurrency,
|
||||||
|
samplesPerTick: runtime.samplesPerTick,
|
||||||
|
listIntervalMs: runtime.listIntervalMs,
|
||||||
|
})
|
||||||
|
// Force fresh roster on first tick after resume
|
||||||
|
rosterFetchedAt = 0
|
||||||
|
runLoopTick()
|
||||||
}
|
}
|
||||||
|
|
||||||
function onPeerCountChange(size, prevSize) {
|
function onPeerCountChange(size, prevSize) {
|
||||||
@@ -842,9 +955,9 @@ export function stopStatsBroadcast() {
|
|||||||
pauseStatsCollection()
|
pauseStatsCollection()
|
||||||
}
|
}
|
||||||
|
|
||||||
/** @returns {boolean} whether the collect interval is currently running */
|
/** @returns {boolean} whether the collect loop is currently armed */
|
||||||
export function isStatsCollectionActive() {
|
export function isStatsCollectionActive() {
|
||||||
return Boolean(intervalHandle)
|
return collectionLoopActive
|
||||||
}
|
}
|
||||||
|
|
||||||
/** Test/ops helpers — same shape as getStatsConfig subset */
|
/** Test/ops helpers — same shape as getStatsConfig subset */
|
||||||
@@ -854,6 +967,8 @@ export function getStatsRuntimeConfig() {
|
|||||||
collectIntervalMs: runtime.collectIntervalMs,
|
collectIntervalMs: runtime.collectIntervalMs,
|
||||||
broadcastIntervalMs: runtime.broadcastIntervalMs,
|
broadcastIntervalMs: runtime.broadcastIntervalMs,
|
||||||
concurrency: runtime.concurrency,
|
concurrency: runtime.concurrency,
|
||||||
|
samplesPerTick: runtime.samplesPerTick,
|
||||||
|
listIntervalMs: runtime.listIntervalMs,
|
||||||
cacheTtlMs: runtime.cacheTtlMs,
|
cacheTtlMs: runtime.cacheTtlMs,
|
||||||
warnContainers: runtime.warnContainers,
|
warnContainers: runtime.warnContainers,
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -90,6 +90,49 @@ test('alerts update global settings persists', async (t) => {
|
|||||||
mod2.stopAlerts()
|
mod2.stopAlerts()
|
||||||
})
|
})
|
||||||
|
|
||||||
|
test('alerts poll interval clamps to min 60s', async (t) => {
|
||||||
|
withTempAlerts(t)
|
||||||
|
const mod = await import('../server/services/alerts.js?' + Date.now() + 'pollclamp')
|
||||||
|
mod.startAlerts()
|
||||||
|
mod.updateAlertsConfig({ pollIntervalMs: 5_000 })
|
||||||
|
t.is(mod.getAlertsConfig().pollIntervalMs, 60_000, 'floor is 60s')
|
||||||
|
mod.updateAlertsConfig({ pollIntervalMs: 180_000 })
|
||||||
|
t.is(mod.getAlertsConfig().pollIntervalMs, 180_000)
|
||||||
|
mod.stopAlerts()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('alerts ignore noisy docker events (exec/attach)', async (t) => {
|
||||||
|
withTempAlerts(t)
|
||||||
|
const mod = await import('../server/services/alerts.js?' + Date.now() + 'noisy')
|
||||||
|
mod.startAlerts()
|
||||||
|
// Fire noisy events — should not throw and should not enqueue work
|
||||||
|
mod.onDockerEvent({
|
||||||
|
Type: 'container',
|
||||||
|
Action: 'exec_create',
|
||||||
|
id: 'abc',
|
||||||
|
Actor: { Attributes: { name: 'x' } },
|
||||||
|
})
|
||||||
|
mod.onDockerEvent({
|
||||||
|
Type: 'container',
|
||||||
|
Action: 'attach',
|
||||||
|
id: 'abc',
|
||||||
|
Actor: { Attributes: { name: 'x' } },
|
||||||
|
})
|
||||||
|
const st = mod.getAlertsStatus()
|
||||||
|
t.is(st.eventQueueDepth, 0, 'noisy events dropped before queue')
|
||||||
|
// Relevant event should queue / process
|
||||||
|
mod.onDockerEvent({
|
||||||
|
Type: 'container',
|
||||||
|
Action: 'die',
|
||||||
|
id: 'dead1',
|
||||||
|
Actor: { Attributes: { name: 'boom' } },
|
||||||
|
})
|
||||||
|
// Allow microtask drain
|
||||||
|
await new Promise((r) => setTimeout(r, 30))
|
||||||
|
t.ok(mod.listAlertHistory(5).length >= 1, 'die event fires alert history')
|
||||||
|
mod.stopAlerts()
|
||||||
|
})
|
||||||
|
|
||||||
test('protocol exposes alert methods and push', async (t) => {
|
test('protocol exposes alert methods and push', async (t) => {
|
||||||
const { Methods, MethodRoles, Pushes, Roles } = await import('../shared/protocol.js')
|
const { Methods, MethodRoles, Pushes, Roles } = await import('../shared/protocol.js')
|
||||||
t.is(Methods.getAlertsConfig, 'getAlertsConfig')
|
t.is(Methods.getAlertsConfig, 'getAlertsConfig')
|
||||||
|
|||||||
@@ -17,12 +17,16 @@ test('normalizeStatsConfig clamps out-of-range values', (t) => {
|
|||||||
collectIntervalMs: 50,
|
collectIntervalMs: 50,
|
||||||
broadcastIntervalMs: 999_999,
|
broadcastIntervalMs: 999_999,
|
||||||
concurrency: 100,
|
concurrency: 100,
|
||||||
|
samplesPerTick: 0,
|
||||||
|
listIntervalMs: 100,
|
||||||
cacheTtlMs: -5,
|
cacheTtlMs: -5,
|
||||||
warnContainers: 1,
|
warnContainers: 1,
|
||||||
})
|
})
|
||||||
t.is(n.collectIntervalMs, STATS_LIMITS.collectIntervalMs.min)
|
t.is(n.collectIntervalMs, STATS_LIMITS.collectIntervalMs.min)
|
||||||
t.is(n.broadcastIntervalMs, STATS_LIMITS.broadcastIntervalMs.max)
|
t.is(n.broadcastIntervalMs, STATS_LIMITS.broadcastIntervalMs.max)
|
||||||
t.is(n.concurrency, STATS_LIMITS.concurrency.max)
|
t.is(n.concurrency, STATS_LIMITS.concurrency.max)
|
||||||
|
t.is(n.samplesPerTick, STATS_LIMITS.samplesPerTick.min)
|
||||||
|
t.is(n.listIntervalMs, STATS_LIMITS.listIntervalMs.min)
|
||||||
t.is(n.cacheTtlMs, STATS_LIMITS.cacheTtlMs.min)
|
t.is(n.cacheTtlMs, STATS_LIMITS.cacheTtlMs.min)
|
||||||
t.is(n.warnContainers, STATS_LIMITS.warnContainers.min)
|
t.is(n.warnContainers, STATS_LIMITS.warnContainers.min)
|
||||||
})
|
})
|
||||||
@@ -31,11 +35,15 @@ test('normalizeStatsConfig accepts seconds aliases from UI', (t) => {
|
|||||||
const n = normalizeStatsConfig({
|
const n = normalizeStatsConfig({
|
||||||
collectIntervalSec: 5,
|
collectIntervalSec: 5,
|
||||||
broadcastIntervalSec: 3,
|
broadcastIntervalSec: 3,
|
||||||
|
listIntervalSec: 20,
|
||||||
concurrency: 4,
|
concurrency: 4,
|
||||||
|
samplesPerTick: 8,
|
||||||
})
|
})
|
||||||
t.is(n.collectIntervalMs, 5000)
|
t.is(n.collectIntervalMs, 5000)
|
||||||
t.is(n.broadcastIntervalMs, 3000)
|
t.is(n.broadcastIntervalMs, 3000)
|
||||||
|
t.is(n.listIntervalMs, 20000)
|
||||||
t.is(n.concurrency, 4)
|
t.is(n.concurrency, 4)
|
||||||
|
t.is(n.samplesPerTick, 8)
|
||||||
})
|
})
|
||||||
|
|
||||||
test('defaultStatsConfig is within limits', (t) => {
|
test('defaultStatsConfig is within limits', (t) => {
|
||||||
|
|||||||
@@ -100,7 +100,7 @@ function renderAlertsGlobal(config, status) {
|
|||||||
const min = document.getElementById('alerts-min-severity')
|
const min = document.getElementById('alerts-min-severity')
|
||||||
if (min) min.value = config.minSeverity || 'info'
|
if (min) min.value = config.minSeverity || 'info'
|
||||||
const poll = document.getElementById('alerts-poll-ms')
|
const poll = document.getElementById('alerts-poll-ms')
|
||||||
if (poll) poll.value = String(Math.round((config.pollIntervalMs || 60000) / 1000))
|
if (poll) poll.value = String(Math.round((config.pollIntervalMs || 120000) / 1000))
|
||||||
const rate = document.getElementById('alerts-rate-limit')
|
const rate = document.getElementById('alerts-rate-limit')
|
||||||
if (rate) rate.value = String(config.rateLimitPerMinute || 40)
|
if (rate) rate.value = String(config.rateLimitPerMinute || 40)
|
||||||
const qh = document.getElementById('alerts-quiet-enabled')
|
const qh = document.getElementById('alerts-quiet-enabled')
|
||||||
@@ -120,10 +120,14 @@ function renderAlertsGlobal(config, status) {
|
|||||||
|
|
||||||
const statusEl = document.getElementById('alerts-engine-status')
|
const statusEl = document.getElementById('alerts-engine-status')
|
||||||
if (statusEl && status) {
|
if (statusEl && status) {
|
||||||
|
const pollLabel = status.pollActive
|
||||||
|
? `poll ${Math.round((status.pollIntervalMs || 0) / 1000)}s`
|
||||||
|
: 'poll off'
|
||||||
const parts = [
|
const parts = [
|
||||||
status.enabled ? 'Enabled' : 'Disabled',
|
status.enabled ? 'Enabled' : 'Disabled',
|
||||||
`${status.channelCount} channel(s)`,
|
`${status.channelCount} channel(s)`,
|
||||||
`${status.ruleCount} rule(s)`,
|
`${status.ruleCount} rule(s)`,
|
||||||
|
pollLabel,
|
||||||
status.lastDaemonOk ? 'Docker OK' : 'Docker DOWN',
|
status.lastDaemonOk ? 'Docker OK' : 'Docker DOWN',
|
||||||
status.quietHoursActive ? 'Quiet hours' : null,
|
status.quietHoursActive ? 'Quiet hours' : null,
|
||||||
].filter(Boolean)
|
].filter(Boolean)
|
||||||
@@ -373,7 +377,7 @@ function readGlobalConfigFromUi() {
|
|||||||
enabled: document.getElementById('alerts-enabled')?.value !== '0',
|
enabled: document.getElementById('alerts-enabled')?.value !== '0',
|
||||||
minSeverity: document.getElementById('alerts-min-severity')?.value || 'info',
|
minSeverity: document.getElementById('alerts-min-severity')?.value || 'info',
|
||||||
pollIntervalMs:
|
pollIntervalMs:
|
||||||
Math.max(15, Number(document.getElementById('alerts-poll-ms')?.value) || 60) * 1000,
|
Math.max(60, Number(document.getElementById('alerts-poll-ms')?.value) || 120) * 1000,
|
||||||
rateLimitPerMinute: Number(document.getElementById('alerts-rate-limit')?.value) || 40,
|
rateLimitPerMinute: Number(document.getElementById('alerts-rate-limit')?.value) || 40,
|
||||||
includeHostname: document.getElementById('alerts-include-hostname')?.value !== '0',
|
includeHostname: document.getElementById('alerts-include-hostname')?.value !== '0',
|
||||||
quietHours: {
|
quietHours: {
|
||||||
|
|||||||
@@ -2518,7 +2518,9 @@ function isServerStatsLiveField(el) {
|
|||||||
return (
|
return (
|
||||||
id === 'stats-collect-sec' ||
|
id === 'stats-collect-sec' ||
|
||||||
id === 'stats-broadcast-sec' ||
|
id === 'stats-broadcast-sec' ||
|
||||||
|
id === 'stats-samples-per-tick' ||
|
||||||
id === 'stats-concurrency' ||
|
id === 'stats-concurrency' ||
|
||||||
|
id === 'stats-list-sec' ||
|
||||||
id === 'stats-cache-ttl-ms' ||
|
id === 'stats-cache-ttl-ms' ||
|
||||||
id === 'stats-warn-containers'
|
id === 'stats-warn-containers'
|
||||||
)
|
)
|
||||||
|
|||||||
+22
-8
@@ -75,34 +75,44 @@ function renderStatsForm(config) {
|
|||||||
const collect = document.getElementById('stats-collect-sec')
|
const collect = document.getElementById('stats-collect-sec')
|
||||||
if (collect) {
|
if (collect) {
|
||||||
collect.value = String(
|
collect.value = String(
|
||||||
config.collectIntervalSec ?? Math.round((config.collectIntervalMs || 2000) / 1000)
|
config.collectIntervalSec ?? Math.round((config.collectIntervalMs || 5000) / 1000)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
const broadcast = document.getElementById('stats-broadcast-sec')
|
const broadcast = document.getElementById('stats-broadcast-sec')
|
||||||
if (broadcast) {
|
if (broadcast) {
|
||||||
broadcast.value = String(
|
broadcast.value = String(
|
||||||
config.broadcastIntervalSec ?? Math.round((config.broadcastIntervalMs || 2000) / 1000)
|
config.broadcastIntervalSec ?? Math.round((config.broadcastIntervalMs || 5000) / 1000)
|
||||||
)
|
)
|
||||||
}
|
}
|
||||||
|
const samples = document.getElementById('stats-samples-per-tick')
|
||||||
|
if (samples) samples.value = String(config.samplesPerTick ?? 4)
|
||||||
const conc = document.getElementById('stats-concurrency')
|
const conc = document.getElementById('stats-concurrency')
|
||||||
if (conc) conc.value = String(config.concurrency ?? 6)
|
if (conc) conc.value = String(config.concurrency ?? 2)
|
||||||
|
const listSec = document.getElementById('stats-list-sec')
|
||||||
|
if (listSec) {
|
||||||
|
listSec.value = String(
|
||||||
|
config.listIntervalSec ?? Math.round((config.listIntervalMs || 15000) / 1000)
|
||||||
|
)
|
||||||
|
}
|
||||||
const cache = document.getElementById('stats-cache-ttl-ms')
|
const cache = document.getElementById('stats-cache-ttl-ms')
|
||||||
if (cache) cache.value = String(config.cacheTtlMs ?? 1000)
|
if (cache) cache.value = String(config.cacheTtlMs ?? 2000)
|
||||||
const warn = document.getElementById('stats-warn-containers')
|
const warn = document.getElementById('stats-warn-containers')
|
||||||
if (warn) warn.value = String(config.warnContainers ?? 80)
|
if (warn) warn.value = String(config.warnContainers ?? 80)
|
||||||
|
|
||||||
const active = config.active ? 'collecting' : 'idle (no peers or paused)'
|
const active = config.active ? 'collecting' : 'idle (no peers or paused)'
|
||||||
setEngineStatus(
|
setEngineStatus(
|
||||||
`${active} · every ${config.collectIntervalSec ?? '?'}s · ×${config.concurrency ?? '?'}`
|
`${active} · gap ${config.collectIntervalSec ?? '?'}s · ${config.samplesPerTick ?? '?'} samples · ×${config.concurrency ?? '?'}`
|
||||||
)
|
)
|
||||||
const meta = document.getElementById('stats-runtime-meta')
|
const meta = document.getElementById('stats-runtime-meta')
|
||||||
if (meta) {
|
if (meta) {
|
||||||
const parts = [
|
const parts = [
|
||||||
`peers=${config.peers ?? 0}`,
|
`peers=${config.peers ?? 0}`,
|
||||||
`collect=${config.collectIntervalMs}ms`,
|
`running=${config.runningTracked ?? '?'}`,
|
||||||
`broadcast=${config.broadcastIntervalMs}ms`,
|
`gap=${config.collectIntervalMs}ms`,
|
||||||
|
`samples/tick=${config.samplesPerTick}`,
|
||||||
`concurrency=${config.concurrency}`,
|
`concurrency=${config.concurrency}`,
|
||||||
`cacheTtl=${config.cacheTtlMs}ms`,
|
`list=${config.listIntervalMs}ms`,
|
||||||
|
`broadcast=${config.broadcastIntervalMs}ms`,
|
||||||
]
|
]
|
||||||
if (config.path) parts.push(`file=${config.path}`)
|
if (config.path) parts.push(`file=${config.path}`)
|
||||||
meta.textContent = parts.join(' · ')
|
meta.textContent = parts.join(' · ')
|
||||||
@@ -117,13 +127,17 @@ function renderStatsForm(config) {
|
|||||||
function readFormPartial() {
|
function readFormPartial() {
|
||||||
const collectSec = Number(document.getElementById('stats-collect-sec')?.value)
|
const collectSec = Number(document.getElementById('stats-collect-sec')?.value)
|
||||||
const broadcastSec = Number(document.getElementById('stats-broadcast-sec')?.value)
|
const broadcastSec = Number(document.getElementById('stats-broadcast-sec')?.value)
|
||||||
|
const samplesPerTick = Number(document.getElementById('stats-samples-per-tick')?.value)
|
||||||
const concurrency = Number(document.getElementById('stats-concurrency')?.value)
|
const concurrency = Number(document.getElementById('stats-concurrency')?.value)
|
||||||
|
const listSec = Number(document.getElementById('stats-list-sec')?.value)
|
||||||
const cacheTtlMs = Number(document.getElementById('stats-cache-ttl-ms')?.value)
|
const cacheTtlMs = Number(document.getElementById('stats-cache-ttl-ms')?.value)
|
||||||
const warnContainers = Number(document.getElementById('stats-warn-containers')?.value)
|
const warnContainers = Number(document.getElementById('stats-warn-containers')?.value)
|
||||||
return {
|
return {
|
||||||
collectIntervalSec: Number.isFinite(collectSec) ? collectSec : undefined,
|
collectIntervalSec: Number.isFinite(collectSec) ? collectSec : undefined,
|
||||||
broadcastIntervalSec: Number.isFinite(broadcastSec) ? broadcastSec : undefined,
|
broadcastIntervalSec: Number.isFinite(broadcastSec) ? broadcastSec : undefined,
|
||||||
|
samplesPerTick: Number.isFinite(samplesPerTick) ? samplesPerTick : undefined,
|
||||||
concurrency: Number.isFinite(concurrency) ? concurrency : undefined,
|
concurrency: Number.isFinite(concurrency) ? concurrency : undefined,
|
||||||
|
listIntervalSec: Number.isFinite(listSec) ? listSec : undefined,
|
||||||
cacheTtlMs: Number.isFinite(cacheTtlMs) ? cacheTtlMs : undefined,
|
cacheTtlMs: Number.isFinite(cacheTtlMs) ? cacheTtlMs : undefined,
|
||||||
warnContainers: Number.isFinite(warnContainers) ? warnContainers : undefined,
|
warnContainers: Number.isFinite(warnContainers) ? warnContainers : undefined,
|
||||||
}
|
}
|
||||||
|
|||||||
Reference in New Issue
Block a user