Add storage stack modules and track them in git

The modules workspace previously ignored every path matching `storage-*`, which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out of version control. Narrow .gitignore to test-artifact patterns only so production category trees are committed. Storage packages (23 total): - storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync, priority-fetch, bitfield-scheduler, audit-chain - storage-hyperbee (5): batch-write, diff-follow, range-watch, secondary-index, tombstone-gc - storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge, version-snapshot, watch-notify - storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus, light-writer Implementation highlights: - Shared attach/gossip helpers in _shared/storage-gossip-base.js - P2P modules use Protomux gossip via p2p-bare; local planners omit swarm - Recent deepen pass: priority queues, batch caps, audit export/import, watch/notify aliases on range-watch, fork/view lease helpers, etc. - Per-package README, docs/api.md, docs/architecture.md, tests, examples Also update modules/README.md doc hub links for the full storage stack (hypercore, hyperbee, hyperdrive, autobase). Co-authored-by: Cursor <[email protected]>
2026-05-21 00:17:51 -04:00
parent b10507b060
commit a11e22badc
214 changed files with 57132 additions and 5 deletions
@@ -0,0 +1,5 @@
+# Changelog
+
+## [0.0.0-scaffold]
+
+- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
+# hyper-p2p-drive-watch-notify
+
+Path watchers with local callbacks plus gossip `drive-change` notifications; bridge to `hyper-p2p-distributed-event-bus` for app-wide fanout. Hyperswarm gossip when `topic` is set.
+
+**Category:** Storage (Hyperdrive)
+
+**Composes with:** `hyper-p2p-distributed-event-bus`
+
+**Protocol:** `drive-watch-notify/v1`
+
+## When to use
+
+Drive tree changes must reach remote peers subscribed on the same topic.
+
+## When not to use
+
+Local-only file notifications without swarm.
+
+## Quick start
+
+```js
+const { HyperP2PDriveWatchNotify } = require('hyper-p2p-drive-watch-notify')
+const topic = process.argv[2] // 64-char hex or string
+const mod = new HyperP2PDriveWatchNotify({ topic })
+await mod.ready()   // joins swarm when topic set
+// ... application logic ...
+await mod.close()
+```
+
+## Docs
+
+- [docs/api.md](docs/api.md) — constructor, methods, events, errors
+- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
+- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
+- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
+
+## Test
+
+```bash
+npm install && npm test
+```
@@ -0,0 +1,110 @@
+# API: hyper-p2p-drive-watch-notify
+
+**Protocol:** `drive-watch-notify/v1`
+
+**Export:** `HyperP2PDriveWatchNotify`
+
+## Overview
+
+Path watchers with local callbacks plus gossip `drive-change` notifications; bridge to `hyper-p2p-distributed-event-bus` for app-wide fanout.
+
+## Constructor
+
+```js
+const mod = new HyperP2PDriveWatchNotify(opts)
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
+| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
+| `drive` | object \| null | null | Attached drive instance (`attach()` also supported) |
+
+## Methods
+
+### `attach(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `watchPath(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:**
+  - `Error: callback required`
+
+### `notifyChange(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `unwatch(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `getStats(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `ready(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `close(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+## Events
+
+| Event | Payload |
+|-------|---------|
+| `change` | `{ key, value, op, delivered }` |
+| `closed` | no payload |
+
+## getStats()
+
+Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
+Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
+
+## Errors
+
+Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
+
+Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
+
+### Documented `throw new Error(...)` strings
+
+- `callback required`
+
+## P2P
+
+When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `drive-watch-notify/v1`.
+Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
+
+### Gossip message types
+
+- `drive-change`
+
+## Testing
+
+```bash
+npm install && npm test
+```
+
+## Common flows
+
+1. `watchPath(path, cb)` — prefix-aware callbacks.
+2. `notifyChange(path, op)` — local delivery + gossip `drive-change`.
+3. Remote gossip replays into matching watches with `remote: true`.
+
@@ -0,0 +1,44 @@
+# Architecture: hyper-p2p-drive-watch-notify
+
+**Category:** Storage (Hyperdrive)
+
+```mermaid
+flowchart LR
+  App[Application] --> Mod[HyperP2PDriveWatchNotify]
+  Mod --> Gossip[Hyperswarm gossip]
+  Gossip --> Peer[Remote peers]
+```
+
+## Sequence (P2P)
+
+```mermaid
+sequenceDiagram
+  participant App
+  participant Mod as Module
+  participant SW as Hyperswarm
+  participant Peer
+  App->>Mod: ready(topic)
+  Mod->>SW: join(topic)
+  SW->>Peer: connection
+  Mod->>Peer: gossip payload
+  Peer-->>Mod: onmessage
+  Mod-->>App: emit(event)
+```
+
+## Wire messages
+
+| type | fields | direction | behavior |
+|------|--------|-----------|----------|
+| `drive-change` | path, op, at | gossip | fanout path change |
+
+## State model
+
+- In-memory `Map` / `Set` / array structures for hot path
+- Optional attached HyperDrive via `attach()`
+- `close()` clears ephemeral state and destroys swarm when P2P
+
+## Composition
+
+Composes with: `hyper-p2p-distributed-event-bus`.
+
+Storage gossip helpers: [`../../_shared/storage-gossip-base.js`](../../_shared/storage-gossip-base.js).
@@ -0,0 +1,14 @@
+require('bare-process/global')
+const { HyperP2PDriveWatchNotify } = require('../index.js')
+
+async function main () {
+  const topic = process.argv[2] || null
+  const m = new HyperP2PDriveWatchNotify({ topic })
+  m.attach({ version: 0 })
+  await m.ready()
+  m.watchPath('/docs', () => {})
+  m.notifyChange('/docs/a.txt')
+  await m.close()
+  console.log('done')
+}
+main().catch(console.error)
@@ -0,0 +1,89 @@
+require('bare-process/global')
+const EventEmitter = require('bare-events')
+const { assertNonEmpty } = require('../../_shared/lib/errors.js')
+const { assertDrive, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
+
+const PROTOCOL = 'drive-watch-notify/v1'
+
+class HyperP2PDriveWatchNotify extends EventEmitter {
+  constructor (opts = {}) {
+    super()
+    this.topic = opts.topic || null
+    this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
+    this.drive = opts.drive || null
+    this._watches = new Map()
+    this._nextId = 1
+    this._stats = { watches: 0, notified: 0, gossipIn: 0, gossipOut: 0 }
+    this.swarm = null
+  }
+
+  attach (drive) {
+    assertDrive(drive)
+    this.drive = drive
+    return this
+  }
+
+  watchPath (path, cb) {
+    assertNonEmpty(path, 'path')
+    if (typeof cb !== 'function') throw new Error('callback required')
+    const id = this._nextId++
+    this._watches.set(id, { path, cb })
+    this._stats.watches++
+    return id
+  }
+
+  notifyChange (path, op = 'update') {
+    assertNonEmpty(path, 'path')
+    let delivered = 0
+    for (const w of this._watches.values()) {
+      if (path === w.path || path.startsWith(w.path + '/')) {
+        w.cb({ path, op, at: Date.now() })
+        delivered++
+      }
+    }
+    this._stats.notified += delivered
+    const msg = { type: 'drive-change', path, op, at: Date.now() }
+    sendGossip(this, msg)
+    this._stats.gossipOut++
+    this.emit('change', msg)
+    return delivered
+  }
+
+  unwatch (id) {
+    return this._watches.delete(id)
+  }
+
+  _onGossip (d) {
+    if (!d || d.type !== 'drive-change') return
+    this._stats.gossipIn++
+    for (const w of this._watches.values()) {
+      if (d.path === w.path || d.path.startsWith(w.path + '/')) {
+        w.cb({ path: d.path, op: d.op, remote: true, at: d.at })
+      }
+    }
+  }
+
+  getStats () {
+    return { ...this._stats, active: this._watches.size, protocol: PROTOCOL }
+  }
+
+  async ready () {
+    if (this.swarm || !this.topic) return this
+    await attachGossip(this, {
+      keyPair: this.keyPair,
+      topic: this.topic,
+      protocol: PROTOCOL,
+      onmessage: (d) => this._onGossip(d)
+    })
+    return this
+  }
+
+  async close () {
+    if (this.swarm) await this.swarm.destroy().catch(() => {})
+    this.swarm = null
+    this._watches.clear()
+    this.emit('closed')
+  }
+}
+
+module.exports = { HyperP2PDriveWatchNotify, PROTOCOL }
@@ -0,0 +1,27 @@
+{
+  "name": "hyper-p2p-drive-watch-notify",
+  "version": "0.3.1",
+  "description": "File watch notifications.",
+  "main": "index.js",
+  "type": "commonjs",
+  "license": "Apache-2.0",
+  "scripts": { "test": "brittle-bare test/test.js" },
+  "dependencies": {
+    "bare-events": "^2.8.0",
+    "bare-process": "^4.4.0",
+    "b4a": "^1.6.7",
+    "hypercore-crypto": "^3.0.0",
+    "protomux": "^3.0.0",
+    "compact-encoding": "^2.16.0"
+  },
+  "peerDependencies": {
+    "hyperswarm": "^4.0.0",
+    "bare": ">=1.0.0",
+    "hyperdrive": "^11.0.0"
+  },
+  "devDependencies": { "brittle": "^3.0.0" },
+  "imports": {
+    "process": { "bare": "bare-process", "default": "process" },
+    "events": { "bare": "bare-events", "default": "events" }
+  }
+}
@@ -0,0 +1,37 @@
+require('bare-process/global')
+const test = require('brittle')
+const { HyperP2PDriveWatchNotify, PROTOCOL } = require('../index.js')
+
+test('exports', (t) => {
+  t.ok(HyperP2PDriveWatchNotify)
+  t.is(PROTOCOL, 'drive-watch-notify/v1')
+})
+
+test('watchPath notifyChange', async (t) => {
+  const m = new HyperP2PDriveWatchNotify()
+  let got = null
+  m.watchPath('/data', (e) => { got = e })
+  const n = m.notifyChange('/data/file', 'update')
+  t.is(n, 1)
+  t.is(got.op, 'update')
+  await m.close()
+})
+
+test('unwatch', async (t) => {
+  const m = new HyperP2PDriveWatchNotify()
+  const id = m.watchPath('/x', () => {})
+  t.ok(m.unwatch(id))
+  await m.close()
+})
+
+test('validation', async (t) => {
+  const m = new HyperP2PDriveWatchNotify()
+  try { m.watchPath('/a', null) } catch (e) { t.ok(e) }
+  await m.close()
+})
+
+test('getStats', async (t) => {
+  const m = new HyperP2PDriveWatchNotify()
+  t.is(m.getStats().protocol, 'drive-watch-notify/v1')
+  await m.close()
+})