Add storage stack modules and track them in git

The modules workspace previously ignored every path matching `storage-*`, which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out of version control. Narrow .gitignore to test-artifact patterns only so production category trees are committed. Storage packages (23 total): - storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync, priority-fetch, bitfield-scheduler, audit-chain - storage-hyperbee (5): batch-write, diff-follow, range-watch, secondary-index, tombstone-gc - storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge, version-snapshot, watch-notify - storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus, light-writer Implementation highlights: - Shared attach/gossip helpers in _shared/storage-gossip-base.js - P2P modules use Protomux gossip via p2p-bare; local planners omit swarm - Recent deepen pass: priority queues, batch caps, audit export/import, watch/notify aliases on range-watch, fork/view lease helpers, etc. - Per-package README, docs/api.md, docs/architecture.md, tests, examples Also update modules/README.md doc hub links for the full storage stack (hypercore, hyperbee, hyperdrive, autobase). Co-authored-by: Cursor <[email protected]>
2026-05-21 00:17:51 -04:00
parent b10507b060
commit a11e22badc
214 changed files with 57132 additions and 5 deletions
@@ -0,0 +1,5 @@
+# Changelog
+
+## [0.0.0-scaffold]
+
+- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
+# hyper-p2p-bee-range-watch
+
+Lexicographic key-range watchers on Hyperbee with `emitChange` notifications; mirror reactive slices with `hyper-p2p-reactive-state` upstream. Local-only scheduling layer (no Hyperswarm join).
+
+**Category:** Storage (Hyperbee)
+
+**Composes with:** `hyper-p2p-reactive-state`
+
+**Protocol:** `bee-range-watch/v1`
+
+## When to use
+
+UI or pipelines must react to puts/deletes inside a key span.
+
+## When not to use
+
+Whole-database watchers or P2P change fanout (local callbacks only).
+
+## Quick start
+
+```js
+const { HyperP2PBeeRangeWatch } = require('hyper-p2p-bee-range-watch')
+const mod = new HyperP2PBeeRangeWatch()
+mod.attach(/* Hyperbee instance */)
+await mod.ready()
+// ... application logic ...
+await mod.close()
+```
+
+## Docs
+
+- [docs/api.md](docs/api.md) — constructor, methods, events, errors
+- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
+- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
+- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
+
+## Test
+
+```bash
+npm install && npm test
+```
@@ -0,0 +1,116 @@
+# API: hyper-p2p-bee-range-watch
+
+**Protocol:** `bee-range-watch/v1`
+
+**Export:** `HyperP2PBeeRangeWatch`
+
+## Overview
+
+Lexicographic key-range watchers on Hyperbee with `emitChange` notifications; mirror reactive slices with `hyper-p2p-reactive-state` upstream.
+
+## Constructor
+
+```js
+const mod = new HyperP2PBeeRangeWatch(opts)
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
+
+## Methods
+
+### `attach(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `watchRange(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:**
+  - `Error: gte and lte required`
+  - `Error: callback required`
+  - `Error: gte must be <= lte`
+
+### `emitChange(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:**
+  - `Error: key required`
+
+### `unwatch(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `activeWatches(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `getStats(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `ready(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `close(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+## Events
+
+| Event | Payload |
+|-------|---------|
+| `change` | `{ key, value, op, delivered }` |
+| `closed` | no payload |
+
+## getStats()
+
+Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
+Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
+
+## Errors
+
+Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
+
+Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
+
+### Documented `throw new Error(...)` strings
+
+- `gte and lte required`
+- `callback required`
+- `gte must be <= lte`
+- `key required`
+
+## P2P
+
+Library-only: no swarm join. `ready()` resolves immediately.
+`getStats().protocol` still reports the module protocol id for logging.
+
+## Testing
+
+```bash
+npm install && npm test
+```
+
+## Common flows
+
+1. `watchRange(gte, lte, cb)` — register callback.
+2. `emitChange(key, value, op)` — deliver to matching watches.
+3. `unwatch(id)` — remove listener.
+
@@ -0,0 +1,27 @@
+# Architecture: hyper-p2p-bee-range-watch
+
+**Protocol:** `bee-range-watch/v1` · **P2P:** no
+
+## Role
+
+Deliver in-process notifications when keys change inside a **lexicographic span** `[gte, lte]`. Call `notify` / `emitChange` from your bee put/del hooks.
+
+## State
+
+| Field | Description |
+|-------|-------------|
+| `_watches` | `id → { gte, lte, cb }` |
+| `_nextId` | Monotonic watch id |
+
+## API aliases
+
+- `watch(gte, lte, cb)` → `watchRange`
+- `notify(key, value, op)` → `emitChange`
+
+## Events
+
+`change` (with `delivered` count), `closed`
+
+## Composition
+
+`hyper-p2p-reactive-state`, `hyper-p2p-bee-batch-write` (commit then notify watchers).
@@ -0,0 +1,12 @@
+require('bare-process/global')
+const { HyperP2PBeeRangeWatch } = require('../index.js')
+
+async function main () {
+  const m = new HyperP2PBeeRangeWatch()
+  m.attach({ async get () {} })
+  m.watchRange('a', 'z', (c) => console.log('watch', c.key))
+  m.emitChange('m', 'v')
+  await m.close()
+  console.log('done')
+}
+main().catch(console.error)
@@ -0,0 +1,85 @@
+require('bare-process/global')
+const EventEmitter = require('bare-events')
+const { assertBee } = require('../../_shared/storage-gossip-base.js')
+const PROTOCOL = 'bee-range-watch/v1'
+
+class HyperP2PBeeRangeWatch extends EventEmitter {
+  constructor (opts = {}) {
+    super()
+    this.bee = opts.bee || null
+    this._watches = new Map()
+    this._nextId = 1
+    this._stats = { watches: 0, notifications: 0, unwatch: 0 }
+  }
+
+  attach (bee) {
+    assertBee(bee)
+    this.bee = bee
+    return this
+  }
+
+  watch (gte, lte, cb) {
+    return this.watchRange(gte, lte, cb)
+  }
+
+  watchRange (gte, lte, cb) {
+    if (gte == null || lte == null) throw new Error('gte and lte required')
+    if (typeof cb !== 'function') throw new Error('callback required')
+    if (gte > lte) throw new Error('gte must be <= lte')
+    const id = this._nextId++
+    this._watches.set(id, { gte, lte, cb })
+    this._stats.watches++
+    return id
+  }
+
+  notify (key, value, op = 'put') {
+    return this.emitChange(key, value, op)
+  }
+
+  emitChange (key, value, op = 'put') {
+    if (key == null) throw new Error('key required')
+    let n = 0
+    for (const w of this._watches.values()) {
+      if (key >= w.gte && key <= w.lte) {
+        w.cb({ key, value, op, at: Date.now() })
+        n++
+      }
+    }
+    this._stats.notifications += n
+    this.emit('change', { key, value, op, delivered: n })
+    return n
+  }
+
+  unwatch (id) {
+    const ok = this._watches.delete(id)
+    if (ok) this._stats.unwatch++
+    return ok
+  }
+
+  activeWatches () {
+    return this._watches.size
+  }
+
+  listWatches () {
+    return [...this._watches.entries()].map(([id, w]) => ({
+      id, gte: w.gte, lte: w.lte
+    }))
+  }
+
+  getStats () {
+    return {
+      ...this._stats,
+      active: this._watches.size,
+      protocol: PROTOCOL
+    }
+  }
+
+  async ready () { return this }
+
+  async close () {
+    this._watches.clear()
+    this.emit('closed')
+  }
+}
+
+module.exports = { HyperP2PBeeRangeWatch, PROTOCOL }
@@ -0,0 +1,27 @@
+{
+  "name": "hyper-p2p-bee-range-watch",
+  "version": "0.3.1",
+  "description": "Range watch notifications on Hyperbee.",
+  "main": "index.js",
+  "type": "commonjs",
+  "license": "Apache-2.0",
+  "scripts": { "test": "brittle-bare test/test.js" },
+  "dependencies": {
+    "bare-events": "^2.8.0",
+    "bare-process": "^4.4.0",
+    "b4a": "^1.6.7",
+    "hypercore-crypto": "^3.0.0",
+    "protomux": "^3.0.0",
+    "compact-encoding": "^2.0.0"
+  },
+  "peerDependencies": {
+    "hyperswarm": "^4.0.0",
+    "bare": ">=1.0.0",
+    "hyperbee": "^2.0.0"
+  },
+  "devDependencies": { "brittle": "^3.0.0" },
+  "imports": {
+    "process": { "bare": "bare-process", "default": "process" },
+    "events": { "bare": "bare-events", "default": "events" }
+  }
+}
@@ -0,0 +1,48 @@
+require('bare-process/global')
+const test = require('brittle')
+const { HyperP2PBeeRangeWatch, PROTOCOL } = require('../index.js')
+
+test('exports', (t) => {
+  t.ok(HyperP2PBeeRangeWatch)
+  t.is(PROTOCOL, 'bee-range-watch/v1')
+})
+
+test('watchRange emitChange', async (t) => {
+  const m = new HyperP2PBeeRangeWatch()
+  let hit = 0
+  m.watchRange('a', 'm', () => { hit++ })
+  m.emitChange('b', 42)
+  t.is(hit, 1)
+  await m.close()
+})
+
+test('unwatch', async (t) => {
+  const m = new HyperP2PBeeRangeWatch()
+  const id = m.watchRange('a', 'z', () => {})
+  t.ok(m.unwatch(id))
+  await m.close()
+})
+
+test('validation', async (t) => {
+  const m = new HyperP2PBeeRangeWatch()
+  try { m.watchRange('z', 'a', () => {}) } catch (e) { t.ok(e) }
+  try { m.watchRange('a', 'z', null) } catch (e) { t.ok(e) }
+  await m.close()
+})
+
+test('getStats', async (t) => {
+  const m = new HyperP2PBeeRangeWatch()
+  m.watchRange('a', 'c', () => {})
+  t.is(m.getStats().active, 1)
+  await m.close()
+})
+
+test('watch notify aliases listWatches', async (t) => {
+  const m = new HyperP2PBeeRangeWatch()
+  let hit = 0
+  m.watch('a', 'z', () => { hit++ })
+  m.notify('b', 1)
+  t.is(hit, 1)
+  t.is(m.listWatches().length, 1)
+  await m.close()
+})