Add storage stack modules and track them in git

The modules workspace previously ignored every path matching `storage-*`, which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out of version control. Narrow .gitignore to test-artifact patterns only so production category trees are committed. Storage packages (23 total): - storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync, priority-fetch, bitfield-scheduler, audit-chain - storage-hyperbee (5): batch-write, diff-follow, range-watch, secondary-index, tombstone-gc - storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge, version-snapshot, watch-notify - storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus, light-writer Implementation highlights: - Shared attach/gossip helpers in _shared/storage-gossip-base.js - P2P modules use Protomux gossip via p2p-bare; local planners omit swarm - Recent deepen pass: priority queues, batch caps, audit export/import, watch/notify aliases on range-watch, fork/view lease helpers, etc. - Per-package README, docs/api.md, docs/architecture.md, tests, examples Also update modules/README.md doc hub links for the full storage stack (hypercore, hyperbee, hyperdrive, autobase). Co-authored-by: Cursor <[email protected]>
2026-05-21 00:17:51 -04:00
parent b10507b060
commit a11e22badc
214 changed files with 57132 additions and 5 deletions
@@ -0,0 +1,5 @@
+# Changelog
+
+## [0.0.0-scaffold]
+
+- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
+# hyper-p2p-core-bitfield-scheduler
+
+Schedules Hypercore bitfield download ranges by priority so replication can respect bandwidth budgets when composed with `hyper-p2p-bandwidth-broker`. Local-only scheduling layer (no Hyperswarm join).
+
+**Category:** Storage (Hypercore)
+
+**Composes with:** `hyper-p2p-bandwidth-broker`
+
+**Protocol:** `core-bitfield-scheduler/v1`
+
+## When to use
+
+You need ordered, deduplicated range requests over a Hypercore bitfield.
+
+## When not to use
+
+You replicate whole cores without range scheduling, or you need cross-peer gossip (this module is local-only).
+
+## Quick start
+
+```js
+const { HyperP2PCoreBitfieldScheduler } = require('hyper-p2p-core-bitfield-scheduler')
+const mod = new HyperP2PCoreBitfieldScheduler()
+mod.attach(/* Hypercore instance */)
+await mod.ready()
+// ... application logic ...
+await mod.close()
+```
+
+## Docs
+
+- [docs/api.md](docs/api.md) — constructor, methods, events, errors
+- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
+- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
+- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
+
+## Test
+
+```bash
+npm install && npm test
+```
@@ -0,0 +1,113 @@
+# API: hyper-p2p-core-bitfield-scheduler
+
+**Protocol:** `core-bitfield-scheduler/v1`
+
+**Export:** `HyperP2PCoreBitfieldScheduler`
+
+## Overview
+
+Schedules Hypercore bitfield download ranges by priority so replication can respect bandwidth budgets when composed with `hyper-p2p-bandwidth-broker`.
+
+## Constructor
+
+```js
+const mod = new HyperP2PCoreBitfieldScheduler(opts)
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `core` | object \| null | null | Attached core instance (`attach()` also supported) |
+
+## Methods
+
+### `attach(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `scheduleRange(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:**
+  - `Error: start must be non-negative`
+  - `Error: len must be positive`
+
+### `nextRange(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `markRangeDone(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `pendingCount(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `getStats(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `ready(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `close(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+## Events
+
+| Event | Payload |
+|-------|---------|
+| `scheduled` | item `{ start, len, end, priority, at, key }` |
+| `range` | scheduled range item |
+| `closed` | no payload |
+
+## getStats()
+
+Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
+Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
+
+## Errors
+
+Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
+
+Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
+
+### Documented `throw new Error(...)` strings
+
+- `start must be non-negative`
+- `len must be positive`
+
+## P2P
+
+Library-only: no swarm join. `ready()` resolves immediately.
+`getStats().protocol` still reports the module protocol id for logging.
+
+## Testing
+
+```bash
+npm install && npm test
+```
+
+## Common flows
+
+1. `attach(core)` then `scheduleRange(start, len, priority)` to enqueue ranges.
+2. `nextRange()` drains the priority queue and emits `range`.
+3. `markRangeDone(start, len)` prevents re-serving the same range.
+
@@ -0,0 +1,59 @@
+# Architecture: hyper-p2p-core-bitfield-scheduler
+
+**Category:** Storage (Hypercore) · **Protocol:** `core-bitfield-scheduler/v1` · **P2P:** no (local planner)
+
+## Role
+
+Schedules **contiguous byte ranges** on a Hypercore for bitfield-driven download or upload work. Higher `priority` ranges dequeue first. Tracks completed ranges in `_served` so duplicate schedules are skipped.
+
+Use with `hyper-p2p-core-priority-fetch` when you split work by **block index** vs **byte span**.
+
+## State model
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `_queue` | `RangeItem[]` | Pending `{ start, len, end, priority, key, at }` |
+| `_served` | `Set<string>` | Keys `start:len` already handed to replication |
+| `core` | Hypercore? | Optional; `coreLength` in stats |
+
+## Operations
+
+1. `scheduleRange(start, len, priority)` — push + sort queue; emit `scheduled`
+2. `peekNext()` — highest priority pending without dequeue
+3. `nextRange()` — shift first unserved item; emit `range`
+4. `markRangeDone(start, len)` — mark served, purge queue entries
+5. `listPending()` — snapshot of unserved queue
+
+## Wire messages
+
+Local only — no Protomux channel. Pair with gossip modules on the same process when peers need policy agreement before scheduling.
+
+## Events
+
+| Event | Payload |
+|-------|---------|
+| `scheduled` | `RangeItem` |
+| `range` | `RangeItem` dequeued |
+| `closed` | — |
+
+## Composition
+
+- **Upstream:** Hypercore replication / custom fetcher consuming `nextRange()`
+- **Peers:** `hyper-p2p-core-replicator`, `hyper-p2p-bandwidth-broker` (rate limits per range)
+- **Downstream:** Holepunch `hypercore` bitfield updates (application applies ranges)
+
+## Failure modes
+
+- `start < 0` or `len <= 0` throws
+- Re-scheduling a served key returns `null` and increments `skipped` stat
+
+## Diagram
+
+```mermaid
+flowchart LR
+  App[Fetcher] --> Sched[BitfieldScheduler]
+  Sched --> Core[Hypercore]
+  Sched --> Q[Priority queue]
+```
+
+Shared: [`../../../_shared/storage-gossip-base.js`](../../../_shared/storage-gossip-base.js).
@@ -0,0 +1,13 @@
+require('bare-process/global')
+const { HyperP2PCoreBitfieldScheduler } = require('../index.js')
+
+async function main () {
+  const m = new HyperP2PCoreBitfieldScheduler()
+  m.attach({ length: 0 })
+  m.scheduleRange(0, 8, 1)
+  const r = m.nextRange()
+  console.log('range', r)
+  await m.close()
+  console.log('done')
+}
+main().catch(console.error)
@@ -0,0 +1,89 @@
+require('bare-process/global')
+const EventEmitter = require('bare-events')
+const { assertCore } = require('../../_shared/storage-gossip-base.js')
+const PROTOCOL = 'core-bitfield-scheduler/v1'
+
+class HyperP2PCoreBitfieldScheduler extends EventEmitter {
+  constructor (opts = {}) {
+    super()
+    this.core = opts.core || null
+    this._queue = []
+    this._served = new Set()
+    this._stats = { scheduled: 0, served: 0, skipped: 0 }
+  }
+
+  attach (core) {
+    assertCore(core)
+    this.core = core
+    return this
+  }
+
+  scheduleRange (start, len, priority = 0) {
+    if (start < 0) throw new Error('start must be non-negative')
+    if (len <= 0) throw new Error('len must be positive')
+    const key = `${start}:${len}`
+    if (this._served.has(key)) {
+      this._stats.skipped++
+      return null
+    }
+    const item = { start, len, end: start + len - 1, priority, at: Date.now(), key }
+    this._queue.push(item)
+    this._queue.sort((a, b) => b.priority - a.priority || a.start - b.start)
+    this._stats.scheduled++
+    this.emit('scheduled', item)
+    return item
+  }
+
+  peekNext () {
+    return this._queue.find((r) => !this._served.has(r.key)) || null
+  }
+
+  listPending () {
+    return this._queue.filter((r) => !this._served.has(r.key)).map((r) => ({ ...r }))
+  }
+
+  nextRange () {
+    while (this._queue.length) {
+      const item = this._queue.shift()
+      if (this._served.has(item.key)) {
+        this._stats.skipped++
+        continue
+      }
+      this._stats.served++
+      this.emit('range', item)
+      return item
+    }
+    return null
+  }
+
+  markRangeDone (start, len) {
+    const key = `${start}:${len}`
+    this._served.add(key)
+    this._queue = this._queue.filter((r) => r.key !== key)
+    return true
+  }
+
+  pendingCount () {
+    return this._queue.length
+  }
+
+  getStats () {
+    return {
+      ...this._stats,
+      pending: this._queue.length,
+      servedKeys: this._served.size,
+      coreLength: this.core ? this.core.length : 0,
+      protocol: PROTOCOL
+    }
+  }
+
+  async ready () { return this }
+
+  async close () {
+    this._queue = []
+    this._served.clear()
+    this.emit('closed')
+  }
+}
+
+module.exports = { HyperP2PCoreBitfieldScheduler, PROTOCOL }
@@ -0,0 +1,27 @@
+{
+  "name": "hyper-p2p-core-bitfield-scheduler",
+  "version": "0.3.1",
+  "description": "Bitfield request scheduling policy.",
+  "main": "index.js",
+  "type": "commonjs",
+  "license": "Apache-2.0",
+  "scripts": { "test": "brittle-bare test/test.js" },
+  "dependencies": {
+    "bare-events": "^2.8.0",
+    "bare-process": "^4.4.0",
+    "b4a": "^1.6.7",
+    "hypercore-crypto": "^3.0.0",
+    "protomux": "^3.0.0",
+    "compact-encoding": "^2.0.0"
+  },
+  "peerDependencies": {
+    "hyperswarm": "^4.0.0",
+    "bare": ">=1.0.0",
+    "hypercore": "^10.0.0"
+  },
+  "devDependencies": { "brittle": "^3.0.0" },
+  "imports": {
+    "process": { "bare": "bare-process", "default": "process" },
+    "events": { "bare": "bare-events", "default": "events" }
+  }
+}
@@ -0,0 +1,50 @@
+require('bare-process/global')
+const test = require('brittle')
+const { HyperP2PCoreBitfieldScheduler, PROTOCOL } = require('../index.js')
+
+test('exports', (t) => {
+  t.ok(HyperP2PCoreBitfieldScheduler)
+  t.is(PROTOCOL, 'core-bitfield-scheduler/v1')
+})
+
+test('scheduleRange and nextRange', async (t) => {
+  const m = new HyperP2PCoreBitfieldScheduler()
+  m.attach({ length: 100 })
+  m.scheduleRange(0, 10, 2)
+  m.scheduleRange(20, 5, 1)
+  const r = m.nextRange()
+  t.is(r.start, 0)
+  t.is(r.len, 10)
+  await m.close()
+})
+
+test('validation', async (t) => {
+  const m = new HyperP2PCoreBitfieldScheduler()
+  try { m.scheduleRange(-1, 5) } catch (e) { t.ok(e) }
+  try { m.scheduleRange(0, 0) } catch (e) { t.ok(e) }
+  await m.close()
+})
+
+test('markRangeDone', async (t) => {
+  const m = new HyperP2PCoreBitfieldScheduler()
+  m.scheduleRange(0, 4, 1)
+  m.markRangeDone(0, 4)
+  t.is(m.nextRange(), null)
+  await m.close()
+})
+
+test('peekNext listPending', async (t) => {
+  const m = new HyperP2PCoreBitfieldScheduler()
+  m.scheduleRange(0, 4, 1)
+  t.ok(m.peekNext())
+  t.is(m.listPending().length, 1)
+  m.nextRange()
+  await m.close()
+})
+
+test('getStats', async (t) => {
+  const m = new HyperP2PCoreBitfieldScheduler()
+  m.scheduleRange(1, 3, 0)
+  t.is(m.getStats().protocol, 'core-bitfield-scheduler/v1')
+  await m.close()
+})