Add storage stack modules and track them in git

The modules workspace previously ignored every path matching `storage-*`, which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out of version control. Narrow .gitignore to test-artifact patterns only so production category trees are committed. Storage packages (23 total): - storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync, priority-fetch, bitfield-scheduler, audit-chain - storage-hyperbee (5): batch-write, diff-follow, range-watch, secondary-index, tombstone-gc - storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge, version-snapshot, watch-notify - storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus, light-writer Implementation highlights: - Shared attach/gossip helpers in _shared/storage-gossip-base.js - P2P modules use Protomux gossip via p2p-bare; local planners omit swarm - Recent deepen pass: priority queues, batch caps, audit export/import, watch/notify aliases on range-watch, fork/view lease helpers, etc. - Per-package README, docs/api.md, docs/architecture.md, tests, examples Also update modules/README.md doc hub links for the full storage stack (hypercore, hyperbee, hyperdrive, autobase). Co-authored-by: Cursor <[email protected]>
2026-05-21 00:17:51 -04:00
parent b10507b060
commit a11e22badc
214 changed files with 57132 additions and 5 deletions
@@ -0,0 +1,5 @@
+# Changelog
+
+## [0.0.0-scaffold]
+
+- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
+# hyper-p2p-bee-secondary-index
+
+In-memory secondary-key → primary-key index for Hyperbee; route lookup intents through `hyper-p2p-intent-router` when queries cross peers. Local-only scheduling layer (no Hyperswarm join).
+
+**Category:** Storage (Hyperbee)
+
+**Composes with:** `hyper-p2p-intent-router`
+
+**Protocol:** `bee-secondary-index/v1`
+
+## When to use
+
+Queries need a non-primary key path into bee records.
+
+## When not to use
+
+Primary-key-only access patterns or indexes persisted inside bee itself.
+
+## Quick start
+
+```js
+const { HyperP2PBeeSecondaryIndex } = require('hyper-p2p-bee-secondary-index')
+const mod = new HyperP2PBeeSecondaryIndex()
+mod.attach(/* Hyperbee instance */)
+await mod.ready()
+// ... application logic ...
+await mod.close()
+```
+
+## Docs
+
+- [docs/api.md](docs/api.md) — constructor, methods, events, errors
+- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
+- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
+- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
+
+## Test
+
+```bash
+npm install && npm test
+```
@@ -0,0 +1,118 @@
+# API: hyper-p2p-bee-secondary-index
+
+**Protocol:** `bee-secondary-index/v1`
+
+**Export:** `HyperP2PBeeSecondaryIndex`
+
+## Overview
+
+In-memory secondary-key → primary-key index for Hyperbee; route lookup intents through `hyper-p2p-intent-router` when queries cross peers.
+
+## Constructor
+
+```js
+const mod = new HyperP2PBeeSecondaryIndex(opts)
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
+| `indexName` | varies | 'default' | Constructor option `indexName` |
+| `keyPrefix` | varies | `@idx/${this.indexName}/` | Constructor option `keyPrefix` |
+
+## Methods
+
+### `attach(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `indexPut(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `indexLookup(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `indexRemove(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `rebuildIndex(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:**
+  - `Error: pairs must be an array`
+
+### `listSecondaryKeys(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `getStats(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `ready(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+### `close(…)`
+
+- **Returns:** module-specific (see implementation)
+
+- **Throws:** — (none in method body)
+
+## Events
+
+| Event | Payload |
+|-------|---------|
+| `index` | indexer event |
+| `closed` | no payload |
+
+## getStats()
+
+Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
+Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
+
+## Errors
+
+Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
+
+Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
+
+### Documented `throw new Error(...)` strings
+
+- `pairs must be an array`
+
+## P2P
+
+Library-only: no swarm join. `ready()` resolves immediately.
+`getStats().protocol` still reports the module protocol id for logging.
+
+## Testing
+
+```bash
+npm install && npm test
+```
+
+## Common flows
+
+1. `indexPut(secondaryKey, primaryKey)` — maintain Set per secondary key.
+2. `indexLookup(secondaryKey)` — return primary keys.
+3. `rebuildIndex(pairs)` — cold rebuild from `{ secondaryKey, primaryKey }` rows.
+
@@ -0,0 +1,20 @@
+# Architecture: hyper-p2p-bee-secondary-index
+
+**Protocol:** `bee-secondary-index/v1` · **P2P:** no
+
+## Role
+
+In-memory **secondary → Set(primary)** index with optional `keyPrefix` for bee key namespacing (`@idx/<name>/`).
+
+## Operations
+
+- `indexPut` / `indexLookup` / `indexRemove`
+- `rebuildIndex(pairs)` — bulk load
+- `hasSecondary` / `countPrimaries` — introspection
+- `listSecondaryKeys`
+
+Persist to Hyperbee in your app by writing `_secKey(secondary)` entries after `indexPut`.
+
+## Composition
+
+Pair with `bee-batch-write` for atomic index + row updates.
@@ -0,0 +1,12 @@
+require('bare-process/global')
+const { HyperP2PBeeSecondaryIndex } = require('../index.js')
+
+async function main () {
+  const m = new HyperP2PBeeSecondaryIndex()
+  m.attach({ async get () {} })
+  m.indexPut('sec', 'pri')
+  console.log(m.indexLookup('sec'))
+  await m.close()
+  console.log('done')
+}
+main().catch(console.error)
@@ -0,0 +1,93 @@
+require('bare-process/global')
+const EventEmitter = require('bare-events')
+const { assertBee } = require('../../_shared/storage-gossip-base.js')
+const { assertNonEmpty } = require('../../_shared/lib/errors.js')
+const PROTOCOL = 'bee-secondary-index/v1'
+
+class HyperP2PBeeSecondaryIndex extends EventEmitter {
+  constructor (opts = {}) {
+    super()
+    this.bee = opts.bee || null
+    this.indexName = opts.indexName || 'default'
+    this.keyPrefix = opts.keyPrefix || `@idx/${this.indexName}/`
+    this._index = new Map()
+    this._stats = { indexed: 0, lookups: 0, removed: 0, rebuilt: 0 }
+  }
+
+  attach (bee) {
+    assertBee(bee)
+    this.bee = bee
+    return this
+  }
+
+  _secKey (secondaryKey) {
+    return `${this.keyPrefix}${secondaryKey}`
+  }
+
+  indexPut (secondaryKey, primaryKey) {
+    assertNonEmpty(secondaryKey, 'secondaryKey')
+    assertNonEmpty(primaryKey, 'primaryKey')
+    const set = this._index.get(secondaryKey) || new Set()
+    set.add(primaryKey)
+    this._index.set(secondaryKey, set)
+    this._stats.indexed++
+    this.emit('index', { secondaryKey, primaryKey })
+    return true
+  }
+
+  indexLookup (secondaryKey) {
+    this._stats.lookups++
+    const set = this._index.get(secondaryKey)
+    return set ? [...set] : []
+  }
+
+  indexRemove (secondaryKey, primaryKey) {
+    const set = this._index.get(secondaryKey)
+    if (!set || !set.has(primaryKey)) return false
+    set.delete(primaryKey)
+    if (!set.size) this._index.delete(secondaryKey)
+    this._stats.removed++
+    return true
+  }
+
+  async rebuildIndex (pairs) {
+    if (!Array.isArray(pairs)) throw new Error('pairs must be an array')
+    this._index.clear()
+    for (const { secondaryKey, primaryKey } of pairs) {
+      this.indexPut(secondaryKey, primaryKey)
+    }
+    this._stats.rebuilt++
+    return this._index.size
+  }
+
+  listSecondaryKeys () {
+    return [...this._index.keys()]
+  }
+
+  hasSecondary (secondaryKey) {
+    return this._index.has(secondaryKey)
+  }
+
+  countPrimaries (secondaryKey) {
+    const set = this._index.get(secondaryKey)
+    return set ? set.size : 0
+  }
+
+  getStats () {
+    return {
+      ...this._stats,
+      secondaryKeys: this._index.size,
+      indexName: this.indexName,
+      protocol: PROTOCOL
+    }
+  }
+
+  async ready () { return this }
+
+  async close () {
+    this._index.clear()
+    this.emit('closed')
+  }
+}
+
+module.exports = { HyperP2PBeeSecondaryIndex, PROTOCOL }
@@ -0,0 +1,27 @@
+{
+  "name": "hyper-p2p-bee-secondary-index",
+  "version": "0.3.1",
+  "description": "Secondary index maintenance.",
+  "main": "index.js",
+  "type": "commonjs",
+  "license": "Apache-2.0",
+  "scripts": { "test": "brittle-bare test/test.js" },
+  "dependencies": {
+    "bare-events": "^2.8.0",
+    "bare-process": "^4.4.0",
+    "b4a": "^1.6.7",
+    "hypercore-crypto": "^3.0.0",
+    "protomux": "^3.0.0",
+    "compact-encoding": "^2.0.0"
+  },
+  "peerDependencies": {
+    "hyperswarm": "^4.0.0",
+    "bare": ">=1.0.0",
+    "hyperbee": "^2.0.0"
+  },
+  "devDependencies": { "brittle": "^3.0.0" },
+  "imports": {
+    "process": { "bare": "bare-process", "default": "process" },
+    "events": { "bare": "bare-events", "default": "events" }
+  }
+}
@@ -0,0 +1,38 @@
+require('bare-process/global')
+const test = require('brittle')
+const { HyperP2PBeeSecondaryIndex, PROTOCOL } = require('../index.js')
+
+test('exports', (t) => {
+  t.ok(HyperP2PBeeSecondaryIndex)
+  t.is(PROTOCOL, 'bee-secondary-index/v1')
+})
+
+test('indexPut indexLookup', async (t) => {
+  const m = new HyperP2PBeeSecondaryIndex()
+  m.attach({ get: async () => null })
+  m.indexPut('tag:a', 'pk-1')
+  m.indexPut('tag:a', 'pk-2')
+  t.is(m.indexLookup('tag:a').length, 2)
+  await m.close()
+})
+
+test('indexRemove', async (t) => {
+  const m = new HyperP2PBeeSecondaryIndex()
+  m.indexPut('s', 'p')
+  t.ok(m.indexRemove('s', 'p'))
+  t.is(m.indexLookup('s').length, 0)
+  await m.close()
+})
+
+test('validation', async (t) => {
+  const m = new HyperP2PBeeSecondaryIndex()
+  try { m.indexPut(null, 'p') } catch (e) { t.ok(e) }
+  await m.close()
+})
+
+test('rebuildIndex', async (t) => {
+  const m = new HyperP2PBeeSecondaryIndex()
+  const n = await m.rebuildIndex([{ secondaryKey: 'b', primaryKey: '1' }])
+  t.is(n, 1)
+  await m.close()
+})