Add storage stack modules and track them in git

The modules workspace previously ignored every path matching `storage-*`,
which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out
of version control. Narrow .gitignore to test-artifact patterns only so
production category trees are committed.

Storage packages (23 total):
- storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync,
  priority-fetch, bitfield-scheduler, audit-chain
- storage-hyperbee (5): batch-write, diff-follow, range-watch,
  secondary-index, tombstone-gc
- storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge,
  version-snapshot, watch-notify
- storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus,
  light-writer

Implementation highlights:
- Shared attach/gossip helpers in _shared/storage-gossip-base.js
- P2P modules use Protomux gossip via p2p-bare; local planners omit swarm
- Recent deepen pass: priority queues, batch caps, audit export/import,
  watch/notify aliases on range-watch, fork/view lease helpers, etc.
- Per-package README, docs/api.md, docs/architecture.md, tests, examples

Also update modules/README.md doc hub links for the full storage stack
(hypercore, hyperbee, hyperdrive, autobase).

Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Raven Scott
2026-05-21 00:17:51 -04:00
co-authored by Cursor
parent b10507b060
commit a11e22badc
214 changed files with 57132 additions and 5 deletions
+2 -4
View File
@@ -5,12 +5,10 @@ node_modules/
real_tests-integration-storage/ real_tests-integration-storage/
**/*.log **/*.log
# Test / example artifacts # Test / example artifacts (do not ignore storage-hypercore|hyperbee|hyperdrive|autobase categories)
test-storage-* test-storage-*
test-*-storage-* test-*-storage-*
example-*-storage* example-*-storage-*
storage-*
*-storage/
coverage/ coverage/
.nyc_output/ .nyc_output/
tmp/ tmp/
+1 -1
View File
@@ -82,7 +82,7 @@ Most networked apps compose layers rather than importing everything:
Reference compositions live under `examples/` (network stack demo, full messaging/collab stack, encoding stack). See the examples index at the repository root. Reference compositions live under `examples/` (network stack demo, full messaging/collab stack, encoding stack). See the examples index at the repository root.
Category guides under `docs/` at the repository root: network stack, experimental, messaging, storage-hypercore, storage-hyperdrive, applications-collab, observability (see `docs/README.md`). Category guides under `docs/` at the repository root: network stack, experimental, messaging, storage (hypercore, hyperbee, hyperdrive, autobase), applications-collab, observability (see `docs/README.md`).
## Working on one module ## Working on one module
+49
View File
@@ -0,0 +1,49 @@
const { initModuleSwarm, gossipSend } = require('./p2p-bare.js')
const { assertNonEmpty } = require('./lib/errors.js')
function assertCore (core) {
assertNonEmpty(core, 'core')
if (typeof core.length !== 'number' && typeof core.byteLength !== 'number') {
throw new Error('core must be a Hypercore instance')
}
}
function assertBee (bee) {
assertNonEmpty(bee, 'bee')
if (typeof bee.get !== 'function') {
throw new Error('bee must be a Hyperbee instance')
}
}
function assertDrive (drive) {
assertNonEmpty(drive, 'drive')
if (typeof drive.get !== 'function' && typeof drive.readdir !== 'function') {
throw new Error('drive must be a Hyperdrive instance')
}
}
function assertAutobase (autobase) {
assertNonEmpty(autobase, 'autobase')
if (typeof autobase.open !== 'function' && typeof autobase.append !== 'function') {
throw new Error('autobase must be an Autobase instance')
}
}
async function attachGossip (instance, opts) {
const { keyPair, topic, protocol, onmessage } = opts
if (!topic) return null
return initModuleSwarm(instance, { keyPair, topic, protocol, onmessage })
}
function sendGossip (instance, payload) {
if (instance._peerMsgs) gossipSend(instance, payload)
}
module.exports = {
assertCore,
assertBee,
assertDrive,
assertAutobase,
attachGossip,
sendGossip
}
+33
View File
@@ -0,0 +1,33 @@
# Storage (Autobase)
**Path:** `modules/storage-autobase/` · **Modules:** 5 (all production)
Doc hub: [`docs/storage-autobase/README.md`](../../docs/storage-autobase/README.md)
Autobase **coordination** for multi-writer logs: fork selection, view sync gossip, leases, indexer bus, and local write batching.
## Packages
| Module | Protocol | P2P |
|--------|----------|:---:|
| [hyper-p2p-autobase-fork-choice](./hyper-p2p-autobase-fork-choice/) | `autobase-fork-choice/v1` | yes |
| [hyper-p2p-autobase-view-sync](./hyper-p2p-autobase-view-sync/) | `autobase-view-sync/v1` | yes |
| [hyper-p2p-autobase-writer-lease](./hyper-p2p-autobase-writer-lease/) | `autobase-writer-lease/v1` | yes |
| [hyper-p2p-autobase-indexer-bus](./hyper-p2p-autobase-indexer-bus/) | `autobase-indexer-bus/v1` | yes |
| [hyper-p2p-autobase-light-writer](./hyper-p2p-autobase-light-writer/) | `autobase-light-writer/v1` | no |
## Composition
```text
fork-choice → pick canonical fork
view-sync → align version/hash across peers
writer-lease → gate who may append
light-writer → batch proposals → autobase.append
indexer-bus → fan-out index jobs after apply
```
## Test
```bash
cd hyper-p2p-autobase-fork-choice && npm test
```
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-autobase-fork-choice
Registers Autobase view forks over gossip and picks the best by weight/version; align with `hyper-p2p-causal-consensus` for writer ordering. Hyperswarm gossip when `topic` is set.
**Category:** Storage (Autobase)
**Composes with:** `hyper-p2p-causal-consensus`
**Protocol:** `autobase-fork-choice/v1`
## When to use
Multiple Autobase views compete and peers must converge on one fork id.
## When not to use
Single linear Autobase with no fork hints.
## Quick start
```js
const { HyperP2PAutobaseForkChoice } = require('hyper-p2p-autobase-fork-choice')
const topic = process.argv[2] // 64-char hex or string
const mod = new HyperP2PAutobaseForkChoice({ topic })
await mod.ready() // joins swarm when topic set
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,111 @@
# API: hyper-p2p-autobase-fork-choice
**Protocol:** `autobase-fork-choice/v1`
**Export:** `HyperP2PAutobaseForkChoice`
## Overview
Registers Autobase view forks over gossip and picks the best by weight/version; align with `hyper-p2p-causal-consensus` for writer ordering.
## Constructor
```js
const mod = new HyperP2PAutobaseForkChoice(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `registerView(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: version must be non-negative`
### `pickBestFork(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `listViews(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `view` | view/fork record |
| `picked` | best fork hint or null |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `version must be non-negative`
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `autobase-fork-choice/v1`.
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
### Gossip message types
- `view-register`
## Testing
```bash
npm install && npm test
```
## Common flows
1. `registerView(forkId, version, meta)` — gossip `view-register`.
2. `pickBestFork()` — deterministic winner.
3. `listViews()` — sorted registry.
@@ -0,0 +1,21 @@
# Architecture: hyper-p2p-autobase-fork-choice
**Protocol:** `autobase-fork-choice/v1` · **P2P:** yes
## Wire
| type | payload |
|------|---------|
| `view-register` | `{ view: { forkId, version, weight, peer, at } }` |
## Selection
`pickBestFork()` — max `weight`, tie-break on `version`. `getChosenFork()` returns winning view.
## Merge
Remote view replaces local when `remote.at >= local.at` for same `forkId`.
## Composition
Run before `autobase-light-writer.flushOps` and `autobase-indexer-bus` publish.
@@ -0,0 +1,14 @@
require('bare-process/global')
const { HyperP2PAutobaseForkChoice } = require('../index.js')
async function main () {
const topic = process.argv[2] || null
const m = new HyperP2PAutobaseForkChoice({ topic })
m.attach({ async append () {} })
await m.ready()
m.registerView('fork-a', 3)
console.log(m.pickBestFork())
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,102 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
const { assertAutobase, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'autobase-fork-choice/v1'
class HyperP2PAutobaseForkChoice extends EventEmitter {
constructor (opts = {}) {
super()
this.topic = opts.topic || null
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
this.autobase = opts.autobase || null
this._views = new Map()
this._chosen = null
this._stats = { registered: 0, picks: 0, gossipIn: 0, gossipOut: 0 }
this.swarm = null
}
attach (autobase) {
assertAutobase(autobase)
this.autobase = autobase
return this
}
registerView (forkId, version, meta = {}) {
assertNonEmpty(forkId, 'forkId')
if (version < 0) throw new Error('version must be non-negative')
const view = {
forkId,
version,
weight: meta.weight != null ? meta.weight : version,
peer: meta.peer || null,
at: Date.now()
}
this._views.set(forkId, view)
this._stats.registered++
sendGossip(this, { type: 'view-register', view })
this._stats.gossipOut++
this.emit('view', view)
return view
}
pickBestFork () {
let best = null
for (const v of this._views.values()) {
if (!best || v.weight > best.weight || (v.weight === best.weight && v.version > best.version)) {
best = v
}
}
this._chosen = best ? best.forkId : null
this._stats.picks++
this.emit('picked', best)
return best
}
listViews () {
return [...this._views.values()].sort((a, b) => b.weight - a.weight)
}
getChosenFork () {
if (!this._chosen) return null
return this._views.get(this._chosen) || null
}
_onGossip (d) {
if (!d || d.type !== 'view-register' || !d.view) return
this._stats.gossipIn++
const existing = this._views.get(d.view.forkId)
if (!existing || (d.view.at || 0) >= (existing.at || 0)) {
this._views.set(d.view.forkId, d.view)
}
}
getStats () {
return {
...this._stats,
views: this._views.size,
chosen: this._chosen,
protocol: PROTOCOL
}
}
async ready () {
if (this.swarm || !this.topic) return this
await attachGossip(this, {
keyPair: this.keyPair,
topic: this.topic,
protocol: PROTOCOL,
onmessage: (d) => this._onGossip(d)
})
return this
}
async close () {
if (this.swarm) await this.swarm.destroy().catch(() => {})
this.swarm = null
this.emit('closed')
}
}
module.exports = { HyperP2PAutobaseForkChoice, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-autobase-fork-choice",
"version": "0.3.1",
"description": "Autobase fork choice helper.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.16.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"autobase": "^7.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,38 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PAutobaseForkChoice, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PAutobaseForkChoice)
t.is(PROTOCOL, 'autobase-fork-choice/v1')
})
test('registerView pickBestFork', async (t) => {
const m = new HyperP2PAutobaseForkChoice()
m.attach({ open: async () => {} })
m.registerView('f1', 2, { weight: 1 })
m.registerView('f2', 5, { weight: 10 })
const best = m.pickBestFork()
t.is(best.forkId, 'f2')
t.is(m.getChosenFork().forkId, 'f2')
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PAutobaseForkChoice()
try { m.registerView(null, 0) } catch (e) { t.ok(e) }
await m.close()
})
test('listViews', async (t) => {
const m = new HyperP2PAutobaseForkChoice()
m.registerView('a', 1)
t.is(m.listViews().length, 1)
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PAutobaseForkChoice()
t.is(m.getStats().protocol, 'autobase-fork-choice/v1')
await m.close()
})
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-autobase-indexer-bus
Indexer event bus with local subscribers and gossip `index-event` replication; compose with `hyper-p2p-distributed-event-bus` for cross-module routing. Hyperswarm gossip when `topic` is set.
**Category:** Storage (Autobase)
**Composes with:** `hyper-p2p-distributed-event-bus`
**Protocol:** `autobase-indexer-bus/v1`
## When to use
Derived index events must be published to all Autobase indexer peers.
## When not to use
Indexing is local-only with no shared topic.
## Quick start
```js
const { HyperP2PAutobaseIndexerBus } = require('hyper-p2p-autobase-indexer-bus')
const topic = process.argv[2] // 64-char hex or string
const mod = new HyperP2PAutobaseIndexerBus({ topic })
await mod.ready() // joins swarm when topic set
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,118 @@
# API: hyper-p2p-autobase-indexer-bus
**Protocol:** `autobase-indexer-bus/v1`
**Export:** `HyperP2PAutobaseIndexerBus`
## Overview
Indexer event bus with local subscribers and gossip `index-event` replication; compose with `hyper-p2p-distributed-event-bus` for cross-module routing.
## Constructor
```js
const mod = new HyperP2PAutobaseIndexerBus(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `publishIndexEvent(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `subscribeIndex(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: handler must be a function`
### `drainEvents(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: max must be non-negative`
### `pendingEvents(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `index` | indexer event |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `handler must be a function`
- `max must be non-negative`
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `autobase-indexer-bus/v1`.
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
### Gossip message types
- `index-event`
## Testing
```bash
npm install && npm test
```
## Common flows
1. `publishIndexEvent(type, payload)` — queue + gossip + deliver.
2. `subscribeIndex(handler)` — local fanout.
3. `drainEvents(max)` — batch dequeue for processors.
@@ -0,0 +1,23 @@
# Architecture: hyper-p2p-autobase-indexer-bus
**Protocol:** `autobase-indexer-bus/v1` · **P2P:** yes
## Wire
| type | payload |
|------|---------|
| `index-event` | `{ evt: { type, payload, at, id } }` |
## Flow
`publishIndexEvent` → local subscribers + gossip → remote `_deliver`.
| Method | Purpose |
|--------|---------|
| `subscribeIndex(handler)` | Returns unsubscribe fn |
| `drainEvents(max)` | Remove from queue |
| `peekEvents(n)` | Non-destructive read |
## Events
`index`, `closed`
@@ -0,0 +1,14 @@
require('bare-process/global')
const { HyperP2PAutobaseIndexerBus } = require('../index.js')
async function main () {
const topic = process.argv[2] || null
const m = new HyperP2PAutobaseIndexerBus({ topic })
m.attach({ async append () {} })
await m.ready()
m.publishIndexEvent('indexed', { n: 1 })
console.log('pending', m.pendingEvents())
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,103 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
const { assertAutobase, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'autobase-indexer-bus/v1'
class HyperP2PAutobaseIndexerBus extends EventEmitter {
constructor (opts = {}) {
super()
this.topic = opts.topic || null
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
this.autobase = opts.autobase || null
this._queue = []
this._subs = new Set()
this._stats = { published: 0, delivered: 0, drained: 0, gossipIn: 0, gossipOut: 0 }
this.swarm = null
}
attach (autobase) {
assertAutobase(autobase)
this.autobase = autobase
return this
}
publishIndexEvent (type, payload = {}) {
assertNonEmpty(type, 'type')
const evt = { type, payload, at: Date.now(), id: this._queue.length + 1 }
this._queue.push(evt)
this._stats.published++
sendGossip(this, { type: 'index-event', evt })
this._stats.gossipOut++
this._deliver(evt)
return evt
}
subscribeIndex (handler) {
if (typeof handler !== 'function') throw new Error('handler must be a function')
this._subs.add(handler)
return () => this._subs.delete(handler)
}
_deliver (evt) {
for (const h of this._subs) {
h(evt)
this._stats.delivered++
}
this.emit('index', evt)
}
drainEvents (max = 64) {
if (max < 0) throw new Error('max must be non-negative')
const out = this._queue.splice(0, max)
this._stats.drained += out.length
return out
}
pendingEvents () {
return this._queue.length
}
peekEvents (n = 16) {
if (n < 0) throw new Error('n must be non-negative')
return this._queue.slice(0, n).map((e) => ({ ...e }))
}
_onGossip (d) {
if (!d || d.type !== 'index-event' || !d.evt) return
this._stats.gossipIn++
this._queue.push(d.evt)
this._deliver(d.evt)
}
getStats () {
return {
...this._stats,
pending: this._queue.length,
subscribers: this._subs.size,
protocol: PROTOCOL
}
}
async ready () {
if (this.swarm || !this.topic) return this
await attachGossip(this, {
keyPair: this.keyPair,
topic: this.topic,
protocol: PROTOCOL,
onmessage: (d) => this._onGossip(d)
})
return this
}
async close () {
if (this.swarm) await this.swarm.destroy().catch(() => {})
this.swarm = null
this._subs.clear()
this._queue = []
this.emit('closed')
}
}
module.exports = { HyperP2PAutobaseIndexerBus, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-autobase-indexer-bus",
"version": "0.3.1",
"description": "Indexer event bus for autobase.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.16.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"autobase": "^7.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,39 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PAutobaseIndexerBus, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PAutobaseIndexerBus)
t.is(PROTOCOL, 'autobase-indexer-bus/v1')
})
test('publishIndexEvent drainEvents', async (t) => {
const m = new HyperP2PAutobaseIndexerBus()
let seen = 0
m.subscribeIndex(() => { seen++ })
m.publishIndexEvent('block', { n: 1 })
const drained = m.drainEvents(10)
t.ok(drained.length >= 0)
t.ok(seen >= 1)
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PAutobaseIndexerBus()
try { m.subscribeIndex(null) } catch (e) { t.ok(e) }
try { m.publishIndexEvent(null) } catch (e) { t.ok(e) }
await m.close()
})
test('pendingEvents', async (t) => {
const m = new HyperP2PAutobaseIndexerBus()
m.publishIndexEvent('x')
t.is(m.pendingEvents(), 1)
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PAutobaseIndexerBus()
t.is(m.getStats().protocol, 'autobase-indexer-bus/v1')
await m.close()
})
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-autobase-light-writer
Batches lightweight Autobase append ops before flush; throttle bursts via `hyper-p2p-activity-queue` at the coordination layer. Local-only scheduling layer (no Hyperswarm join).
**Category:** Storage (Autobase)
**Composes with:** `hyper-p2p-activity-queue`
**Protocol:** `autobase-light-writer/v1`
## When to use
Clients propose many small ops that should append in bounded batches.
## When not to use
Each op must commit immediately or you need cross-peer writer leases.
## Quick start
```js
const { HyperP2PAutobaseLightWriter } = require('hyper-p2p-autobase-light-writer')
const mod = new HyperP2PAutobaseLightWriter()
mod.attach(/* Hyperautobase instance */)
await mod.ready()
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,116 @@
# API: hyper-p2p-autobase-light-writer
**Protocol:** `autobase-light-writer/v1`
**Export:** `HyperP2PAutobaseLightWriter`
## Overview
Batches lightweight Autobase append ops before flush; throttle bursts via `hyper-p2p-activity-queue` at the coordination layer.
## Constructor
```js
const mod = new HyperP2PAutobaseLightWriter(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
| `maxBatch` | varies | 64 | Constructor option `maxBatch` |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `proposeOp(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: op object required`
- `Error: batch full`
### `flushOps(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: autobase is required`
### `pendingOps(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `peekOps(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `propose` | staged op |
| `flush` | `{ count }` |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `op object required`
- `batch full`
- `autobase is required`
## P2P
Library-only: no swarm join. `ready()` resolves immediately.
`getStats().protocol` still reports the module protocol id for logging.
## Testing
```bash
npm install && npm test
```
## Common flows
1. `proposeOp(op)` — stage until `maxBatch`.
2. `flushOps()` — append all staged ops to Autobase.
3. `peekOps()` / `pendingOps()` — inspect queue.
@@ -0,0 +1,26 @@
# Architecture: hyper-p2p-autobase-light-writer
**Protocol:** `autobase-light-writer/v1` · **P2P:** no
## Role
Buffer append **proposals** then `flushOps()` to `autobase.append(op)` when attached.
## Options
`maxBatch` (default 64) — max pending ops; `proposeOp` throws when full.
## Methods
| Method | Effect |
|--------|--------|
| `proposeOp(op)` | Queue with `seq`, `at` |
| `flushOps()` | Append all via autobase |
| `rollbackOps()` | Clear queue |
| `peekOps` / `pendingOps` | Inspect |
## Events
`propose`, `flush`, `rollback`, `closed`
Requires `writer-lease` when multiple writers share one autobase.
@@ -0,0 +1,12 @@
require('bare-process/global')
const { HyperP2PAutobaseLightWriter } = require('../index.js')
async function main () {
const m = new HyperP2PAutobaseLightWriter()
m.attach({ async append () {} })
m.proposeOp({ type: 'put', key: 'x' })
console.log('pending', m.pendingOps())
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,77 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertAutobase } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'autobase-light-writer/v1'
class HyperP2PAutobaseLightWriter extends EventEmitter {
constructor (opts = {}) {
super()
this.autobase = opts.autobase || null
this.maxBatch = opts.maxBatch || 64
this._ops = []
this._stats = { proposed: 0, flushed: 0, dropped: 0 }
}
attach (autobase) {
assertAutobase(autobase)
this.autobase = autobase
return this
}
proposeOp (op) {
if (!op || typeof op !== 'object') throw new Error('op object required')
if (this._ops.length >= this.maxBatch) {
this._stats.dropped++
throw new Error('batch full')
}
const entry = { ...op, at: Date.now(), seq: this._ops.length }
this._ops.push(entry)
this._stats.proposed++
this.emit('propose', entry)
return entry
}
async flushOps () {
if (!this.autobase) throw new Error('autobase is required')
const ops = this._ops.splice(0, this.maxBatch)
for (const op of ops) {
if (this.autobase.append) await this.autobase.append(op)
}
this._stats.flushed += ops.length
this.emit('flush', { count: ops.length })
return ops.length
}
pendingOps () {
return this._ops.length
}
peekOps () {
return [...this._ops]
}
rollbackOps () {
const n = this._ops.length
this._ops = []
this.emit('rollback', { count: n })
return n
}
getStats () {
return {
...this._stats,
pending: this._ops.length,
maxBatch: this.maxBatch,
protocol: PROTOCOL
}
}
async ready () { return this }
async close () {
this._ops = []
this.emit('closed')
}
}
module.exports = { HyperP2PAutobaseLightWriter, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-autobase-light-writer",
"version": "0.3.1",
"description": "Light writer pattern wrapper.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.16.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"autobase": "^7.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,39 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PAutobaseLightWriter, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PAutobaseLightWriter)
t.is(PROTOCOL, 'autobase-light-writer/v1')
})
test('proposeOp flushOps', async (t) => {
const appended = []
const m = new HyperP2PAutobaseLightWriter({ maxBatch: 4 })
m.attach({ append: async (op) => { appended.push(op) } })
m.proposeOp({ type: 'put' })
const n = await m.flushOps()
t.is(n, 1)
t.is(appended.length, 1)
await m.close()
})
test('batch full', async (t) => {
const m = new HyperP2PAutobaseLightWriter({ maxBatch: 1 })
m.proposeOp({ a: 1 })
try { m.proposeOp({ b: 2 }) } catch (e) { t.ok(e) }
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PAutobaseLightWriter()
try { m.proposeOp(null) } catch (e) { t.ok(e) }
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PAutobaseLightWriter()
m.proposeOp({ x: 1 })
t.is(m.getStats().pending, 1)
await m.close()
})
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-autobase-view-sync
Gossip-synced Autobase view version + content hash; reactive UIs can mirror via `hyper-p2p-reactive-state`. Hyperswarm gossip when `topic` is set.
**Category:** Storage (Autobase)
**Composes with:** `hyper-p2p-reactive-state`
**Protocol:** `autobase-view-sync/v1`
## When to use
Peers must agree on the latest materialized view version/hash.
## When not to use
View state is local-only or full Autobase sync replaces hash checks.
## Quick start
```js
const { HyperP2PAutobaseViewSync } = require('hyper-p2p-autobase-view-sync')
const topic = process.argv[2] // 64-char hex or string
const mod = new HyperP2PAutobaseViewSync({ topic })
await mod.ready() // joins swarm when topic set
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,111 @@
# API: hyper-p2p-autobase-view-sync
**Protocol:** `autobase-view-sync/v1`
**Export:** `HyperP2PAutobaseViewSync`
## Overview
Gossip-synced Autobase view version + content hash; reactive UIs can mirror via `hyper-p2p-reactive-state`.
## Constructor
```js
const mod = new HyperP2PAutobaseViewSync(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `publishView(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: version must be non-negative`
### `mergeRemoteView(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `currentView(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `publish` | published view |
| `merged` | merged view state |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `version must be non-negative`
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `autobase-view-sync/v1`.
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
### Gossip message types
- `view-sync`
## Testing
```bash
npm install && npm test
```
## Common flows
1. `publishView(version)` — gossip `view-sync`.
2. `mergeRemoteView(view)` — monotonic version merge.
3. `currentView()` — local snapshot.
@@ -0,0 +1,21 @@
# Architecture: hyper-p2p-autobase-view-sync
**Protocol:** `autobase-view-sync/v1` · **P2P:** yes
## Wire
| type | fields |
|------|--------|
| `view-sync` | `{ view: { version, hash, at } }` |
`hash` = `sha256("view:"+version)` (coordination digest, not Autobase proof).
## Helpers
- `isAtLeast(version)` — local caught up?
- `behindBy(version)` — blocks/versions behind target
- `mergeRemoteView` — monotonic version + hash match
## Events
`publish`, `merged`, `closed`
@@ -0,0 +1,13 @@
require('bare-process/global')
const { HyperP2PAutobaseViewSync } = require('../index.js')
async function main () {
const topic = process.argv[2] || null
const m = new HyperP2PAutobaseViewSync({ topic })
m.attach({ async append () {} })
await m.ready()
console.log(m.publishView(1))
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,98 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const b4a = require('b4a')
const crypto = require('hypercore-crypto')
const { assertAutobase, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'autobase-view-sync/v1'
class HyperP2PAutobaseViewSync extends EventEmitter {
constructor (opts = {}) {
super()
this.topic = opts.topic || null
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
this.autobase = opts.autobase || null
this._view = { version: 0, hash: null, at: 0 }
this._stats = { published: 0, merged: 0, gossipIn: 0, gossipOut: 0 }
this.swarm = null
}
attach (autobase) {
assertAutobase(autobase)
this.autobase = autobase
return this
}
_hashView (version) {
return b4a.toString(crypto.hash(b4a.from(`view:${version}`)), 'hex')
}
publishView (version) {
if (version < 0) throw new Error('version must be non-negative')
const view = {
version,
hash: this._hashView(version),
at: Date.now()
}
if (version >= this._view.version) this._view = view
sendGossip(this, { type: 'view-sync', view })
this._stats.published++
this._stats.gossipOut++
this.emit('publish', view)
return view
}
mergeRemoteView (view) {
if (!view || typeof view.version !== 'number') return false
if (view.version < this._view.version) return false
if (view.version === this._view.version && view.hash !== this._view.hash) {
return false
}
this._view = { ...view, mergedAt: Date.now() }
this._stats.merged++
this.emit('merged', this._view)
return true
}
currentView () {
return { ...this._view }
}
isAtLeast (version) {
return this._view.version >= version
}
behindBy (version) {
if (version < 0) throw new Error('version must be non-negative')
return Math.max(0, version - this._view.version)
}
_onGossip (d) {
if (!d || d.type !== 'view-sync' || !d.view) return
this._stats.gossipIn++
this.mergeRemoteView(d.view)
}
getStats () {
return { ...this._stats, view: this._view, protocol: PROTOCOL }
}
async ready () {
if (this.swarm || !this.topic) return this
await attachGossip(this, {
keyPair: this.keyPair,
topic: this.topic,
protocol: PROTOCOL,
onmessage: (d) => this._onGossip(d)
})
return this
}
async close () {
if (this.swarm) await this.swarm.destroy().catch(() => {})
this.swarm = null
this.emit('closed')
}
}
module.exports = { HyperP2PAutobaseViewSync, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-autobase-view-sync",
"version": "0.3.1",
"description": "Autobase view synchronization.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.16.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"autobase": "^7.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,38 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PAutobaseViewSync, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PAutobaseViewSync)
t.is(PROTOCOL, 'autobase-view-sync/v1')
})
test('publishView mergeRemoteView', async (t) => {
const m = new HyperP2PAutobaseViewSync()
m.attach({ open: async () => {} })
m.publishView(3)
t.is(m.currentView().version, 3)
const v5 = m.publishView(5)
t.ok(m.mergeRemoteView({ version: 5, hash: v5.hash, at: Date.now() }))
t.is(m.currentView().version, 5)
await m.close()
})
test('merge rejects regression', async (t) => {
const m = new HyperP2PAutobaseViewSync()
m.publishView(10)
t.not(m.mergeRemoteView({ version: 2, hash: 'x', at: 1 }))
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PAutobaseViewSync()
try { m.publishView(-1) } catch (e) { t.ok(e) }
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PAutobaseViewSync()
t.is(m.getStats().protocol, 'autobase-view-sync/v1')
await m.close()
})
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-autobase-writer-lease
TTL writer leases gossiped across peers; hard mutual exclusion layers on `hyper-p2p-distributed-lock` when needed. Hyperswarm gossip when `topic` is set.
**Category:** Storage (Autobase)
**Composes with:** `hyper-p2p-distributed-lock`
**Protocol:** `autobase-writer-lease/v1`
## When to use
Only one writer should append to Autobase for a window of time.
## When not to use
Autobase already serializes writers or leases are enforced off-chain.
## Quick start
```js
const { HyperP2PAutobaseWriterLease } = require('hyper-p2p-autobase-writer-lease')
const topic = process.argv[2] // 64-char hex or string
const mod = new HyperP2PAutobaseWriterLease({ topic })
await mod.ready() // joins swarm when topic set
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,119 @@
# API: hyper-p2p-autobase-writer-lease
**Protocol:** `autobase-writer-lease/v1`
**Export:** `HyperP2PAutobaseWriterLease`
## Overview
TTL writer leases gossiped across peers; hard mutual exclusion layers on `hyper-p2p-distributed-lock` when needed.
## Constructor
```js
const mod = new HyperP2PAutobaseWriterLease(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
| `localWriterId` | varies | null | Constructor option `localWriterId` |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `acquireLease(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: ttlMs must be positive`
### `renewLease(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `hasLease(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `releaseLease(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `lease` | lease record |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `ttlMs must be positive`
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `autobase-writer-lease/v1`.
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
### Gossip message types
- `lease-acquire`
- `lease-renew`
- `lease-release`
## Testing
```bash
npm install && npm test
```
## Common flows
1. `acquireLease(writerId, ttlMs)` — gossip `lease-acquire`.
2. `renewLease` / `releaseLease` — extend or drop lease.
3. `hasLease(writerId)` — check non-expired holder.
@@ -0,0 +1,21 @@
# Architecture: hyper-p2p-autobase-writer-lease
**Protocol:** `autobase-writer-lease/v1` · **P2P:** yes
## Wire
| type | purpose |
|------|---------|
| `lease-acquire` | New lease with `expiresAt` |
| `lease-renew` | Extend TTL |
| `lease-release` | Drop by `writerId` |
## Local API
`acquireLease`, `renewLease`, `hasLease`, `releaseLease`, `listActiveLeases`, `expireStaleLeases`
## Events
`lease`, `expired`, `closed`
Gate `light-writer.proposeOp` on `hasLease(localWriterId)`.
@@ -0,0 +1,13 @@
require('bare-process/global')
const { HyperP2PAutobaseWriterLease } = require('../index.js')
async function main () {
const topic = process.argv[2] || null
const m = new HyperP2PAutobaseWriterLease({ topic })
m.attach({ async append () {} })
await m.ready()
console.log(m.acquireLease('writer-1', 5000))
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,137 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
const { assertAutobase, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'autobase-writer-lease/v1'
class HyperP2PAutobaseWriterLease extends EventEmitter {
constructor (opts = {}) {
super()
this.topic = opts.topic || null
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
this.autobase = opts.autobase || null
this.localWriterId = opts.writerId || null
this._leases = new Map()
this._stats = { acquired: 0, renewed: 0, released: 0, gossipIn: 0, gossipOut: 0 }
this.swarm = null
}
attach (autobase) {
assertAutobase(autobase)
this.autobase = autobase
return this
}
acquireLease (writerId, ttlMs = 30000) {
assertNonEmpty(writerId, 'writerId')
if (ttlMs <= 0) throw new Error('ttlMs must be positive')
const now = Date.now()
const existing = this._leases.get(writerId)
if (existing && existing.expiresAt > now && existing.holder !== writerId) {
return false
}
const lease = { holder: writerId, acquiredAt: now, expiresAt: now + ttlMs, ttlMs }
this._leases.set(writerId, lease)
this._stats.acquired++
sendGossip(this, { type: 'lease-acquire', lease })
this._stats.gossipOut++
this.emit('lease', lease)
return lease
}
renewLease (writerId, ttlMs = 30000) {
const lease = this._leases.get(writerId)
if (!lease || lease.holder !== writerId) return false
lease.expiresAt = Date.now() + ttlMs
lease.ttlMs = ttlMs
this._stats.renewed++
sendGossip(this, { type: 'lease-renew', lease })
this._stats.gossipOut++
return lease
}
hasLease (writerId) {
const lease = this._leases.get(writerId)
if (!lease) return false
if (lease.expiresAt <= Date.now()) {
this._leases.delete(writerId)
return false
}
return lease.holder === writerId
}
listActiveLeases () {
const now = Date.now()
const out = []
for (const [id, lease] of this._leases) {
if (lease.expiresAt > now) out.push({ writerId: id, ...lease })
}
return out
}
expireStaleLeases () {
const now = Date.now()
let n = 0
for (const [id, lease] of this._leases) {
if (lease.expiresAt <= now) {
this._leases.delete(id)
n++
}
}
if (n) this.emit('expired', { count: n })
return n
}
releaseLease (writerId) {
const ok = this._leases.delete(writerId)
if (ok) {
this._stats.released++
sendGossip(this, { type: 'lease-release', writerId })
this._stats.gossipOut++
}
return ok
}
_onGossip (d) {
if (!d) return
this._stats.gossipIn++
if (d.type === 'lease-acquire' && d.lease) {
const l = d.lease
const existing = this._leases.get(l.holder)
if (!existing || l.acquiredAt >= existing.acquiredAt) {
this._leases.set(l.holder, l)
}
}
if (d.type === 'lease-renew' && d.lease) this._leases.set(d.lease.holder, d.lease)
if (d.type === 'lease-release') this._leases.delete(d.writerId)
}
getStats () {
return {
...this._stats,
activeLeases: this._leases.size,
protocol: PROTOCOL
}
}
async ready () {
if (this.swarm || !this.topic) return this
await attachGossip(this, {
keyPair: this.keyPair,
topic: this.topic,
protocol: PROTOCOL,
onmessage: (d) => this._onGossip(d)
})
return this
}
async close () {
if (this.swarm) await this.swarm.destroy().catch(() => {})
this.swarm = null
this._leases.clear()
this.emit('closed')
}
}
module.exports = { HyperP2PAutobaseWriterLease, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-autobase-writer-lease",
"version": "0.3.1",
"description": "Single-writer lease for autobase.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.16.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"autobase": "^7.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,38 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PAutobaseWriterLease, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PAutobaseWriterLease)
t.is(PROTOCOL, 'autobase-writer-lease/v1')
})
test('acquireLease hasLease releaseLease', async (t) => {
const m = new HyperP2PAutobaseWriterLease()
m.attach({ open: async () => {} })
t.ok(m.acquireLease('w1', 60000))
t.ok(m.hasLease('w1'))
t.ok(m.releaseLease('w1'))
t.not(m.hasLease('w1'))
await m.close()
})
test('renewLease', async (t) => {
const m = new HyperP2PAutobaseWriterLease()
m.acquireLease('w2', 5000)
t.ok(m.renewLease('w2', 8000))
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PAutobaseWriterLease()
try { m.acquireLease(null) } catch (e) { t.ok(e) }
try { m.acquireLease('w', 0) } catch (e) { t.ok(e) }
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PAutobaseWriterLease()
t.is(m.getStats().protocol, 'autobase-writer-lease/v1')
await m.close()
})
+34
View File
@@ -0,0 +1,34 @@
# Storage (Hyperbee)
**Path:** `modules/storage-hyperbee/` · **Modules:** 5 (all production)
Doc hub: [`docs/storage-hyperbee/README.md`](../../docs/storage-hyperbee/README.md)
Hyperbee **application helpers** — batching, watches, indexes, tombstones, and diff cursors. These modules do not replace Hyperbee; they wrap `attach(bee)` and optional gossip via `_shared/storage-gossip-base.js`.
## Packages
| Module | Protocol | Summary |
|--------|----------|---------|
| [hyper-p2p-bee-batch-write](./hyper-p2p-bee-batch-write/) | `bee-batch-write/v1` | Buffered put/del; `maxBatch` cap |
| [hyper-p2p-bee-diff-follow](./hyper-p2p-bee-diff-follow/) | `bee-diff-follow/v1` | Version cursor + `applyBatch` |
| [hyper-p2p-bee-range-watch](./hyper-p2p-bee-range-watch/) | `bee-range-watch/v1` | Lexicographic range callbacks |
| [hyper-p2p-bee-secondary-index](./hyper-p2p-bee-secondary-index/) | `bee-secondary-index/v1` | Secondary → primary map |
| [hyper-p2p-bee-tombstone-gc](./hyper-p2p-bee-tombstone-gc/) | `bee-tombstone-gc/v1` | Tombstone + time sweep |
## Stack
```text
storage-hypercore (replicate cores backing the bee)
storage-hyperbee (this category)
applications / collab modules
```
## Test & demo
```bash
cd hyper-p2p-bee-range-watch && npm test
bare ../../examples/demo-storage-bee/index.js
```
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-bee-batch-write
Buffers Hyperbee put/delete ops and commits them in one batch; compose with `hyper-p2p-crdt-map` when merging batched CRDT updates across peers. Local-only scheduling layer (no Hyperswarm join).
**Category:** Storage (Hyperbee)
**Composes with:** `hyper-p2p-crdt-map`
**Protocol:** `bee-batch-write/v1`
## When to use
Many small bee writes should flush atomically to reduce IO churn.
## When not to use
Single-key latency-sensitive writes without batching.
## Quick start
```js
const { HyperP2PBeeBatchWrite } = require('hyper-p2p-bee-batch-write')
const mod = new HyperP2PBeeBatchWrite()
mod.attach(/* Hyperbee instance */)
await mod.ready()
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,126 @@
# API: hyper-p2p-bee-batch-write
**Protocol:** `bee-batch-write/v1`
**Export:** `HyperP2PBeeBatchWrite`
## Overview
Buffers Hyperbee put/delete ops and commits them in one batch; compose with `hyper-p2p-crdt-map` when merging batched CRDT updates across peers.
## Constructor
```js
const mod = new HyperP2PBeeBatchWrite(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `bufferPut(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: key required`
### `bufferDel(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: key required`
### `commitBatch(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: bee is required`
### `rollbackBatch(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `pendingCount(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `peekOps(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `commit` | `{ count }` |
| `rollback` | `{ count }` |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `key required`
- `bee is required`
## P2P
Library-only: no swarm join. `ready()` resolves immediately.
`getStats().protocol` still reports the module protocol id for logging.
## Testing
```bash
npm install && npm test
```
## Common flows
1. `bufferPut` / `bufferDel` — stage ops.
2. `commitBatch()` — flush to bee (or bee.batch()).
3. `rollbackBatch()` — discard staged ops.
@@ -0,0 +1,23 @@
# Architecture: hyper-p2p-bee-batch-write
**Protocol:** `bee-batch-write/v1` · **P2P:** no
## Role
Accumulate put/del operations then flush via Hyperbee `batch()` when available, else sequential `put`/`del`.
## Options
| Option | Default | Effect |
|--------|---------|--------|
| `maxBatch` | `0` | Max buffered ops; `0` = unlimited |
## Lifecycle
1. `bufferPut` / `bufferDel` — queue ops (may return `false` when full)
2. `commitBatch` — flush, emit `commit`
3. `rollbackBatch` — discard, emit `rollback`
## Composition
Use before `bee-range-watch.notify` so watchers see committed keys only.
@@ -0,0 +1,12 @@
require('bare-process/global')
const { HyperP2PBeeBatchWrite } = require('../index.js')
async function main () {
const m = new HyperP2PBeeBatchWrite()
m.attach({ async get () {} })
m.bufferPut('k', 'v')
console.log('pending', m.pendingCount())
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,97 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertBee } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'bee-batch-write/v1'
class HyperP2PBeeBatchWrite extends EventEmitter {
constructor (opts = {}) {
super()
this.bee = opts.bee || null
this._ops = []
this.maxBatch = opts.maxBatch != null ? opts.maxBatch : 0
this._stats = { puts: 0, dels: 0, committed: 0, rolledBack: 0, rejected: 0 }
}
attach (bee) {
assertBee(bee)
this.bee = bee
return this
}
_canBuffer () {
if (this.maxBatch > 0 && this._ops.length >= this.maxBatch) {
this._stats.rejected++
return false
}
return true
}
bufferPut (key, value) {
if (key == null) throw new Error('key required')
if (!this._canBuffer()) return false
this._ops.push({ op: 'put', key, value })
this._stats.puts++
return this._ops.length
}
bufferDel (key) {
if (key == null) throw new Error('key required')
if (!this._canBuffer()) return false
this._ops.push({ op: 'del', key })
this._stats.dels++
return this._ops.length
}
async commitBatch () {
if (!this.bee) throw new Error('bee is required')
if (!this._ops.length) return 0
const batch = this.bee.batch ? this.bee.batch() : null
const target = batch || this.bee
for (const op of this._ops) {
if (op.op === 'put') {
if (target.put) await target.put(op.key, op.value)
} else if (op.op === 'del' && target.del) {
await target.del(op.key)
}
}
if (batch && batch.flush) await batch.flush()
const n = this._ops.length
this._ops = []
this._stats.committed += n
this.emit('commit', { count: n })
return n
}
rollbackBatch () {
const n = this._ops.length
this._ops = []
this._stats.rolledBack += n
this.emit('rollback', { count: n })
return n
}
pendingCount () {
return this._ops.length
}
peekOps () {
return [...this._ops]
}
getStats () {
return {
...this._stats,
pending: this._ops.length,
protocol: PROTOCOL
}
}
async ready () { return this }
async close () {
this._ops = []
this.emit('closed')
}
}
module.exports = { HyperP2PBeeBatchWrite, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-bee-batch-write",
"version": "0.3.1",
"description": "Batch write coalescing for Hyperbee.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.0.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"hyperbee": "^2.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,58 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PBeeBatchWrite, PROTOCOL } = require('../index.js')
const mockBee = {
get: async () => null,
batch () {
const puts = []
return {
put: async (k, v) => { puts.push([k, v]) },
flush: async () => puts.length
}
}
}
test('exports', (t) => {
t.ok(HyperP2PBeeBatchWrite)
t.is(PROTOCOL, 'bee-batch-write/v1')
})
test('bufferPut commitBatch', async (t) => {
const m = new HyperP2PBeeBatchWrite()
m.attach(mockBee)
m.bufferPut('a', 1)
m.bufferDel('b')
const n = await m.commitBatch()
t.is(n, 2)
t.is(m.pendingCount(), 0)
await m.close()
})
test('rollbackBatch', async (t) => {
const m = new HyperP2PBeeBatchWrite()
m.bufferPut('x', 1)
t.is(m.rollbackBatch(), 1)
t.is(m.pendingCount(), 0)
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PBeeBatchWrite()
try { m.bufferPut(null, 1) } catch (e) { t.ok(e) }
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PBeeBatchWrite()
m.bufferPut('k', 'v')
t.is(m.getStats().pending, 1)
await m.close()
})
test('maxBatch rejects overflow', async (t) => {
const m = new HyperP2PBeeBatchWrite({ maxBatch: 1 })
m.bufferPut('a', 1)
t.is(m.bufferPut('b', 2), false)
await m.close()
})
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-bee-diff-follow
Tracks a version cursor and buffered diffs for Hyperbee followers; fan out applied entries via `hyper-p2p-distributed-event-bus` at the app layer. Local-only scheduling layer (no Hyperswarm join).
**Category:** Storage (Hyperbee)
**Composes with:** `hyper-p2p-distributed-event-bus`
**Protocol:** `bee-diff-follow/v1`
## When to use
Incremental bee replication from a known version watermark.
## When not to use
Full snapshot sync only, or P2P diff transport (local buffer only).
## Quick start
```js
const { HyperP2PBeeDiffFollow } = require('hyper-p2p-bee-diff-follow')
const mod = new HyperP2PBeeDiffFollow()
mod.attach(/* Hyperbee instance */)
await mod.ready()
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,122 @@
# API: hyper-p2p-bee-diff-follow
**Protocol:** `bee-diff-follow/v1`
**Export:** `HyperP2PBeeDiffFollow`
## Overview
Tracks a version cursor and buffered diffs for Hyperbee followers; fan out applied entries via `hyper-p2p-distributed-event-bus` at the app layer.
## Constructor
```js
const mod = new HyperP2PBeeDiffFollow(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `followSince(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: version must be non-negative`
### `recordDiff(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: entry with version required`
### `pullDiff(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: max must be non-negative`
### `applyDiff(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: entry with version required`
### `diffCursor(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `follow` | cursor version number |
| `applied` | range entry |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `version must be non-negative`
- `entry with version required`
- `max must be non-negative`
## P2P
Library-only: no swarm join. `ready()` resolves immediately.
`getStats().protocol` still reports the module protocol id for logging.
## Testing
```bash
npm install && npm test
```
## Common flows
1. `followSince(version)` — reset cursor.
2. `recordDiff(entry)` — buffer if version > cursor.
3. `pullDiff(max)` / `applyDiff(entry)` — drain and advance cursor.
@@ -0,0 +1,24 @@
# Architecture: hyper-p2p-bee-diff-follow
**Protocol:** `bee-diff-follow/v1` · **P2P:** no
## Role
Track a **version cursor** and an ordered buffer of diff entries for incremental sync.
## Flow
```text
followSince(v) → recordDiff(entry) → pullDiff(max) → applyDiff / applyBatch
```
| Method | Purpose |
|--------|---------|
| `recordDiff` | Insert if `entry.version > cursor` |
| `pullDiff` | Drain buffer up to `max` |
| `applyBatch` | Apply many entries, advance cursor |
| `clearBuffer` | Drop buffered entries without applying |
## Events
`follow`, `applied`, `closed`
@@ -0,0 +1,13 @@
require('bare-process/global')
const { HyperP2PBeeDiffFollow } = require('../index.js')
async function main () {
const m = new HyperP2PBeeDiffFollow()
m.attach({ async get () {} })
m.followSince(0)
m.recordDiff({ version: 1, key: 'a' })
console.log(m.pullDiff(10))
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,101 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertBee } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'bee-diff-follow/v1'
class HyperP2PBeeDiffFollow extends EventEmitter {
constructor (opts = {}) {
super()
this.bee = opts.bee || null
this._cursor = opts.sinceVersion || 0
this._buffer = []
this._stats = { follows: 0, pulled: 0, applied: 0 }
}
attach (bee) {
assertBee(bee)
this.bee = bee
return this
}
followSince (version = 0) {
if (version < 0) throw new Error('version must be non-negative')
this._cursor = version
this._stats.follows++
this.emit('follow', this._cursor)
return this._cursor
}
recordDiff (entry) {
if (!entry || typeof entry.version !== 'number') {
throw new Error('entry with version required')
}
if (entry.version <= this._cursor) return false
this._buffer.push(entry)
this._buffer.sort((a, b) => a.version - b.version)
return true
}
pullDiff (max = 100) {
if (max < 0) throw new Error('max must be non-negative')
const out = []
while (this._buffer.length && out.length < max) {
const next = this._buffer[0]
if (next.version <= this._cursor) {
this._buffer.shift()
continue
}
out.push(this._buffer.shift())
}
this._stats.pulled += out.length
return out
}
applyDiff (entry) {
if (!entry || typeof entry.version !== 'number') {
throw new Error('entry with version required')
}
if (entry.version < this._cursor) return false
this._cursor = entry.version
this._stats.applied++
this.emit('applied', entry)
return true
}
diffCursor () {
return this._cursor
}
applyBatch (entries) {
if (!Array.isArray(entries)) throw new Error('entries must be an array')
let n = 0
for (const entry of entries) {
if (this.applyDiff(entry)) n++
}
return n
}
clearBuffer () {
const n = this._buffer.length
this._buffer = []
return n
}
getStats () {
return {
...this._stats,
cursor: this._cursor,
buffered: this._buffer.length,
protocol: PROTOCOL
}
}
async ready () { return this }
async close () {
this._buffer = []
this.emit('closed')
}
}
module.exports = { HyperP2PBeeDiffFollow, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-bee-diff-follow",
"version": "0.3.1",
"description": "Diff stream follower.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.0.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"hyperbee": "^2.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,40 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PBeeDiffFollow, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PBeeDiffFollow)
t.is(PROTOCOL, 'bee-diff-follow/v1')
})
test('followSince pullDiff applyDiff', async (t) => {
const m = new HyperP2PBeeDiffFollow()
m.attach({ get: async () => null })
m.followSince(2)
m.recordDiff({ version: 3, key: 'a' })
const pulled = m.pullDiff(10)
t.is(pulled.length, 1)
m.applyDiff({ version: 3 })
t.is(m.diffCursor(), 3)
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PBeeDiffFollow()
try { m.followSince(-1) } catch (e) { t.ok(e) }
try { m.applyDiff({}) } catch (e) { t.ok(e) }
await m.close()
})
test('recordDiff skips old', async (t) => {
const m = new HyperP2PBeeDiffFollow()
m.followSince(5)
t.not(m.recordDiff({ version: 3 }))
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PBeeDiffFollow()
t.is(m.getStats().protocol, 'bee-diff-follow/v1')
await m.close()
})
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-bee-range-watch
Lexicographic key-range watchers on Hyperbee with `emitChange` notifications; mirror reactive slices with `hyper-p2p-reactive-state` upstream. Local-only scheduling layer (no Hyperswarm join).
**Category:** Storage (Hyperbee)
**Composes with:** `hyper-p2p-reactive-state`
**Protocol:** `bee-range-watch/v1`
## When to use
UI or pipelines must react to puts/deletes inside a key span.
## When not to use
Whole-database watchers or P2P change fanout (local callbacks only).
## Quick start
```js
const { HyperP2PBeeRangeWatch } = require('hyper-p2p-bee-range-watch')
const mod = new HyperP2PBeeRangeWatch()
mod.attach(/* Hyperbee instance */)
await mod.ready()
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,116 @@
# API: hyper-p2p-bee-range-watch
**Protocol:** `bee-range-watch/v1`
**Export:** `HyperP2PBeeRangeWatch`
## Overview
Lexicographic key-range watchers on Hyperbee with `emitChange` notifications; mirror reactive slices with `hyper-p2p-reactive-state` upstream.
## Constructor
```js
const mod = new HyperP2PBeeRangeWatch(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `watchRange(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: gte and lte required`
- `Error: callback required`
- `Error: gte must be <= lte`
### `emitChange(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: key required`
### `unwatch(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `activeWatches(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `change` | `{ key, value, op, delivered }` |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `gte and lte required`
- `callback required`
- `gte must be <= lte`
- `key required`
## P2P
Library-only: no swarm join. `ready()` resolves immediately.
`getStats().protocol` still reports the module protocol id for logging.
## Testing
```bash
npm install && npm test
```
## Common flows
1. `watchRange(gte, lte, cb)` — register callback.
2. `emitChange(key, value, op)` — deliver to matching watches.
3. `unwatch(id)` — remove listener.
@@ -0,0 +1,27 @@
# Architecture: hyper-p2p-bee-range-watch
**Protocol:** `bee-range-watch/v1` · **P2P:** no
## Role
Deliver in-process notifications when keys change inside a **lexicographic span** `[gte, lte]`. Call `notify` / `emitChange` from your bee put/del hooks.
## State
| Field | Description |
|-------|-------------|
| `_watches` | `id → { gte, lte, cb }` |
| `_nextId` | Monotonic watch id |
## API aliases
- `watch(gte, lte, cb)``watchRange`
- `notify(key, value, op)``emitChange`
## Events
`change` (with `delivered` count), `closed`
## Composition
`hyper-p2p-reactive-state`, `hyper-p2p-bee-batch-write` (commit then notify watchers).
@@ -0,0 +1,12 @@
require('bare-process/global')
const { HyperP2PBeeRangeWatch } = require('../index.js')
async function main () {
const m = new HyperP2PBeeRangeWatch()
m.attach({ async get () {} })
m.watchRange('a', 'z', (c) => console.log('watch', c.key))
m.emitChange('m', 'v')
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,85 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertBee } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'bee-range-watch/v1'
class HyperP2PBeeRangeWatch extends EventEmitter {
constructor (opts = {}) {
super()
this.bee = opts.bee || null
this._watches = new Map()
this._nextId = 1
this._stats = { watches: 0, notifications: 0, unwatch: 0 }
}
attach (bee) {
assertBee(bee)
this.bee = bee
return this
}
watch (gte, lte, cb) {
return this.watchRange(gte, lte, cb)
}
watchRange (gte, lte, cb) {
if (gte == null || lte == null) throw new Error('gte and lte required')
if (typeof cb !== 'function') throw new Error('callback required')
if (gte > lte) throw new Error('gte must be <= lte')
const id = this._nextId++
this._watches.set(id, { gte, lte, cb })
this._stats.watches++
return id
}
notify (key, value, op = 'put') {
return this.emitChange(key, value, op)
}
emitChange (key, value, op = 'put') {
if (key == null) throw new Error('key required')
let n = 0
for (const w of this._watches.values()) {
if (key >= w.gte && key <= w.lte) {
w.cb({ key, value, op, at: Date.now() })
n++
}
}
this._stats.notifications += n
this.emit('change', { key, value, op, delivered: n })
return n
}
unwatch (id) {
const ok = this._watches.delete(id)
if (ok) this._stats.unwatch++
return ok
}
activeWatches () {
return this._watches.size
}
listWatches () {
return [...this._watches.entries()].map(([id, w]) => ({
id, gte: w.gte, lte: w.lte
}))
}
getStats () {
return {
...this._stats,
active: this._watches.size,
protocol: PROTOCOL
}
}
async ready () { return this }
async close () {
this._watches.clear()
this.emit('closed')
}
}
module.exports = { HyperP2PBeeRangeWatch, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-bee-range-watch",
"version": "0.3.1",
"description": "Range watch notifications on Hyperbee.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.0.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"hyperbee": "^2.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,48 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PBeeRangeWatch, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PBeeRangeWatch)
t.is(PROTOCOL, 'bee-range-watch/v1')
})
test('watchRange emitChange', async (t) => {
const m = new HyperP2PBeeRangeWatch()
let hit = 0
m.watchRange('a', 'm', () => { hit++ })
m.emitChange('b', 42)
t.is(hit, 1)
await m.close()
})
test('unwatch', async (t) => {
const m = new HyperP2PBeeRangeWatch()
const id = m.watchRange('a', 'z', () => {})
t.ok(m.unwatch(id))
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PBeeRangeWatch()
try { m.watchRange('z', 'a', () => {}) } catch (e) { t.ok(e) }
try { m.watchRange('a', 'z', null) } catch (e) { t.ok(e) }
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PBeeRangeWatch()
m.watchRange('a', 'c', () => {})
t.is(m.getStats().active, 1)
await m.close()
})
test('watch notify aliases listWatches', async (t) => {
const m = new HyperP2PBeeRangeWatch()
let hit = 0
m.watch('a', 'z', () => { hit++ })
m.notify('b', 1)
t.is(hit, 1)
t.is(m.listWatches().length, 1)
await m.close()
})
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-bee-secondary-index
In-memory secondary-key → primary-key index for Hyperbee; route lookup intents through `hyper-p2p-intent-router` when queries cross peers. Local-only scheduling layer (no Hyperswarm join).
**Category:** Storage (Hyperbee)
**Composes with:** `hyper-p2p-intent-router`
**Protocol:** `bee-secondary-index/v1`
## When to use
Queries need a non-primary key path into bee records.
## When not to use
Primary-key-only access patterns or indexes persisted inside bee itself.
## Quick start
```js
const { HyperP2PBeeSecondaryIndex } = require('hyper-p2p-bee-secondary-index')
const mod = new HyperP2PBeeSecondaryIndex()
mod.attach(/* Hyperbee instance */)
await mod.ready()
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,118 @@
# API: hyper-p2p-bee-secondary-index
**Protocol:** `bee-secondary-index/v1`
**Export:** `HyperP2PBeeSecondaryIndex`
## Overview
In-memory secondary-key → primary-key index for Hyperbee; route lookup intents through `hyper-p2p-intent-router` when queries cross peers.
## Constructor
```js
const mod = new HyperP2PBeeSecondaryIndex(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
| `indexName` | varies | 'default' | Constructor option `indexName` |
| `keyPrefix` | varies | `@idx/${this.indexName}/` | Constructor option `keyPrefix` |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `indexPut(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `indexLookup(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `indexRemove(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `rebuildIndex(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: pairs must be an array`
### `listSecondaryKeys(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `index` | indexer event |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `pairs must be an array`
## P2P
Library-only: no swarm join. `ready()` resolves immediately.
`getStats().protocol` still reports the module protocol id for logging.
## Testing
```bash
npm install && npm test
```
## Common flows
1. `indexPut(secondaryKey, primaryKey)` — maintain Set per secondary key.
2. `indexLookup(secondaryKey)` — return primary keys.
3. `rebuildIndex(pairs)` — cold rebuild from `{ secondaryKey, primaryKey }` rows.
@@ -0,0 +1,20 @@
# Architecture: hyper-p2p-bee-secondary-index
**Protocol:** `bee-secondary-index/v1` · **P2P:** no
## Role
In-memory **secondary → Set(primary)** index with optional `keyPrefix` for bee key namespacing (`@idx/<name>/`).
## Operations
- `indexPut` / `indexLookup` / `indexRemove`
- `rebuildIndex(pairs)` — bulk load
- `hasSecondary` / `countPrimaries` — introspection
- `listSecondaryKeys`
Persist to Hyperbee in your app by writing `_secKey(secondary)` entries after `indexPut`.
## Composition
Pair with `bee-batch-write` for atomic index + row updates.
@@ -0,0 +1,12 @@
require('bare-process/global')
const { HyperP2PBeeSecondaryIndex } = require('../index.js')
async function main () {
const m = new HyperP2PBeeSecondaryIndex()
m.attach({ async get () {} })
m.indexPut('sec', 'pri')
console.log(m.indexLookup('sec'))
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,93 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertBee } = require('../../_shared/storage-gossip-base.js')
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
const PROTOCOL = 'bee-secondary-index/v1'
class HyperP2PBeeSecondaryIndex extends EventEmitter {
constructor (opts = {}) {
super()
this.bee = opts.bee || null
this.indexName = opts.indexName || 'default'
this.keyPrefix = opts.keyPrefix || `@idx/${this.indexName}/`
this._index = new Map()
this._stats = { indexed: 0, lookups: 0, removed: 0, rebuilt: 0 }
}
attach (bee) {
assertBee(bee)
this.bee = bee
return this
}
_secKey (secondaryKey) {
return `${this.keyPrefix}${secondaryKey}`
}
indexPut (secondaryKey, primaryKey) {
assertNonEmpty(secondaryKey, 'secondaryKey')
assertNonEmpty(primaryKey, 'primaryKey')
const set = this._index.get(secondaryKey) || new Set()
set.add(primaryKey)
this._index.set(secondaryKey, set)
this._stats.indexed++
this.emit('index', { secondaryKey, primaryKey })
return true
}
indexLookup (secondaryKey) {
this._stats.lookups++
const set = this._index.get(secondaryKey)
return set ? [...set] : []
}
indexRemove (secondaryKey, primaryKey) {
const set = this._index.get(secondaryKey)
if (!set || !set.has(primaryKey)) return false
set.delete(primaryKey)
if (!set.size) this._index.delete(secondaryKey)
this._stats.removed++
return true
}
async rebuildIndex (pairs) {
if (!Array.isArray(pairs)) throw new Error('pairs must be an array')
this._index.clear()
for (const { secondaryKey, primaryKey } of pairs) {
this.indexPut(secondaryKey, primaryKey)
}
this._stats.rebuilt++
return this._index.size
}
listSecondaryKeys () {
return [...this._index.keys()]
}
hasSecondary (secondaryKey) {
return this._index.has(secondaryKey)
}
countPrimaries (secondaryKey) {
const set = this._index.get(secondaryKey)
return set ? set.size : 0
}
getStats () {
return {
...this._stats,
secondaryKeys: this._index.size,
indexName: this.indexName,
protocol: PROTOCOL
}
}
async ready () { return this }
async close () {
this._index.clear()
this.emit('closed')
}
}
module.exports = { HyperP2PBeeSecondaryIndex, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-bee-secondary-index",
"version": "0.3.1",
"description": "Secondary index maintenance.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.0.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"hyperbee": "^2.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,38 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PBeeSecondaryIndex, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PBeeSecondaryIndex)
t.is(PROTOCOL, 'bee-secondary-index/v1')
})
test('indexPut indexLookup', async (t) => {
const m = new HyperP2PBeeSecondaryIndex()
m.attach({ get: async () => null })
m.indexPut('tag:a', 'pk-1')
m.indexPut('tag:a', 'pk-2')
t.is(m.indexLookup('tag:a').length, 2)
await m.close()
})
test('indexRemove', async (t) => {
const m = new HyperP2PBeeSecondaryIndex()
m.indexPut('s', 'p')
t.ok(m.indexRemove('s', 'p'))
t.is(m.indexLookup('s').length, 0)
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PBeeSecondaryIndex()
try { m.indexPut(null, 'p') } catch (e) { t.ok(e) }
await m.close()
})
test('rebuildIndex', async (t) => {
const m = new HyperP2PBeeSecondaryIndex()
const n = await m.rebuildIndex([{ secondaryKey: 'b', primaryKey: '1' }])
t.is(n, 1)
await m.close()
})
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,41 @@
# hyper-p2p-bee-tombstone-gc
Soft-delete tombstones with time-based `gcSweep`; pair with `hyper-p2p-temporal-index` when retention policies are time-ordered. Local-only scheduling layer (no Hyperswarm join).
**Category:** Storage (Hyperbee)
**Composes with:** `hyper-p2p-temporal-index`
**Protocol:** `bee-tombstone-gc/v1`
## When to use
Deletes should linger for undo/audit before physical removal.
## When not to use
Immediate hard deletes or distributed GC coordination (local map only).
## Quick start
```js
const { HyperP2PBeeTombstoneGc } = require('hyper-p2p-bee-tombstone-gc')
const mod = new HyperP2PBeeTombstoneGc()
mod.attach(/* Hyperbee instance */)
await mod.ready()
// ... application logic ...
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,120 @@
# API: hyper-p2p-bee-tombstone-gc
**Protocol:** `bee-tombstone-gc/v1`
**Export:** `HyperP2PBeeTombstoneGc`
## Overview
Soft-delete tombstones with time-based `gcSweep`; pair with `hyper-p2p-temporal-index` when retention policies are time-ordered.
## Constructor
```js
const mod = new HyperP2PBeeTombstoneGc(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `tombstone(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: key required`
### `gcSweep(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: olderThanMs must be non-negative`
### `listTombstones(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `restore(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `pendingTombstones(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `tombstone` | `{ key }` |
| `sweep` | `{ removed }` |
| `restore` | `{ key }` |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `key required`
- `olderThanMs must be non-negative`
## P2P
Library-only: no swarm join. `ready()` resolves immediately.
`getStats().protocol` still reports the module protocol id for logging.
## Testing
```bash
npm install && npm test
```
## Common flows
1. `tombstone(key, meta)` — mark pending delete.
2. `gcSweep(olderThanMs)` — purge expired tombstones.
3. `restore(key)` — cancel tombstone before sweep.
@@ -0,0 +1,26 @@
# Architecture: hyper-p2p-bee-tombstone-gc
**Protocol:** `bee-tombstone-gc/v1` · **P2P:** no
## Role
Soft-delete registry: mark keys tombstoned, sweep after `olderThanMs`, optional `restore`.
## State
`Map<key, { at, meta }>`
## Methods
| Method | Effect |
|--------|--------|
| `tombstone(key, meta?)` | Mark deleted |
| `isTombstoned(key)` | Query |
| `gcSweep(olderThanMs)` | Remove stale entries |
| `listTombstones` / `restore` | Admin |
## Events
`tombstone`, `sweep`, `restore`, `closed`
Run physical `bee.del` in your app after sweep returns keys.
@@ -0,0 +1,12 @@
require('bare-process/global')
const { HyperP2PBeeTombstoneGc } = require('../index.js')
async function main () {
const m = new HyperP2PBeeTombstoneGc()
m.attach({ async get () {} })
m.tombstone('old-key')
console.log('pending', m.pendingTombstones())
await m.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,79 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const { assertBee } = require('../../_shared/storage-gossip-base.js')
const PROTOCOL = 'bee-tombstone-gc/v1'
class HyperP2PBeeTombstoneGc extends EventEmitter {
constructor (opts = {}) {
super()
this.bee = opts.bee || null
this._tombstones = new Map()
this._stats = { tombstoned: 0, swept: 0, restored: 0 }
}
attach (bee) {
assertBee(bee)
this.bee = bee
return this
}
tombstone (key, meta = {}) {
if (key == null) throw new Error('key required')
this._tombstones.set(key, { at: Date.now(), meta })
this._stats.tombstoned++
this.emit('tombstone', { key })
return true
}
gcSweep (olderThanMs = 86400000) {
if (olderThanMs < 0) throw new Error('olderThanMs must be non-negative')
const now = Date.now()
const removed = []
for (const [key, entry] of this._tombstones) {
if (now - entry.at >= olderThanMs) {
this._tombstones.delete(key)
removed.push(key)
}
}
this._stats.swept += removed.length
this.emit('sweep', { removed, count: removed.length })
return removed
}
listTombstones () {
return [...this._tombstones.entries()].map(([key, entry]) => ({ key, ...entry }))
}
restore (key) {
if (!this._tombstones.has(key)) return false
this._tombstones.delete(key)
this._stats.restored++
this.emit('restore', { key })
return true
}
pendingTombstones () {
return this._tombstones.size
}
isTombstoned (key) {
return this._tombstones.has(key)
}
getStats () {
return {
...this._stats,
pending: this._tombstones.size,
protocol: PROTOCOL
}
}
async ready () { return this }
async close () {
this._tombstones.clear()
this.emit('closed')
}
}
module.exports = { HyperP2PBeeTombstoneGc, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,27 @@
{
"name": "hyper-p2p-bee-tombstone-gc",
"version": "0.3.1",
"description": "Tombstone garbage collection policy.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.0.0"
},
"peerDependencies": {
"hyperswarm": "^4.0.0",
"bare": ">=1.0.0",
"hyperbee": "^2.0.0"
},
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,39 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PBeeTombstoneGc, PROTOCOL } = require('../index.js')
test('exports', (t) => {
t.ok(HyperP2PBeeTombstoneGc)
t.is(PROTOCOL, 'bee-tombstone-gc/v1')
})
test('tombstone gcSweep', async (t) => {
const m = new HyperP2PBeeTombstoneGc()
m.tombstone('old-key')
m._tombstones.set('gone', { at: 0 })
const removed = m.gcSweep(0)
t.is(removed.length, 2)
await m.close()
})
test('restore', async (t) => {
const m = new HyperP2PBeeTombstoneGc()
m.tombstone('k')
t.ok(m.restore('k'))
t.is(m.pendingTombstones(), 0)
await m.close()
})
test('validation', async (t) => {
const m = new HyperP2PBeeTombstoneGc()
try { m.tombstone(null) } catch (e) { t.ok(e) }
try { m.gcSweep(-1) } catch (e) { t.ok(e) }
await m.close()
})
test('getStats', async (t) => {
const m = new HyperP2PBeeTombstoneGc()
m.tombstone('x')
t.is(m.getStats().pending, 1)
await m.close()
})
+49
View File
@@ -0,0 +1,49 @@
# Storage (Hypercore)
**Path:** `modules/storage-hypercore/` · **Modules:** 7 (all production)
Doc hub: [`docs/storage-hypercore/README.md`](../../docs/storage-hypercore/README.md) · Index: [MODULE_CATEGORIES.md](../MODULE_CATEGORIES.md#storage-hypercore)
Hypercore **coordination** primitives — not a replacement for `hypercore` replication. Modules either gossip policy/range/fork state on a Hyperswarm topic or run **local planners** (priority fetch, bitfield scheduler, audit chain) that your downloader attaches to a core instance.
## Packages
| Module | Protocol | P2P gossip | Summary |
|--------|----------|:----------:|---------|
| [hyper-p2p-core-replicator](./hyper-p2p-core-replicator/) | `core-replicator/v1` | yes | Per-peer `maxBytes`, pause flags; `policy-sync` merge |
| [hyper-p2p-core-seed-policy](./hyper-p2p-core-seed-policy/) | `core-seed-policy/v1` | yes | Allow/deny seeding, `maxBlocks`, default deny |
| [hyper-p2p-core-fork-picker](./hyper-p2p-core-fork-picker/) | `core-fork-picker/v1` | yes | Fork hints by weight/length; `fork-register` gossip |
| [hyper-p2p-core-merkle-sync](./hyper-p2p-core-merkle-sync/) | `core-merkle-sync/v1` | yes | Range roots, `diffRanges`, `merkle-ranges` publish |
| [hyper-p2p-core-priority-fetch](./hyper-p2p-core-priority-fetch/) | `core-priority-fetch/v1` | no | Max-heap block index queue with inflight tracking |
| [hyper-p2p-core-bitfield-scheduler](./hyper-p2p-core-bitfield-scheduler/) | `core-bitfield-scheduler/v1` | no | Priority ranges `[start, len)` for bitfield work |
| [hyper-p2p-core-audit-chain](./hyper-p2p-core-audit-chain/) | `core-audit-chain/v1` | no | Hash-linked audit entries; `verifyChain` / export |
## Stack placement
```text
Application / drive layer
storage-hyperdrive / storage-hyperbee (optional)
storage-hypercore (this category) — policy + planners
hypercore instance + Hyperswarm (via _shared/p2p-bare.js)
```
## Shared runtime
- [`../_shared/storage-gossip-base.js`](../_shared/storage-gossip-base.js) — `attach(core)`, `attachGossip`, `sendGossip`
- [`../_shared/p2p-bare.js`](../_shared/p2p-bare.js) — swarm + Protomux when `topic` is set
## Quick test
```bash
cd hyper-p2p-core-replicator && npm install && npm test
bare ../../examples/demo-storage-core/index.js
```
## Related categories
- [storage-hyperbee](../storage-hyperbee/) — bee batch, range watch, tombstones
- [storage-hyperdrive](../storage-hyperdrive/) — drive catalog, mirror, GC
- [network-stack](../network-stack/) — transport underlay for large transfers
@@ -0,0 +1,5 @@
# Changelog
## [0.0.0-scaffold]
- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,42 @@
# hyper-p2p-core-audit-chain
Append-only hash-linked audit log over a Hypercore. Local-only (no Hyperswarm topic).
**Category:** Storage (Hypercore)
**Composes with:** `hyper-p2p-attestation-chain`
**Protocol:** `core-audit-chain/v1`
## When to use
Apps that need tamper-evident audit trails tied to core length (compliance, replication forensics).
## When not to use
When you only need unstructured logs without hash chaining (use observability modules).
## Quick start
```js
const { HyperP2PCoreAuditChain } = require('hyper-p2p-core-audit-chain')
const mod = new HyperP2PCoreAuditChain()
mod.attach(core)
mod.appendAudit({ op: 'write', actor: 'peer-1' })
console.log(mod.verifyChain())
console.log(mod.tail(5))
await mod.close()
```
## Docs
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
- [docs/architecture.md](docs/architecture.md) — chain model, verification
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,85 @@
# API: hyper-p2p-core-audit-chain
**Protocol:** `core-audit-chain/v1`
**Export:** `HyperP2PCoreAuditChain`
## Overview
Append-only hash-linked audit log over a Hypercore. Each entry includes `prevHash`, `hash`, `seq`, and optional `coreLength` snapshot.
## Constructor
```js
const mod = new HyperP2PCoreAuditChain(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `core` | Hypercore | null | Optional core attached at construct time |
## Methods
### `attach(core)`
- **Returns:** `this`
- **Throws:** `Error: core must be a Hypercore instance`
### `appendAudit(event)`
- **Parameters:** `event` — plain object (domain fields)
- **Returns:** entry with `seq`, `hash`, `prevHash`, `at`
- **Throws:** `Error: event object required`
### `verifyChain()`
- **Returns:** `{ ok: true, length, head }` or `{ ok: false, seq, reason }`
- **Throws:** —
### `tail(n = 10)`
- **Returns:** last `n` entries
- **Throws:** `Error: n must be non-negative`
### `getStats()`
- **Returns:** `{ appended, verified, failures, length, head, coreLength, protocol }`
### `ready()`
- **Returns:** `Promise<this>` (no-op)
### `close()`
- **Returns:** `Promise<void>` — clears chain
## Events
| Event | Payload |
|-------|---------|
| `audit` | appended entry |
| `closed` | no payload |
## getStats()
See fields above. `protocol` is always `core-audit-chain/v1`.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
## P2P
Local-only — no `topic` or gossip wire messages.
## Testing
```bash
npm install && npm test
```
## Common flows
1. `attach(core)``appendAudit` for each mutation → periodic `verifyChain()`
2. `tail(n)` for dashboards without scanning full chain
3. `close()` on shutdown to drop in-memory chain
@@ -0,0 +1,48 @@
# Architecture: hyper-p2p-core-audit-chain
**Category:** Storage (Hypercore) · **Protocol:** `core-audit-chain/v1` · **P2P:** no
## Role
Append-only **hash-linked audit log** for operations on a Hypercore (replication decisions, admin actions, policy changes). Each entry stores `prevHash`, `hash`, `seq`, and `coreLength` at append time for correlation with chain state.
## Hash algorithm
`hash = sha256(JSON.stringify({ prevHash, event }))` via `hypercore-crypto` (hex string). Verification walks the chain recomputing hashes; mismatch returns `{ ok: false, seq, reason }`.
## State model
| Field | Description |
|-------|-------------|
| `_chain` | Ordered entries with metadata |
| `_headHash` | Latest hash or `null` |
## API surface
| Method | Purpose |
|--------|---------|
| `appendAudit(event)` | Push linked entry; emit `audit` |
| `verifyChain()` | Full integrity check |
| `tail(n)` | Last `n` entries |
| `exportChain()` / `importChain(entries)` | Snapshot / restore for persistence |
## Wire messages
None. Export JSON and ship over your app channel if peers need shared audit history.
## Events
| Event | Payload |
|-------|---------|
| `audit` | New entry |
| `closed` | Chain cleared |
## Composition
- Log replication policy changes from `hyper-p2p-core-replicator`
- Log fork picks from `hyper-p2p-core-fork-picker`
- Persist `exportChain()` to Hyperbee or Hyperdrive for durability
## Threat model
Detects tampering of in-memory chain only. Does not sign entries — add application-level signatures if non-repudiation is required.

Some files were not shown because too many files have changed in this diff Show More