Add storage stack modules and track them in git
The modules workspace previously ignored every path matching `storage-*`, which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out of version control. Narrow .gitignore to test-artifact patterns only so production category trees are committed. Storage packages (23 total): - storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync, priority-fetch, bitfield-scheduler, audit-chain - storage-hyperbee (5): batch-write, diff-follow, range-watch, secondary-index, tombstone-gc - storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge, version-snapshot, watch-notify - storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus, light-writer Implementation highlights: - Shared attach/gossip helpers in _shared/storage-gossip-base.js - P2P modules use Protomux gossip via p2p-bare; local planners omit swarm - Recent deepen pass: priority queues, batch caps, audit export/import, watch/notify aliases on range-watch, fork/view lease helpers, etc. - Per-package README, docs/api.md, docs/architecture.md, tests, examples Also update modules/README.md doc hub links for the full storage stack (hypercore, hyperbee, hyperdrive, autobase). Co-authored-by: Cursor <[email protected]>
This commit is contained in:
+2
-4
@@ -5,12 +5,10 @@ node_modules/
|
|||||||
real_tests-integration-storage/
|
real_tests-integration-storage/
|
||||||
**/*.log
|
**/*.log
|
||||||
|
|
||||||
# Test / example artifacts
|
# Test / example artifacts (do not ignore storage-hypercore|hyperbee|hyperdrive|autobase categories)
|
||||||
test-storage-*
|
test-storage-*
|
||||||
test-*-storage-*
|
test-*-storage-*
|
||||||
example-*-storage*
|
example-*-storage-*
|
||||||
storage-*
|
|
||||||
*-storage/
|
|
||||||
coverage/
|
coverage/
|
||||||
.nyc_output/
|
.nyc_output/
|
||||||
tmp/
|
tmp/
|
||||||
|
|||||||
@@ -82,7 +82,7 @@ Most networked apps compose layers rather than importing everything:
|
|||||||
|
|
||||||
Reference compositions live under `examples/` (network stack demo, full messaging/collab stack, encoding stack). See the examples index at the repository root.
|
Reference compositions live under `examples/` (network stack demo, full messaging/collab stack, encoding stack). See the examples index at the repository root.
|
||||||
|
|
||||||
Category guides under `docs/` at the repository root: network stack, experimental, messaging, storage-hypercore, storage-hyperdrive, applications-collab, observability (see `docs/README.md`).
|
Category guides under `docs/` at the repository root: network stack, experimental, messaging, storage (hypercore, hyperbee, hyperdrive, autobase), applications-collab, observability (see `docs/README.md`).
|
||||||
|
|
||||||
## Working on one module
|
## Working on one module
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
const { initModuleSwarm, gossipSend } = require('./p2p-bare.js')
|
||||||
|
const { assertNonEmpty } = require('./lib/errors.js')
|
||||||
|
|
||||||
|
function assertCore (core) {
|
||||||
|
assertNonEmpty(core, 'core')
|
||||||
|
if (typeof core.length !== 'number' && typeof core.byteLength !== 'number') {
|
||||||
|
throw new Error('core must be a Hypercore instance')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function assertBee (bee) {
|
||||||
|
assertNonEmpty(bee, 'bee')
|
||||||
|
if (typeof bee.get !== 'function') {
|
||||||
|
throw new Error('bee must be a Hyperbee instance')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function assertDrive (drive) {
|
||||||
|
assertNonEmpty(drive, 'drive')
|
||||||
|
if (typeof drive.get !== 'function' && typeof drive.readdir !== 'function') {
|
||||||
|
throw new Error('drive must be a Hyperdrive instance')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function assertAutobase (autobase) {
|
||||||
|
assertNonEmpty(autobase, 'autobase')
|
||||||
|
if (typeof autobase.open !== 'function' && typeof autobase.append !== 'function') {
|
||||||
|
throw new Error('autobase must be an Autobase instance')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function attachGossip (instance, opts) {
|
||||||
|
const { keyPair, topic, protocol, onmessage } = opts
|
||||||
|
if (!topic) return null
|
||||||
|
return initModuleSwarm(instance, { keyPair, topic, protocol, onmessage })
|
||||||
|
}
|
||||||
|
|
||||||
|
function sendGossip (instance, payload) {
|
||||||
|
if (instance._peerMsgs) gossipSend(instance, payload)
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = {
|
||||||
|
assertCore,
|
||||||
|
assertBee,
|
||||||
|
assertDrive,
|
||||||
|
assertAutobase,
|
||||||
|
attachGossip,
|
||||||
|
sendGossip
|
||||||
|
}
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# Storage (Autobase)
|
||||||
|
|
||||||
|
**Path:** `modules/storage-autobase/` · **Modules:** 5 (all production)
|
||||||
|
|
||||||
|
Doc hub: [`docs/storage-autobase/README.md`](../../docs/storage-autobase/README.md)
|
||||||
|
|
||||||
|
Autobase **coordination** for multi-writer logs: fork selection, view sync gossip, leases, indexer bus, and local write batching.
|
||||||
|
|
||||||
|
## Packages
|
||||||
|
|
||||||
|
| Module | Protocol | P2P |
|
||||||
|
|--------|----------|:---:|
|
||||||
|
| [hyper-p2p-autobase-fork-choice](./hyper-p2p-autobase-fork-choice/) | `autobase-fork-choice/v1` | yes |
|
||||||
|
| [hyper-p2p-autobase-view-sync](./hyper-p2p-autobase-view-sync/) | `autobase-view-sync/v1` | yes |
|
||||||
|
| [hyper-p2p-autobase-writer-lease](./hyper-p2p-autobase-writer-lease/) | `autobase-writer-lease/v1` | yes |
|
||||||
|
| [hyper-p2p-autobase-indexer-bus](./hyper-p2p-autobase-indexer-bus/) | `autobase-indexer-bus/v1` | yes |
|
||||||
|
| [hyper-p2p-autobase-light-writer](./hyper-p2p-autobase-light-writer/) | `autobase-light-writer/v1` | no |
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
```text
|
||||||
|
fork-choice → pick canonical fork
|
||||||
|
view-sync → align version/hash across peers
|
||||||
|
writer-lease → gate who may append
|
||||||
|
light-writer → batch proposals → autobase.append
|
||||||
|
indexer-bus → fan-out index jobs after apply
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd hyper-p2p-autobase-fork-choice && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-autobase-fork-choice
|
||||||
|
|
||||||
|
Registers Autobase view forks over gossip and picks the best by weight/version; align with `hyper-p2p-causal-consensus` for writer ordering. Hyperswarm gossip when `topic` is set.
|
||||||
|
|
||||||
|
**Category:** Storage (Autobase)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-causal-consensus`
|
||||||
|
|
||||||
|
**Protocol:** `autobase-fork-choice/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Multiple Autobase views compete and peers must converge on one fork id.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Single linear Autobase with no fork hints.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PAutobaseForkChoice } = require('hyper-p2p-autobase-fork-choice')
|
||||||
|
const topic = process.argv[2] // 64-char hex or string
|
||||||
|
const mod = new HyperP2PAutobaseForkChoice({ topic })
|
||||||
|
await mod.ready() // joins swarm when topic set
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# API: hyper-p2p-autobase-fork-choice
|
||||||
|
|
||||||
|
**Protocol:** `autobase-fork-choice/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PAutobaseForkChoice`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Registers Autobase view forks over gossip and picks the best by weight/version; align with `hyper-p2p-causal-consensus` for writer ordering.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PAutobaseForkChoice(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
|
||||||
|
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
|
||||||
|
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `registerView(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: version must be non-negative`
|
||||||
|
|
||||||
|
### `pickBestFork(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `listViews(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `view` | view/fork record |
|
||||||
|
| `picked` | best fork hint or null |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `version must be non-negative`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `autobase-fork-choice/v1`.
|
||||||
|
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
|
||||||
|
|
||||||
|
### Gossip message types
|
||||||
|
|
||||||
|
- `view-register`
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `registerView(forkId, version, meta)` — gossip `view-register`.
|
||||||
|
2. `pickBestFork()` — deterministic winner.
|
||||||
|
3. `listViews()` — sorted registry.
|
||||||
|
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Architecture: hyper-p2p-autobase-fork-choice
|
||||||
|
|
||||||
|
**Protocol:** `autobase-fork-choice/v1` · **P2P:** yes
|
||||||
|
|
||||||
|
## Wire
|
||||||
|
|
||||||
|
| type | payload |
|
||||||
|
|------|---------|
|
||||||
|
| `view-register` | `{ view: { forkId, version, weight, peer, at } }` |
|
||||||
|
|
||||||
|
## Selection
|
||||||
|
|
||||||
|
`pickBestFork()` — max `weight`, tie-break on `version`. `getChosenFork()` returns winning view.
|
||||||
|
|
||||||
|
## Merge
|
||||||
|
|
||||||
|
Remote view replaces local when `remote.at >= local.at` for same `forkId`.
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
Run before `autobase-light-writer.flushOps` and `autobase-indexer-bus` publish.
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PAutobaseForkChoice } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const topic = process.argv[2] || null
|
||||||
|
const m = new HyperP2PAutobaseForkChoice({ topic })
|
||||||
|
m.attach({ async append () {} })
|
||||||
|
await m.ready()
|
||||||
|
m.registerView('fork-a', 3)
|
||||||
|
console.log(m.pickBestFork())
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,102 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
|
||||||
|
const { assertAutobase, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'autobase-fork-choice/v1'
|
||||||
|
|
||||||
|
class HyperP2PAutobaseForkChoice extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.autobase = opts.autobase || null
|
||||||
|
this._views = new Map()
|
||||||
|
this._chosen = null
|
||||||
|
this._stats = { registered: 0, picks: 0, gossipIn: 0, gossipOut: 0 }
|
||||||
|
this.swarm = null
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (autobase) {
|
||||||
|
assertAutobase(autobase)
|
||||||
|
this.autobase = autobase
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
registerView (forkId, version, meta = {}) {
|
||||||
|
assertNonEmpty(forkId, 'forkId')
|
||||||
|
if (version < 0) throw new Error('version must be non-negative')
|
||||||
|
const view = {
|
||||||
|
forkId,
|
||||||
|
version,
|
||||||
|
weight: meta.weight != null ? meta.weight : version,
|
||||||
|
peer: meta.peer || null,
|
||||||
|
at: Date.now()
|
||||||
|
}
|
||||||
|
this._views.set(forkId, view)
|
||||||
|
this._stats.registered++
|
||||||
|
sendGossip(this, { type: 'view-register', view })
|
||||||
|
this._stats.gossipOut++
|
||||||
|
this.emit('view', view)
|
||||||
|
return view
|
||||||
|
}
|
||||||
|
|
||||||
|
pickBestFork () {
|
||||||
|
let best = null
|
||||||
|
for (const v of this._views.values()) {
|
||||||
|
if (!best || v.weight > best.weight || (v.weight === best.weight && v.version > best.version)) {
|
||||||
|
best = v
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this._chosen = best ? best.forkId : null
|
||||||
|
this._stats.picks++
|
||||||
|
this.emit('picked', best)
|
||||||
|
return best
|
||||||
|
}
|
||||||
|
|
||||||
|
listViews () {
|
||||||
|
return [...this._views.values()].sort((a, b) => b.weight - a.weight)
|
||||||
|
}
|
||||||
|
|
||||||
|
getChosenFork () {
|
||||||
|
if (!this._chosen) return null
|
||||||
|
return this._views.get(this._chosen) || null
|
||||||
|
}
|
||||||
|
|
||||||
|
_onGossip (d) {
|
||||||
|
if (!d || d.type !== 'view-register' || !d.view) return
|
||||||
|
this._stats.gossipIn++
|
||||||
|
const existing = this._views.get(d.view.forkId)
|
||||||
|
if (!existing || (d.view.at || 0) >= (existing.at || 0)) {
|
||||||
|
this._views.set(d.view.forkId, d.view)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
views: this._views.size,
|
||||||
|
chosen: this._chosen,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await attachGossip(this, {
|
||||||
|
keyPair: this.keyPair,
|
||||||
|
topic: this.topic,
|
||||||
|
protocol: PROTOCOL,
|
||||||
|
onmessage: (d) => this._onGossip(d)
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PAutobaseForkChoice, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-autobase-fork-choice",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Autobase fork choice helper.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.16.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"autobase": "^7.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PAutobaseForkChoice, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PAutobaseForkChoice)
|
||||||
|
t.is(PROTOCOL, 'autobase-fork-choice/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('registerView pickBestFork', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseForkChoice()
|
||||||
|
m.attach({ open: async () => {} })
|
||||||
|
m.registerView('f1', 2, { weight: 1 })
|
||||||
|
m.registerView('f2', 5, { weight: 10 })
|
||||||
|
const best = m.pickBestFork()
|
||||||
|
t.is(best.forkId, 'f2')
|
||||||
|
t.is(m.getChosenFork().forkId, 'f2')
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseForkChoice()
|
||||||
|
try { m.registerView(null, 0) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('listViews', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseForkChoice()
|
||||||
|
m.registerView('a', 1)
|
||||||
|
t.is(m.listViews().length, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseForkChoice()
|
||||||
|
t.is(m.getStats().protocol, 'autobase-fork-choice/v1')
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-autobase-indexer-bus
|
||||||
|
|
||||||
|
Indexer event bus with local subscribers and gossip `index-event` replication; compose with `hyper-p2p-distributed-event-bus` for cross-module routing. Hyperswarm gossip when `topic` is set.
|
||||||
|
|
||||||
|
**Category:** Storage (Autobase)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-distributed-event-bus`
|
||||||
|
|
||||||
|
**Protocol:** `autobase-indexer-bus/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Derived index events must be published to all Autobase indexer peers.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Indexing is local-only with no shared topic.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PAutobaseIndexerBus } = require('hyper-p2p-autobase-indexer-bus')
|
||||||
|
const topic = process.argv[2] // 64-char hex or string
|
||||||
|
const mod = new HyperP2PAutobaseIndexerBus({ topic })
|
||||||
|
await mod.ready() // joins swarm when topic set
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
# API: hyper-p2p-autobase-indexer-bus
|
||||||
|
|
||||||
|
**Protocol:** `autobase-indexer-bus/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PAutobaseIndexerBus`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Indexer event bus with local subscribers and gossip `index-event` replication; compose with `hyper-p2p-distributed-event-bus` for cross-module routing.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PAutobaseIndexerBus(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
|
||||||
|
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
|
||||||
|
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `publishIndexEvent(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `subscribeIndex(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: handler must be a function`
|
||||||
|
|
||||||
|
### `drainEvents(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: max must be non-negative`
|
||||||
|
|
||||||
|
### `pendingEvents(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `index` | indexer event |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `handler must be a function`
|
||||||
|
- `max must be non-negative`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `autobase-indexer-bus/v1`.
|
||||||
|
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
|
||||||
|
|
||||||
|
### Gossip message types
|
||||||
|
|
||||||
|
- `index-event`
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `publishIndexEvent(type, payload)` — queue + gossip + deliver.
|
||||||
|
2. `subscribeIndex(handler)` — local fanout.
|
||||||
|
3. `drainEvents(max)` — batch dequeue for processors.
|
||||||
|
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Architecture: hyper-p2p-autobase-indexer-bus
|
||||||
|
|
||||||
|
**Protocol:** `autobase-indexer-bus/v1` · **P2P:** yes
|
||||||
|
|
||||||
|
## Wire
|
||||||
|
|
||||||
|
| type | payload |
|
||||||
|
|------|---------|
|
||||||
|
| `index-event` | `{ evt: { type, payload, at, id } }` |
|
||||||
|
|
||||||
|
## Flow
|
||||||
|
|
||||||
|
`publishIndexEvent` → local subscribers + gossip → remote `_deliver`.
|
||||||
|
|
||||||
|
| Method | Purpose |
|
||||||
|
|--------|---------|
|
||||||
|
| `subscribeIndex(handler)` | Returns unsubscribe fn |
|
||||||
|
| `drainEvents(max)` | Remove from queue |
|
||||||
|
| `peekEvents(n)` | Non-destructive read |
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
`index`, `closed`
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PAutobaseIndexerBus } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const topic = process.argv[2] || null
|
||||||
|
const m = new HyperP2PAutobaseIndexerBus({ topic })
|
||||||
|
m.attach({ async append () {} })
|
||||||
|
await m.ready()
|
||||||
|
m.publishIndexEvent('indexed', { n: 1 })
|
||||||
|
console.log('pending', m.pendingEvents())
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,103 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
|
||||||
|
const { assertAutobase, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'autobase-indexer-bus/v1'
|
||||||
|
|
||||||
|
class HyperP2PAutobaseIndexerBus extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.autobase = opts.autobase || null
|
||||||
|
this._queue = []
|
||||||
|
this._subs = new Set()
|
||||||
|
this._stats = { published: 0, delivered: 0, drained: 0, gossipIn: 0, gossipOut: 0 }
|
||||||
|
this.swarm = null
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (autobase) {
|
||||||
|
assertAutobase(autobase)
|
||||||
|
this.autobase = autobase
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
publishIndexEvent (type, payload = {}) {
|
||||||
|
assertNonEmpty(type, 'type')
|
||||||
|
const evt = { type, payload, at: Date.now(), id: this._queue.length + 1 }
|
||||||
|
this._queue.push(evt)
|
||||||
|
this._stats.published++
|
||||||
|
sendGossip(this, { type: 'index-event', evt })
|
||||||
|
this._stats.gossipOut++
|
||||||
|
this._deliver(evt)
|
||||||
|
return evt
|
||||||
|
}
|
||||||
|
|
||||||
|
subscribeIndex (handler) {
|
||||||
|
if (typeof handler !== 'function') throw new Error('handler must be a function')
|
||||||
|
this._subs.add(handler)
|
||||||
|
return () => this._subs.delete(handler)
|
||||||
|
}
|
||||||
|
|
||||||
|
_deliver (evt) {
|
||||||
|
for (const h of this._subs) {
|
||||||
|
h(evt)
|
||||||
|
this._stats.delivered++
|
||||||
|
}
|
||||||
|
this.emit('index', evt)
|
||||||
|
}
|
||||||
|
|
||||||
|
drainEvents (max = 64) {
|
||||||
|
if (max < 0) throw new Error('max must be non-negative')
|
||||||
|
const out = this._queue.splice(0, max)
|
||||||
|
this._stats.drained += out.length
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
pendingEvents () {
|
||||||
|
return this._queue.length
|
||||||
|
}
|
||||||
|
|
||||||
|
peekEvents (n = 16) {
|
||||||
|
if (n < 0) throw new Error('n must be non-negative')
|
||||||
|
return this._queue.slice(0, n).map((e) => ({ ...e }))
|
||||||
|
}
|
||||||
|
|
||||||
|
_onGossip (d) {
|
||||||
|
if (!d || d.type !== 'index-event' || !d.evt) return
|
||||||
|
this._stats.gossipIn++
|
||||||
|
this._queue.push(d.evt)
|
||||||
|
this._deliver(d.evt)
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
pending: this._queue.length,
|
||||||
|
subscribers: this._subs.size,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await attachGossip(this, {
|
||||||
|
keyPair: this.keyPair,
|
||||||
|
topic: this.topic,
|
||||||
|
protocol: PROTOCOL,
|
||||||
|
onmessage: (d) => this._onGossip(d)
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._subs.clear()
|
||||||
|
this._queue = []
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PAutobaseIndexerBus, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-autobase-indexer-bus",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Indexer event bus for autobase.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.16.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"autobase": "^7.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PAutobaseIndexerBus, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PAutobaseIndexerBus)
|
||||||
|
t.is(PROTOCOL, 'autobase-indexer-bus/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('publishIndexEvent drainEvents', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseIndexerBus()
|
||||||
|
let seen = 0
|
||||||
|
m.subscribeIndex(() => { seen++ })
|
||||||
|
m.publishIndexEvent('block', { n: 1 })
|
||||||
|
const drained = m.drainEvents(10)
|
||||||
|
t.ok(drained.length >= 0)
|
||||||
|
t.ok(seen >= 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseIndexerBus()
|
||||||
|
try { m.subscribeIndex(null) } catch (e) { t.ok(e) }
|
||||||
|
try { m.publishIndexEvent(null) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('pendingEvents', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseIndexerBus()
|
||||||
|
m.publishIndexEvent('x')
|
||||||
|
t.is(m.pendingEvents(), 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseIndexerBus()
|
||||||
|
t.is(m.getStats().protocol, 'autobase-indexer-bus/v1')
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-autobase-light-writer
|
||||||
|
|
||||||
|
Batches lightweight Autobase append ops before flush; throttle bursts via `hyper-p2p-activity-queue` at the coordination layer. Local-only scheduling layer (no Hyperswarm join).
|
||||||
|
|
||||||
|
**Category:** Storage (Autobase)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-activity-queue`
|
||||||
|
|
||||||
|
**Protocol:** `autobase-light-writer/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Clients propose many small ops that should append in bounded batches.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Each op must commit immediately or you need cross-peer writer leases.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PAutobaseLightWriter } = require('hyper-p2p-autobase-light-writer')
|
||||||
|
const mod = new HyperP2PAutobaseLightWriter()
|
||||||
|
mod.attach(/* Hyperautobase instance */)
|
||||||
|
await mod.ready()
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
# API: hyper-p2p-autobase-light-writer
|
||||||
|
|
||||||
|
**Protocol:** `autobase-light-writer/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PAutobaseLightWriter`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Batches lightweight Autobase append ops before flush; throttle bursts via `hyper-p2p-activity-queue` at the coordination layer.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PAutobaseLightWriter(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
|
||||||
|
| `maxBatch` | varies | 64 | Constructor option `maxBatch` |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `proposeOp(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: op object required`
|
||||||
|
- `Error: batch full`
|
||||||
|
|
||||||
|
### `flushOps(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: autobase is required`
|
||||||
|
|
||||||
|
### `pendingOps(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `peekOps(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `propose` | staged op |
|
||||||
|
| `flush` | `{ count }` |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `op object required`
|
||||||
|
- `batch full`
|
||||||
|
- `autobase is required`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
Library-only: no swarm join. `ready()` resolves immediately.
|
||||||
|
`getStats().protocol` still reports the module protocol id for logging.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `proposeOp(op)` — stage until `maxBatch`.
|
||||||
|
2. `flushOps()` — append all staged ops to Autobase.
|
||||||
|
3. `peekOps()` / `pendingOps()` — inspect queue.
|
||||||
|
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Architecture: hyper-p2p-autobase-light-writer
|
||||||
|
|
||||||
|
**Protocol:** `autobase-light-writer/v1` · **P2P:** no
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Buffer append **proposals** then `flushOps()` to `autobase.append(op)` when attached.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
`maxBatch` (default 64) — max pending ops; `proposeOp` throws when full.
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
| Method | Effect |
|
||||||
|
|--------|--------|
|
||||||
|
| `proposeOp(op)` | Queue with `seq`, `at` |
|
||||||
|
| `flushOps()` | Append all via autobase |
|
||||||
|
| `rollbackOps()` | Clear queue |
|
||||||
|
| `peekOps` / `pendingOps` | Inspect |
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
`propose`, `flush`, `rollback`, `closed`
|
||||||
|
|
||||||
|
Requires `writer-lease` when multiple writers share one autobase.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PAutobaseLightWriter } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const m = new HyperP2PAutobaseLightWriter()
|
||||||
|
m.attach({ async append () {} })
|
||||||
|
m.proposeOp({ type: 'put', key: 'x' })
|
||||||
|
console.log('pending', m.pendingOps())
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,77 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertAutobase } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
const PROTOCOL = 'autobase-light-writer/v1'
|
||||||
|
|
||||||
|
class HyperP2PAutobaseLightWriter extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.autobase = opts.autobase || null
|
||||||
|
this.maxBatch = opts.maxBatch || 64
|
||||||
|
this._ops = []
|
||||||
|
this._stats = { proposed: 0, flushed: 0, dropped: 0 }
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (autobase) {
|
||||||
|
assertAutobase(autobase)
|
||||||
|
this.autobase = autobase
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
proposeOp (op) {
|
||||||
|
if (!op || typeof op !== 'object') throw new Error('op object required')
|
||||||
|
if (this._ops.length >= this.maxBatch) {
|
||||||
|
this._stats.dropped++
|
||||||
|
throw new Error('batch full')
|
||||||
|
}
|
||||||
|
const entry = { ...op, at: Date.now(), seq: this._ops.length }
|
||||||
|
this._ops.push(entry)
|
||||||
|
this._stats.proposed++
|
||||||
|
this.emit('propose', entry)
|
||||||
|
return entry
|
||||||
|
}
|
||||||
|
|
||||||
|
async flushOps () {
|
||||||
|
if (!this.autobase) throw new Error('autobase is required')
|
||||||
|
const ops = this._ops.splice(0, this.maxBatch)
|
||||||
|
for (const op of ops) {
|
||||||
|
if (this.autobase.append) await this.autobase.append(op)
|
||||||
|
}
|
||||||
|
this._stats.flushed += ops.length
|
||||||
|
this.emit('flush', { count: ops.length })
|
||||||
|
return ops.length
|
||||||
|
}
|
||||||
|
|
||||||
|
pendingOps () {
|
||||||
|
return this._ops.length
|
||||||
|
}
|
||||||
|
|
||||||
|
peekOps () {
|
||||||
|
return [...this._ops]
|
||||||
|
}
|
||||||
|
|
||||||
|
rollbackOps () {
|
||||||
|
const n = this._ops.length
|
||||||
|
this._ops = []
|
||||||
|
this.emit('rollback', { count: n })
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
pending: this._ops.length,
|
||||||
|
maxBatch: this.maxBatch,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () { return this }
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
this._ops = []
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PAutobaseLightWriter, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-autobase-light-writer",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Light writer pattern wrapper.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.16.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"autobase": "^7.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PAutobaseLightWriter, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PAutobaseLightWriter)
|
||||||
|
t.is(PROTOCOL, 'autobase-light-writer/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('proposeOp flushOps', async (t) => {
|
||||||
|
const appended = []
|
||||||
|
const m = new HyperP2PAutobaseLightWriter({ maxBatch: 4 })
|
||||||
|
m.attach({ append: async (op) => { appended.push(op) } })
|
||||||
|
m.proposeOp({ type: 'put' })
|
||||||
|
const n = await m.flushOps()
|
||||||
|
t.is(n, 1)
|
||||||
|
t.is(appended.length, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('batch full', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseLightWriter({ maxBatch: 1 })
|
||||||
|
m.proposeOp({ a: 1 })
|
||||||
|
try { m.proposeOp({ b: 2 }) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseLightWriter()
|
||||||
|
try { m.proposeOp(null) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseLightWriter()
|
||||||
|
m.proposeOp({ x: 1 })
|
||||||
|
t.is(m.getStats().pending, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-autobase-view-sync
|
||||||
|
|
||||||
|
Gossip-synced Autobase view version + content hash; reactive UIs can mirror via `hyper-p2p-reactive-state`. Hyperswarm gossip when `topic` is set.
|
||||||
|
|
||||||
|
**Category:** Storage (Autobase)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-reactive-state`
|
||||||
|
|
||||||
|
**Protocol:** `autobase-view-sync/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Peers must agree on the latest materialized view version/hash.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
View state is local-only or full Autobase sync replaces hash checks.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PAutobaseViewSync } = require('hyper-p2p-autobase-view-sync')
|
||||||
|
const topic = process.argv[2] // 64-char hex or string
|
||||||
|
const mod = new HyperP2PAutobaseViewSync({ topic })
|
||||||
|
await mod.ready() // joins swarm when topic set
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,111 @@
|
|||||||
|
# API: hyper-p2p-autobase-view-sync
|
||||||
|
|
||||||
|
**Protocol:** `autobase-view-sync/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PAutobaseViewSync`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Gossip-synced Autobase view version + content hash; reactive UIs can mirror via `hyper-p2p-reactive-state`.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PAutobaseViewSync(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
|
||||||
|
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
|
||||||
|
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `publishView(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: version must be non-negative`
|
||||||
|
|
||||||
|
### `mergeRemoteView(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `currentView(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `publish` | published view |
|
||||||
|
| `merged` | merged view state |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `version must be non-negative`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `autobase-view-sync/v1`.
|
||||||
|
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
|
||||||
|
|
||||||
|
### Gossip message types
|
||||||
|
|
||||||
|
- `view-sync`
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `publishView(version)` — gossip `view-sync`.
|
||||||
|
2. `mergeRemoteView(view)` — monotonic version merge.
|
||||||
|
3. `currentView()` — local snapshot.
|
||||||
|
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Architecture: hyper-p2p-autobase-view-sync
|
||||||
|
|
||||||
|
**Protocol:** `autobase-view-sync/v1` · **P2P:** yes
|
||||||
|
|
||||||
|
## Wire
|
||||||
|
|
||||||
|
| type | fields |
|
||||||
|
|------|--------|
|
||||||
|
| `view-sync` | `{ view: { version, hash, at } }` |
|
||||||
|
|
||||||
|
`hash` = `sha256("view:"+version)` (coordination digest, not Autobase proof).
|
||||||
|
|
||||||
|
## Helpers
|
||||||
|
|
||||||
|
- `isAtLeast(version)` — local caught up?
|
||||||
|
- `behindBy(version)` — blocks/versions behind target
|
||||||
|
- `mergeRemoteView` — monotonic version + hash match
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
`publish`, `merged`, `closed`
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PAutobaseViewSync } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const topic = process.argv[2] || null
|
||||||
|
const m = new HyperP2PAutobaseViewSync({ topic })
|
||||||
|
m.attach({ async append () {} })
|
||||||
|
await m.ready()
|
||||||
|
console.log(m.publishView(1))
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,98 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const crypto = require('hypercore-crypto')
|
||||||
|
const { assertAutobase, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'autobase-view-sync/v1'
|
||||||
|
|
||||||
|
class HyperP2PAutobaseViewSync extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.autobase = opts.autobase || null
|
||||||
|
this._view = { version: 0, hash: null, at: 0 }
|
||||||
|
this._stats = { published: 0, merged: 0, gossipIn: 0, gossipOut: 0 }
|
||||||
|
this.swarm = null
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (autobase) {
|
||||||
|
assertAutobase(autobase)
|
||||||
|
this.autobase = autobase
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
_hashView (version) {
|
||||||
|
return b4a.toString(crypto.hash(b4a.from(`view:${version}`)), 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
publishView (version) {
|
||||||
|
if (version < 0) throw new Error('version must be non-negative')
|
||||||
|
const view = {
|
||||||
|
version,
|
||||||
|
hash: this._hashView(version),
|
||||||
|
at: Date.now()
|
||||||
|
}
|
||||||
|
if (version >= this._view.version) this._view = view
|
||||||
|
sendGossip(this, { type: 'view-sync', view })
|
||||||
|
this._stats.published++
|
||||||
|
this._stats.gossipOut++
|
||||||
|
this.emit('publish', view)
|
||||||
|
return view
|
||||||
|
}
|
||||||
|
|
||||||
|
mergeRemoteView (view) {
|
||||||
|
if (!view || typeof view.version !== 'number') return false
|
||||||
|
if (view.version < this._view.version) return false
|
||||||
|
if (view.version === this._view.version && view.hash !== this._view.hash) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
this._view = { ...view, mergedAt: Date.now() }
|
||||||
|
this._stats.merged++
|
||||||
|
this.emit('merged', this._view)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
currentView () {
|
||||||
|
return { ...this._view }
|
||||||
|
}
|
||||||
|
|
||||||
|
isAtLeast (version) {
|
||||||
|
return this._view.version >= version
|
||||||
|
}
|
||||||
|
|
||||||
|
behindBy (version) {
|
||||||
|
if (version < 0) throw new Error('version must be non-negative')
|
||||||
|
return Math.max(0, version - this._view.version)
|
||||||
|
}
|
||||||
|
|
||||||
|
_onGossip (d) {
|
||||||
|
if (!d || d.type !== 'view-sync' || !d.view) return
|
||||||
|
this._stats.gossipIn++
|
||||||
|
this.mergeRemoteView(d.view)
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, view: this._view, protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await attachGossip(this, {
|
||||||
|
keyPair: this.keyPair,
|
||||||
|
topic: this.topic,
|
||||||
|
protocol: PROTOCOL,
|
||||||
|
onmessage: (d) => this._onGossip(d)
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PAutobaseViewSync, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-autobase-view-sync",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Autobase view synchronization.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.16.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"autobase": "^7.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PAutobaseViewSync, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PAutobaseViewSync)
|
||||||
|
t.is(PROTOCOL, 'autobase-view-sync/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('publishView mergeRemoteView', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseViewSync()
|
||||||
|
m.attach({ open: async () => {} })
|
||||||
|
m.publishView(3)
|
||||||
|
t.is(m.currentView().version, 3)
|
||||||
|
const v5 = m.publishView(5)
|
||||||
|
t.ok(m.mergeRemoteView({ version: 5, hash: v5.hash, at: Date.now() }))
|
||||||
|
t.is(m.currentView().version, 5)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('merge rejects regression', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseViewSync()
|
||||||
|
m.publishView(10)
|
||||||
|
t.not(m.mergeRemoteView({ version: 2, hash: 'x', at: 1 }))
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseViewSync()
|
||||||
|
try { m.publishView(-1) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseViewSync()
|
||||||
|
t.is(m.getStats().protocol, 'autobase-view-sync/v1')
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-autobase-writer-lease
|
||||||
|
|
||||||
|
TTL writer leases gossiped across peers; hard mutual exclusion layers on `hyper-p2p-distributed-lock` when needed. Hyperswarm gossip when `topic` is set.
|
||||||
|
|
||||||
|
**Category:** Storage (Autobase)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-distributed-lock`
|
||||||
|
|
||||||
|
**Protocol:** `autobase-writer-lease/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Only one writer should append to Autobase for a window of time.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Autobase already serializes writers or leases are enforced off-chain.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PAutobaseWriterLease } = require('hyper-p2p-autobase-writer-lease')
|
||||||
|
const topic = process.argv[2] // 64-char hex or string
|
||||||
|
const mod = new HyperP2PAutobaseWriterLease({ topic })
|
||||||
|
await mod.ready() // joins swarm when topic set
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,119 @@
|
|||||||
|
# API: hyper-p2p-autobase-writer-lease
|
||||||
|
|
||||||
|
**Protocol:** `autobase-writer-lease/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PAutobaseWriterLease`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
TTL writer leases gossiped across peers; hard mutual exclusion layers on `hyper-p2p-distributed-lock` when needed.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PAutobaseWriterLease(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
|
||||||
|
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
|
||||||
|
| `autobase` | object \| null | null | Attached autobase instance (`attach()` also supported) |
|
||||||
|
| `localWriterId` | varies | null | Constructor option `localWriterId` |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `acquireLease(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: ttlMs must be positive`
|
||||||
|
|
||||||
|
### `renewLease(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `hasLease(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `releaseLease(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `lease` | lease record |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `ttlMs must be positive`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `autobase-writer-lease/v1`.
|
||||||
|
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
|
||||||
|
|
||||||
|
### Gossip message types
|
||||||
|
|
||||||
|
- `lease-acquire`
|
||||||
|
- `lease-renew`
|
||||||
|
- `lease-release`
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `acquireLease(writerId, ttlMs)` — gossip `lease-acquire`.
|
||||||
|
2. `renewLease` / `releaseLease` — extend or drop lease.
|
||||||
|
3. `hasLease(writerId)` — check non-expired holder.
|
||||||
|
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
# Architecture: hyper-p2p-autobase-writer-lease
|
||||||
|
|
||||||
|
**Protocol:** `autobase-writer-lease/v1` · **P2P:** yes
|
||||||
|
|
||||||
|
## Wire
|
||||||
|
|
||||||
|
| type | purpose |
|
||||||
|
|------|---------|
|
||||||
|
| `lease-acquire` | New lease with `expiresAt` |
|
||||||
|
| `lease-renew` | Extend TTL |
|
||||||
|
| `lease-release` | Drop by `writerId` |
|
||||||
|
|
||||||
|
## Local API
|
||||||
|
|
||||||
|
`acquireLease`, `renewLease`, `hasLease`, `releaseLease`, `listActiveLeases`, `expireStaleLeases`
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
`lease`, `expired`, `closed`
|
||||||
|
|
||||||
|
Gate `light-writer.proposeOp` on `hasLease(localWriterId)`.
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PAutobaseWriterLease } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const topic = process.argv[2] || null
|
||||||
|
const m = new HyperP2PAutobaseWriterLease({ topic })
|
||||||
|
m.attach({ async append () {} })
|
||||||
|
await m.ready()
|
||||||
|
console.log(m.acquireLease('writer-1', 5000))
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,137 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
|
||||||
|
const { assertAutobase, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'autobase-writer-lease/v1'
|
||||||
|
|
||||||
|
class HyperP2PAutobaseWriterLease extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.autobase = opts.autobase || null
|
||||||
|
this.localWriterId = opts.writerId || null
|
||||||
|
this._leases = new Map()
|
||||||
|
this._stats = { acquired: 0, renewed: 0, released: 0, gossipIn: 0, gossipOut: 0 }
|
||||||
|
this.swarm = null
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (autobase) {
|
||||||
|
assertAutobase(autobase)
|
||||||
|
this.autobase = autobase
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
acquireLease (writerId, ttlMs = 30000) {
|
||||||
|
assertNonEmpty(writerId, 'writerId')
|
||||||
|
if (ttlMs <= 0) throw new Error('ttlMs must be positive')
|
||||||
|
const now = Date.now()
|
||||||
|
const existing = this._leases.get(writerId)
|
||||||
|
if (existing && existing.expiresAt > now && existing.holder !== writerId) {
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
const lease = { holder: writerId, acquiredAt: now, expiresAt: now + ttlMs, ttlMs }
|
||||||
|
this._leases.set(writerId, lease)
|
||||||
|
this._stats.acquired++
|
||||||
|
sendGossip(this, { type: 'lease-acquire', lease })
|
||||||
|
this._stats.gossipOut++
|
||||||
|
this.emit('lease', lease)
|
||||||
|
return lease
|
||||||
|
}
|
||||||
|
|
||||||
|
renewLease (writerId, ttlMs = 30000) {
|
||||||
|
const lease = this._leases.get(writerId)
|
||||||
|
if (!lease || lease.holder !== writerId) return false
|
||||||
|
lease.expiresAt = Date.now() + ttlMs
|
||||||
|
lease.ttlMs = ttlMs
|
||||||
|
this._stats.renewed++
|
||||||
|
sendGossip(this, { type: 'lease-renew', lease })
|
||||||
|
this._stats.gossipOut++
|
||||||
|
return lease
|
||||||
|
}
|
||||||
|
|
||||||
|
hasLease (writerId) {
|
||||||
|
const lease = this._leases.get(writerId)
|
||||||
|
if (!lease) return false
|
||||||
|
if (lease.expiresAt <= Date.now()) {
|
||||||
|
this._leases.delete(writerId)
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return lease.holder === writerId
|
||||||
|
}
|
||||||
|
|
||||||
|
listActiveLeases () {
|
||||||
|
const now = Date.now()
|
||||||
|
const out = []
|
||||||
|
for (const [id, lease] of this._leases) {
|
||||||
|
if (lease.expiresAt > now) out.push({ writerId: id, ...lease })
|
||||||
|
}
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
expireStaleLeases () {
|
||||||
|
const now = Date.now()
|
||||||
|
let n = 0
|
||||||
|
for (const [id, lease] of this._leases) {
|
||||||
|
if (lease.expiresAt <= now) {
|
||||||
|
this._leases.delete(id)
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (n) this.emit('expired', { count: n })
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
releaseLease (writerId) {
|
||||||
|
const ok = this._leases.delete(writerId)
|
||||||
|
if (ok) {
|
||||||
|
this._stats.released++
|
||||||
|
sendGossip(this, { type: 'lease-release', writerId })
|
||||||
|
this._stats.gossipOut++
|
||||||
|
}
|
||||||
|
return ok
|
||||||
|
}
|
||||||
|
|
||||||
|
_onGossip (d) {
|
||||||
|
if (!d) return
|
||||||
|
this._stats.gossipIn++
|
||||||
|
if (d.type === 'lease-acquire' && d.lease) {
|
||||||
|
const l = d.lease
|
||||||
|
const existing = this._leases.get(l.holder)
|
||||||
|
if (!existing || l.acquiredAt >= existing.acquiredAt) {
|
||||||
|
this._leases.set(l.holder, l)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (d.type === 'lease-renew' && d.lease) this._leases.set(d.lease.holder, d.lease)
|
||||||
|
if (d.type === 'lease-release') this._leases.delete(d.writerId)
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
activeLeases: this._leases.size,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await attachGossip(this, {
|
||||||
|
keyPair: this.keyPair,
|
||||||
|
topic: this.topic,
|
||||||
|
protocol: PROTOCOL,
|
||||||
|
onmessage: (d) => this._onGossip(d)
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._leases.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PAutobaseWriterLease, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-autobase-writer-lease",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Single-writer lease for autobase.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.16.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"autobase": "^7.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PAutobaseWriterLease, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PAutobaseWriterLease)
|
||||||
|
t.is(PROTOCOL, 'autobase-writer-lease/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('acquireLease hasLease releaseLease', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseWriterLease()
|
||||||
|
m.attach({ open: async () => {} })
|
||||||
|
t.ok(m.acquireLease('w1', 60000))
|
||||||
|
t.ok(m.hasLease('w1'))
|
||||||
|
t.ok(m.releaseLease('w1'))
|
||||||
|
t.not(m.hasLease('w1'))
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('renewLease', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseWriterLease()
|
||||||
|
m.acquireLease('w2', 5000)
|
||||||
|
t.ok(m.renewLease('w2', 8000))
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseWriterLease()
|
||||||
|
try { m.acquireLease(null) } catch (e) { t.ok(e) }
|
||||||
|
try { m.acquireLease('w', 0) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PAutobaseWriterLease()
|
||||||
|
t.is(m.getStats().protocol, 'autobase-writer-lease/v1')
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,34 @@
|
|||||||
|
# Storage (Hyperbee)
|
||||||
|
|
||||||
|
**Path:** `modules/storage-hyperbee/` · **Modules:** 5 (all production)
|
||||||
|
|
||||||
|
Doc hub: [`docs/storage-hyperbee/README.md`](../../docs/storage-hyperbee/README.md)
|
||||||
|
|
||||||
|
Hyperbee **application helpers** — batching, watches, indexes, tombstones, and diff cursors. These modules do not replace Hyperbee; they wrap `attach(bee)` and optional gossip via `_shared/storage-gossip-base.js`.
|
||||||
|
|
||||||
|
## Packages
|
||||||
|
|
||||||
|
| Module | Protocol | Summary |
|
||||||
|
|--------|----------|---------|
|
||||||
|
| [hyper-p2p-bee-batch-write](./hyper-p2p-bee-batch-write/) | `bee-batch-write/v1` | Buffered put/del; `maxBatch` cap |
|
||||||
|
| [hyper-p2p-bee-diff-follow](./hyper-p2p-bee-diff-follow/) | `bee-diff-follow/v1` | Version cursor + `applyBatch` |
|
||||||
|
| [hyper-p2p-bee-range-watch](./hyper-p2p-bee-range-watch/) | `bee-range-watch/v1` | Lexicographic range callbacks |
|
||||||
|
| [hyper-p2p-bee-secondary-index](./hyper-p2p-bee-secondary-index/) | `bee-secondary-index/v1` | Secondary → primary map |
|
||||||
|
| [hyper-p2p-bee-tombstone-gc](./hyper-p2p-bee-tombstone-gc/) | `bee-tombstone-gc/v1` | Tombstone + time sweep |
|
||||||
|
|
||||||
|
## Stack
|
||||||
|
|
||||||
|
```text
|
||||||
|
storage-hypercore (replicate cores backing the bee)
|
||||||
|
↓
|
||||||
|
storage-hyperbee (this category)
|
||||||
|
↓
|
||||||
|
applications / collab modules
|
||||||
|
```
|
||||||
|
|
||||||
|
## Test & demo
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd hyper-p2p-bee-range-watch && npm test
|
||||||
|
bare ../../examples/demo-storage-bee/index.js
|
||||||
|
```
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-bee-batch-write
|
||||||
|
|
||||||
|
Buffers Hyperbee put/delete ops and commits them in one batch; compose with `hyper-p2p-crdt-map` when merging batched CRDT updates across peers. Local-only scheduling layer (no Hyperswarm join).
|
||||||
|
|
||||||
|
**Category:** Storage (Hyperbee)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-crdt-map`
|
||||||
|
|
||||||
|
**Protocol:** `bee-batch-write/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Many small bee writes should flush atomically to reduce IO churn.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Single-key latency-sensitive writes without batching.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PBeeBatchWrite } = require('hyper-p2p-bee-batch-write')
|
||||||
|
const mod = new HyperP2PBeeBatchWrite()
|
||||||
|
mod.attach(/* Hyperbee instance */)
|
||||||
|
await mod.ready()
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,126 @@
|
|||||||
|
# API: hyper-p2p-bee-batch-write
|
||||||
|
|
||||||
|
**Protocol:** `bee-batch-write/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PBeeBatchWrite`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Buffers Hyperbee put/delete ops and commits them in one batch; compose with `hyper-p2p-crdt-map` when merging batched CRDT updates across peers.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PBeeBatchWrite(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `bufferPut(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: key required`
|
||||||
|
|
||||||
|
### `bufferDel(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: key required`
|
||||||
|
|
||||||
|
### `commitBatch(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: bee is required`
|
||||||
|
|
||||||
|
### `rollbackBatch(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `pendingCount(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `peekOps(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `commit` | `{ count }` |
|
||||||
|
| `rollback` | `{ count }` |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `key required`
|
||||||
|
- `bee is required`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
Library-only: no swarm join. `ready()` resolves immediately.
|
||||||
|
`getStats().protocol` still reports the module protocol id for logging.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `bufferPut` / `bufferDel` — stage ops.
|
||||||
|
2. `commitBatch()` — flush to bee (or bee.batch()).
|
||||||
|
3. `rollbackBatch()` — discard staged ops.
|
||||||
|
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
# Architecture: hyper-p2p-bee-batch-write
|
||||||
|
|
||||||
|
**Protocol:** `bee-batch-write/v1` · **P2P:** no
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Accumulate put/del operations then flush via Hyperbee `batch()` when available, else sequential `put`/`del`.
|
||||||
|
|
||||||
|
## Options
|
||||||
|
|
||||||
|
| Option | Default | Effect |
|
||||||
|
|--------|---------|--------|
|
||||||
|
| `maxBatch` | `0` | Max buffered ops; `0` = unlimited |
|
||||||
|
|
||||||
|
## Lifecycle
|
||||||
|
|
||||||
|
1. `bufferPut` / `bufferDel` — queue ops (may return `false` when full)
|
||||||
|
2. `commitBatch` — flush, emit `commit`
|
||||||
|
3. `rollbackBatch` — discard, emit `rollback`
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
Use before `bee-range-watch.notify` so watchers see committed keys only.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PBeeBatchWrite } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const m = new HyperP2PBeeBatchWrite()
|
||||||
|
m.attach({ async get () {} })
|
||||||
|
m.bufferPut('k', 'v')
|
||||||
|
console.log('pending', m.pendingCount())
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,97 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertBee } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
const PROTOCOL = 'bee-batch-write/v1'
|
||||||
|
|
||||||
|
class HyperP2PBeeBatchWrite extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.bee = opts.bee || null
|
||||||
|
this._ops = []
|
||||||
|
this.maxBatch = opts.maxBatch != null ? opts.maxBatch : 0
|
||||||
|
this._stats = { puts: 0, dels: 0, committed: 0, rolledBack: 0, rejected: 0 }
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (bee) {
|
||||||
|
assertBee(bee)
|
||||||
|
this.bee = bee
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
_canBuffer () {
|
||||||
|
if (this.maxBatch > 0 && this._ops.length >= this.maxBatch) {
|
||||||
|
this._stats.rejected++
|
||||||
|
return false
|
||||||
|
}
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
bufferPut (key, value) {
|
||||||
|
if (key == null) throw new Error('key required')
|
||||||
|
if (!this._canBuffer()) return false
|
||||||
|
this._ops.push({ op: 'put', key, value })
|
||||||
|
this._stats.puts++
|
||||||
|
return this._ops.length
|
||||||
|
}
|
||||||
|
|
||||||
|
bufferDel (key) {
|
||||||
|
if (key == null) throw new Error('key required')
|
||||||
|
if (!this._canBuffer()) return false
|
||||||
|
this._ops.push({ op: 'del', key })
|
||||||
|
this._stats.dels++
|
||||||
|
return this._ops.length
|
||||||
|
}
|
||||||
|
|
||||||
|
async commitBatch () {
|
||||||
|
if (!this.bee) throw new Error('bee is required')
|
||||||
|
if (!this._ops.length) return 0
|
||||||
|
const batch = this.bee.batch ? this.bee.batch() : null
|
||||||
|
const target = batch || this.bee
|
||||||
|
for (const op of this._ops) {
|
||||||
|
if (op.op === 'put') {
|
||||||
|
if (target.put) await target.put(op.key, op.value)
|
||||||
|
} else if (op.op === 'del' && target.del) {
|
||||||
|
await target.del(op.key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (batch && batch.flush) await batch.flush()
|
||||||
|
const n = this._ops.length
|
||||||
|
this._ops = []
|
||||||
|
this._stats.committed += n
|
||||||
|
this.emit('commit', { count: n })
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
rollbackBatch () {
|
||||||
|
const n = this._ops.length
|
||||||
|
this._ops = []
|
||||||
|
this._stats.rolledBack += n
|
||||||
|
this.emit('rollback', { count: n })
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
pendingCount () {
|
||||||
|
return this._ops.length
|
||||||
|
}
|
||||||
|
|
||||||
|
peekOps () {
|
||||||
|
return [...this._ops]
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
pending: this._ops.length,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () { return this }
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
this._ops = []
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PBeeBatchWrite, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-bee-batch-write",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Batch write coalescing for Hyperbee.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"hyperbee": "^2.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,58 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PBeeBatchWrite, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
const mockBee = {
|
||||||
|
get: async () => null,
|
||||||
|
batch () {
|
||||||
|
const puts = []
|
||||||
|
return {
|
||||||
|
put: async (k, v) => { puts.push([k, v]) },
|
||||||
|
flush: async () => puts.length
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PBeeBatchWrite)
|
||||||
|
t.is(PROTOCOL, 'bee-batch-write/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('bufferPut commitBatch', async (t) => {
|
||||||
|
const m = new HyperP2PBeeBatchWrite()
|
||||||
|
m.attach(mockBee)
|
||||||
|
m.bufferPut('a', 1)
|
||||||
|
m.bufferDel('b')
|
||||||
|
const n = await m.commitBatch()
|
||||||
|
t.is(n, 2)
|
||||||
|
t.is(m.pendingCount(), 0)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('rollbackBatch', async (t) => {
|
||||||
|
const m = new HyperP2PBeeBatchWrite()
|
||||||
|
m.bufferPut('x', 1)
|
||||||
|
t.is(m.rollbackBatch(), 1)
|
||||||
|
t.is(m.pendingCount(), 0)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PBeeBatchWrite()
|
||||||
|
try { m.bufferPut(null, 1) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PBeeBatchWrite()
|
||||||
|
m.bufferPut('k', 'v')
|
||||||
|
t.is(m.getStats().pending, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('maxBatch rejects overflow', async (t) => {
|
||||||
|
const m = new HyperP2PBeeBatchWrite({ maxBatch: 1 })
|
||||||
|
m.bufferPut('a', 1)
|
||||||
|
t.is(m.bufferPut('b', 2), false)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-bee-diff-follow
|
||||||
|
|
||||||
|
Tracks a version cursor and buffered diffs for Hyperbee followers; fan out applied entries via `hyper-p2p-distributed-event-bus` at the app layer. Local-only scheduling layer (no Hyperswarm join).
|
||||||
|
|
||||||
|
**Category:** Storage (Hyperbee)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-distributed-event-bus`
|
||||||
|
|
||||||
|
**Protocol:** `bee-diff-follow/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Incremental bee replication from a known version watermark.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Full snapshot sync only, or P2P diff transport (local buffer only).
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PBeeDiffFollow } = require('hyper-p2p-bee-diff-follow')
|
||||||
|
const mod = new HyperP2PBeeDiffFollow()
|
||||||
|
mod.attach(/* Hyperbee instance */)
|
||||||
|
await mod.ready()
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,122 @@
|
|||||||
|
# API: hyper-p2p-bee-diff-follow
|
||||||
|
|
||||||
|
**Protocol:** `bee-diff-follow/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PBeeDiffFollow`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Tracks a version cursor and buffered diffs for Hyperbee followers; fan out applied entries via `hyper-p2p-distributed-event-bus` at the app layer.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PBeeDiffFollow(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `followSince(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: version must be non-negative`
|
||||||
|
|
||||||
|
### `recordDiff(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: entry with version required`
|
||||||
|
|
||||||
|
### `pullDiff(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: max must be non-negative`
|
||||||
|
|
||||||
|
### `applyDiff(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: entry with version required`
|
||||||
|
|
||||||
|
### `diffCursor(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `follow` | cursor version number |
|
||||||
|
| `applied` | range entry |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `version must be non-negative`
|
||||||
|
- `entry with version required`
|
||||||
|
- `max must be non-negative`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
Library-only: no swarm join. `ready()` resolves immediately.
|
||||||
|
`getStats().protocol` still reports the module protocol id for logging.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `followSince(version)` — reset cursor.
|
||||||
|
2. `recordDiff(entry)` — buffer if version > cursor.
|
||||||
|
3. `pullDiff(max)` / `applyDiff(entry)` — drain and advance cursor.
|
||||||
|
|
||||||
@@ -0,0 +1,24 @@
|
|||||||
|
# Architecture: hyper-p2p-bee-diff-follow
|
||||||
|
|
||||||
|
**Protocol:** `bee-diff-follow/v1` · **P2P:** no
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Track a **version cursor** and an ordered buffer of diff entries for incremental sync.
|
||||||
|
|
||||||
|
## Flow
|
||||||
|
|
||||||
|
```text
|
||||||
|
followSince(v) → recordDiff(entry) → pullDiff(max) → applyDiff / applyBatch
|
||||||
|
```
|
||||||
|
|
||||||
|
| Method | Purpose |
|
||||||
|
|--------|---------|
|
||||||
|
| `recordDiff` | Insert if `entry.version > cursor` |
|
||||||
|
| `pullDiff` | Drain buffer up to `max` |
|
||||||
|
| `applyBatch` | Apply many entries, advance cursor |
|
||||||
|
| `clearBuffer` | Drop buffered entries without applying |
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
`follow`, `applied`, `closed`
|
||||||
@@ -0,0 +1,13 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PBeeDiffFollow } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const m = new HyperP2PBeeDiffFollow()
|
||||||
|
m.attach({ async get () {} })
|
||||||
|
m.followSince(0)
|
||||||
|
m.recordDiff({ version: 1, key: 'a' })
|
||||||
|
console.log(m.pullDiff(10))
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertBee } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
const PROTOCOL = 'bee-diff-follow/v1'
|
||||||
|
|
||||||
|
class HyperP2PBeeDiffFollow extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.bee = opts.bee || null
|
||||||
|
this._cursor = opts.sinceVersion || 0
|
||||||
|
this._buffer = []
|
||||||
|
this._stats = { follows: 0, pulled: 0, applied: 0 }
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (bee) {
|
||||||
|
assertBee(bee)
|
||||||
|
this.bee = bee
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
followSince (version = 0) {
|
||||||
|
if (version < 0) throw new Error('version must be non-negative')
|
||||||
|
this._cursor = version
|
||||||
|
this._stats.follows++
|
||||||
|
this.emit('follow', this._cursor)
|
||||||
|
return this._cursor
|
||||||
|
}
|
||||||
|
|
||||||
|
recordDiff (entry) {
|
||||||
|
if (!entry || typeof entry.version !== 'number') {
|
||||||
|
throw new Error('entry with version required')
|
||||||
|
}
|
||||||
|
if (entry.version <= this._cursor) return false
|
||||||
|
this._buffer.push(entry)
|
||||||
|
this._buffer.sort((a, b) => a.version - b.version)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
pullDiff (max = 100) {
|
||||||
|
if (max < 0) throw new Error('max must be non-negative')
|
||||||
|
const out = []
|
||||||
|
while (this._buffer.length && out.length < max) {
|
||||||
|
const next = this._buffer[0]
|
||||||
|
if (next.version <= this._cursor) {
|
||||||
|
this._buffer.shift()
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
out.push(this._buffer.shift())
|
||||||
|
}
|
||||||
|
this._stats.pulled += out.length
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
applyDiff (entry) {
|
||||||
|
if (!entry || typeof entry.version !== 'number') {
|
||||||
|
throw new Error('entry with version required')
|
||||||
|
}
|
||||||
|
if (entry.version < this._cursor) return false
|
||||||
|
this._cursor = entry.version
|
||||||
|
this._stats.applied++
|
||||||
|
this.emit('applied', entry)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
diffCursor () {
|
||||||
|
return this._cursor
|
||||||
|
}
|
||||||
|
|
||||||
|
applyBatch (entries) {
|
||||||
|
if (!Array.isArray(entries)) throw new Error('entries must be an array')
|
||||||
|
let n = 0
|
||||||
|
for (const entry of entries) {
|
||||||
|
if (this.applyDiff(entry)) n++
|
||||||
|
}
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
clearBuffer () {
|
||||||
|
const n = this._buffer.length
|
||||||
|
this._buffer = []
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
cursor: this._cursor,
|
||||||
|
buffered: this._buffer.length,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () { return this }
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
this._buffer = []
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PBeeDiffFollow, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-bee-diff-follow",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Diff stream follower.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"hyperbee": "^2.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,40 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PBeeDiffFollow, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PBeeDiffFollow)
|
||||||
|
t.is(PROTOCOL, 'bee-diff-follow/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('followSince pullDiff applyDiff', async (t) => {
|
||||||
|
const m = new HyperP2PBeeDiffFollow()
|
||||||
|
m.attach({ get: async () => null })
|
||||||
|
m.followSince(2)
|
||||||
|
m.recordDiff({ version: 3, key: 'a' })
|
||||||
|
const pulled = m.pullDiff(10)
|
||||||
|
t.is(pulled.length, 1)
|
||||||
|
m.applyDiff({ version: 3 })
|
||||||
|
t.is(m.diffCursor(), 3)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PBeeDiffFollow()
|
||||||
|
try { m.followSince(-1) } catch (e) { t.ok(e) }
|
||||||
|
try { m.applyDiff({}) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('recordDiff skips old', async (t) => {
|
||||||
|
const m = new HyperP2PBeeDiffFollow()
|
||||||
|
m.followSince(5)
|
||||||
|
t.not(m.recordDiff({ version: 3 }))
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PBeeDiffFollow()
|
||||||
|
t.is(m.getStats().protocol, 'bee-diff-follow/v1')
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-bee-range-watch
|
||||||
|
|
||||||
|
Lexicographic key-range watchers on Hyperbee with `emitChange` notifications; mirror reactive slices with `hyper-p2p-reactive-state` upstream. Local-only scheduling layer (no Hyperswarm join).
|
||||||
|
|
||||||
|
**Category:** Storage (Hyperbee)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-reactive-state`
|
||||||
|
|
||||||
|
**Protocol:** `bee-range-watch/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
UI or pipelines must react to puts/deletes inside a key span.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Whole-database watchers or P2P change fanout (local callbacks only).
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PBeeRangeWatch } = require('hyper-p2p-bee-range-watch')
|
||||||
|
const mod = new HyperP2PBeeRangeWatch()
|
||||||
|
mod.attach(/* Hyperbee instance */)
|
||||||
|
await mod.ready()
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,116 @@
|
|||||||
|
# API: hyper-p2p-bee-range-watch
|
||||||
|
|
||||||
|
**Protocol:** `bee-range-watch/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PBeeRangeWatch`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Lexicographic key-range watchers on Hyperbee with `emitChange` notifications; mirror reactive slices with `hyper-p2p-reactive-state` upstream.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PBeeRangeWatch(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `watchRange(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: gte and lte required`
|
||||||
|
- `Error: callback required`
|
||||||
|
- `Error: gte must be <= lte`
|
||||||
|
|
||||||
|
### `emitChange(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: key required`
|
||||||
|
|
||||||
|
### `unwatch(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `activeWatches(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `change` | `{ key, value, op, delivered }` |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `gte and lte required`
|
||||||
|
- `callback required`
|
||||||
|
- `gte must be <= lte`
|
||||||
|
- `key required`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
Library-only: no swarm join. `ready()` resolves immediately.
|
||||||
|
`getStats().protocol` still reports the module protocol id for logging.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `watchRange(gte, lte, cb)` — register callback.
|
||||||
|
2. `emitChange(key, value, op)` — deliver to matching watches.
|
||||||
|
3. `unwatch(id)` — remove listener.
|
||||||
|
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Architecture: hyper-p2p-bee-range-watch
|
||||||
|
|
||||||
|
**Protocol:** `bee-range-watch/v1` · **P2P:** no
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Deliver in-process notifications when keys change inside a **lexicographic span** `[gte, lte]`. Call `notify` / `emitChange` from your bee put/del hooks.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
|-------|-------------|
|
||||||
|
| `_watches` | `id → { gte, lte, cb }` |
|
||||||
|
| `_nextId` | Monotonic watch id |
|
||||||
|
|
||||||
|
## API aliases
|
||||||
|
|
||||||
|
- `watch(gte, lte, cb)` → `watchRange`
|
||||||
|
- `notify(key, value, op)` → `emitChange`
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
`change` (with `delivered` count), `closed`
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
`hyper-p2p-reactive-state`, `hyper-p2p-bee-batch-write` (commit then notify watchers).
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PBeeRangeWatch } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const m = new HyperP2PBeeRangeWatch()
|
||||||
|
m.attach({ async get () {} })
|
||||||
|
m.watchRange('a', 'z', (c) => console.log('watch', c.key))
|
||||||
|
m.emitChange('m', 'v')
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertBee } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
const PROTOCOL = 'bee-range-watch/v1'
|
||||||
|
|
||||||
|
class HyperP2PBeeRangeWatch extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.bee = opts.bee || null
|
||||||
|
this._watches = new Map()
|
||||||
|
this._nextId = 1
|
||||||
|
this._stats = { watches: 0, notifications: 0, unwatch: 0 }
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (bee) {
|
||||||
|
assertBee(bee)
|
||||||
|
this.bee = bee
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
watch (gte, lte, cb) {
|
||||||
|
return this.watchRange(gte, lte, cb)
|
||||||
|
}
|
||||||
|
|
||||||
|
watchRange (gte, lte, cb) {
|
||||||
|
if (gte == null || lte == null) throw new Error('gte and lte required')
|
||||||
|
if (typeof cb !== 'function') throw new Error('callback required')
|
||||||
|
if (gte > lte) throw new Error('gte must be <= lte')
|
||||||
|
const id = this._nextId++
|
||||||
|
this._watches.set(id, { gte, lte, cb })
|
||||||
|
this._stats.watches++
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
|
||||||
|
notify (key, value, op = 'put') {
|
||||||
|
return this.emitChange(key, value, op)
|
||||||
|
}
|
||||||
|
|
||||||
|
emitChange (key, value, op = 'put') {
|
||||||
|
if (key == null) throw new Error('key required')
|
||||||
|
let n = 0
|
||||||
|
for (const w of this._watches.values()) {
|
||||||
|
if (key >= w.gte && key <= w.lte) {
|
||||||
|
w.cb({ key, value, op, at: Date.now() })
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this._stats.notifications += n
|
||||||
|
this.emit('change', { key, value, op, delivered: n })
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
unwatch (id) {
|
||||||
|
const ok = this._watches.delete(id)
|
||||||
|
if (ok) this._stats.unwatch++
|
||||||
|
return ok
|
||||||
|
}
|
||||||
|
|
||||||
|
activeWatches () {
|
||||||
|
return this._watches.size
|
||||||
|
}
|
||||||
|
|
||||||
|
listWatches () {
|
||||||
|
return [...this._watches.entries()].map(([id, w]) => ({
|
||||||
|
id, gte: w.gte, lte: w.lte
|
||||||
|
}))
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
active: this._watches.size,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () { return this }
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
this._watches.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PBeeRangeWatch, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-bee-range-watch",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Range watch notifications on Hyperbee.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"hyperbee": "^2.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PBeeRangeWatch, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PBeeRangeWatch)
|
||||||
|
t.is(PROTOCOL, 'bee-range-watch/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('watchRange emitChange', async (t) => {
|
||||||
|
const m = new HyperP2PBeeRangeWatch()
|
||||||
|
let hit = 0
|
||||||
|
m.watchRange('a', 'm', () => { hit++ })
|
||||||
|
m.emitChange('b', 42)
|
||||||
|
t.is(hit, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('unwatch', async (t) => {
|
||||||
|
const m = new HyperP2PBeeRangeWatch()
|
||||||
|
const id = m.watchRange('a', 'z', () => {})
|
||||||
|
t.ok(m.unwatch(id))
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PBeeRangeWatch()
|
||||||
|
try { m.watchRange('z', 'a', () => {}) } catch (e) { t.ok(e) }
|
||||||
|
try { m.watchRange('a', 'z', null) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PBeeRangeWatch()
|
||||||
|
m.watchRange('a', 'c', () => {})
|
||||||
|
t.is(m.getStats().active, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('watch notify aliases listWatches', async (t) => {
|
||||||
|
const m = new HyperP2PBeeRangeWatch()
|
||||||
|
let hit = 0
|
||||||
|
m.watch('a', 'z', () => { hit++ })
|
||||||
|
m.notify('b', 1)
|
||||||
|
t.is(hit, 1)
|
||||||
|
t.is(m.listWatches().length, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-bee-secondary-index
|
||||||
|
|
||||||
|
In-memory secondary-key → primary-key index for Hyperbee; route lookup intents through `hyper-p2p-intent-router` when queries cross peers. Local-only scheduling layer (no Hyperswarm join).
|
||||||
|
|
||||||
|
**Category:** Storage (Hyperbee)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-intent-router`
|
||||||
|
|
||||||
|
**Protocol:** `bee-secondary-index/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Queries need a non-primary key path into bee records.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Primary-key-only access patterns or indexes persisted inside bee itself.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PBeeSecondaryIndex } = require('hyper-p2p-bee-secondary-index')
|
||||||
|
const mod = new HyperP2PBeeSecondaryIndex()
|
||||||
|
mod.attach(/* Hyperbee instance */)
|
||||||
|
await mod.ready()
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,118 @@
|
|||||||
|
# API: hyper-p2p-bee-secondary-index
|
||||||
|
|
||||||
|
**Protocol:** `bee-secondary-index/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PBeeSecondaryIndex`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
In-memory secondary-key → primary-key index for Hyperbee; route lookup intents through `hyper-p2p-intent-router` when queries cross peers.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PBeeSecondaryIndex(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
|
||||||
|
| `indexName` | varies | 'default' | Constructor option `indexName` |
|
||||||
|
| `keyPrefix` | varies | `@idx/${this.indexName}/` | Constructor option `keyPrefix` |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `indexPut(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `indexLookup(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `indexRemove(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `rebuildIndex(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: pairs must be an array`
|
||||||
|
|
||||||
|
### `listSecondaryKeys(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `index` | indexer event |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `pairs must be an array`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
Library-only: no swarm join. `ready()` resolves immediately.
|
||||||
|
`getStats().protocol` still reports the module protocol id for logging.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `indexPut(secondaryKey, primaryKey)` — maintain Set per secondary key.
|
||||||
|
2. `indexLookup(secondaryKey)` — return primary keys.
|
||||||
|
3. `rebuildIndex(pairs)` — cold rebuild from `{ secondaryKey, primaryKey }` rows.
|
||||||
|
|
||||||
@@ -0,0 +1,20 @@
|
|||||||
|
# Architecture: hyper-p2p-bee-secondary-index
|
||||||
|
|
||||||
|
**Protocol:** `bee-secondary-index/v1` · **P2P:** no
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
In-memory **secondary → Set(primary)** index with optional `keyPrefix` for bee key namespacing (`@idx/<name>/`).
|
||||||
|
|
||||||
|
## Operations
|
||||||
|
|
||||||
|
- `indexPut` / `indexLookup` / `indexRemove`
|
||||||
|
- `rebuildIndex(pairs)` — bulk load
|
||||||
|
- `hasSecondary` / `countPrimaries` — introspection
|
||||||
|
- `listSecondaryKeys`
|
||||||
|
|
||||||
|
Persist to Hyperbee in your app by writing `_secKey(secondary)` entries after `indexPut`.
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
Pair with `bee-batch-write` for atomic index + row updates.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PBeeSecondaryIndex } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const m = new HyperP2PBeeSecondaryIndex()
|
||||||
|
m.attach({ async get () {} })
|
||||||
|
m.indexPut('sec', 'pri')
|
||||||
|
console.log(m.indexLookup('sec'))
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertBee } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
|
||||||
|
const PROTOCOL = 'bee-secondary-index/v1'
|
||||||
|
|
||||||
|
class HyperP2PBeeSecondaryIndex extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.bee = opts.bee || null
|
||||||
|
this.indexName = opts.indexName || 'default'
|
||||||
|
this.keyPrefix = opts.keyPrefix || `@idx/${this.indexName}/`
|
||||||
|
this._index = new Map()
|
||||||
|
this._stats = { indexed: 0, lookups: 0, removed: 0, rebuilt: 0 }
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (bee) {
|
||||||
|
assertBee(bee)
|
||||||
|
this.bee = bee
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
_secKey (secondaryKey) {
|
||||||
|
return `${this.keyPrefix}${secondaryKey}`
|
||||||
|
}
|
||||||
|
|
||||||
|
indexPut (secondaryKey, primaryKey) {
|
||||||
|
assertNonEmpty(secondaryKey, 'secondaryKey')
|
||||||
|
assertNonEmpty(primaryKey, 'primaryKey')
|
||||||
|
const set = this._index.get(secondaryKey) || new Set()
|
||||||
|
set.add(primaryKey)
|
||||||
|
this._index.set(secondaryKey, set)
|
||||||
|
this._stats.indexed++
|
||||||
|
this.emit('index', { secondaryKey, primaryKey })
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
indexLookup (secondaryKey) {
|
||||||
|
this._stats.lookups++
|
||||||
|
const set = this._index.get(secondaryKey)
|
||||||
|
return set ? [...set] : []
|
||||||
|
}
|
||||||
|
|
||||||
|
indexRemove (secondaryKey, primaryKey) {
|
||||||
|
const set = this._index.get(secondaryKey)
|
||||||
|
if (!set || !set.has(primaryKey)) return false
|
||||||
|
set.delete(primaryKey)
|
||||||
|
if (!set.size) this._index.delete(secondaryKey)
|
||||||
|
this._stats.removed++
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
async rebuildIndex (pairs) {
|
||||||
|
if (!Array.isArray(pairs)) throw new Error('pairs must be an array')
|
||||||
|
this._index.clear()
|
||||||
|
for (const { secondaryKey, primaryKey } of pairs) {
|
||||||
|
this.indexPut(secondaryKey, primaryKey)
|
||||||
|
}
|
||||||
|
this._stats.rebuilt++
|
||||||
|
return this._index.size
|
||||||
|
}
|
||||||
|
|
||||||
|
listSecondaryKeys () {
|
||||||
|
return [...this._index.keys()]
|
||||||
|
}
|
||||||
|
|
||||||
|
hasSecondary (secondaryKey) {
|
||||||
|
return this._index.has(secondaryKey)
|
||||||
|
}
|
||||||
|
|
||||||
|
countPrimaries (secondaryKey) {
|
||||||
|
const set = this._index.get(secondaryKey)
|
||||||
|
return set ? set.size : 0
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
secondaryKeys: this._index.size,
|
||||||
|
indexName: this.indexName,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () { return this }
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
this._index.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PBeeSecondaryIndex, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-bee-secondary-index",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Secondary index maintenance.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"hyperbee": "^2.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,38 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PBeeSecondaryIndex, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PBeeSecondaryIndex)
|
||||||
|
t.is(PROTOCOL, 'bee-secondary-index/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('indexPut indexLookup', async (t) => {
|
||||||
|
const m = new HyperP2PBeeSecondaryIndex()
|
||||||
|
m.attach({ get: async () => null })
|
||||||
|
m.indexPut('tag:a', 'pk-1')
|
||||||
|
m.indexPut('tag:a', 'pk-2')
|
||||||
|
t.is(m.indexLookup('tag:a').length, 2)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('indexRemove', async (t) => {
|
||||||
|
const m = new HyperP2PBeeSecondaryIndex()
|
||||||
|
m.indexPut('s', 'p')
|
||||||
|
t.ok(m.indexRemove('s', 'p'))
|
||||||
|
t.is(m.indexLookup('s').length, 0)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PBeeSecondaryIndex()
|
||||||
|
try { m.indexPut(null, 'p') } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('rebuildIndex', async (t) => {
|
||||||
|
const m = new HyperP2PBeeSecondaryIndex()
|
||||||
|
const n = await m.rebuildIndex([{ secondaryKey: 'b', primaryKey: '1' }])
|
||||||
|
t.is(n, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,41 @@
|
|||||||
|
# hyper-p2p-bee-tombstone-gc
|
||||||
|
|
||||||
|
Soft-delete tombstones with time-based `gcSweep`; pair with `hyper-p2p-temporal-index` when retention policies are time-ordered. Local-only scheduling layer (no Hyperswarm join).
|
||||||
|
|
||||||
|
**Category:** Storage (Hyperbee)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-temporal-index`
|
||||||
|
|
||||||
|
**Protocol:** `bee-tombstone-gc/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Deletes should linger for undo/audit before physical removal.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Immediate hard deletes or distributed GC coordination (local map only).
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PBeeTombstoneGc } = require('hyper-p2p-bee-tombstone-gc')
|
||||||
|
const mod = new HyperP2PBeeTombstoneGc()
|
||||||
|
mod.attach(/* Hyperbee instance */)
|
||||||
|
await mod.ready()
|
||||||
|
// ... application logic ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,120 @@
|
|||||||
|
# API: hyper-p2p-bee-tombstone-gc
|
||||||
|
|
||||||
|
**Protocol:** `bee-tombstone-gc/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PBeeTombstoneGc`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Soft-delete tombstones with time-based `gcSweep`; pair with `hyper-p2p-temporal-index` when retention policies are time-ordered.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PBeeTombstoneGc(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `bee` | object \| null | null | Attached bee instance (`attach()` also supported) |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `tombstone(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: key required`
|
||||||
|
|
||||||
|
### `gcSweep(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:**
|
||||||
|
- `Error: olderThanMs must be non-negative`
|
||||||
|
|
||||||
|
### `listTombstones(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `restore(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `pendingTombstones(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `getStats(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `ready(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
### `close(…)`
|
||||||
|
|
||||||
|
- **Returns:** module-specific (see implementation)
|
||||||
|
|
||||||
|
- **Throws:** — (none in method body)
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `tombstone` | `{ key }` |
|
||||||
|
| `sweep` | `{ removed }` |
|
||||||
|
| `restore` | `{ key }` |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||||
|
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||||
|
|
||||||
|
### Documented `throw new Error(...)` strings
|
||||||
|
|
||||||
|
- `key required`
|
||||||
|
- `olderThanMs must be non-negative`
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
Library-only: no swarm join. `ready()` resolves immediately.
|
||||||
|
`getStats().protocol` still reports the module protocol id for logging.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `tombstone(key, meta)` — mark pending delete.
|
||||||
|
2. `gcSweep(olderThanMs)` — purge expired tombstones.
|
||||||
|
3. `restore(key)` — cancel tombstone before sweep.
|
||||||
|
|
||||||
@@ -0,0 +1,26 @@
|
|||||||
|
# Architecture: hyper-p2p-bee-tombstone-gc
|
||||||
|
|
||||||
|
**Protocol:** `bee-tombstone-gc/v1` · **P2P:** no
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Soft-delete registry: mark keys tombstoned, sweep after `olderThanMs`, optional `restore`.
|
||||||
|
|
||||||
|
## State
|
||||||
|
|
||||||
|
`Map<key, { at, meta }>`
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
| Method | Effect |
|
||||||
|
|--------|--------|
|
||||||
|
| `tombstone(key, meta?)` | Mark deleted |
|
||||||
|
| `isTombstoned(key)` | Query |
|
||||||
|
| `gcSweep(olderThanMs)` | Remove stale entries |
|
||||||
|
| `listTombstones` / `restore` | Admin |
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
`tombstone`, `sweep`, `restore`, `closed`
|
||||||
|
|
||||||
|
Run physical `bee.del` in your app after sweep returns keys.
|
||||||
@@ -0,0 +1,12 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PBeeTombstoneGc } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const m = new HyperP2PBeeTombstoneGc()
|
||||||
|
m.attach({ async get () {} })
|
||||||
|
m.tombstone('old-key')
|
||||||
|
console.log('pending', m.pendingTombstones())
|
||||||
|
await m.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,79 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const { assertBee } = require('../../_shared/storage-gossip-base.js')
|
||||||
|
const PROTOCOL = 'bee-tombstone-gc/v1'
|
||||||
|
|
||||||
|
class HyperP2PBeeTombstoneGc extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this.bee = opts.bee || null
|
||||||
|
this._tombstones = new Map()
|
||||||
|
this._stats = { tombstoned: 0, swept: 0, restored: 0 }
|
||||||
|
}
|
||||||
|
|
||||||
|
attach (bee) {
|
||||||
|
assertBee(bee)
|
||||||
|
this.bee = bee
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
tombstone (key, meta = {}) {
|
||||||
|
if (key == null) throw new Error('key required')
|
||||||
|
this._tombstones.set(key, { at: Date.now(), meta })
|
||||||
|
this._stats.tombstoned++
|
||||||
|
this.emit('tombstone', { key })
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
gcSweep (olderThanMs = 86400000) {
|
||||||
|
if (olderThanMs < 0) throw new Error('olderThanMs must be non-negative')
|
||||||
|
const now = Date.now()
|
||||||
|
const removed = []
|
||||||
|
for (const [key, entry] of this._tombstones) {
|
||||||
|
if (now - entry.at >= olderThanMs) {
|
||||||
|
this._tombstones.delete(key)
|
||||||
|
removed.push(key)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
this._stats.swept += removed.length
|
||||||
|
this.emit('sweep', { removed, count: removed.length })
|
||||||
|
return removed
|
||||||
|
}
|
||||||
|
|
||||||
|
listTombstones () {
|
||||||
|
return [...this._tombstones.entries()].map(([key, entry]) => ({ key, ...entry }))
|
||||||
|
}
|
||||||
|
|
||||||
|
restore (key) {
|
||||||
|
if (!this._tombstones.has(key)) return false
|
||||||
|
this._tombstones.delete(key)
|
||||||
|
this._stats.restored++
|
||||||
|
this.emit('restore', { key })
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
pendingTombstones () {
|
||||||
|
return this._tombstones.size
|
||||||
|
}
|
||||||
|
|
||||||
|
isTombstoned (key) {
|
||||||
|
return this._tombstones.has(key)
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
pending: this._tombstones.size,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () { return this }
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
this._tombstones.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PBeeTombstoneGc, PROTOCOL }
|
||||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-bee-tombstone-gc",
|
||||||
|
"version": "0.3.1",
|
||||||
|
"description": "Tombstone garbage collection policy.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": {
|
||||||
|
"hyperswarm": "^4.0.0",
|
||||||
|
"bare": ">=1.0.0",
|
||||||
|
"hyperbee": "^2.0.0"
|
||||||
|
},
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PBeeTombstoneGc, PROTOCOL } = require('../index.js')
|
||||||
|
|
||||||
|
test('exports', (t) => {
|
||||||
|
t.ok(HyperP2PBeeTombstoneGc)
|
||||||
|
t.is(PROTOCOL, 'bee-tombstone-gc/v1')
|
||||||
|
})
|
||||||
|
|
||||||
|
test('tombstone gcSweep', async (t) => {
|
||||||
|
const m = new HyperP2PBeeTombstoneGc()
|
||||||
|
m.tombstone('old-key')
|
||||||
|
m._tombstones.set('gone', { at: 0 })
|
||||||
|
const removed = m.gcSweep(0)
|
||||||
|
t.is(removed.length, 2)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('restore', async (t) => {
|
||||||
|
const m = new HyperP2PBeeTombstoneGc()
|
||||||
|
m.tombstone('k')
|
||||||
|
t.ok(m.restore('k'))
|
||||||
|
t.is(m.pendingTombstones(), 0)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('validation', async (t) => {
|
||||||
|
const m = new HyperP2PBeeTombstoneGc()
|
||||||
|
try { m.tombstone(null) } catch (e) { t.ok(e) }
|
||||||
|
try { m.gcSweep(-1) } catch (e) { t.ok(e) }
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('getStats', async (t) => {
|
||||||
|
const m = new HyperP2PBeeTombstoneGc()
|
||||||
|
m.tombstone('x')
|
||||||
|
t.is(m.getStats().pending, 1)
|
||||||
|
await m.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,49 @@
|
|||||||
|
# Storage (Hypercore)
|
||||||
|
|
||||||
|
**Path:** `modules/storage-hypercore/` · **Modules:** 7 (all production)
|
||||||
|
|
||||||
|
Doc hub: [`docs/storage-hypercore/README.md`](../../docs/storage-hypercore/README.md) · Index: [MODULE_CATEGORIES.md](../MODULE_CATEGORIES.md#storage-hypercore)
|
||||||
|
|
||||||
|
Hypercore **coordination** primitives — not a replacement for `hypercore` replication. Modules either gossip policy/range/fork state on a Hyperswarm topic or run **local planners** (priority fetch, bitfield scheduler, audit chain) that your downloader attaches to a core instance.
|
||||||
|
|
||||||
|
## Packages
|
||||||
|
|
||||||
|
| Module | Protocol | P2P gossip | Summary |
|
||||||
|
|--------|----------|:----------:|---------|
|
||||||
|
| [hyper-p2p-core-replicator](./hyper-p2p-core-replicator/) | `core-replicator/v1` | yes | Per-peer `maxBytes`, pause flags; `policy-sync` merge |
|
||||||
|
| [hyper-p2p-core-seed-policy](./hyper-p2p-core-seed-policy/) | `core-seed-policy/v1` | yes | Allow/deny seeding, `maxBlocks`, default deny |
|
||||||
|
| [hyper-p2p-core-fork-picker](./hyper-p2p-core-fork-picker/) | `core-fork-picker/v1` | yes | Fork hints by weight/length; `fork-register` gossip |
|
||||||
|
| [hyper-p2p-core-merkle-sync](./hyper-p2p-core-merkle-sync/) | `core-merkle-sync/v1` | yes | Range roots, `diffRanges`, `merkle-ranges` publish |
|
||||||
|
| [hyper-p2p-core-priority-fetch](./hyper-p2p-core-priority-fetch/) | `core-priority-fetch/v1` | no | Max-heap block index queue with inflight tracking |
|
||||||
|
| [hyper-p2p-core-bitfield-scheduler](./hyper-p2p-core-bitfield-scheduler/) | `core-bitfield-scheduler/v1` | no | Priority ranges `[start, len)` for bitfield work |
|
||||||
|
| [hyper-p2p-core-audit-chain](./hyper-p2p-core-audit-chain/) | `core-audit-chain/v1` | no | Hash-linked audit entries; `verifyChain` / export |
|
||||||
|
|
||||||
|
## Stack placement
|
||||||
|
|
||||||
|
```text
|
||||||
|
Application / drive layer
|
||||||
|
↓
|
||||||
|
storage-hyperdrive / storage-hyperbee (optional)
|
||||||
|
↓
|
||||||
|
storage-hypercore (this category) — policy + planners
|
||||||
|
↓
|
||||||
|
hypercore instance + Hyperswarm (via _shared/p2p-bare.js)
|
||||||
|
```
|
||||||
|
|
||||||
|
## Shared runtime
|
||||||
|
|
||||||
|
- [`../_shared/storage-gossip-base.js`](../_shared/storage-gossip-base.js) — `attach(core)`, `attachGossip`, `sendGossip`
|
||||||
|
- [`../_shared/p2p-bare.js`](../_shared/p2p-bare.js) — swarm + Protomux when `topic` is set
|
||||||
|
|
||||||
|
## Quick test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd hyper-p2p-core-replicator && npm install && npm test
|
||||||
|
bare ../../examples/demo-storage-core/index.js
|
||||||
|
```
|
||||||
|
|
||||||
|
## Related categories
|
||||||
|
|
||||||
|
- [storage-hyperbee](../storage-hyperbee/) — bee batch, range watch, tombstones
|
||||||
|
- [storage-hyperdrive](../storage-hyperdrive/) — drive catalog, mirror, GC
|
||||||
|
- [network-stack](../network-stack/) — transport underlay for large transfers
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## [0.0.0-scaffold]
|
||||||
|
|
||||||
|
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# hyper-p2p-core-audit-chain
|
||||||
|
|
||||||
|
Append-only hash-linked audit log over a Hypercore. Local-only (no Hyperswarm topic).
|
||||||
|
|
||||||
|
**Category:** Storage (Hypercore)
|
||||||
|
|
||||||
|
**Composes with:** `hyper-p2p-attestation-chain`
|
||||||
|
|
||||||
|
**Protocol:** `core-audit-chain/v1`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Apps that need tamper-evident audit trails tied to core length (compliance, replication forensics).
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
When you only need unstructured logs without hash chaining (use observability modules).
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PCoreAuditChain } = require('hyper-p2p-core-audit-chain')
|
||||||
|
const mod = new HyperP2PCoreAuditChain()
|
||||||
|
mod.attach(core)
|
||||||
|
mod.appendAudit({ op: 'write', actor: 'peer-1' })
|
||||||
|
console.log(mod.verifyChain())
|
||||||
|
console.log(mod.tail(5))
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||||
|
- [docs/architecture.md](docs/architecture.md) — chain model, verification
|
||||||
|
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||||
|
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,85 @@
|
|||||||
|
# API: hyper-p2p-core-audit-chain
|
||||||
|
|
||||||
|
**Protocol:** `core-audit-chain/v1`
|
||||||
|
|
||||||
|
**Export:** `HyperP2PCoreAuditChain`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Append-only hash-linked audit log over a Hypercore. Each entry includes `prevHash`, `hash`, `seq`, and optional `coreLength` snapshot.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PCoreAuditChain(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `core` | Hypercore | null | Optional core attached at construct time |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
### `attach(core)`
|
||||||
|
|
||||||
|
- **Returns:** `this`
|
||||||
|
- **Throws:** `Error: core must be a Hypercore instance`
|
||||||
|
|
||||||
|
### `appendAudit(event)`
|
||||||
|
|
||||||
|
- **Parameters:** `event` — plain object (domain fields)
|
||||||
|
- **Returns:** entry with `seq`, `hash`, `prevHash`, `at`
|
||||||
|
- **Throws:** `Error: event object required`
|
||||||
|
|
||||||
|
### `verifyChain()`
|
||||||
|
|
||||||
|
- **Returns:** `{ ok: true, length, head }` or `{ ok: false, seq, reason }`
|
||||||
|
- **Throws:** —
|
||||||
|
|
||||||
|
### `tail(n = 10)`
|
||||||
|
|
||||||
|
- **Returns:** last `n` entries
|
||||||
|
- **Throws:** `Error: n must be non-negative`
|
||||||
|
|
||||||
|
### `getStats()`
|
||||||
|
|
||||||
|
- **Returns:** `{ appended, verified, failures, length, head, coreLength, protocol }`
|
||||||
|
|
||||||
|
### `ready()`
|
||||||
|
|
||||||
|
- **Returns:** `Promise<this>` (no-op)
|
||||||
|
|
||||||
|
### `close()`
|
||||||
|
|
||||||
|
- **Returns:** `Promise<void>` — clears chain
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `audit` | appended entry |
|
||||||
|
| `closed` | no payload |
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
See fields above. `protocol` is always `core-audit-chain/v1`.
|
||||||
|
|
||||||
|
## Errors
|
||||||
|
|
||||||
|
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
Local-only — no `topic` or gossip wire messages.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
|
|
||||||
|
## Common flows
|
||||||
|
|
||||||
|
1. `attach(core)` → `appendAudit` for each mutation → periodic `verifyChain()`
|
||||||
|
2. `tail(n)` for dashboards without scanning full chain
|
||||||
|
3. `close()` on shutdown to drop in-memory chain
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# Architecture: hyper-p2p-core-audit-chain
|
||||||
|
|
||||||
|
**Category:** Storage (Hypercore) · **Protocol:** `core-audit-chain/v1` · **P2P:** no
|
||||||
|
|
||||||
|
## Role
|
||||||
|
|
||||||
|
Append-only **hash-linked audit log** for operations on a Hypercore (replication decisions, admin actions, policy changes). Each entry stores `prevHash`, `hash`, `seq`, and `coreLength` at append time for correlation with chain state.
|
||||||
|
|
||||||
|
## Hash algorithm
|
||||||
|
|
||||||
|
`hash = sha256(JSON.stringify({ prevHash, event }))` via `hypercore-crypto` (hex string). Verification walks the chain recomputing hashes; mismatch returns `{ ok: false, seq, reason }`.
|
||||||
|
|
||||||
|
## State model
|
||||||
|
|
||||||
|
| Field | Description |
|
||||||
|
|-------|-------------|
|
||||||
|
| `_chain` | Ordered entries with metadata |
|
||||||
|
| `_headHash` | Latest hash or `null` |
|
||||||
|
|
||||||
|
## API surface
|
||||||
|
|
||||||
|
| Method | Purpose |
|
||||||
|
|--------|---------|
|
||||||
|
| `appendAudit(event)` | Push linked entry; emit `audit` |
|
||||||
|
| `verifyChain()` | Full integrity check |
|
||||||
|
| `tail(n)` | Last `n` entries |
|
||||||
|
| `exportChain()` / `importChain(entries)` | Snapshot / restore for persistence |
|
||||||
|
|
||||||
|
## Wire messages
|
||||||
|
|
||||||
|
None. Export JSON and ship over your app channel if peers need shared audit history.
|
||||||
|
|
||||||
|
## Events
|
||||||
|
|
||||||
|
| Event | Payload |
|
||||||
|
|-------|---------|
|
||||||
|
| `audit` | New entry |
|
||||||
|
| `closed` | Chain cleared |
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
- Log replication policy changes from `hyper-p2p-core-replicator`
|
||||||
|
- Log fork picks from `hyper-p2p-core-fork-picker`
|
||||||
|
- Persist `exportChain()` to Hyperbee or Hyperdrive for durability
|
||||||
|
|
||||||
|
## Threat model
|
||||||
|
|
||||||
|
Detects tampering of in-memory chain only. Does not sign entries — add application-level signatures if non-repudiation is required.
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user