Add storage stack modules and track them in git
The modules workspace previously ignored every path matching `storage-*`, which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out of version control. Narrow .gitignore to test-artifact patterns only so production category trees are committed. Storage packages (23 total): - storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync, priority-fetch, bitfield-scheduler, audit-chain - storage-hyperbee (5): batch-write, diff-follow, range-watch, secondary-index, tombstone-gc - storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge, version-snapshot, watch-notify - storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus, light-writer Implementation highlights: - Shared attach/gossip helpers in _shared/storage-gossip-base.js - P2P modules use Protomux gossip via p2p-bare; local planners omit swarm - Recent deepen pass: priority queues, batch caps, audit export/import, watch/notify aliases on range-watch, fork/view lease helpers, etc. - Per-package README, docs/api.md, docs/architecture.md, tests, examples Also update modules/README.md doc hub links for the full storage stack (hypercore, hyperbee, hyperdrive, autobase). Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Raven Scott
@@ -0,0 +1,5 @@
|
||||
# Changelog
|
||||
|
||||
## [0.0.0-scaffold]
|
||||
|
||||
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||
@@ -0,0 +1,41 @@
|
||||
# hyper-p2p-core-seed-policy
|
||||
|
||||
Per-peer seeding policies (allow, max blocks, priority) synced over gossip; use with `hyper-p2p-presence` to map policy peers to live connections. Hyperswarm gossip when `topic` is set.
|
||||
|
||||
**Category:** Storage (Hypercore)
|
||||
|
||||
**Composes with:** `hyper-p2p-presence`
|
||||
|
||||
**Protocol:** `core-seed-policy/v1`
|
||||
|
||||
## When to use
|
||||
|
||||
You must gate which peers may seed blocks and cap their contribution.
|
||||
|
||||
## When not to use
|
||||
|
||||
Open replication to all peers or policies managed entirely off-core.
|
||||
|
||||
## Quick start
|
||||
|
||||
```js
|
||||
const { HyperP2PCoreSeedPolicy } = require('hyper-p2p-core-seed-policy')
|
||||
const topic = process.argv[2] // 64-char hex or string
|
||||
const mod = new HyperP2PCoreSeedPolicy({ topic })
|
||||
await mod.ready() // joins swarm when topic set
|
||||
// ... application logic ...
|
||||
await mod.close()
|
||||
```
|
||||
|
||||
## Docs
|
||||
|
||||
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
npm install && npm test
|
||||
```
|
||||
@@ -0,0 +1,117 @@
|
||||
# API: hyper-p2p-core-seed-policy
|
||||
|
||||
**Protocol:** `core-seed-policy/v1`
|
||||
|
||||
**Export:** `HyperP2PCoreSeedPolicy`
|
||||
|
||||
## Overview
|
||||
|
||||
Per-peer seeding policies (allow, max blocks, priority) synced over gossip; use with `hyper-p2p-presence` to map policy peers to live connections.
|
||||
|
||||
## Constructor
|
||||
|
||||
```js
|
||||
const mod = new HyperP2PCoreSeedPolicy(opts)
|
||||
```
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
|
||||
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
|
||||
| `core` | object \| null | null | Attached core instance (`attach()` also supported) |
|
||||
| `defaultPolicy` | varies | { allow: false, maxBlocks: 0 } | Constructor option `defaultPolicy` |
|
||||
|
||||
## Methods
|
||||
|
||||
### `attach(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `setPolicy(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:**
|
||||
- `Error: policy object required`
|
||||
|
||||
### `allowed(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `getPolicy(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `syncPolicies(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `getStats(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `ready(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `close(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
## Events
|
||||
|
||||
| Event | Payload |
|
||||
|-------|---------|
|
||||
| `policy` | `{ peer, policy }` |
|
||||
| `closed` | no payload |
|
||||
|
||||
## getStats()
|
||||
|
||||
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||
|
||||
## Errors
|
||||
|
||||
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||
|
||||
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||
|
||||
### Documented `throw new Error(...)` strings
|
||||
|
||||
- `policy object required`
|
||||
|
||||
## P2P
|
||||
|
||||
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `core-seed-policy/v1`.
|
||||
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
|
||||
|
||||
### Gossip message types
|
||||
|
||||
- `seed-policy-sync`
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
npm install && npm test
|
||||
```
|
||||
|
||||
## Common flows
|
||||
|
||||
1. `setPolicy(peer, policy)` — local map + `policy` event.
|
||||
2. `allowed(peer)` — consult map or `defaultPolicy`.
|
||||
3. `syncPolicies()` — gossip `seed-policy-sync`.
|
||||
|
||||
@@ -0,0 +1,50 @@
|
||||
# Architecture: hyper-p2p-core-seed-policy
|
||||
|
||||
**Category:** Storage (Hypercore) · **Protocol:** `core-seed-policy/v1` · **P2P:** yes
|
||||
|
||||
## Role
|
||||
|
||||
Control **which peers may seed** blocks from this core and optional `maxBlocks` caps. Default policy applies when a peer has no explicit entry (`defaultPolicy.allow`).
|
||||
|
||||
## Wire messages
|
||||
|
||||
| type | fields | behavior |
|
||||
|------|--------|----------|
|
||||
| `seed-policy-sync` | `policies`, `defaultPolicy` | Merge by `updatedAt` per peer |
|
||||
|
||||
### Policy entry
|
||||
|
||||
```json
|
||||
{
|
||||
"allow": true,
|
||||
"maxBlocks": 1024,
|
||||
"priority": 0,
|
||||
"updatedAt": 1710000000000
|
||||
}
|
||||
```
|
||||
|
||||
## State model
|
||||
|
||||
| Field | Description |
|
||||
|-------|-------------|
|
||||
| `_policies` | `Map<peer, entry>` |
|
||||
| `defaultPolicy` | Constructor default |
|
||||
|
||||
## Methods
|
||||
|
||||
- `setPolicy(peer, policy)` — local update
|
||||
- `allowed(peer)` — boolean; increments `denied` stat when false
|
||||
- `getPolicy` / `listPeers`
|
||||
- `syncPolicies()` — gossip full map
|
||||
|
||||
## Events
|
||||
|
||||
`policy`, `remote-policy` (after merge), `closed`
|
||||
|
||||
## Composition
|
||||
|
||||
Use **before** `hyper-p2p-core-replicator`: seed-policy gates **who**, replicator defines **how much** per allowed peer.
|
||||
|
||||
## Merge rules
|
||||
|
||||
Same last-write-wins on `updatedAt` as replicator policies.
|
||||
@@ -0,0 +1,14 @@
|
||||
require('bare-process/global')
|
||||
const { HyperP2PCoreSeedPolicy } = require('../index.js')
|
||||
|
||||
async function main () {
|
||||
const topic = process.argv[2] || null
|
||||
const m = new HyperP2PCoreSeedPolicy({ topic })
|
||||
m.attach({ length: 0 })
|
||||
await m.ready()
|
||||
m.setPolicy('peer-1', { allow: true, maxBlocks: 100 })
|
||||
console.log(m.allowed('peer-1'))
|
||||
await m.close()
|
||||
console.log('done')
|
||||
}
|
||||
main().catch(console.error)
|
||||
@@ -0,0 +1,111 @@
|
||||
require('bare-process/global')
|
||||
const EventEmitter = require('bare-events')
|
||||
const { assertCore, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
|
||||
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
|
||||
const PROTOCOL = 'core-seed-policy/v1'
|
||||
|
||||
class HyperP2PCoreSeedPolicy extends EventEmitter {
|
||||
constructor (opts = {}) {
|
||||
super()
|
||||
this.topic = opts.topic || null
|
||||
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||
this.core = opts.core || null
|
||||
this.defaultPolicy = opts.defaultPolicy || { allow: false, maxBlocks: 0 }
|
||||
this._policies = new Map()
|
||||
this._stats = { set: 0, syncs: 0, denied: 0 }
|
||||
this.swarm = null
|
||||
}
|
||||
|
||||
attach (core) {
|
||||
assertCore(core)
|
||||
this.core = core
|
||||
return this
|
||||
}
|
||||
|
||||
setPolicy (peer, policy) {
|
||||
assertNonEmpty(peer, 'peer')
|
||||
if (!policy || typeof policy !== 'object') throw new Error('policy object required')
|
||||
const entry = {
|
||||
allow: !!policy.allow,
|
||||
maxBlocks: policy.maxBlocks != null ? policy.maxBlocks : this.defaultPolicy.maxBlocks,
|
||||
priority: policy.priority || 0,
|
||||
updatedAt: Date.now()
|
||||
}
|
||||
this._policies.set(peer, entry)
|
||||
this._stats.set++
|
||||
this.emit('policy', { peer, policy: entry })
|
||||
return entry
|
||||
}
|
||||
|
||||
allowed (peer) {
|
||||
assertNonEmpty(peer, 'peer')
|
||||
const entry = this._policies.get(peer)
|
||||
if (!entry) {
|
||||
this._stats.denied++
|
||||
return !!this.defaultPolicy.allow
|
||||
}
|
||||
if (!entry.allow) this._stats.denied++
|
||||
return !!entry.allow
|
||||
}
|
||||
|
||||
getPolicy (peer) {
|
||||
return this._policies.get(peer) || { ...this.defaultPolicy }
|
||||
}
|
||||
|
||||
listPeers () {
|
||||
return [...this._policies.keys()]
|
||||
}
|
||||
|
||||
syncPolicies () {
|
||||
const payload = {
|
||||
type: 'seed-policy-sync',
|
||||
policies: Object.fromEntries(this._policies),
|
||||
defaultPolicy: this.defaultPolicy
|
||||
}
|
||||
sendGossip(this, payload)
|
||||
this._stats.syncs++
|
||||
return payload
|
||||
}
|
||||
|
||||
_mergePolicies (policies) {
|
||||
const merged = []
|
||||
for (const [peer, policy] of Object.entries(policies || {})) {
|
||||
const existing = this._policies.get(peer)
|
||||
if (!existing || (policy.updatedAt || 0) >= (existing.updatedAt || 0)) {
|
||||
this._policies.set(peer, policy)
|
||||
merged.push(peer)
|
||||
}
|
||||
}
|
||||
if (merged.length) this.emit('remote-policy', { peers: merged })
|
||||
}
|
||||
|
||||
getStats () {
|
||||
return {
|
||||
...this._stats,
|
||||
policyCount: this._policies.size,
|
||||
coreLength: this.core ? this.core.length : 0,
|
||||
protocol: PROTOCOL
|
||||
}
|
||||
}
|
||||
|
||||
async ready () {
|
||||
if (this.swarm || !this.topic) return this
|
||||
await attachGossip(this, {
|
||||
keyPair: this.keyPair,
|
||||
topic: this.topic,
|
||||
protocol: PROTOCOL,
|
||||
onmessage: (d) => {
|
||||
if (d && d.type === 'seed-policy-sync') this._mergePolicies(d.policies)
|
||||
}
|
||||
})
|
||||
return this
|
||||
}
|
||||
|
||||
async close () {
|
||||
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||
this.swarm = null
|
||||
this.emit('closed')
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { HyperP2PCoreSeedPolicy, PROTOCOL }
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "hyper-p2p-core-seed-policy",
|
||||
"version": "0.3.1",
|
||||
"description": "Seeder policy and announce hints.",
|
||||
"main": "index.js",
|
||||
"type": "commonjs",
|
||||
"license": "Apache-2.0",
|
||||
"scripts": { "test": "brittle-bare test/test.js" },
|
||||
"dependencies": {
|
||||
"bare-events": "^2.8.0",
|
||||
"bare-process": "^4.4.0",
|
||||
"b4a": "^1.6.7",
|
||||
"hypercore-crypto": "^3.0.0",
|
||||
"protomux": "^3.0.0",
|
||||
"compact-encoding": "^2.0.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"hyperswarm": "^4.0.0",
|
||||
"bare": ">=1.0.0",
|
||||
"hypercore": "^10.0.0"
|
||||
},
|
||||
"devDependencies": { "brittle": "^3.0.0" },
|
||||
"imports": {
|
||||
"process": { "bare": "bare-process", "default": "process" },
|
||||
"events": { "bare": "bare-events", "default": "events" }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,46 @@
|
||||
require('bare-process/global')
|
||||
const test = require('brittle')
|
||||
const { HyperP2PCoreSeedPolicy, PROTOCOL } = require('../index.js')
|
||||
|
||||
test('exports', (t) => {
|
||||
t.ok(HyperP2PCoreSeedPolicy)
|
||||
t.is(PROTOCOL, 'core-seed-policy/v1')
|
||||
})
|
||||
|
||||
test('setPolicy allowed', async (t) => {
|
||||
const m = new HyperP2PCoreSeedPolicy({ defaultPolicy: { allow: false } })
|
||||
m.setPolicy('peer-a', { allow: true })
|
||||
t.ok(m.allowed('peer-a'))
|
||||
t.not(m.allowed('peer-b'))
|
||||
await m.close()
|
||||
})
|
||||
|
||||
test('validation', async (t) => {
|
||||
const m = new HyperP2PCoreSeedPolicy()
|
||||
try { m.setPolicy(null, {}) } catch (e) { t.ok(e) }
|
||||
try { m.setPolicy('p', null) } catch (e) { t.ok(e) }
|
||||
await m.close()
|
||||
})
|
||||
|
||||
test('syncPolicies payload', async (t) => {
|
||||
const m = new HyperP2PCoreSeedPolicy()
|
||||
m.setPolicy('p1', { allow: true })
|
||||
const p = m.syncPolicies()
|
||||
t.is(p.type, 'seed-policy-sync')
|
||||
await m.close()
|
||||
})
|
||||
|
||||
test('getStats', async (t) => {
|
||||
const m = new HyperP2PCoreSeedPolicy()
|
||||
t.is(m.getStats().protocol, 'core-seed-policy/v1')
|
||||
await m.close()
|
||||
})
|
||||
|
||||
test('listPeers remote merge', async (t) => {
|
||||
const m = new HyperP2PCoreSeedPolicy()
|
||||
m.setPolicy('p1', { allow: true })
|
||||
t.is(m.listPeers().length, 1)
|
||||
m._mergePolicies({ p2: { allow: false, updatedAt: 2 } })
|
||||
t.is(m.listPeers().length, 2)
|
||||
await m.close()
|
||||
})
|
||||
Reference in New Issue
Block a user