Add storage stack modules and track them in git
The modules workspace previously ignored every path matching `storage-*`, which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out of version control. Narrow .gitignore to test-artifact patterns only so production category trees are committed. Storage packages (23 total): - storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync, priority-fetch, bitfield-scheduler, audit-chain - storage-hyperbee (5): batch-write, diff-follow, range-watch, secondary-index, tombstone-gc - storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge, version-snapshot, watch-notify - storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus, light-writer Implementation highlights: - Shared attach/gossip helpers in _shared/storage-gossip-base.js - P2P modules use Protomux gossip via p2p-bare; local planners omit swarm - Recent deepen pass: priority queues, batch caps, audit export/import, watch/notify aliases on range-watch, fork/view lease helpers, etc. - Per-package README, docs/api.md, docs/architecture.md, tests, examples Also update modules/README.md doc hub links for the full storage stack (hypercore, hyperbee, hyperdrive, autobase). Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Raven Scott
@@ -0,0 +1,5 @@
|
||||
# Changelog
|
||||
|
||||
## [0.0.0-scaffold]
|
||||
|
||||
- Registry scaffold: file tree, load smoke tests, docs stubs
|
||||
@@ -0,0 +1,41 @@
|
||||
# hyper-p2p-drive-entry-catalog
|
||||
|
||||
Gossip-replicated path catalog with metadata for Hyperdrive entries; merge remote catalogs via `hyper-p2p-merge-registry` policies. Hyperswarm gossip when `topic` is set.
|
||||
|
||||
**Category:** Storage (Hyperdrive)
|
||||
|
||||
**Composes with:** `hyper-p2p-merge-registry`
|
||||
|
||||
**Protocol:** `drive-entry-catalog/v1`
|
||||
|
||||
## When to use
|
||||
|
||||
Peers need a searchable index of drive paths without full directory walks.
|
||||
|
||||
## When not to use
|
||||
|
||||
Single-writer drives with no shared topic.
|
||||
|
||||
## Quick start
|
||||
|
||||
```js
|
||||
const { HyperP2PDriveEntryCatalog } = require('hyper-p2p-drive-entry-catalog')
|
||||
const topic = process.argv[2] // 64-char hex or string
|
||||
const mod = new HyperP2PDriveEntryCatalog({ topic })
|
||||
await mod.ready() // joins swarm when topic set
|
||||
// ... application logic ...
|
||||
await mod.close()
|
||||
```
|
||||
|
||||
## Docs
|
||||
|
||||
- [docs/api.md](docs/api.md) — constructor, methods, events, errors
|
||||
- [docs/architecture.md](docs/architecture.md) — wire types, state, composition
|
||||
- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
|
||||
- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
|
||||
|
||||
## Test
|
||||
|
||||
```bash
|
||||
npm install && npm test
|
||||
```
|
||||
@@ -0,0 +1,119 @@
|
||||
# API: hyper-p2p-drive-entry-catalog
|
||||
|
||||
**Protocol:** `drive-entry-catalog/v1`
|
||||
|
||||
**Export:** `HyperP2PDriveEntryCatalog`
|
||||
|
||||
## Overview
|
||||
|
||||
Gossip-replicated path catalog with metadata for Hyperdrive entries; merge remote catalogs via `hyper-p2p-merge-registry` policies.
|
||||
|
||||
## Constructor
|
||||
|
||||
```js
|
||||
const mod = new HyperP2PDriveEntryCatalog(opts)
|
||||
```
|
||||
|
||||
| Option | Type | Default | Description |
|
||||
|--------|------|---------|-------------|
|
||||
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
|
||||
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
|
||||
| `drive` | object \| null | null | Attached drive instance (`attach()` also supported) |
|
||||
|
||||
## Methods
|
||||
|
||||
### `attach(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `catalogEntry(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `lookupEntry(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `searchPrefix(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `listEntries(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `syncCatalog(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `getStats(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `ready(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
### `close(…)`
|
||||
|
||||
- **Returns:** module-specific (see implementation)
|
||||
|
||||
- **Throws:** — (none in method body)
|
||||
|
||||
## Events
|
||||
|
||||
| Event | Payload |
|
||||
|-------|---------|
|
||||
| `catalog` | catalog entry |
|
||||
| `remote-catalog` | gossip catalog payload |
|
||||
| `closed` | no payload |
|
||||
|
||||
## getStats()
|
||||
|
||||
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
|
||||
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
|
||||
|
||||
## Errors
|
||||
|
||||
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
|
||||
|
||||
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
|
||||
|
||||
## P2P
|
||||
|
||||
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `drive-entry-catalog/v1`.
|
||||
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
|
||||
|
||||
### Gossip message types
|
||||
|
||||
- `catalog-entry`
|
||||
- `catalog-sync`
|
||||
|
||||
## Testing
|
||||
|
||||
```bash
|
||||
npm install && npm test
|
||||
```
|
||||
|
||||
## Common flows
|
||||
|
||||
1. `catalogEntry(path, meta)` — gossip `catalog-entry`.
|
||||
2. `syncCatalog()` — push full `catalog-sync` snapshot.
|
||||
3. `searchPrefix(prefix)` — local prefix listing.
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
# Architecture: hyper-p2p-drive-entry-catalog
|
||||
|
||||
**Category:** Storage (Hyperdrive)
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
App[Application] --> Mod[HyperP2PDriveEntryCatalog]
|
||||
Mod --> Gossip[Hyperswarm gossip]
|
||||
Gossip --> Peer[Remote peers]
|
||||
```
|
||||
|
||||
## Sequence (P2P)
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
participant App
|
||||
participant Mod as Module
|
||||
participant SW as Hyperswarm
|
||||
participant Peer
|
||||
App->>Mod: ready(topic)
|
||||
Mod->>SW: join(topic)
|
||||
SW->>Peer: connection
|
||||
Mod->>Peer: gossip payload
|
||||
Peer-->>Mod: onmessage
|
||||
Mod-->>App: emit(event)
|
||||
```
|
||||
|
||||
## Wire messages
|
||||
|
||||
| type | fields | direction | behavior |
|
||||
|------|--------|-----------|----------|
|
||||
| `catalog-entry` | path, meta | gossip | upsert catalog path |
|
||||
| `catalog-sync` | entries | gossip | bulk catalog snapshot |
|
||||
|
||||
## State model
|
||||
|
||||
- In-memory `Map` / `Set` / array structures for hot path
|
||||
- Optional attached HyperDrive via `attach()`
|
||||
- `close()` clears ephemeral state and destroys swarm when P2P
|
||||
|
||||
## Composition
|
||||
|
||||
Composes with: `hyper-p2p-merge-registry`.
|
||||
|
||||
Storage gossip helpers: [`../../_shared/storage-gossip-base.js`](../../_shared/storage-gossip-base.js).
|
||||
@@ -0,0 +1,14 @@
|
||||
require('bare-process/global')
|
||||
const { HyperP2PDriveEntryCatalog } = require('../index.js')
|
||||
|
||||
async function main () {
|
||||
const topic = process.argv[2] || null
|
||||
const m = new HyperP2PDriveEntryCatalog({ topic })
|
||||
m.attach({ version: 0 })
|
||||
await m.ready()
|
||||
m.catalogEntry('/readme.md', { size: 12 })
|
||||
console.log(m.lookupEntry('/readme.md'))
|
||||
await m.close()
|
||||
console.log('done')
|
||||
}
|
||||
main().catch(console.error)
|
||||
@@ -0,0 +1,91 @@
|
||||
require('bare-process/global')
|
||||
const EventEmitter = require('bare-events')
|
||||
const { assertNonEmpty } = require('../../_shared/lib/errors.js')
|
||||
const { assertDrive, attachGossip, sendGossip } = require('../../_shared/storage-gossip-base.js')
|
||||
|
||||
const PROTOCOL = 'drive-entry-catalog/v1'
|
||||
|
||||
class HyperP2PDriveEntryCatalog extends EventEmitter {
|
||||
constructor (opts = {}) {
|
||||
super()
|
||||
this.topic = opts.topic || null
|
||||
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||
this.drive = opts.drive || null
|
||||
this._catalog = new Map()
|
||||
this._stats = { cataloged: 0, lookups: 0, gossipIn: 0, gossipOut: 0 }
|
||||
this.swarm = null
|
||||
}
|
||||
|
||||
attach (drive) {
|
||||
assertDrive(drive)
|
||||
this.drive = drive
|
||||
return this
|
||||
}
|
||||
|
||||
catalogEntry (path, meta = {}) {
|
||||
assertNonEmpty(path, 'path')
|
||||
const entry = { ...meta, path, at: Date.now() }
|
||||
this._catalog.set(path, entry)
|
||||
this._stats.cataloged++
|
||||
sendGossip(this, { type: 'catalog-entry', path, meta: entry })
|
||||
this._stats.gossipOut++
|
||||
this.emit('catalog', entry)
|
||||
return entry
|
||||
}
|
||||
|
||||
lookupEntry (path) {
|
||||
assertNonEmpty(path, 'path')
|
||||
this._stats.lookups++
|
||||
return this._catalog.get(path) || null
|
||||
}
|
||||
|
||||
searchPrefix (prefix) {
|
||||
assertNonEmpty(prefix, 'prefix')
|
||||
const out = []
|
||||
for (const [path, meta] of this._catalog) {
|
||||
if (path.startsWith(prefix)) out.push({ path, meta })
|
||||
}
|
||||
return out.sort((a, b) => a.path.localeCompare(b.path))
|
||||
}
|
||||
|
||||
listEntries () {
|
||||
return [...this._catalog.entries()].map(([path, meta]) => ({ path, meta }))
|
||||
}
|
||||
|
||||
_onGossip (d) {
|
||||
if (!d || d.type !== 'catalog-entry') return
|
||||
this._stats.gossipIn++
|
||||
if (d.path) this._catalog.set(d.path, d.meta || { at: Date.now() })
|
||||
this.emit('remote-catalog', d)
|
||||
}
|
||||
|
||||
syncCatalog () {
|
||||
const entries = this.listEntries()
|
||||
sendGossip(this, { type: 'catalog-sync', entries })
|
||||
this._stats.gossipOut++
|
||||
return entries.length
|
||||
}
|
||||
|
||||
getStats () {
|
||||
return { ...this._stats, entries: this._catalog.size, protocol: PROTOCOL }
|
||||
}
|
||||
|
||||
async ready () {
|
||||
if (this.swarm || !this.topic) return this
|
||||
await attachGossip(this, {
|
||||
keyPair: this.keyPair,
|
||||
topic: this.topic,
|
||||
protocol: PROTOCOL,
|
||||
onmessage: (d) => this._onGossip(d)
|
||||
})
|
||||
return this
|
||||
}
|
||||
|
||||
async close () {
|
||||
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||
this.swarm = null
|
||||
this.emit('closed')
|
||||
}
|
||||
}
|
||||
|
||||
module.exports = { HyperP2PDriveEntryCatalog, PROTOCOL }
|
||||
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,27 @@
|
||||
{
|
||||
"name": "hyper-p2p-drive-entry-catalog",
|
||||
"version": "0.3.1",
|
||||
"description": "Drive entry catalog and metadata.",
|
||||
"main": "index.js",
|
||||
"type": "commonjs",
|
||||
"license": "Apache-2.0",
|
||||
"scripts": { "test": "brittle-bare test/test.js" },
|
||||
"dependencies": {
|
||||
"bare-events": "^2.8.0",
|
||||
"bare-process": "^4.4.0",
|
||||
"b4a": "^1.6.7",
|
||||
"hypercore-crypto": "^3.0.0",
|
||||
"protomux": "^3.0.0",
|
||||
"compact-encoding": "^2.16.0"
|
||||
},
|
||||
"peerDependencies": {
|
||||
"hyperswarm": "^4.0.0",
|
||||
"bare": ">=1.0.0",
|
||||
"hyperdrive": "^11.0.0"
|
||||
},
|
||||
"devDependencies": { "brittle": "^3.0.0" },
|
||||
"imports": {
|
||||
"process": { "bare": "bare-process", "default": "process" },
|
||||
"events": { "bare": "bare-events", "default": "events" }
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,40 @@
|
||||
require('bare-process/global')
|
||||
const test = require('brittle')
|
||||
const { HyperP2PDriveEntryCatalog, PROTOCOL } = require('../index.js')
|
||||
|
||||
const mockDrive = { get: async () => null, readdir: async () => [] }
|
||||
|
||||
test('exports', (t) => {
|
||||
t.ok(HyperP2PDriveEntryCatalog)
|
||||
t.is(PROTOCOL, 'drive-entry-catalog/v1')
|
||||
})
|
||||
|
||||
test('catalogEntry lookupEntry searchPrefix', async (t) => {
|
||||
const m = new HyperP2PDriveEntryCatalog()
|
||||
m.attach(mockDrive)
|
||||
m.catalogEntry('/docs/a.txt', { size: 10 })
|
||||
m.catalogEntry('/docs/b.txt', { size: 20 })
|
||||
t.ok(m.lookupEntry('/docs/a.txt'))
|
||||
t.is(m.searchPrefix('/docs').length, 2)
|
||||
await m.close()
|
||||
})
|
||||
|
||||
test('validation', async (t) => {
|
||||
const m = new HyperP2PDriveEntryCatalog()
|
||||
try { m.attach(null) } catch (e) { t.ok(e) }
|
||||
try { m.catalogEntry(null) } catch (e) { t.ok(e) }
|
||||
await m.close()
|
||||
})
|
||||
|
||||
test('listEntries', async (t) => {
|
||||
const m = new HyperP2PDriveEntryCatalog()
|
||||
m.catalogEntry('/x', {})
|
||||
t.is(m.listEntries().length, 1)
|
||||
await m.close()
|
||||
})
|
||||
|
||||
test('getStats', async (t) => {
|
||||
const m = new HyperP2PDriveEntryCatalog()
|
||||
t.is(m.getStats().protocol, 'drive-entry-catalog/v1')
|
||||
await m.close()
|
||||
})
|
||||
Reference in New Issue
Block a user