Add storage stack modules and track them in git

The modules workspace previously ignored every path matching `storage-*`, which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out of version control. Narrow .gitignore to test-artifact patterns only so production category trees are committed. Storage packages (23 total): - storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync, priority-fetch, bitfield-scheduler, audit-chain - storage-hyperbee (5): batch-write, diff-follow, range-watch, secondary-index, tombstone-gc - storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge, version-snapshot, watch-notify - storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus, light-writer Implementation highlights: - Shared attach/gossip helpers in _shared/storage-gossip-base.js - P2P modules use Protomux gossip via p2p-bare; local planners omit swarm - Recent deepen pass: priority queues, batch caps, audit export/import, watch/notify aliases on range-watch, fork/view lease helpers, etc. - Per-package README, docs/api.md, docs/architecture.md, tests, examples Also update modules/README.md doc hub links for the full storage stack (hypercore, hyperbee, hyperdrive, autobase). Co-authored-by: Cursor <[email protected]>
2026-05-21 00:17:51 -04:00
parent b10507b060
commit a11e22badc
214 changed files with 57132 additions and 5 deletions
@@ -0,0 +1,5 @@
+# Changelog
+
+## [0.0.0-scaffold]
+
+- Registry scaffold: file tree, load smoke tests, docs stubs
@@ -0,0 +1,42 @@
+# hyper-p2p-core-audit-chain
+
+Append-only hash-linked audit log over a Hypercore. Local-only (no Hyperswarm topic).
+
+**Category:** Storage (Hypercore)
+
+**Composes with:** `hyper-p2p-attestation-chain`
+
+**Protocol:** `core-audit-chain/v1`
+
+## When to use
+
+Apps that need tamper-evident audit trails tied to core length (compliance, replication forensics).
+
+## When not to use
+
+When you only need unstructured logs without hash chaining (use observability modules).
+
+## Quick start
+
+```js
+const { HyperP2PCoreAuditChain } = require('hyper-p2p-core-audit-chain')
+const mod = new HyperP2PCoreAuditChain()
+mod.attach(core)
+mod.appendAudit({ op: 'write', actor: 'peer-1' })
+console.log(mod.verifyChain())
+console.log(mod.tail(5))
+await mod.close()
+```
+
+## Docs
+
+- [docs/api.md](docs/api.md) — constructor, methods, events, errors
+- [docs/architecture.md](docs/architecture.md) — chain model, verification
+- [../_shared/PRODUCTION.md](../../_shared/PRODUCTION.md) — production checklist
+- [../_shared/DOC_STANDARDS.md](../../_shared/DOC_STANDARDS.md) — documentation standards
+
+## Test
+
+```bash
+npm install && npm test
+```
@@ -0,0 +1,85 @@
+# API: hyper-p2p-core-audit-chain
+
+**Protocol:** `core-audit-chain/v1`
+
+**Export:** `HyperP2PCoreAuditChain`
+
+## Overview
+
+Append-only hash-linked audit log over a Hypercore. Each entry includes `prevHash`, `hash`, `seq`, and optional `coreLength` snapshot.
+
+## Constructor
+
+```js
+const mod = new HyperP2PCoreAuditChain(opts)
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `core` | Hypercore | null | Optional core attached at construct time |
+
+## Methods
+
+### `attach(core)`
+
+- **Returns:** `this`
+- **Throws:** `Error: core must be a Hypercore instance`
+
+### `appendAudit(event)`
+
+- **Parameters:** `event` — plain object (domain fields)
+- **Returns:** entry with `seq`, `hash`, `prevHash`, `at`
+- **Throws:** `Error: event object required`
+
+### `verifyChain()`
+
+- **Returns:** `{ ok: true, length, head }` or `{ ok: false, seq, reason }`
+- **Throws:** —
+
+### `tail(n = 10)`
+
+- **Returns:** last `n` entries
+- **Throws:** `Error: n must be non-negative`
+
+### `getStats()`
+
+- **Returns:** `{ appended, verified, failures, length, head, coreLength, protocol }`
+
+### `ready()`
+
+- **Returns:** `Promise<this>` (no-op)
+
+### `close()`
+
+- **Returns:** `Promise<void>` — clears chain
+
+## Events
+
+| Event | Payload |
+|-------|---------|
+| `audit` | appended entry |
+| `closed` | no payload |
+
+## getStats()
+
+See fields above. `protocol` is always `core-audit-chain/v1`.
+
+## Errors
+
+Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
+
+## P2P
+
+Local-only — no `topic` or gossip wire messages.
+
+## Testing
+
+```bash
+npm install && npm test
+```
+
+## Common flows
+
+1. `attach(core)` → `appendAudit` for each mutation → periodic `verifyChain()`
+2. `tail(n)` for dashboards without scanning full chain
+3. `close()` on shutdown to drop in-memory chain
@@ -0,0 +1,48 @@
+# Architecture: hyper-p2p-core-audit-chain
+
+**Category:** Storage (Hypercore) · **Protocol:** `core-audit-chain/v1` · **P2P:** no
+
+## Role
+
+Append-only **hash-linked audit log** for operations on a Hypercore (replication decisions, admin actions, policy changes). Each entry stores `prevHash`, `hash`, `seq`, and `coreLength` at append time for correlation with chain state.
+
+## Hash algorithm
+
+`hash = sha256(JSON.stringify({ prevHash, event }))` via `hypercore-crypto` (hex string). Verification walks the chain recomputing hashes; mismatch returns `{ ok: false, seq, reason }`.
+
+## State model
+
+| Field | Description |
+|-------|-------------|
+| `_chain` | Ordered entries with metadata |
+| `_headHash` | Latest hash or `null` |
+
+## API surface
+
+| Method | Purpose |
+|--------|---------|
+| `appendAudit(event)` | Push linked entry; emit `audit` |
+| `verifyChain()` | Full integrity check |
+| `tail(n)` | Last `n` entries |
+| `exportChain()` / `importChain(entries)` | Snapshot / restore for persistence |
+
+## Wire messages
+
+None. Export JSON and ship over your app channel if peers need shared audit history.
+
+## Events
+
+| Event | Payload |
+|-------|---------|
+| `audit` | New entry |
+| `closed` | Chain cleared |
+
+## Composition
+
+- Log replication policy changes from `hyper-p2p-core-replicator`
+- Log fork picks from `hyper-p2p-core-fork-picker`
+- Persist `exportChain()` to Hyperbee or Hyperdrive for durability
+
+## Threat model
+
+Detects tampering of in-memory chain only. Does not sign entries — add application-level signatures if non-repudiation is required.
@@ -0,0 +1,16 @@
+require('bare-process/global')
+const { HyperP2PCoreAuditChain } = require('../index.js')
+
+async function main () {
+  const mod = new HyperP2PCoreAuditChain()
+  mod.attach({ length: 0 })
+  mod.appendAudit({ op: 'bootstrap' })
+  mod.appendAudit({ op: 'append', block: 1 })
+  console.log('verify', mod.verifyChain())
+  console.log('tail', mod.tail(2))
+  console.log('stats', mod.getStats())
+  await mod.close()
+  console.log('done')
+}
+
+main().catch(console.error)
@@ -0,0 +1,106 @@
+require('bare-process/global')
+const EventEmitter = require('bare-events')
+const b4a = require('b4a')
+const crypto = require('hypercore-crypto')
+const { assertCore } = require('../../_shared/storage-gossip-base.js')
+const PROTOCOL = 'core-audit-chain/v1'
+
+function hashEntry (prevHash, event) {
+  const body = JSON.stringify({ prevHash, event })
+  return b4a.toString(crypto.hash(b4a.from(body)), 'hex')
+}
+
+class HyperP2PCoreAuditChain extends EventEmitter {
+  constructor (opts = {}) {
+    super()
+    this.core = opts.core || null
+    this._chain = []
+    this._headHash = null
+    this._stats = { appended: 0, verified: 0, failures: 0 }
+  }
+
+  attach (core) {
+    assertCore(core)
+    this.core = core
+    return this
+  }
+
+  appendAudit (event) {
+    if (!event || typeof event !== 'object') throw new Error('event object required')
+    const prevHash = this._headHash
+    const hash = hashEntry(prevHash, event)
+    const entry = {
+      ...event,
+      seq: this._chain.length,
+      prevHash,
+      hash,
+      coreLength: this.core ? this.core.length : 0,
+      at: Date.now()
+    }
+    this._chain.push(entry)
+    this._headHash = hash
+    this._stats.appended++
+    this.emit('audit', entry)
+    return entry
+  }
+
+  verifyChain () {
+    let prev = null
+    for (const entry of this._chain) {
+      const expected = hashEntry(prev, stripMeta(entry))
+      if (entry.hash !== expected || entry.prevHash !== prev) {
+        this._stats.failures++
+        return { ok: false, seq: entry.seq, reason: 'hash mismatch' }
+      }
+      prev = entry.hash
+    }
+    this._stats.verified++
+    return { ok: true, length: this._chain.length, head: this._headHash }
+  }
+
+  tail (n = 10) {
+    if (n < 0) throw new Error('n must be non-negative')
+    if (n === 0) return []
+    return this._chain.slice(-n)
+  }
+
+  exportChain () {
+    return this._chain.map((e) => ({ ...e }))
+  }
+
+  importChain (entries) {
+    if (!Array.isArray(entries)) throw new Error('entries must be an array')
+    this._chain = []
+    this._headHash = null
+    for (const raw of entries) {
+      const event = stripMeta(raw)
+      this.appendAudit(event)
+    }
+    return this.verifyChain()
+  }
+
+  getStats () {
+    return {
+      ...this._stats,
+      length: this._chain.length,
+      head: this._headHash,
+      coreLength: this.core ? this.core.length : 0,
+      protocol: PROTOCOL
+    }
+  }
+
+  async ready () { return this }
+
+  async close () {
+    this._chain = []
+    this._headHash = null
+    this.emit('closed')
+  }
+}
+
+function stripMeta (entry) {
+  const { seq, prevHash, hash, coreLength, at, ...event } = entry
+  return event
+}
+
+module.exports = { HyperP2PCoreAuditChain, PROTOCOL }
@@ -0,0 +1,27 @@
+{
+  "name": "hyper-p2p-core-audit-chain",
+  "version": "0.3.1",
+  "description": "Append-only audit chain on cores.",
+  "main": "index.js",
+  "type": "commonjs",
+  "license": "Apache-2.0",
+  "scripts": { "test": "brittle-bare test/test.js" },
+  "dependencies": {
+    "bare-events": "^2.8.0",
+    "bare-process": "^4.4.0",
+    "b4a": "^1.6.7",
+    "hypercore-crypto": "^3.0.0",
+    "protomux": "^3.0.0",
+    "compact-encoding": "^2.0.0"
+  },
+  "peerDependencies": {
+    "hyperswarm": "^4.0.0",
+    "bare": ">=1.0.0",
+    "hypercore": "^10.0.0"
+  },
+  "devDependencies": { "brittle": "^3.0.0" },
+  "imports": {
+    "process": { "bare": "bare-process", "default": "process" },
+    "events": { "bare": "bare-events", "default": "events" }
+  }
+}
@@ -0,0 +1,61 @@
+require('bare-process/global')
+const test = require('brittle')
+const { HyperP2PCoreAuditChain, PROTOCOL } = require('../index.js')
+
+const mockCore = { length: 3 }
+
+test('exports', (t) => {
+  t.ok(HyperP2PCoreAuditChain)
+  t.is(PROTOCOL, 'core-audit-chain/v1')
+})
+
+test('appendAudit verifyChain tail', async (t) => {
+  const m = new HyperP2PCoreAuditChain()
+  m.attach(mockCore)
+  m.appendAudit({ op: 'init' })
+  m.appendAudit({ op: 'write' })
+  const v = m.verifyChain()
+  t.ok(v.ok)
+  t.is(m.tail(1).length, 1)
+  t.is(m.tail(2).length, 2)
+  await m.close()
+})
+
+test('validation', async (t) => {
+  const m = new HyperP2PCoreAuditChain()
+  try { m.appendAudit(null) } catch (e) { t.ok(e) }
+  try { m.attach(null) } catch (e) { t.ok(e) }
+  try { m.tail(-1) } catch (e) { t.ok(e) }
+  await m.close()
+})
+
+test('getStats', async (t) => {
+  const m = new HyperP2PCoreAuditChain()
+  m.appendAudit({ x: 1 })
+  const s = m.getStats()
+  t.is(s.protocol, 'core-audit-chain/v1')
+  t.is(s.length, 1)
+  await m.close()
+})
+
+test('exportChain importChain', async (t) => {
+  const m = new HyperP2PCoreAuditChain()
+  m.appendAudit({ op: 'a' })
+  m.appendAudit({ op: 'b' })
+  const exported = m.exportChain()
+  const m2 = new HyperP2PCoreAuditChain()
+  const v = m2.importChain(exported)
+  t.ok(v.ok)
+  t.is(m2.getStats().length, 2)
+  await m.close()
+  await m2.close()
+})
+
+test('broken chain detected', async (t) => {
+  const m = new HyperP2PCoreAuditChain()
+  m.appendAudit({ a: 1 })
+  m._chain[0].hash = 'bad'
+  const v = m.verifyChain()
+  t.not(v.ok)
+  await m.close()
+})