Add storage stack modules and track them in git

The modules workspace previously ignored every path matching `storage-*`,
which kept all Hypercore, Hyperbee, Hyperdrive, and Autobase packages out
of version control. Narrow .gitignore to test-artifact patterns only so
production category trees are committed.

Storage packages (23 total):
- storage-hypercore (7): replicator, seed-policy, fork-picker, merkle-sync,
  priority-fetch, bitfield-scheduler, audit-chain
- storage-hyperbee (5): batch-write, diff-follow, range-watch,
  secondary-index, tombstone-gc
- storage-hyperdrive (6): entry-catalog, gc-sweep, mirror-sync, mount-bridge,
  version-snapshot, watch-notify
- storage-autobase (5): fork-choice, view-sync, writer-lease, indexer-bus,
  light-writer

Implementation highlights:
- Shared attach/gossip helpers in _shared/storage-gossip-base.js
- P2P modules use Protomux gossip via p2p-bare; local planners omit swarm
- Recent deepen pass: priority queues, batch caps, audit export/import,
  watch/notify aliases on range-watch, fork/view lease helpers, etc.
- Per-package README, docs/api.md, docs/architecture.md, tests, examples

Also update modules/README.md doc hub links for the full storage stack
(hypercore, hyperbee, hyperdrive, autobase).

Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Raven Scott
2026-05-21 00:17:51 -04:00
co-authored by Cursor
parent b10507b060
commit a11e22badc
214 changed files with 57132 additions and 5 deletions
@@ -0,0 +1,117 @@
# API: hyper-p2p-drive-gc-sweep
**Protocol:** `drive-gc-sweep/v1`
**Export:** `HyperP2PDriveGcSweep`
## Overview
Marks stale drive paths and sweeps them in batches over gossip; coordinate exclusive sweep windows with `hyper-p2p-topic-lease`.
## Constructor
```js
const mod = new HyperP2PDriveGcSweep(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | string \| Buffer \| null | null | Hyperswarm topic; gossip attaches when set |
| `keyPair` | Ed25519 KeyPair | random | Signing identity for swarm |
| `drive` | object \| null | null | Attached drive instance (`attach()` also supported) |
## Methods
### `attach(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `markStale(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `sweepBatch(…)`
- **Returns:** module-specific (see implementation)
- **Throws:**
- `Error: max must be non-negative`
### `staleCount(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `listStale(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `getStats(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `ready(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
### `close(…)`
- **Returns:** module-specific (see implementation)
- **Throws:** — (none in method body)
## Events
| Event | Payload |
|-------|---------|
| `sweep` | `{ removed }` |
| `closed` | no payload |
## getStats()
Returns `{ ...this._stats, protocol }` plus module-specific counters (pending queues, registry sizes, gossip in/out when P2P).
Local modules report hot-path counters only; P2P modules include gossip traffic when `topic` is set.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
Validation helpers may throw `ValidationError` (e.g. `peer is required`, `path is required`).
### Documented `throw new Error(...)` strings
- `max must be non-negative`
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens gossip for `drive-gc-sweep/v1`.
Outbound payloads use `sendGossip` (Protomux peer map); inbound handled in `_onGossip`.
### Gossip message types
- `stale-mark`
- `sweep-done`
## Testing
```bash
npm install && npm test
```
## Common flows
1. `markStale(path, reason)` — gossip `stale-mark`.
2. `sweepBatch(max)` — remove up to `max` paths, gossip `sweep-done`.
3. `listStale()` — inspect pending set.
@@ -0,0 +1,45 @@
# Architecture: hyper-p2p-drive-gc-sweep
**Category:** Storage (Hyperdrive)
```mermaid
flowchart LR
App[Application] --> Mod[HyperP2PDriveGcSweep]
Mod --> Gossip[Hyperswarm gossip]
Gossip --> Peer[Remote peers]
```
## Sequence (P2P)
```mermaid
sequenceDiagram
participant App
participant Mod as Module
participant SW as Hyperswarm
participant Peer
App->>Mod: ready(topic)
Mod->>SW: join(topic)
SW->>Peer: connection
Mod->>Peer: gossip payload
Peer-->>Mod: onmessage
Mod-->>App: emit(event)
```
## Wire messages
| type | fields | direction | behavior |
|------|--------|-----------|----------|
| `stale-mark` | path, reason | gossip | mark path stale |
| `sweep-done` | paths[] | gossip | confirm batch removal |
## State model
- In-memory `Map` / `Set` / array structures for hot path
- Optional attached HyperDrive via `attach()`
- `close()` clears ephemeral state and destroys swarm when P2P
## Composition
Composes with: `hyper-p2p-topic-lease`.
Storage gossip helpers: [`../../_shared/storage-gossip-base.js`](../../_shared/storage-gossip-base.js).