This commit is contained in:
Raven Scott
2026-05-20 23:36:32 -04:00
parent a020270cb1
commit be94546cd3
218 changed files with 9189 additions and 3078 deletions
@@ -1,92 +1,83 @@
# API: hyper-p2p-stream-backpressure
**Protocol:** `stream-backpressure/v1`
**Protocol:** `stream-backpressure/v1` (local buffer)
**Export:** `HyperP2PStreamBackpressure`
**Export:** `HyperP2PStreamBackpressure`, `PROTOCOL`
## Overview
Production p2p module: Hyperswarm discovery + Protomux when `topic` is set.
In-memory buffered stream with high-water-mark backpressure. `write` returns `false` when paused or over limit; emits `pause`, `resume`, `backpressure`, and `data`.
## Constructor
```js
const mod = new HyperP2PStreamBackpressure(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | `string` | `Buffer` | `null` | Hyperswarm topic; required for P2P `ready()` |
| `keyPair` | KeyPair | random | Ed25519 key pair |
| `highWaterMark` | number | 65536 | highWaterMark |
| Option | Default | Description |
|--------|---------|-------------|
| `highWaterMark` | `65536` | Max buffered bytes before pause/drop |
## Methods
### `write(chunk)`
- **Returns:** `value`
- **Throws:**
- `Error: chunk required`
- **Returns:** `true` if accepted, `false` if dropped (increments `dropped`)
- **Throws:** `chunk required`
### `pause()`
### `pause()` / `resume()`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
Manual flow control; `resume` emits `resume`.
### `resume(—)`
### `read()`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
FIFO shift from buffer; may clear `_paused` when below watermark.
### `read(—)`
### `pending()`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
Queue depth (chunk count).
### `pending(—)`
### `getStats()`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
`written`, `dropped`, `paused`, `bytes`, `paused` flag, `protocol`.
### `getStats(—)`
### `ready()` / `close()`
- **Returns:** `object`
- **Throws:** — (none documented in method body)
### `ready(—)`
- **Returns:** `Promise`
- **Throws:** — (none documented in method body)
### `close(—)`
- **Returns:** `Promise<void>`
- **Throws:** — (none documented in method body)
Clears buffer on close.
## Events
| Event | Payload |
|-------|---------|
| `backpressure` | bytes |
| `data` | buf |
| `pause` | no payload |
| `resume` | no payload |
| Event | When |
|-------|------|
| `data` | Chunk accepted |
| `pause` | Watermark hit |
| `resume` | Manual resume |
| `backpressure` | Write rejected |
## getStats()
## Wire
Returns `{ ...this._stats }` — typically `ops`, `errors`, and module-specific counters (`created`, `relays`, `open`, `peers`, etc.).
Library-only modules may include `mode: 'local'`.
None — local-only.
## Errors
## Composition
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `stream-backpressure/v1`.
Upstream of `hyper-p2p-stream-multiplex` or chunker pipelines.
## Testing
```bash
npm install && npm test
```
`npm test`
## Example
`examples/basic.js`
## State model
FIFO `_buffer` with `_bytes` accounting; `_paused` blocks writes until `read()` drains below watermark.
## Performance
Dropped writes increment `dropped` — monitor via `getStats()` in production pipelines.
## Versioning
Local-only module; protocol id for registry only.
## See also
[`docs/architecture.md`](architecture.md)
@@ -1,73 +1,84 @@
# API: hyper-p2p-stream-chunker
**Protocol:** `stream-chunker/v1`
**Protocol:** `stream-chunker/v1` (local transform; no Hyperswarm)
**Export:** `HyperP2PStreamChunker`
**Export:** `HyperP2PStreamChunker`, `PROTOCOL`
## Overview
Production p2p module: Hyperswarm discovery + Protomux when `topic` is set.
`HyperP2PStreamChunker` is a local byte-buffer utility for Bare/Pear stream pipelines. It accumulates inbound `Buffer` or string data, emits fixed-size slices via `push()` and `flush()`, and reports stats. There is no P2P wire — use with `hyper-p2p-stream-multiplex` or `hyper-p2p-stream-backpressure` for networked framing.
Extends `bare-events` `EventEmitter`.
## Constructor
```js
const mod = new HyperP2PStreamChunker(opts)
const chunker = new HyperP2PStreamChunker(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | `string` | `Buffer` | `null` | Hyperswarm topic; required for P2P `ready()` |
| `keyPair` | KeyPair | random | Ed25519 key pair |
| `chunkSize` | number | 4096 | chunkSize |
| `chunkSize` | `number` | `4096` | Maximum bytes per emitted chunk |
Internal state: `_pending` buffer, `_stats.chunks`, `_stats.bytes`.
## Methods
### `push(data)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
Appends data and emits complete chunks.
### `flush(—)`
- **Parameters:** `data``string` (UTF-8 via `b4a.from`) or `Buffer`
- **Returns:** `Array<Buffer>` — slices emitted in this call (may be empty)
- **Throws:** — (invalid buffer types may fail in `b4a.concat`)
- **Returns:** `value`
- **Throws:** — (none documented in method body)
While `_pending.length >= chunkSize`, takes `subarray(0, chunkSize)`, advances pending, increments `chunks`, emits `chunk` event.
### `getStats(—)`
### `flush()`
- **Returns:** `object`
- **Throws:** — (none documented in method body)
Emits any remaining pending bytes as one final chunk.
### `ready(—)`
- **Returns:** `Buffer | null` — tail buffer or `null` if nothing pending
- **Emits:** `chunk` when tail non-empty
- **Returns:** `Promise`
- **Throws:** — (none documented in method body)
### `getStats()`
### `close(—)`
- **Returns:** `{ chunks, bytes, pending, protocol }``pending` is current buffer length
- **Returns:** `Promise<void>`
- **Throws:** — (none documented in method body)
### `ready()` / `close()`
- **Returns:** `Promise<this>` — no-op ready; `close()` zeroes pending buffer
## Events
| Event | Payload |
|-------|---------|
| `chunk` | tail |
| Event | Payload | When |
|-------|---------|------|
| `chunk` | `Buffer` | Each slice from `push` or `flush` |
## getStats()
## Wire / P2P
Returns `{ ...this._stats }` — typically `ops`, `errors`, and module-specific counters (`created`, `relays`, `open`, `peers`, etc.).
Library-only modules may include `mode: 'local'`.
None. Protocol constant exists for registry and composition docs only.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
No validation errors on normal use. Empty `push` is allowed.
## P2P
## Composition
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `stream-chunker/v1`.
| Module | Role |
|--------|------|
| `hyper-p2p-stream-multiplex` | Frame chunked slices per stream id |
| `hyper-p2p-stream-backpressure` | Gate `push` when high water mark hit |
| `hyper-p2p-stream-transform` | Map/filter between chunker and mux |
## Testing
```bash
npm install && npm test
```
`npm test` — multi-chunk `push`, partial tail `flush`, stats.
## Example
`examples/basic.js`
## See also
[`docs/architecture.md`](architecture.md)
@@ -1,99 +1,92 @@
# API: hyper-p2p-stream-multiplex
**Protocol:** `stream-multiplex/v1`
**Protocol:** `stream-multiplex/v1` (local framing; wire via app transport)
**Export:** `StreamHandle`
**Export:** `HyperP2PStreamMultiplex`, `PROTOCOL`, `StreamHandle`
## Overview
Production p2p module: Hyperswarm discovery + Protomux when `topic` is set.
Multiplex many logical byte streams over a single frame channel. Each stream has an id, `write`/`end`, and `ondata` listeners. Frames are `{ type: 'frame', streamId, chunk, fin }` emitted on the mux `EventEmitter` for bridging to Protomux or UDX.
## Constructor
```js
const mod = new StreamHandle(opts)
```
| Option | Default | Description |
|--------|---------|-------------|
| `highWaterMark` | `65536` | Total bytes before `_sendFrame` returns false |
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | `string` | `Buffer` | `null` | Hyperswarm topic; required for P2P `ready()` |
| `keyPair` | KeyPair | random | Ed25519 key pair |
| `highWaterMark` | number | 65536 | highWaterMark |
## StreamHandle
## Methods
Created by `openStream(id?)`.
### `write(chunk)`
| Method | Description |
|--------|-------------|
| `write(chunk)` | Sends non-fin frame; throws if closed |
| `end(chunk?)` | Optional final chunk + FIN |
| `ondata(fn)` | Returns unsubscribe function |
- **Returns:** `value`
- **Throws:**
- `Error: stream closed`
## HyperP2PStreamMultiplex methods
### `end(chunk)`
### `openStream(id?)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
### `ondata(fn)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
### `openStream(id = null)`
- **Returns:** `value`
- **Throws:**
- `Error: stream id already open`
Opens stream; auto-increments id if omitted. Throws if id exists.
### `receiveFrame(frame)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
Delivers inbound frame to local handle; deletes on `fin`.
### `closeStream(streamId)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
Ends and removes stream.
### `getStats()`
### `getStats()`
- **Returns:** `object`
- **Throws:** — (none documented in method body)
`streams`, `frames`, `bytes`, `protocol`.
### `ready()`
### `ready()` / `close()`
- **Returns:** `Promise`
- **Throws:** — (none documented in method body)
Clears all streams on close.
### `close(—)`
## Frame wire shape
- **Returns:** `Promise<void>`
- **Throws:** — (none documented in method body)
| Field | Type | Description |
|-------|------|-------------|
| `type` | `'frame'` | Discriminator |
| `streamId` | `string` | Stream key |
| `chunk` | `Buffer` \| `null` | Payload |
| `fin` | `boolean` | End of stream |
## Events
| Event | Payload |
|-------|---------|
| `closed` | no payload |
| `frame` | frame |
| `open` | streamId |
## getStats()
Returns `{ ...this._stats }` — typically `ops`, `errors`, and module-specific counters (`created`, `relays`, `open`, `peers`, etc.).
Library-only modules may include `mode: 'local'`.
`open`, `frame`.
## Errors
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
`stream closed`, `stream id already open`.
## P2P
## Composition
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `stream-multiplex/v1`.
`hyper-p2p-stream-chunker`, `hyper-p2p-stream-backpressure`, Wave 9 `stream-multiplex-two-node` integration.
## Testing
```bash
npm install && npm test
```
`npm test`, `real_tests/integration/stream-multiplex-two-node.js`
Integration: [`../../../real_tests/integration/stream-multiplex-two-node.js`](../../../real_tests/integration/stream-multiplex-two-node.js)
## Example
`examples/basic.js`
## State model
`_streams` map id → `StreamHandle`; global `bytes` stat enforces `highWaterMark`.
## Performance
Frame emission is sync; bridge to network in app transport layer.
## Versioning
Frame shape stable for `stream-multiplex/v1`.
## See also
[`docs/architecture.md`](architecture.md)
@@ -1,86 +1,80 @@
# API: hyper-p2p-stream-resume-token
**Protocol:** `stream-resume-token/v1`
**Protocol:** `stream-resume-token/v1` (local checkpoints)
**Export:** `HyperP2PStreamResumeToken`
**Export:** `HyperP2PStreamResumeToken`, `PROTOCOL`
## Overview
Production p2p module: Hyperswarm discovery + Protomux when `topic` is set.
## Constructor
```js
const mod = new HyperP2PStreamResumeToken(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | `string` | `Buffer` | `null` | Hyperswarm topic; required for P2P `ready()` |
| `keyPair` | KeyPair | random | Ed25519 key pair |
Buffer stream chunks and issue byte-offset resume tokens for crash-safe replay. `readFromOffset()` returns chunks after a token offset.
## Methods
### `write(chunk)`
- **Returns:** `value`
- **Throws:**
- `Error: chunk required`
Appends to internal chunk list; updates `bytes` stat.
### `checkpoint()`
### `checkpoint()`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
- **Returns:** `{ id, offset, at }`
- **Emits:** `checkpoint`
### `resume(tokenId)`
- **Returns:** `value`
- **Throws:**
- `Error: unknown resume token`
- **Throws:** `unknown resume token`
- **Sets:** `_offset` to token byte position
### `readFromOffset()`
### `readFromOffset()`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
Returns chunks spanning bytes after current offset.
### `getStats()`
- **Returns:** `object`
- **Throws:** — (none documented in method body)
### `ready(—)`
- **Returns:** `Promise`
- **Throws:** — (none documented in method body)
### `close(—)`
- **Returns:** `Promise<void>`
- **Throws:** — (none documented in method body)
### `getStats()` / `ready()` / `close()`
## Events
| Event | Payload |
|-------|---------|
| `checkpoint` | token |
| `data` | buf |
| `resume` | token |
`data`, `checkpoint`, `resume`.
## getStats()
## Wire
Returns `{ ...this._stats }` — typically `ops`, `errors`, and module-specific counters (`created`, `relays`, `open`, `peers`, etc.).
Library-only modules may include `mode: 'local'`.
Gossip optional at app layer by serializing token ids; module is local buffer only.
## Errors
## Composition
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `stream-resume-token/v1`.
`hyper-p2p-stream-multiplex` FIN boundaries + Hyperdrive blob stores.
## Testing
```bash
npm install && npm test
```
`npm test`
## Example
`examples/basic.js`
## State model
`_chunks` list plus `_tokens` map; `_offset` byte position for replay.
## Performance
`readFromOffset` scans all chunks — compact periodically in app layer.
## Versioning
Token ids are local strings; persist offset externally for crash recovery.
## Security
Resume tokens do not authenticate peers; bind tokens to session secrets in app code.
## Related modules
- `hyper-p2p-stream-multiplex` — stream FIN boundaries
## Changelog
- Wave 89: byte-offset checkpoints for stream replay demos.
- `readFromOffset` replays chunk list after checkpoint byte position.
## See also
[`docs/architecture.md`](architecture.md)
@@ -1,84 +1,77 @@
# API: hyper-p2p-stream-tee
**Protocol:** `stream-tee/v1`
**Protocol:** `stream-tee/v1` (local fan-out)
**Export:** `HyperP2PStreamTee`
**Export:** `HyperP2PStreamTee`, `PROTOCOL`
## Overview
Production p2p module: Hyperswarm discovery + Protomux when `topic` is set.
## Constructor
```js
const mod = new HyperP2PStreamTee(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | `string` | `Buffer` | `null` | Hyperswarm topic; required for P2P `ready()` |
| `keyPair` | KeyPair | random | Ed25519 key pair |
Fan-out `write(chunk)` to named branches, each with its own FIFO chunk queue. Useful for teeing one ingress stream to logging and processing paths.
## Methods
### `addBranch(name)`
- **Returns:** `value`
- **Throws:**
- `Error: branch name required`
- **Returns:** unsubscribe function
- **Throws:** `branch name required`
### `write(chunk)`
### `write(chunk)` / `readBranch(name)` / `pending(name)`
- **Returns:** `value`
- **Throws:**
- `Error: chunk required`
### `readBranch(name)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
### `pending(name)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
### `getStats(—)`
- **Returns:** `object`
- **Throws:** — (none documented in method body)
### `ready(—)`
- **Returns:** `Promise`
- **Throws:** — (none documented in method body)
### `close(—)`
- **Returns:** `Promise<void>`
- **Throws:** — (none documented in method body)
### `getStats()` / `ready()` / `close()`
## Events
| Event | Payload |
|-------|---------|
| `data` | buf |
`data` on every write.
## getStats()
## Wire
Returns `{ ...this._stats }` — typically `ops`, `errors`, and module-specific counters (`created`, `relays`, `open`, `peers`, etc.).
Library-only modules may include `mode: 'local'`.
None.
## Errors
## Composition
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `stream-tee/v1`.
`hyper-p2p-stream-transform`, `hyper-p2p-stream-chunker`.
## Testing
```bash
npm install && npm test
```
`npm test`
## Example
`examples/basic.js`
## State model
Each branch stores chunk array; `write` duplicates buffer reference to all branches.
## Performance
Memory multiplies by branch count × queued chunks.
## Versioning
Local-only `stream-tee/v1` id.
## Security
Branches share buffer references; treat as trusted in-process only.
## Related modules
- `hyper-p2p-stream-transform` — per-branch mapping
## Changelog
- Wave 89: local tee for stream pipeline examples and integration tests.
- `addBranch` returns unsubscribe to drop a branch without closing tee.
## Integration checklist
1. `addBranch('audit')` and `addBranch('process')`.
2. `write(chunk)` fans out to both queues.
3. `readBranch('process')` consumes on worker tick.
4. `close()` clears all branches on shutdown.
5. Use `pending(name)` to detect backlog before applying backpressure upstream.
## See also
[`docs/architecture.md`](architecture.md)
@@ -1,84 +1,83 @@
# API: hyper-p2p-stream-transform
**Protocol:** `stream-transform/v1`
**Protocol:** `stream-transform/v1` (local map)
**Export:** `HyperP2PStreamTransform`
**Export:** `HyperP2PStreamTransform`, `PROTOCOL`
## Overview
Production p2p module: Hyperswarm discovery + Protomux when `topic` is set.
Buffered transform stage: optional `transform(buf)` on each `write`, output queue via `read()`.
## Constructor
```js
const mod = new HyperP2PStreamTransform(opts)
```
| Option | Type | Default | Description |
|--------|------|---------|-------------|
| `topic` | `string` | `Buffer` | `null` | Hyperswarm topic; required for P2P `ready()` |
| `keyPair` | KeyPair | random | Ed25519 key pair |
| Option | Description |
|--------|-------------|
| `transform` | `(buf) => Buffer \| string \| null` |
## Methods
### `setTransform(fn)`
- **Returns:** `value`
- **Throws:**
- `Error: transform must be a function`
Replace transform at runtime.
### `write(chunk)`
### `write(chunk)` / `read()` / `pending()`
- **Returns:** `value`
- **Throws:**
- `Error: chunk required`
### `getStats()`
### `read(—)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
### `pending(—)`
- **Returns:** `value`
- **Throws:** — (none documented in method body)
### `getStats(—)`
- **Returns:** `object`
- **Throws:** — (none documented in method body)
### `ready(—)`
- **Returns:** `Promise`
- **Throws:** — (none documented in method body)
### `close(—)`
- **Returns:** `Promise<void>`
- **Throws:** — (none documented in method body)
`in`, `out`, `protocol`.
## Events
| Event | Payload |
|-------|---------|
| `data` | out |
`data` when transform returns non-null.
## getStats()
## Wire
Returns `{ ...this._stats }` — typically `ops`, `errors`, and module-specific counters (`created`, `relays`, `open`, `peers`, etc.).
Library-only modules may include `mode: 'local'`.
None.
## Errors
## Composition
Stable message substrings: see [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
## P2P
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `stream-transform/v1`.
Between chunker and tee branches.
## Testing
```bash
npm install && npm test
```
`npm test`
## Example
`examples/basic.js`
## State model
Output queue `_out` after optional transform; null transform result skips enqueue.
## Performance
Transform should be fast; blocks `write` caller.
## Versioning
Local pipeline stage only.
## Security
Transforms run synchronously; untrusted transform functions can block the event loop.
## Related modules
- `hyper-p2p-stream-chunker` — fixed-size output
## Changelog
- Wave 89: pluggable transform fn with output queue.
- `setTransform` allows hot-swapping mapping without new instance.
## Integration checklist
1. Pass `transform` in constructor or `setTransform` later.
2. `write` upstream chunks; `read` downstream in loop.
3. Monitor `getStats().out` vs `in` for transform drops.
4. Pair with chunker for fixed frame sizes.
## See also
[`docs/architecture.md`](architecture.md)