Updates

state-crdts/hyper-p2p-reactive-state/docs/api.md

@@ -0,0 +1,238 @@
+# API: hyper-p2p-reactive-state
+
+**Protocol:** `hyper-p2p-reactive-state/v1` (`REACTIVE_STATE_PROTOCOL`)
+
+**Export:** `{ HyperP2PReactiveState, PROTOCOL }` from `index.js`
+
+## Overview
+
+`HyperP2PReactiveState` is a Bare/Pear–compatible reactive key–value store that merges concurrent writes with an **LWW-Register** CRDT (last-writer-wins by `timestamp`, with deterministic **`peerId` lexicographic tie-break**). Local changes emit fine-grained events (`change`, `set`, `delete`). When a Hyperswarm `topic` is configured, peers exchange state over a dedicated **Protomux** channel using `REACTIVE_STATE_PROTOCOL`. Optional **Hyperbee** persistence (backed by **Hypercore**) provides offline-first reload of all `lww:` keys.
+
+The class extends `bare-events` `EventEmitter`. Call `ready()` before relying on storage or swarm membership; call `close()` to tear down timers, swarm, and Hyperbee.
+
+## Constructor
+
+```js
+const { HyperP2PReactiveState, PROTOCOL } = require('hyper-p2p-reactive-state')
+const state = new HyperP2PReactiveState(opts)
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `keyPair` | Hypercore `KeyPair` | `hypercore-crypto.keyPair()` | Ed25519 identity for Hypercore and swarm |
+| `topic` | `string` \| `Buffer` \| `null` | `null` | Hyperswarm discovery topic. `null` skips swarm join (local + persistence only) |
+| `storageDir` | `string` | `path.join(process.cwd(), 'hyper-p2p-reactive-state-storage')` | Hypercore storage directory |
+| `syncInterval` | `number` | `15000` | Interval (ms) for background sync tick when `enableBackgroundTimers` is true |
+| `expiry` | `number` | `300000` | Peer presence TTL (ms) before `peer-expired` |
+| `metadata` | `object` | `{ agent: 'hyper-p2p-reactive-state' }` | Opaque metadata attached to peer records (not sent on wire by default) |
+| `enableBackgroundTimers` | `boolean` | `false` | When `true`, starts periodic sync broadcast intent and peer cleanup |
+| `memoryOnly` | `boolean` | `false` | When `true`, skips Hypercore/Hyperbee; in-memory `state` Map only |
+
+### Read-only properties
+
+| Property | Type | Description |
+|----------|------|-------------|
+| `publicKey` | `Buffer` | `keyPair.publicKey` |
+| `publicKeyHex` | `string` | Hex-encoded public key; used as default `peerId` for local writes |
+
+## Lifecycle
+
+### `async ready() → HyperP2PReactiveState`
+
+Idempotent. Initializes Hyperbee (unless `memoryOnly`), joins Hyperswarm when `topic` is set, optionally starts background timers, sets `_joined`, emits `ready`.
+
+Does not throw module-specific errors; underlying `fs.mkdir`, Hypercore, or Hyperswarm failures propagate as standard `Error`.
+
+### `async close() → void`
+
+Clears sync/cleanup intervals, destroys swarm, closes Hyperbee and Hypercore, sets `_joined = false`, emits `closed`. Errors during teardown are swallowed with `.catch(() => {})`.
+
+## Methods
+
+### `async set(key, value, opts = {}) → { key, value, timestamp, peerId }`
+
+Applies LWW merge locally. If the write wins:
+
+1. Updates in-memory `state` Map
+2. Emits `change` and `set`
+3. Persists to Hyperbee at `lww:{key}` when `bee` is active
+4. Broadcasts `{ type: 'update', data: { key, value, timestamp, peerId } }` to all open peer channels
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `key` | `string` | — | Logical state key |
+| `value` | `any` | — | JSON-serializable value stored in LWW entry |
+| `opts.timestamp` | `number` | `Date.now()` | LWW clock; higher wins |
+| `opts.peerId` | `string` | `publicKeyHex` | Writer id for tie-break and attribution |
+
+**Returns:** `{ key, value, timestamp, peerId }` whether or not the write changed local state (losing concurrent writes still return the attempted metadata).
+
+**Stored entry shape:**
+
+```json
+{
+  "value": "<any>",
+  "timestamp": 1710000000000,
+  "peerId": "<hex>",
+  "updatedAt": 1710000000123
+}
+```
+
+### `get(key) → object | null`
+
+Returns the full LWW entry `{ value, timestamp, peerId, updatedAt, deleted? }` or `null` if absent. Use `.value` for the payload; check `.deleted === true` after `delete()`.
+
+### `async delete(key, opts = {}) → true`
+
+Writes a tombstone: `value: null`, `deleted: true`, `timestamp: (opts.timestamp || Date.now()) + 1` so delete wins over a prior set at the same wall clock. Emits `change` (type `delete`) and `delete`, persists, broadcasts update.
+
+| Parameter | Type | Default | Description |
+|-----------|------|---------|-------------|
+| `opts.timestamp` | `number` | `Date.now()` | Base timestamp before `+1` bump |
+| `opts.peerId` | `string` | `publicKeyHex` | Writer id |
+
+### `subscribe(keyOrKeys, callback) → unsubscribeFn`
+
+| Parameter | Type | Description |
+|-----------|------|-------------|
+| `keyOrKeys` | `string` \| `string[]` | Key to watch, or `'*'` for all keys |
+| `callback` | `(change) => void` | Invoked on matching `change` events |
+
+**`change` payload:** `{ key, value, peer, type }` where `type` is `'set'` or `'delete'`, `peer` is hex public key of the writer, `value` is the new value (or `null` when deleted).
+
+Returns a function that removes all handlers registered for this subscription.
+
+### `query(filter = {}) → array`
+
+Scans in-memory `state`. Returns `[{ key, value, timestamp, peerId, updatedAt, deleted? }, ...]`.
+
+| Filter field | Type | Behavior |
+|--------------|------|----------|
+| `keyPrefix` | `string` | Include keys where `key.startsWith(keyPrefix)` |
+| `minTimestamp` | `number` | Exclude entries with `timestamp < minTimestamp` |
+
+### `toJSON() → object`
+
+Plain map `key → value` (tombstones appear as `null` values).
+
+### `getPeers() → array`
+
+`[{ publicKey, lastSeen, metadata, msg? }, ...]` from the live peer registry.
+
+### `getStats() → { ops: number, errors: number }`
+
+Shallow copy of internal counters (`ops` / `errors` are reserved; not incremented in current `index.js`).
+
+## LWW conflict resolution
+
+For a given `key`, remote entry **R** replaces local **L** when:
+
+1. No local entry exists, or
+2. `R.timestamp > L.timestamp`, or
+3. `R.timestamp === L.timestamp` and `R.peerId > L.peerId` (lexicographic string compare)
+
+This matches `_shouldUpdate()` and is covered by unit tests (`test/test.js`).
+
+## Events
+
+| Event | When | Payload fields |
+|-------|------|----------------|
+| `ready` | After `ready()` completes | — |
+| `loaded` | Hyperbee load finished | `{ count: number }` — number of `lww:` keys restored |
+| `swarm-ready` | Hyperswarm wired (topic set) | — |
+| `peer-connected` | Protomux channel open | `{ peer: string, metadata: object }` |
+| `peer-disconnected` | Channel closed | `{ peer: string }` |
+| `peer-expired` | `lastSeen` older than `expiry` | `{ peer: string }` |
+| `change` | Any winning local or remote write | `{ key, value, peer, type: 'set' \| 'delete' }` |
+| `set` | Local or remote set won merge | `{ key, value, peer }` |
+| `delete` | Local delete | `{ key, peer }` |
+| `sync` | Snapshot merge changed state | `{ peer, type: 'snapshot' }` |
+| `sync-tick` | Background timer (if enabled) | `{ stateSize: number }` |
+| `update-broadcast` | After `_broadcastUpdate` | `{ key, data }` — full LWW entry |
+| `error` | Persistence failure | `Error` |
+| `closed` | After `close()` | — |
+
+## Wire protocol (Protomux)
+
+Encoding: `compact-encoding` **JSON** via shared `protocolChannel()` (`../_shared/p2p-bare.js`). Channel id: **`hyper-p2p-reactive-state/v1`**.
+
+All messages are objects with a `type` field.
+
+### Outbound / inbound message types
+
+| `type` | Fields | Direction | Behavior |
+|--------|--------|-----------|----------|
+| `snapshot` | `data: Record<string, LWWEntry>` | server → client on `onopen`; either peer may send | `_mergeSnapshot`: LWW-merge each key; emit `change` per key; `sync` if any change; `_persistState` |
+| `update` | `data: { key, value, timestamp, peerId }` | either | `_applyUpdate`: build LWW entry, merge, emit `change`/`set`, persist |
+| `query` | (handler not implemented) | either | Routed in `_handleProtocolMessage` to `_handleQuery` (undefined in current code — reserved) |
+
+**`update` example:**
+
+```json
+{
+  "type": "update",
+  "data": {
+    "key": "user:alice",
+    "value": { "name": "Alice", "score": 100 },
+    "timestamp": 1710000000000,
+    "peerId": "a1b2c3..."
+  }
+}
+```
+
+**`snapshot` example:**
+
+```json
+{
+  "type": "snapshot",
+  "data": {
+    "config:theme": {
+      "value": "dark",
+      "timestamp": 1710000000000,
+      "peerId": "deadbeef...",
+      "updatedAt": 1710000000100
+    }
+  }
+}
+```
+
+On each new peer connection, the opener sends a full in-memory snapshot immediately (`_sendStateSnapshot`).
+
+## Hyperbee persistence
+
+| Key prefix | Value | Notes |
+|------------|-------|-------|
+| `lww:{key}` | LWW entry JSON | Loaded on startup with range `[lww:, lww:\xff]` |
+| `meta:` | (constant defined, unused in `index.js`) | Reserved |
+
+Hypercore is opened at `storageDir` with `valueEncoding: 'json'`. `_persistState()` rewrites all in-memory keys (full scan) after snapshot merge; `set`/`delete` write single keys.
+
+## P2P prerequisites
+
+- Shared `topic` (64-char hex or string hashed via `topicToBuffer`)
+- Distinct `storageDir` per peer recommended; same `keyPair` only when intentionally sharing identity
+- Set `enableBackgroundTimers: false` in tests to avoid timer noise
+
+Topic normalization: see `topicToBuffer()` in [`../_shared/p2p-bare.js`](../../_shared/p2p-bare.js).
+
+## Errors
+
+This module does not define a custom error enum. Failures surface as plain `Error` from Bare FS, Hypercore, or Hyperswarm. See [`../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md) for cross-module patterns.
+
+Shared helper `createSwarm` throws `Error('topic is required for createSwarm')` if topic is missing — not used when reactive-state `topic` is `null`.
+
+## Testing
+
+```bash
+cd modules/state-crdts/hyper-p2p-reactive-state
+npm install && npm test
+```
+
+- Unit: `test/test.js` — LWW, persistence, subscribe, lifecycle
+- Integration: [`../../../real_tests/integration/reactive-state-two-node.js`](../../../real_tests/integration/reactive-state-two-node.js)
+
+Run example:
+
+```bash
+bare examples/basic-usage.js
+```

state-crdts/hyper-p2p-reactive-state/docs/architecture.md

@@ -0,0 +1,173 @@
+# Architecture: hyper-p2p-reactive-state
+
+**Category:** State & CRDTs
+
+**Protocol:** `hyper-p2p-reactive-state/v1` (`REACTIVE_STATE_PROTOCOL`)
+
+**Composes with:** `hyper-p2p-crdt-map`, `hyper-p2p-conflict-set`
+
+## Problem and approach
+
+Decentralized Bare/Pear apps need shared mutable state without a central server. This module combines:
+
+1. **LWW-Register CRDT** — one convergent value per key
+2. **Reactive API** — `EventEmitter` + `subscribe()` for UI and pipelines
+3. **Hyperswarm + Protomux** — pairwise replication
+4. **Hyperbee on Hypercore** — durable `lww:` keyspace for restart survival
+
+Unlike a full JSON document CRDT, granularity is **per key**, so peers can update different keys concurrently without conflict.
+
+## Layer diagram
+
+```mermaid
+flowchart TB
+  subgraph app [Application]
+    UI[UI / game logic]
+  end
+  subgraph mod [HyperP2PReactiveState]
+    API[set get delete query subscribe]
+    LWW[LWW merge _shouldUpdate]
+    MEM[(state Map)]
+    PEERS[(peers Map)]
+    MSGS[( _peerMsgs Map)]
+  end
+  subgraph persist [Persistence]
+    HC[Hypercore json]
+    HB[Hyperbee utf-8 keys]
+  end
+  subgraph net [P2P stack]
+    MUX[Protomux channel]
+    SW[Hyperswarm]
+  end
+  UI --> API
+  API --> LWW
+  LWW --> MEM
+  API --> HB
+  HB --> HC
+  LWW --> MUX
+  MUX --> SW
+  SW --> MUX
+  MUX --> LWW
+  MUX --> MSGS
+```
+
+## Primary sequence: ready → peer join → sync
+
+```mermaid
+sequenceDiagram
+  participant App
+  participant RS as HyperP2PReactiveState
+  participant HB as Hyperbee
+  participant SW as Hyperswarm
+  participant Peer as Remote peer
+
+  App->>RS: ready()
+  RS->>HB: createReadStream lww:*
+  HB-->>RS: restore entries
+  RS-->>App: loaded
+  RS->>SW: join(topic)
+  RS-->>App: swarm-ready
+
+  SW->>RS: connection + mux
+  RS->>Peer: Protomux open REACTIVE_STATE_PROTOCOL
+  RS->>Peer: snapshot { data: all keys }
+  Peer-->>RS: snapshot / update messages
+  RS->>RS: _shouldUpdate per key
+  RS->>HB: put lww:key
+  RS-->>App: change / sync
+```
+
+## LWW state model
+
+Each key maps to one register:
+
+```
+Register(key) = { value, timestamp, peerId, updatedAt, deleted? }
+```
+
+| Field | Role |
+|-------|------|
+| `timestamp` | Primary LWW clock (milliseconds; app may override via `opts`) |
+| `peerId` | Tie-break when timestamps equal (hex string; default `publicKeyHex`) |
+| `value` | Application payload; `null` when tombstoned |
+| `deleted` | `true` on `delete()` tombstones |
+| `updatedAt` | Local wall clock of last mutation (not used in merge) |
+
+**Merge rule (deterministic):** accept remote iff newer timestamp, or same timestamp and `remote.peerId > local.peerId` (lexicographic).
+
+**Delete semantics:** `delete()` sets `timestamp` to `base + 1` so tombstones beat a concurrent `set()` at the same base time.
+
+## In-memory structures
+
+| Structure | Type | Purpose |
+|-----------|------|---------|
+| `state` | `Map<key, LWWEntry>` | Authoritative merged view |
+| `peers` | `Map<peerPubHex, { lastSeen, metadata, msg? }>` | Presence; `msg` for outbound sends |
+| `_peerMsgs` | `Map<peerPubHex, ProtomuxMessage>` | Broadcast targets for `update` |
+| `subscriptions` | `Set<key>` | Keys registered via `subscribe()` (tracking only) |
+
+## Wire messages (detailed)
+
+Protomux channel protocol string: **`hyper-p2p-reactive-state/v1`**. Payload encoding: **JSON** (`compact-encoding/json`).
+
+| type | fields | direction | behavior |
+|------|--------|-----------|----------|
+| `snapshot` | `data: { [key: string]: LWWEntry }` | bidirectional; sent on channel `onopen` from acceptor | Iterate keys; `_shouldUpdate`; `state.set`; per-key `change`; batch `sync` + `_persistState` if any change |
+| `update` | `data: { key, value, timestamp, peerId }` | bidirectional; sent after local `set`/`delete` | Construct entry; merge; `change` + `set`; single-key `bee.put` |
+| `query` | (reserved) | — | Parsed in `_handleProtocolMessage` but `_handleQuery` is not implemented — do not rely on query/response in production until added |
+
+Invalid or missing `type` is ignored (`_handleProtocolMessage` early return).
+
+### Connection lifecycle
+
+1. `wireConnection` → `protocolChannel(mux, { protocol: REACTIVE_STATE_PROTOCOL, ... })`
+2. `onopen`: register peer, store `msg`, emit `peer-connected`, **`_sendStateSnapshot(msg)`**
+3. `onmessage`: dispatch by `type`
+4. `onclose`: remove peer maps, emit `peer-disconnected`
+
+## Hyperbee / Hypercore layout
+
+```
+storageDir/
+  Hypercore (keyPair, valueEncoding: json)
+    Hyperbee (keyEncoding: utf-8, valueEncoding: json)
+      lww:{applicationKey}  →  LWWEntry
+```
+
+Startup scan: `gte: 'lww:'`, `lte: 'lww:\xff'`. Keys strip the `LWW_PREFIX` (`lww:`) to recover application key names.
+
+`memoryOnly: true` skips the entire persistence subgraph; P2P still works when `topic` is set.
+
+Constants in `index.js` (for cross-module tooling): `STATE_DB_NAME = 'reactive-state'`, `META_PREFIX = 'meta:'` (unused), `DEFAULT_SYNC_INTERVAL = 15000`, `DEFAULT_EXPIRY = 300000`.
+
+## Timers and background work
+
+| Timer | Interval | When | Action |
+|-------|----------|------|--------|
+| `syncTimer` | `syncInterval` (default 15s) | `enableBackgroundTimers: true` | `_broadcastUpdates` → emits `sync-tick` (full fan-out not implemented beyond connection-time snapshot + per-`set` broadcast) |
+| `cleanupTimer` | 60s | `enableBackgroundTimers: true` | Remove stale `peers` entries past `expiry` |
+
+## Composition in the network stack
+
+Use a **shared Hyperswarm topic** per app session. Pair with:
+
+- **`hyper-p2p-crdt-map`** — multi-key map structures when LWW-per-key is insufficient
+- **`hyper-p2p-conflict-set`** — set CRDT for membership rosters
+- **`hyper-p2p-distributed-event-bus`** — append-only domain events while this module holds latest snapshot state
+
+See [`../_shared/WAVE6_NETWORK_STACK.md`](../../_shared/WAVE6_NETWORK_STACK.md) for overlay, pool, and handshake ordering when stacking modules.
+
+## Operational notes
+
+- **Idempotent `ready()`:** safe to call multiple times
+- **Concurrent writers:** always pass monotonic or wall-clock `timestamp`; use distinct `peerId` per device (default public key satisfies tie-break)
+- **Large state:** snapshot on connect is O(all keys); shard topics or namespaces by key prefix for scale
+- **Security:** wire payloads are unsigned JSON; trust the swarm topic and transport encryption; add app-layer signing if needed
+
+## Testing architecture
+
+| Layer | File |
+|-------|------|
+| Unit | `test/test.js` — LWW tie-break, persistence reload, subscribe |
+| Integration | `real_tests/integration/reactive-state-two-node.js` — snapshot merge path |
+| Example | `examples/basic-usage.js` — local reactive flow |