Raven ScottandCursor
e43e2e83f1
Extend all eight index/search packages with practical helpers for app development and fix semantic-vector getStats() to expose real metrics. Per-module code: - bloom-gossip: listKeys(), clear() - inverted-index: searchAll(and|or), getDocTerms() - fulltext-lite: search(and|or), suggest(prefix) - trie-prefix: countPrefix(), autocomplete(limit) - graph-index: removeEdge(), outDegree(), edgeCount() - similarity-lsh: nearVector(), bucketCount() - semantic-vector-index: getStats() delegates to getMetrics() + PROTOCOL export - spatial-index: listLocalPoints(), pointCount(), protocol in getStats() Tests added for new APIs across six packages; all indexes-search npm test suites pass (8/8). Docs: - Rewrite indexes-search category README (module table, composition) - Expand bloom-gossip and semantic-vector architecture notes - modules/README.md: link indexes-search doc hub Parent workspace docs/indexes-search/README.md and MODULE_DOC_PASS.md updated separately (outside this git root). Co-authored-by: Cursor <[email protected]>
60 lines
1.8 KiB
Markdown
60 lines
1.8 KiB
Markdown
# Architecture: hyper-p2p-bloom-gossip
|
|
|
|
**Category:** indexes-search
|
|
**Protocol:** `bloom-gossip/v1`
|
|
|
|
## Purpose
|
|
|
|
Replicate “seen key” hints across a Hyperswarm topic using a compact bloom filter plus incremental `bloom-add` gossip, avoiding full filter snapshots on every insert.
|
|
|
|
## Component diagram
|
|
|
|
```
|
|
Application
|
|
│
|
|
▼
|
|
HyperP2PBloomGossip
|
|
├── Uint8Array bit field (_bits)
|
|
├── Set _keys (exact membership for stats)
|
|
└── initModuleSwarm → Protomux JSON channel
|
|
```
|
|
|
|
## Wire messages
|
|
|
|
| type | direction | fields | behavior |
|
|
|------|-----------|--------|----------|
|
|
| `bloom-add` | outbound / inbound | `key`, `peer`, `at` | Set bloom bits for `key` if not already in `_keys`; emit `remote-add` on inbound |
|
|
|
|
Ignored: messages with missing `type`, wrong `type`, or missing `key`.
|
|
|
|
## State model
|
|
|
|
| Field | Role |
|
|
|-------|------|
|
|
| `_bits` | Packed bloom array, length `ceil(_size / 8)` |
|
|
| `_keys` | Exact keys seen (local + remote) |
|
|
| `_stats` | Counters: `added`, `queries`, `gossipIn`, `gossipOut` |
|
|
| `peerHex` | Local public key hex on gossip payloads |
|
|
|
|
## Introspection
|
|
|
|
- `listKeys()` — exact key set (for debugging; Bloom `mightContain` remains probabilistic for unknown keys)
|
|
- `clear()` — zero bits and keys
|
|
- `exportBits()` / `importBits()` — hand off filter state to new peers
|
|
|
|
## Lifecycle
|
|
|
|
1. Construct with optional `topic` and `bits`
|
|
2. `await ready()` — joins swarm when `topic` set
|
|
3. `add` / `mightContain` / `exportBits`
|
|
4. `await close()` — zero filter, tear down swarm
|
|
|
|
## Composition
|
|
|
|
Uses `../../_shared/p2p-bare.js` (`initModuleSwarm`, `gossipSend`) and `../../_shared/lib/errors.js` (`assertNonEmpty`).
|
|
|
|
## Trade-offs
|
|
|
|
- **Pros:** Low bandwidth per new key; fast negative lookups
|
|
- **Cons:** False positives; `importBits` does not sync `_keys`; no delete gossip
|