57 lines
1.4 KiB
Markdown
57 lines
1.4 KiB
Markdown
# Architecture: hyper-p2p-inverted-index
|
|
|
|
**Category:** indexes-search
|
|
**Protocol:** `inverted-index/v1`
|
|
|
|
## Purpose
|
|
|
|
Distributed term → document inverted index with incremental gossip sync.
|
|
|
|
## Dual index
|
|
|
|
```
|
|
_terms: Map<term, Set<docId>>
|
|
_docs: Map<docId, Set<term>>
|
|
```
|
|
|
|
Keeps remove and re-index operations consistent.
|
|
|
|
## Wire messages
|
|
|
|
| type | op | fields | behavior |
|
|
|------|-----|--------|----------|
|
|
| `term-index-sync` | `add` | `term`, `docId`, `peer`, `at` | Insert posting; emit `remote-index` |
|
|
| `term-index-sync` | `remove` | `term`, `docId`, `peer`, `at` | Delete posting; prune empty term; emit `remote-remove` |
|
|
|
|
## Gossip flow
|
|
|
|
```
|
|
index(doc, terms)
|
|
→ for new (term, doc) pairs → gossipSend add
|
|
removeDoc(doc)
|
|
→ for each term → gossipSend remove
|
|
_onGossip
|
|
→ merge into _terms / _docs
|
|
```
|
|
|
|
## Swarm integration
|
|
|
|
`initModuleSwarm` from `../../_shared/p2p-bare.js` registers Protomux JSON handler → `_onGossip`.
|
|
|
|
## Lifecycle
|
|
|
|
1. Construct with `topic`
|
|
2. `ready()` — join topic
|
|
3. `index` / `search` / `removeDoc`
|
|
4. `close()` — teardown
|
|
|
|
## Trade-offs
|
|
|
|
- Gossip per term change (not batch snapshots)
|
|
- No ranking or TF-IDF
|
|
- Re-index replaces doc terms locally but only gossips **new** terms on `index` (not automatic diff remove for changed terms — re-call `removeDoc` or manage terms explicitly)
|
|
|
|
## Composition
|
|
|
|
Pair with `hyper-p2p-fulltext-lite` locally for snippets; use Hyperbee at app layer for persistence.
|