Updates

2026-05-20 21:02:45 -04:00
parent 1f3f4b24a2
commit 14d0980b4f
734 changed files with 682 additions and 746 deletions
@@ -0,0 +1,221 @@
+# API: hyper-spatial-index
+
+**Protocol:** `hyper-spatial-index/v1` (optional point gossip only)
+
+**Export:** `SpatialIndex` (class)
+
+## Overview
+
+`SpatialIndex` is a **local-first geospatial index** for Bare/Pear: insert points `(x, y)` with arbitrary `data`, persist them in a **grid-bucketed Hyperbee** store, and run **range**, **nearest-neighbor**, and **radius** queries without requiring peer participation. An in-memory `localPoints` map mirrors recent inserts for fast scans.
+
+The module is **not** a distributed spatial shard or P2P query router. Hyperswarm integration (when `ready()` runs) only **replicates point records** via `{ type: 'point', point }` gossip; all geo algorithms read local Hyperbee + `localPoints`.
+
+## Constructor
+
+```js
+const index = new SpatialIndex(opts)
+```
+
+| Option | Type | Default | Description |
+|--------|------|---------|-------------|
+| `keyPair` | `KeyPair` | `hypercore-crypto.keyPair()` | Identity for Hypercore path (and swarm if used) |
+| `storageDir` | `string` | `{cwd}/hyper-spatial-index-storage` | Root directory; Hypercore at `spatial-core/` |
+| `gridSize` | `number` | `1000` | Cell width/height in coordinate units (meters, abstract units, etc.) |
+| `topic` | `string` \| `Buffer` \| `null` | `null` → `'hyper-spatial-index/v1'` at swarm init | Hyperswarm topic for point gossip; hashed if not 64-char hex |
+
+### Coordinate model
+
+- **Axes:** Cartesian `x`, `y` (no built-in lat/lon projection; use app-level conversion)
+- **Grid key:** `grid:{floor(x/gridSize)}:{floor(y/gridSize)}`
+- **Distance:** Euclidean `sqrt((x-x0)² + (y-y0)²)` in query methods
+
+## Lifecycle
+
+### `async ready()`
+
+Creates storage directory, opens Hyperbee on `spatial-core`, optionally joins Hyperswarm via `initModuleSwarm`, sets `_joined`, emits `ready`.
+
+- **Returns:** `Promise<void>`
+- **Throws:** Hypercore/Hyperbee initialization failures
+- Idempotent if already joined
+
+### `async close()`
+
+Destroys swarm (if any), closes Hyperbee, emits `closed`.
+
+- **Returns:** `Promise<void>`
+- **Throws:** — (swarm destroy is not wrapped in catch)
+
+Safe to call without `ready()` (unit tests do this).
+
+## Point records
+
+### Shape (insert / storage / query results)
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `id` | `string` | Unique point identifier |
+| `x` | `number` | X coordinate |
+| `y` | `number` | Y coordinate |
+| `data` | `object` | Application payload (default `{}`) |
+| `timestamp` | `number` | `Date.now()` at insert |
+
+Query methods may add **`distance`** (number) on returned objects.
+
+### `async insert(id, x, y, data = {})`
+
+Inserts or overwrites by `id` in `localPoints`, appends to the grid bucket in Hyperbee, emits `point-inserted`, gossips if `swarm` is active.
+
+- **Returns:** `Promise<Point>` — full point object
+- **Throws:** — (no validation throws; invalid ids/coords are caller responsibility)
+
+### `async deletePoint(id)`
+
+Removes from `localPoints` and filters the point out of the persisted grid bucket (deletes key if bucket empty).
+
+- **Returns:** `Promise<boolean>` — `true` if existed, `false` otherwise
+- **Throws:** — (none)
+
+## Geo queries (local)
+
+All query methods scan grid cells and filter in-process. No remote RPC or peer query protocol.
+
+### `async rangeQuery(minX, minY, maxX, maxY)`
+
+Axis-aligned bounding box query inclusive on bounds (`p.x >= minX && p.x <= maxX`, same for `y`).
+
+**Algorithm:**
+
+1. Compute grid index ranges from corners and `gridSize`
+2. For each `grid:gx:gy` key, load bucket array from Hyperbee
+3. Filter points inside the rectangle
+
+- **Returns:** `Promise<Point[]>` — unsorted; may include duplicates if same id existed in multiple buckets historically (normally one bucket per point)
+- **Throws:** — (none)
+
+### `async nearestNeighbor(x, y, k = 1)`
+
+Returns up to `k` closest points by Euclidean distance.
+
+**Algorithm:**
+
+1. Collect all `localPoints` with distance
+2. Scan neighboring grids within **±2** cells (`searchRadiusGrids = 2`) from Hyperbee, skip ids already in `localPoints`
+3. Dedupe by `id`, sort by `distance`, `slice(0, k)`
+
+- **Returns:** `Promise<(Point & { distance })[]>` — sorted nearest-first
+- **Throws:** — (none)
+
+### `async findInRadius(x, y, radius)`
+
+All points with `distance <= radius`, sorted nearest-first.
+
+**Algorithm:**
+
+1. Include matching `localPoints`
+2. Expand grid scan: `searchGrids = ceil(radius / gridSize) + 1` in each direction
+3. Load buckets, compute distance, dedupe, sort
+
+- **Returns:** `Promise<(Point & { distance })[]>`
+- **Throws:** — (none)
+
+**Choosing `gridSize`:** Should be on the order of typical query radius for efficiency; tests use `500` with radius `300`.
+
+## Introspection
+
+### `getStats()`
+
+- **Returns:** `{ ops: number, errors: number }` — shallow copy (counters reserved, not fully wired)
+- **Throws:** — (none)
+
+## Events
+
+| Event | Payload | When |
+|-------|---------|------|
+| `ready` | — | `ready()` completed |
+| `closed` | — | `close()` completed |
+| `point-inserted` | `{ id, x, y }` | After successful `insert` |
+| `point-deleted` | `{ id }` | After successful `deletePoint` |
+| `point-received` | `Point` | Inbound gossip `type: 'point'` (optional P2P) |
+
+There is no `error` event on the class; swarm errors are not forwarded in v0.3.1.
+
+## Grid bucket persistence
+
+| Hyperbee key | Value |
+|--------------|-------|
+| `grid:{gx}:{gy}` | `Point[]` — all points whose coordinates fall in that cell |
+
+`insert` **appends** to the array (does not dedupe by id in bucket). `deletePoint` filters by `id` within the point’s cell only.
+
+## getStats()
+
+| Field | Type | Description |
+|-------|------|-------------|
+| `ops` | `number` | Reserved operation counter |
+| `errors` | `number` | Reserved error counter |
+
+## Errors
+
+This module’s public geo API does not throw validation errors. Failures are typically I/O or Hyperbee related during `ready()` / `insert` / queries.
+
+| Scenario | Typical failure |
+|----------|-----------------|
+| Missing storage permissions | `bare-fs` mkdir/read errors |
+| Corrupt Hyperbee value | Runtime errors iterating non-array bucket values |
+
+Cross-module conventions: [`../../_shared/ERROR_CODES.md`](../../_shared/ERROR_CODES.md).
+
+## Optional replication (not geo routing)
+
+When `ready()` initializes Hyperswarm (`topic` defaults to protocol string):
+
+| Wire `type` | Fields | Behavior |
+|-------------|--------|----------|
+| `point` | `point: Point` | Receiver sets `localPoints`, emits `point-received` |
+
+`insert` calls `_gossipPoint` to fan out to connected peers via `gossipSend`. **Queries never consult remote peers** — replicated points land in `localPoints` and/or Hyperbee only after the app inserts or receives gossip.
+
+For **offline / single-process** use: call geo methods after `ready()`; ignore `point-received` unless building a replicated cache.
+
+## Usage patterns
+
+### Local geo service
+
+```js
+const index = new SpatialIndex({ storageDir: './my-geo-db', gridSize: 250 })
+await index.ready()
+await index.insert('node-1', 40.71, -74.00, { label: 'NYC' })
+const nearby = await index.findInRadius(40.71, -74.00, 5000)
+await index.close()
+```
+
+### Range + nearest
+
+```js
+const box = await index.rangeQuery(0, 0, 500, 500)
+const top2 = await index.nearestNeighbor(120, 120, 2)
+```
+
+## Performance notes
+
+- Complexity scales with **number of grid cells intersecting the query region**, not total global points
+- Large `radius` or wide `rangeQuery` spans many cells — tune `gridSize` to workload
+- `nearestNeighbor` only searches ±2 neighbor cells beyond center; distant points outside that window may be **missed** (documented demo limitation; production should widen radius or use hierarchical index)
+
+## Testing
+
+```bash
+cd modules/indexes-search/hyper-spatial-index && npm install && npm test
+```
+
+Unit tests: [`../test/test.js`](../test/test.js) — insert, range, nearest, delete, `findInRadius`.
+
+Integration (swarm smoke only): [`../../../real_tests/integration/spatial-index-two-node.js`](../../../real_tests/integration/spatial-index-two-node.js).
+
+## Related modules
+
+| Module | Relationship |
+|--------|----------------|
+| `hyper-p2p-intent-router` | Route to services discovered by intent, not by coordinates |
+| `hyper-p2p-semantic-vector-index` | Non-geographic similarity search |
@@ -0,0 +1,202 @@
+# Architecture: hyper-spatial-index
+
+**Category:** Indexes & search ([`../../MODULE_CATEGORIES.md`](../../MODULE_CATEGORIES.md))
+
+**Composes with:** `hyper-p2p-semantic-vector-index`, `hyper-p2p-intent-router` (app-level: geo index + intent routing)
+
+**Primary surface:** `SpatialIndex` — grid-backed geospatial queries on local Hyperbee. **Not** a P2P spatial query mesh.
+
+## Layer diagram
+
+```mermaid
+flowchart TB
+  subgraph app [Application]
+    INS[insert / deletePoint]
+    Q1[rangeQuery]
+    Q2[nearestNeighbor]
+    Q3[findInRadius]
+  end
+  subgraph index [SpatialIndex]
+    LP[localPoints Map]
+    GK[_getGridKey]
+  end
+  subgraph store [Persistence]
+    HB[(Hyperbee spatial-core)]
+  end
+  subgraph optional [Optional replication]
+    SW[Hyperswarm]
+    GS[gossipSend point]
+  end
+  INS --> LP
+  INS --> GK --> HB
+  INS -.-> GS
+  Q1 --> HB
+  Q2 --> LP
+  Q2 --> HB
+  Q3 --> LP
+  Q3 --> HB
+  GS --> SW
+  SW -.-> LP
+```
+
+Solid lines: required for geo correctness. Dotted: optional peer point cache, not used by query algorithms.
+
+## Query sequence (local)
+
+```mermaid
+sequenceDiagram
+  participant App
+  participant SI as SpatialIndex
+  participant LP as localPoints
+  participant HB as Hyperbee
+  App->>SI: ready()
+  SI->>HB: open spatial-core
+  App->>SI: insert(id, x, y, data)
+  SI->>LP: set(id, point)
+  SI->>HB: get grid:gx:gy → append → put
+  App->>SI: findInRadius(x, y, r)
+  SI->>LP: scan all local with dist <= r
+  SI->>HB: scan grid cells in expanded window
+  SI->>SI: dedupe, sort by distance
+  SI-->>App: Point[]
+```
+
+No peer participates in the query path.
+
+## Grid indexing model
+
+The implementation uses a **fixed uniform grid** (quadtree-inspired comment in source; structure is flat cells, not a tree).
+
+```
+gridSize = 1000 (default)
+
+gx = floor(x / gridSize)
+gy = floor(y / gridSize)
+key = "grid:" + gx + ":" + gy
+value = [ Point, Point, ... ]
+```
+
+```mermaid
+flowchart LR
+  subgraph cell ["grid:1:2"]
+    P1[p1]
+    P2[p2]
+  end
+  INS[insert at x,y] --> cell
+  cell --> HB[(Hyperbee)]
+```
+
+### rangeQuery cell coverage
+
+```
+minGx..maxGx = floor(minX/gridSize) .. floor(maxX/gridSize)
+minGy..maxGy = floor(minY/gridSize) .. floor(maxY/gridSize)
+nested loops → get each key → bbox filter
+```
+
+### findInRadius cell coverage
+
+```
+searchGrids = ceil(radius / gridSize) + 1
+for dx, dy in [-searchGrids .. +searchGrids]
+  load grid:(gx+dx):(gy+dy)
+  euclidean filter dist <= radius
+```
+
+### nearestNeighbor cell coverage
+
+```
+searchRadiusGrids = 2  (fixed)
+center (gx, gy) from query (x, y)
+scan (gx±2, gy±2) plus full localPoints
+```
+
+## State model
+
+| Structure | Key | Value | Lifecycle |
+|-----------|-----|-------|-----------|
+| `localPoints` | `id` | `Point` | Updated on `insert`, `deletePoint`, inbound gossip |
+| Hyperbee | `grid:gx:gy` | `Point[]` | Append on insert; filter on delete; read on queries |
+| `swarm` | — | Hyperswarm instance | Set in `_initSwarm`; destroyed on `close` |
+| `_peerMsgs` | `peerHex` | Protomux msg | Managed inside `initModuleSwarm` (shared helper) |
+| `_joined` | — | boolean | After first `ready()` |
+
+### Storage layout on disk
+
+```
+{storageDir}/
+  spatial-core/     # Hypercore (default encoding)
+    Hyperbee keys: grid:{gx}:{gy} → JSON Point[]
+```
+
+Directory creation swallows errors in `_initStorage` (empty catch); ensure `storageDir` is writable in production.
+
+## Optional wire: point gossip
+
+Protocol id: `hyper-spatial-index/v1` (`SPATIAL_PROTOCOL`).
+
+| type | fields | direction | behavior |
+|------|--------|-----------|----------|
+| `point` | `point: { id, x, y, data, timestamp }` | peer → peer | `onmessage` stores in `localPoints`, emits `point-received` |
+
+**Direction on insert:** local `insert` → `_gossipPoint` → `gossipSend(this, { type: 'point', point })` to all entries in `_peerMsgs`.
+
+This is **eventual replication of records**, not:
+
+- Partitioned spatial sharding
+- Federated range query
+- Nearest-neighbor across the network
+
+Applications that need cluster-wide geo search must merge peer `point-received` into Hyperbee themselves or run a coordinator.
+
+```mermaid
+sequenceDiagram
+  participant A as Peer A
+  participant B as Peer B
+  A->>A: insert → Hyperbee + localPoints
+  A->>B: gossip { type: point, point }
+  B->>B: localPoints.set (no Hyperbee write in handler)
+  Note over B: Queries still local unless app persists received points
+```
+
+Inbound gossip in v0.3.1 **does not** call `bee.put` — only `localPoints`. For durable shared indexes, mirror `point-received` into `insert` or a custom persistence hook.
+
+## Algorithm comparison
+
+| Method | Grid scan | localPoints | Sort | Dedupe |
+|--------|-----------|-------------|------|--------|
+| `rangeQuery` | bbox cells only | not scanned separately | no | no |
+| `nearestNeighbor` | ±2 cells | yes | by distance | yes |
+| `findInRadius` | expanded by radius | yes | by distance | yes |
+
+## Composition patterns
+
+| Pattern | Modules |
+|---------|---------|
+| Geo-fenced app data | spatial-index only |
+| “Find service near me” | spatial-index for coords + intent-router for capability routing |
+| Embedding search | semantic-vector-index (orthogonal axis) |
+
+Example stack line from categories doc: **Geo / ML app** → `spatial-index` or `semantic-vector-index` + `hyper-p2p-reactive-state`.
+
+## Limits and evolution
+
+| Limit | Detail |
+|-------|--------|
+| Grid not quadtree | No hierarchical split; dense cells degrade to linear scan per cell |
+| `nearestNeighbor` window | ±2 cells may omit global nearest point |
+| Bucket append | Re-insert same `id` without delete can duplicate in Hyperbee array |
+| Gossip → memory only | Received points not auto-persisted to Hyperbee |
+| Stats | `_stats.ops` / `errors` not updated in all paths |
+
+Reasonable upgrades (out of scope for current file): R-tree/quadtree structure, Hyperbee write on gossip, idempotent upsert per cell, configurable neighbor search radius.
+
+## Bare runtime constraints
+
+- Uses `bare-fs`, `bare-path`, `bare-process`, `bare-events`, `bare-crypto`, `bare-timers`
+- No Node.js APIs; compatible with Pear/Bare bundles via `package.json` `imports` map
+
+## Security
+
+- Point `data` is unauthenticated JSON from peers when gossip is enabled
+- Treat `point-received` as untrusted input; validate `id`, bounds, and schema before use in safety-critical geo logic