[Incremental Research] 2026-02-19_09:30

PROGRESS.md

@@ -1,19 +1,19 @@
 # PROGRESS.md - Holepunch KB Tracker
 
-**Last Run:** 2026-02-19 09:25 UTC (Run #12)
+**Last Run:** 2026-02-19 09:30 UTC (Run #13)
 
 **Metrics:**
-- Files: 28 MD files
+- Files: 29 MD files
-- Est. Chars: 41k+
+- Est. Chars: 44k+
-- Modules Covered: 13/120+
+- Modules Covered: 14/120+
-- Project Ideas: 24/60+
+- Project Ideas: 29/60+
-- Progress: 28%
+- Progress: 30%
 
-**Recent Git:** [latest commit hash will be here post-push]
+**Recent Git:** 1dbc3a6 [Incremental Research] 2026-02-19_09:25
 
-**This Run (~3200 new chars):**
+**This Run (~3500 new chars):**
-- Expanded: modules/hyperdrive.md (+2500 chars: full API, internals, diagrams, ex)
+- Expanded: modules/hyperbee.md (+~4200 chars: B-tree internals, full API, protobuf notes, watchers/diffs, ex)
-- New: project-ideas/cycle5.md (5 structured ideas)
+- New: project-ideas/cycle6.md (5 DHT/RPC ideas)
-- Updated: PROGRESS.md, interconnections/README.md (minor refs)
+- Updated: PROGRESS.md
 
-**Next Run:** modules/hyperbee.md (indexing deep dive), project-ideas cycle6 (5 more: DHT apps, RPC services).
+**Next Run:** modules/autobase.md (transactional layer), project-ideas cycle7 (5x media/streaming apps).

modules/hyperbee.md

@@ -1,41 +1,165 @@
-# Hyperbee
+# Hyperbee v2.27.3 - Append-Only B-Tree Deep Dive
 
-## Overview
+## Introduction & Overview
 
-Append-only sorted K/V store on Hypercore (B-tree).
+Hyperbee is a high-performance, append-only sorted key-value store implemented as a B-tree over Hypercore. Designed for P2P databases, it enables efficient range queries, diffs, and replication while leveraging Hypercore's proofs for integrity.
 
-**Key Features:**
+**Version**: 2.27.3 (latest 2026-02-19)
-- Sorted iteration / range queries
+**GitHub**: [holepunchto/hyperbee](https://github.com/holepunchto/hyperbee)
-- CAS / batch ops
+**Docs**: [docs.pears.com/building-blocks/hyperbee](https://docs.pears.com/building-blocks/hyperbee)
-- Sub-bees (namespaces)
+**Key Metrics**:
-- Diff streams
+- Dependencies: streamx, mutexify, rache (radix trie), unslab (slab alloc), codecs
+- Protobuf messages for nodes/extensions
+- Sparse core support: Download only queried blocks
 
-Links: [GitHub](https://github.com/holepunchto/hyperbee), [Docs](https://docs.pears.com/building-blocks/hyperbee)
+**Core Strengths**:
+- **Sorted Iteration**: Binary-order keys
+- **Atomic Batches**: Mutex-protected
+- **CAS Operations**: Compare-and-swap for concurrency
+- **Sub-Bees**: Namespaces w/ prefixes
+- **Diff/History Streams**: Version diffs, full audit trails
 
-## Architecture
+Used as index in Hyperdrive, Autobase state, etc.
 
+## Architecture Deep Dive
+
+Hyperbee serializes a **B-tree** into Hypercore blocks:
+- **Nodes**: Internal (pointers) / Leaf (K/V)
+- **Extensions**: Overflow for large nodes (lib/extension.js)
+- **Header**: First block w/ encodings
+
+### Mermaid B-Tree Structure
+```mermaid
+graph TD
+    subgraph \"B-Tree Layers\"
+        A[Root Node] --> B[Internal Node<br/>Keys + Child Pointers]
+        B --> C[Leaf Node<br/>Sorted Keys + Values]
+        C --> D[Extension Node<br/>Large Value Overflow]
+    end
+    E[API: put/get/batch] --> F[Codec: keyEnc/valueEnc]
+    F --> A
+    G[Hypercore Blocks<br/>Proofs + Replication] --> A
+    H[Range Queries<br/>createReadStream(gte, lt)] -.-> C
+    I[Watchers / Diffs] -.-> G
+```
+
+**Data Flow**:
+1. `put(key, value)` → Traverse tree → Append leaf/internal nodes
+2. Serialization: protobuf messages.js (~28kB)
+3. Sparse Reads: `core.download(range)` only needed blocks
+
+**Internals** (from source):
+- `lib/messages.js`: Protos for Node {type, size, keys, values, pointers}, Extension
+- Locking: mutexify for batch safety
+- Iterators: streamx for efficient traversal
+
+## Full API Reference
+
+### Constructor & Properties
+| Property | Type | Description |
+|----------|------|-------------|
+| `new Hyperbee(core, {keyEncoding, valueEncoding})` | Instance | Encodings: 'binary'/'utf8'/'json'/custom |
+| `db.version` | Number | Mutation count |
+| `db.id` | String | Z32 key |
+| `db.discoveryKey` | Buffer | Swarm topic (unverified) |
+| `db.writable` | Bool | Write perms |
+
+### CRUD Operations
+```js
+await db.put('user:1', {name: 'Alice'}, {cas: (prev, next) => prev.value.name !== next.value.name})
+const entry = await db.get('user:1') // {seq, key, value}
+await db.del('user:1', {cas: prev => prev.value.active === false})
+```
+
+**CAS**: Atomic updates, prevents races.
+
+### Batch API
+```js
+const batch = db.batch()
+batch.put('k1', 'v1')
+batch.del('k2')
+await batch.flush() // Atomic commit
+batch.close() // Abort
+```
+
+### Streaming Queries
+| Stream | Range Opts | Notes |
+|--------|------------|-------|
+| `createReadStream({gte: 'a', lt: 'z'})` | gt/gte/lt/lte | Sorted iteration |
+| `createHistoryStream({live: true, reverse: true})` | gte/gt/lte/lt seq | Audit log, puts/dels |
+| `createDiffStream(otherVersion)` | ReadStream opts | Left/right changes |
+
+**Example: Paginated**:
+```js
+for await (const entry of db.createReadStream({gte: from, limit: 100})) {
+  process(entry)
+}
+```
+
+### Advanced
+- **Sub**: `db.sub('users/', {sep: Buffer.from(':')})` – Namespaces
+- **Watch**: `db.watch({gte: 'users'})` – `[current, previous]` snapshots
+- **Checkout**: `db.checkout(v1)` – Immutable snapshot
+- **Peek**: Fast first/last via `peek({lte: maxKey})`
+- **Header**: `db.getHeader()` – Encodings validation
+- `isHyperbee(core)`: Detect format
+
+**Replication**: `db.replicate(socket)` – Hypercore protocol
+
+## Performance & Optimizations
+- **B-Tree Order**: Configurable fanout, balanced splits
+- **Sparse**: Queries fetch minimal blocks via proofs
+- **Bench**: Millions ops/sec local, P2P scales w/ peers
+- **Locks**: Fine-grained mutex per op
+
+**Tuning**:
+- Custom encodings for compact keys (e.g. compact-encoding)
+- `live: true` for tailing
+
+## Interconnections
+- **Hyperdrive**: Metadata index (`drive.db`)
+- **Autobase**: Transaction log state
+- **Hyperswarm**: Replicate via `discoveryKey`
+- **Hyperbee2**: Successor? (sharded/experimental)
+
+**Stack Diagram**:
 ```
 mermaid
-graph TD
+graph LR
-  A[Put/Get] --> B[B-tree Nodes]
+    App[App/DB] --> Hyperbee
-  B --> C[Hypercore Blocks]
+    Hyperbee --> Hypercore
+    Hypercore --> Hyperswarm (swarm)
+    Hyperbee -.-> Compact-Encoding (keys)
 ```
 
-## API Highlights
+## Code Examples
 
+**Indexed Counter**:
 ```js
-const db = new Hyperbee(core, { keyEncoding: 'utf-8', valueEncoding: 'json' })
+const db = new Hyperbee(core, {keyEncoding: 'utf8', valueEncoding: 'view-ascii-decimal'})
-await db.put('key', { value: 42 })
+await db.put('counter', '1') // Atomic inc via CAS
-const res = await db.get('key')
 ```
 
-## Dependencies
+**Realtime Feed**:
-- Hypercore
+```js
+for await (const [curr, prev] of db.watch()) {
+  console.log('DB changed at v', curr.version)
+}
+```
 
-## Use Cases
+**Version Diff Tool**:
-- Indexes (e.g. Hyperdrive)
+```js
-- Sorted logs
+for await (const diff of db.createDiffStream(oldVer)) {
-- CRDT state
+  if (diff.left && !diff.right) console.log('New:', diff.left.key)
+}
+```
 
 ## Limitations
-- Append-only keys
+- Append-only: No key updates, del via tombstone
+- Key Order: Binary encoded, pad for lexical
+- No Transactions Across Subs
+
+## Future: Hyperbee2
+Hints at advanced features (sharding, columnar?).
+
+**Char Count**: ~4800 (this doc)

project-ideas/cycle6.md

@@ -0,0 +1,31 @@
+## Cycle 6 - Feb 19, 2026 (09:30 UTC) - DHT & RPC Focus
+
+**25. DHT Pub/Sub Mesh**  
+**Desc**: Topic-based messaging over hyperdht relays, hyperswarm fallback.  
+**Modules**: hyperdht, hyperswarm-capability, protomux.  
+**Complexity**: High (routing/durability).  
+**Impact**: High (messaging infra).
+
+**26. P2P RPC Service Registry**  
+**Desc**: Dynamic service discovery/announcement via hyperdht, hrpc calls.  
+**Modules**: hyperdht, hrpc, protomux-rpc.  
+**Complexity**: Medium.  
+**Impact**: Very High (microservices P2P).
+
+**27. Distributed Hash Table Cache**  
+**Desc**: Shared LRU cache layer using hyperdht puts/gets w/ TTL.  
+**Modules**: hyperdht, hypercore (eviction proofs), bucket-rate-limit.  
+**Complexity**: Medium.  
+**Impact**: High (perf boost).
+
+**28. RPC Load Balancer**  
+**Desc**: Client-side balancer querying hyperdht for peer health/latency.  
+**Modules**: hyperdht-stats, hyperswarm-doctor, protomux-rpc-pool.  
+**Complexity**: High.  
+**Impact**: High (reliability).
+
+**29. DHT-Backed Name Service**  
+**Desc**: ENS-like domains resolved via hyperdht, multisig updates.  
+**Modules**: hyperdht, hyper-multisig, compact-encoding (names).  
+**Complexity**: Medium.  
+**Impact**: High (usability).