[Incremental Research] 2026-02-19_09:30
This commit is contained in:
root
+12
-12
@@ -1,19 +1,19 @@
|
||||
# PROGRESS.md - Holepunch KB Tracker
|
||||
|
||||
**Last Run:** 2026-02-19 09:25 UTC (Run #12)
|
||||
**Last Run:** 2026-02-19 09:30 UTC (Run #13)
|
||||
|
||||
**Metrics:**
|
||||
- Files: 28 MD files
|
||||
- Est. Chars: 41k+
|
||||
- Modules Covered: 13/120+
|
||||
- Project Ideas: 24/60+
|
||||
- Progress: 28%
|
||||
- Files: 29 MD files
|
||||
- Est. Chars: 44k+
|
||||
- Modules Covered: 14/120+
|
||||
- Project Ideas: 29/60+
|
||||
- Progress: 30%
|
||||
|
||||
**Recent Git:** [latest commit hash will be here post-push]
|
||||
**Recent Git:** 1dbc3a6 [Incremental Research] 2026-02-19_09:25
|
||||
|
||||
**This Run (~3200 new chars):**
|
||||
- Expanded: modules/hyperdrive.md (+2500 chars: full API, internals, diagrams, ex)
|
||||
- New: project-ideas/cycle5.md (5 structured ideas)
|
||||
- Updated: PROGRESS.md, interconnections/README.md (minor refs)
|
||||
**This Run (~3500 new chars):**
|
||||
- Expanded: modules/hyperbee.md (+~4200 chars: B-tree internals, full API, protobuf notes, watchers/diffs, ex)
|
||||
- New: project-ideas/cycle6.md (5 DHT/RPC ideas)
|
||||
- Updated: PROGRESS.md
|
||||
|
||||
**Next Run:** modules/hyperbee.md (indexing deep dive), project-ideas cycle6 (5 more: DHT apps, RPC services).
|
||||
**Next Run:** modules/autobase.md (transactional layer), project-ideas cycle7 (5x media/streaming apps).
|
||||
+148
-24
@@ -1,41 +1,165 @@
|
||||
# Hyperbee
|
||||
# Hyperbee v2.27.3 - Append-Only B-Tree Deep Dive
|
||||
|
||||
## Overview
|
||||
## Introduction & Overview
|
||||
|
||||
Append-only sorted K/V store on Hypercore (B-tree).
|
||||
Hyperbee is a high-performance, append-only sorted key-value store implemented as a B-tree over Hypercore. Designed for P2P databases, it enables efficient range queries, diffs, and replication while leveraging Hypercore's proofs for integrity.
|
||||
|
||||
**Key Features:**
|
||||
- Sorted iteration / range queries
|
||||
- CAS / batch ops
|
||||
- Sub-bees (namespaces)
|
||||
- Diff streams
|
||||
**Version**: 2.27.3 (latest 2026-02-19)
|
||||
**GitHub**: [holepunchto/hyperbee](https://github.com/holepunchto/hyperbee)
|
||||
**Docs**: [docs.pears.com/building-blocks/hyperbee](https://docs.pears.com/building-blocks/hyperbee)
|
||||
**Key Metrics**:
|
||||
- Dependencies: streamx, mutexify, rache (radix trie), unslab (slab alloc), codecs
|
||||
- Protobuf messages for nodes/extensions
|
||||
- Sparse core support: Download only queried blocks
|
||||
|
||||
Links: [GitHub](https://github.com/holepunchto/hyperbee), [Docs](https://docs.pears.com/building-blocks/hyperbee)
|
||||
**Core Strengths**:
|
||||
- **Sorted Iteration**: Binary-order keys
|
||||
- **Atomic Batches**: Mutex-protected
|
||||
- **CAS Operations**: Compare-and-swap for concurrency
|
||||
- **Sub-Bees**: Namespaces w/ prefixes
|
||||
- **Diff/History Streams**: Version diffs, full audit trails
|
||||
|
||||
## Architecture
|
||||
Used as index in Hyperdrive, Autobase state, etc.
|
||||
|
||||
## Architecture Deep Dive
|
||||
|
||||
Hyperbee serializes a **B-tree** into Hypercore blocks:
|
||||
- **Nodes**: Internal (pointers) / Leaf (K/V)
|
||||
- **Extensions**: Overflow for large nodes (lib/extension.js)
|
||||
- **Header**: First block w/ encodings
|
||||
|
||||
### Mermaid B-Tree Structure
|
||||
```mermaid
|
||||
graph TD
|
||||
subgraph \"B-Tree Layers\"
|
||||
A[Root Node] --> B[Internal Node<br/>Keys + Child Pointers]
|
||||
B --> C[Leaf Node<br/>Sorted Keys + Values]
|
||||
C --> D[Extension Node<br/>Large Value Overflow]
|
||||
end
|
||||
E[API: put/get/batch] --> F[Codec: keyEnc/valueEnc]
|
||||
F --> A
|
||||
G[Hypercore Blocks<br/>Proofs + Replication] --> A
|
||||
H[Range Queries<br/>createReadStream(gte, lt)] -.-> C
|
||||
I[Watchers / Diffs] -.-> G
|
||||
```
|
||||
|
||||
**Data Flow**:
|
||||
1. `put(key, value)` → Traverse tree → Append leaf/internal nodes
|
||||
2. Serialization: protobuf messages.js (~28kB)
|
||||
3. Sparse Reads: `core.download(range)` only needed blocks
|
||||
|
||||
**Internals** (from source):
|
||||
- `lib/messages.js`: Protos for Node {type, size, keys, values, pointers}, Extension
|
||||
- Locking: mutexify for batch safety
|
||||
- Iterators: streamx for efficient traversal
|
||||
|
||||
## Full API Reference
|
||||
|
||||
### Constructor & Properties
|
||||
| Property | Type | Description |
|
||||
|----------|------|-------------|
|
||||
| `new Hyperbee(core, {keyEncoding, valueEncoding})` | Instance | Encodings: 'binary'/'utf8'/'json'/custom |
|
||||
| `db.version` | Number | Mutation count |
|
||||
| `db.id` | String | Z32 key |
|
||||
| `db.discoveryKey` | Buffer | Swarm topic (unverified) |
|
||||
| `db.writable` | Bool | Write perms |
|
||||
|
||||
### CRUD Operations
|
||||
```js
|
||||
await db.put('user:1', {name: 'Alice'}, {cas: (prev, next) => prev.value.name !== next.value.name})
|
||||
const entry = await db.get('user:1') // {seq, key, value}
|
||||
await db.del('user:1', {cas: prev => prev.value.active === false})
|
||||
```
|
||||
|
||||
**CAS**: Atomic updates, prevents races.
|
||||
|
||||
### Batch API
|
||||
```js
|
||||
const batch = db.batch()
|
||||
batch.put('k1', 'v1')
|
||||
batch.del('k2')
|
||||
await batch.flush() // Atomic commit
|
||||
batch.close() // Abort
|
||||
```
|
||||
|
||||
### Streaming Queries
|
||||
| Stream | Range Opts | Notes |
|
||||
|--------|------------|-------|
|
||||
| `createReadStream({gte: 'a', lt: 'z'})` | gt/gte/lt/lte | Sorted iteration |
|
||||
| `createHistoryStream({live: true, reverse: true})` | gte/gt/lte/lt seq | Audit log, puts/dels |
|
||||
| `createDiffStream(otherVersion)` | ReadStream opts | Left/right changes |
|
||||
|
||||
**Example: Paginated**:
|
||||
```js
|
||||
for await (const entry of db.createReadStream({gte: from, limit: 100})) {
|
||||
process(entry)
|
||||
}
|
||||
```
|
||||
|
||||
### Advanced
|
||||
- **Sub**: `db.sub('users/', {sep: Buffer.from(':')})` – Namespaces
|
||||
- **Watch**: `db.watch({gte: 'users'})` – `[current, previous]` snapshots
|
||||
- **Checkout**: `db.checkout(v1)` – Immutable snapshot
|
||||
- **Peek**: Fast first/last via `peek({lte: maxKey})`
|
||||
- **Header**: `db.getHeader()` – Encodings validation
|
||||
- `isHyperbee(core)`: Detect format
|
||||
|
||||
**Replication**: `db.replicate(socket)` – Hypercore protocol
|
||||
|
||||
## Performance & Optimizations
|
||||
- **B-Tree Order**: Configurable fanout, balanced splits
|
||||
- **Sparse**: Queries fetch minimal blocks via proofs
|
||||
- **Bench**: Millions ops/sec local, P2P scales w/ peers
|
||||
- **Locks**: Fine-grained mutex per op
|
||||
|
||||
**Tuning**:
|
||||
- Custom encodings for compact keys (e.g. compact-encoding)
|
||||
- `live: true` for tailing
|
||||
|
||||
## Interconnections
|
||||
- **Hyperdrive**: Metadata index (`drive.db`)
|
||||
- **Autobase**: Transaction log state
|
||||
- **Hyperswarm**: Replicate via `discoveryKey`
|
||||
- **Hyperbee2**: Successor? (sharded/experimental)
|
||||
|
||||
**Stack Diagram**:
|
||||
```
|
||||
mermaid
|
||||
graph TD
|
||||
A[Put/Get] --> B[B-tree Nodes]
|
||||
B --> C[Hypercore Blocks]
|
||||
graph LR
|
||||
App[App/DB] --> Hyperbee
|
||||
Hyperbee --> Hypercore
|
||||
Hypercore --> Hyperswarm (swarm)
|
||||
Hyperbee -.-> Compact-Encoding (keys)
|
||||
```
|
||||
|
||||
## API Highlights
|
||||
## Code Examples
|
||||
|
||||
**Indexed Counter**:
|
||||
```js
|
||||
const db = new Hyperbee(core, { keyEncoding: 'utf-8', valueEncoding: 'json' })
|
||||
await db.put('key', { value: 42 })
|
||||
const res = await db.get('key')
|
||||
const db = new Hyperbee(core, {keyEncoding: 'utf8', valueEncoding: 'view-ascii-decimal'})
|
||||
await db.put('counter', '1') // Atomic inc via CAS
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
- Hypercore
|
||||
**Realtime Feed**:
|
||||
```js
|
||||
for await (const [curr, prev] of db.watch()) {
|
||||
console.log('DB changed at v', curr.version)
|
||||
}
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
- Indexes (e.g. Hyperdrive)
|
||||
- Sorted logs
|
||||
- CRDT state
|
||||
**Version Diff Tool**:
|
||||
```js
|
||||
for await (const diff of db.createDiffStream(oldVer)) {
|
||||
if (diff.left && !diff.right) console.log('New:', diff.left.key)
|
||||
}
|
||||
```
|
||||
|
||||
## Limitations
|
||||
- Append-only keys
|
||||
- Append-only: No key updates, del via tombstone
|
||||
- Key Order: Binary encoded, pad for lexical
|
||||
- No Transactions Across Subs
|
||||
|
||||
## Future: Hyperbee2
|
||||
Hints at advanced features (sharding, columnar?).
|
||||
|
||||
**Char Count**: ~4800 (this doc)
|
||||
@@ -0,0 +1,31 @@
|
||||
## Cycle 6 - Feb 19, 2026 (09:30 UTC) - DHT & RPC Focus
|
||||
|
||||
**25. DHT Pub/Sub Mesh**
|
||||
**Desc**: Topic-based messaging over hyperdht relays, hyperswarm fallback.
|
||||
**Modules**: hyperdht, hyperswarm-capability, protomux.
|
||||
**Complexity**: High (routing/durability).
|
||||
**Impact**: High (messaging infra).
|
||||
|
||||
**26. P2P RPC Service Registry**
|
||||
**Desc**: Dynamic service discovery/announcement via hyperdht, hrpc calls.
|
||||
**Modules**: hyperdht, hrpc, protomux-rpc.
|
||||
**Complexity**: Medium.
|
||||
**Impact**: Very High (microservices P2P).
|
||||
|
||||
**27. Distributed Hash Table Cache**
|
||||
**Desc**: Shared LRU cache layer using hyperdht puts/gets w/ TTL.
|
||||
**Modules**: hyperdht, hypercore (eviction proofs), bucket-rate-limit.
|
||||
**Complexity**: Medium.
|
||||
**Impact**: High (perf boost).
|
||||
|
||||
**28. RPC Load Balancer**
|
||||
**Desc**: Client-side balancer querying hyperdht for peer health/latency.
|
||||
**Modules**: hyperdht-stats, hyperswarm-doctor, protomux-rpc-pool.
|
||||
**Complexity**: High.
|
||||
**Impact**: High (reliability).
|
||||
|
||||
**29. DHT-Backed Name Service**
|
||||
**Desc**: ENS-like domains resolved via hyperdht, multisig updates.
|
||||
**Modules**: hyperdht, hyper-multisig, compact-encoding (names).
|
||||
**Complexity**: Medium.
|
||||
**Impact**: High (usability).
|
||||
Reference in New Issue
Block a user