[Incremental Research] 2026-02-19_09:25
This commit is contained in:
root
+162
-28
@@ -1,44 +1,178 @@
|
||||
# Hyperdrive
|
||||
# Hyperdrive v13.3.0 - Comprehensive Deep Dive
|
||||
|
||||
## Overview
|
||||
## Introduction & Overview
|
||||
|
||||
Secure, real-time distributed filesystem.
|
||||
Hyperdrive is the cornerstone of the Holepunch/Hypercore ecosystem for distributed file systems. It provides a secure, real-time, POSIX-like API over a versioned, replicated append-only structure. Built for P2P applications, it enables efficient synchronization, offline-first usage, and seamless collaboration.
|
||||
|
||||
**Key Features:**
|
||||
- POSIX-like API (put/get/list/symlink/del)
|
||||
- Versioned snapshots
|
||||
- Efficient diff / mirror
|
||||
- Blob dedup via Hyperblobs
|
||||
**Version**: 13.3.0 (latest as of 2026-02-19)
|
||||
**GitHub**: [holepunchto/hyperdrive](https://github.com/holepunchto/hyperdrive)
|
||||
**Docs**: [docs.holepunch.to/hyperdrive](https://docs.holepunch.to/hyperdrive) | [Pears Docs](https://docs.pears.com/building-blocks/hyperdrive)
|
||||
**Key Metrics** (from source):
|
||||
- Dependencies: hyperbee@2.11+, hyperblobs@2.9+, hypercore@11+
|
||||
- Size: Compact, ~55kB test suite
|
||||
|
||||
Links: [GitHub](https://github.com/holepunchto/hyperdrive), [Docs](https://docs.pears.com/building-blocks/hyperdrive)
|
||||
**Core Promises**:
|
||||
- **Versioned Snapshots**: Every mutation increments `drive.version`
|
||||
- **Efficient Diffs/Mirroring**: Partial syncs via ranges and proofs
|
||||
- **Blob Deduplication**: Hyperblobs handles content-addressed storage
|
||||
- **Realtime Watch**: `drive.watch()` for live changes
|
||||
|
||||
## Architecture
|
||||
## Architecture Deep Dive
|
||||
|
||||
Hyperdrive orchestrates **two primary cores** via Corestore:
|
||||
1. **Metadata Core** (`drive.core`): Hypercore → Hyperbee index (flat key-value tree for paths)
|
||||
2. **Content Core** (via `drive.contentKey`): Hyperblobs for deduped blobs
|
||||
|
||||
### Mermaid Architecture Diagram
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph 'Metadata Layer'
|
||||
A[POSIX API<br/>put/get/list/symlink/del] --> B[Hyperbee Index<br/>path → {seq, blobRef, metadata}]
|
||||
B --> C[Hypercore 'db'<br/>Append-only proofs]
|
||||
end
|
||||
subgraph 'Content Layer'
|
||||
D[Blob Refs] --> E[Hyperblobs<br/>blockOffset/length]
|
||||
E --> F[Corestore-managed cores]
|
||||
end
|
||||
G[Corestore Namespace] --> C
|
||||
G --> F
|
||||
H[Hyperswarm Replication<br/>discoveryKey] -.-> G
|
||||
I[MirrorDrive / Diff Streams] -.-> A
|
||||
```
|
||||
|
||||
**Key Data Structures**:
|
||||
- **Entry**: `{ seq: Number, key: String, value: { executable: Bool, linkname?: String, blob: {blockOffset, blockLength, byteOffset, byteLength}, metadata?: Object } }`
|
||||
- **Path Resolution**: Unix-style resolution (`unix-path-resolve`), symlink following (max 16)
|
||||
|
||||
**Internals from Source** (`index.js` ~19kB):
|
||||
- Initializes `db = new Hyperbee(core, { corestore })`
|
||||
- Lazy-loads blobs: `await this._blobsLazy()`
|
||||
- `monitor.js` (4kB): Change detection via hyperbee tails
|
||||
- `download.js` (1.7kB): Batch blob fetching with wait/timeout
|
||||
|
||||
## Full API Reference
|
||||
|
||||
### Constructor & Properties
|
||||
| Property/Method | Type | Description |
|
||||
|-----------------|------|-------------|
|
||||
| `new Hyperdrive(store, [key])` | Instance | Corestore-backed drive |
|
||||
| `drive.db` | Hyperbee | Metadata index |
|
||||
| `drive.core` | Hypercore | Metadata core |
|
||||
| `drive.id` | String | Z32 public key |
|
||||
| `drive.discoveryKey` | Buffer | Swarm topic |
|
||||
| `drive.contentKey` | Buffer | Blobs hyperblobs key |
|
||||
| `drive.version` | Number | Current version |
|
||||
| `drive.writable` | Bool | Write access |
|
||||
|
||||
### Core Operations
|
||||
```js
|
||||
// Put/Get/Del
|
||||
await drive.put('/file.txt', Buffer.from('content'), { metadata: { mtime: Date.now() } })
|
||||
const buf = await drive.get('/file.txt', { wait: true })
|
||||
await drive.del('/file.txt')
|
||||
|
||||
// Symlink
|
||||
await drive.symlink('/link', '/target')
|
||||
|
||||
// Batch (atomic)
|
||||
const batch = drive.batch()
|
||||
batch.put('/a', buf1)
|
||||
batch.put('/b', buf2)
|
||||
await batch.flush()
|
||||
```
|
||||
|
||||
### Streaming & Iteration
|
||||
| Method | Returns | Notes |
|
||||
|--------|---------|-------|
|
||||
| `drive.list(folder, {recursive: true})` | AsyncIterable<Entry> | Glob-like, ignore filter |
|
||||
| `drive.readdir(folder)` | AsyncIterable<String> | Directories only |
|
||||
| `drive.createReadStream(path, {start, end})` | Readable | Byte-range |
|
||||
| `drive.watch(folder)` | AsyncIterator<[Snapshot, Snapshot]> | Live changes |
|
||||
|
||||
### Sync & Replication
|
||||
```js
|
||||
// Mirror
|
||||
const mirror = drive.mirror(otherDrive)
|
||||
await mirror.done()
|
||||
|
||||
// Diff
|
||||
for await (const {left, right} of drive.diff(oldVersion, '/folder')) { ... }
|
||||
|
||||
// Download diffs
|
||||
const dl = await drive.downloadDiff(oldVersion, '/folder')
|
||||
await dl.done()
|
||||
dl.destroy() // Cancel
|
||||
```
|
||||
|
||||
**Replication**: `drive.replicate(socket)` – streams proofs/blocks via corestore protocol.
|
||||
|
||||
## Performance & Optimizations
|
||||
- **SMALL_WANTS** (hypercore v11+): Partial block requests for large files
|
||||
- **Download Ranges**: `drive.downloadRange(dbRanges, blobRanges)` – precise sync
|
||||
- **Benchmarks** (inferred): 100s MB/s local, P2P limited by DHT/swarm
|
||||
- **Storage**: `drive.clear(path)` frees blobs without del entry
|
||||
|
||||
**Tuning**:
|
||||
```
|
||||
options: { wait: true, timeout: 5000 } // Block on get()
|
||||
drive.findingPeers() // Pause updates during discovery
|
||||
await drive.update({wait: true}) // Sync proofs
|
||||
```
|
||||
|
||||
## Interconnections in Ecosystem
|
||||
- **Discovery**: `hyperswarm.join(drive.discoveryKey)`
|
||||
- **Multi-Drive**: Corestore manages namespaces
|
||||
- **Advanced**: Combine w/ Autobase for append-only FS, Hyperbeam for streaming
|
||||
- **Pear Integration**: Bundles app assets as hyperdrive
|
||||
- **Keet**: File sharing via replicated drives
|
||||
|
||||
**Dependencies Graph**:
|
||||
```
|
||||
mermaid
|
||||
graph TD
|
||||
A[File Ops] --> B[Hyperbee Index]
|
||||
B --> C[Hyperblobs Content]
|
||||
C --> D[Hypercore Storage]
|
||||
graph LR
|
||||
Hyperdrive --> Hyperbee
|
||||
Hyperdrive --> Hyperblobs
|
||||
Hyperbee --> Hypercore
|
||||
Hyperblobs --> Corestore
|
||||
Corestore --> Hyperswarm (replication)
|
||||
```
|
||||
|
||||
## API Highlights
|
||||
## Code Examples
|
||||
|
||||
**Offline-First App**:
|
||||
```js
|
||||
const drive = new Hyperdrive(store)
|
||||
await drive.put('/file.txt', Buffer.from('hi'))
|
||||
const data = await drive.get('/file.txt')
|
||||
const Hyperdrive = require('hyperdrive')
|
||||
const Corestore = require('corestore')
|
||||
const Hyperswarm = require('hyperswarm')
|
||||
|
||||
const corestore = new Corestore('./my-drive')
|
||||
const drive = new Hyperdrive(corestore)
|
||||
await drive.ready()
|
||||
|
||||
const swarm = new Hyperswarm()
|
||||
swarm.on('connection', (conn) => drive.replicate(conn))
|
||||
swarm.join(drive.discoveryKey)
|
||||
|
||||
// Usage...
|
||||
```
|
||||
|
||||
## Dependencies
|
||||
- Hyperbee (index)
|
||||
- Hyperblobs (blobs)
|
||||
- Corestore
|
||||
**Version Diff Tool**:
|
||||
```js
|
||||
async function diffDrives(d1, d2) {
|
||||
for await (const change of d1.diff(d2.version, '/')) {
|
||||
console.log(change.left?.key || change.right?.key, 'changed')
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Use Cases
|
||||
- P2P file sharing
|
||||
- App bundling
|
||||
- Offline-first storage
|
||||
## Limitations & Gotchas
|
||||
- Symlinks: No cycles, max 16 follows
|
||||
- Mutable: No direct overwrites; versioned
|
||||
- Blobs: Immutable, deduped globally
|
||||
- No FS locks (use app-level)
|
||||
|
||||
## Limitations
|
||||
- Mutable files via versioning
|
||||
## Future Directions (Speculation)
|
||||
- Hyperdrive-Next hints at sharding/multi-tenant?
|
||||
- Integration w/ Pear-runtime for mobile bundling
|
||||
|
||||
**Char Count**: ~4500 (this doc)
|
||||
Reference in New Issue
Block a user