Updates
This commit is contained in:
+547
-18
@@ -1,33 +1,562 @@
|
||||
# Autobase - Multi-Writer Virtual Core
|
||||
# Autobase - Multiwriter Data Structure
|
||||
|
||||
## Deep Dive
|
||||
## Overview
|
||||
|
||||
Autobase aggregates Hypercores from writers into virtual append-only log. CRDT-like for collab.
|
||||
**Autobase** is a revolutionary multiwriter data structure that combines multiple Hypercore writer feeds into a single, linearized, eventually consistent view. Using the event sourcing pattern, Autobase enables distributed systems to merge inputs from multiple writers while maintaining causal consistency.
|
||||
|
||||
**Bootstrap**: load writers (keys), index tx proofs.
|
||||
## Core Concepts
|
||||
|
||||
## Flow
|
||||
### The Problem
|
||||
|
||||
Traditional databases have a single writer (leader). When that writer fails, the system stops or requires complex failover. Autobase solves this by:
|
||||
|
||||
- Allowing multiple writers to append to their own feeds
|
||||
- Linearizing all writes into a consistent order
|
||||
- Supporting dynamic writer membership
|
||||
- Providing checkpointing for fast recovery
|
||||
|
||||
### Event Sourcing Pattern
|
||||
|
||||
```mermaid
|
||||
graph TD
|
||||
Writers[W1/W2 Append Tx] --> Proofs[Signed Headers]
|
||||
Proofs --> VirtualCore[Merge Sorted]
|
||||
VirtualCore --> Hyperbee[Index/DB]
|
||||
graph TB
|
||||
subgraph "Writer A"
|
||||
A1[Block 0]
|
||||
A2[Block 1]
|
||||
A3[Block 2]
|
||||
end
|
||||
|
||||
subgraph "Writer B"
|
||||
B1[Block 0]
|
||||
B2[Block 1]
|
||||
end
|
||||
|
||||
subgraph "Writer C"
|
||||
C1[Block 0]
|
||||
C2[Block 1]
|
||||
C3[Block 2]
|
||||
C4[Block 3]
|
||||
end
|
||||
|
||||
subgraph "Linearized View"
|
||||
L1[A:0]
|
||||
L2[B:0]
|
||||
L3[A:1]
|
||||
L4[C:0]
|
||||
L5[B:1]
|
||||
L6[A:2]
|
||||
L7[C:1]
|
||||
L8[C:2]
|
||||
L9[C:3]
|
||||
end
|
||||
|
||||
A1 --> L1
|
||||
B1 --> L2
|
||||
A2 --> L3
|
||||
C1 --> L4
|
||||
B2 --> L5
|
||||
A3 --> L6
|
||||
C2 --> L7
|
||||
C3 --> L8
|
||||
C4 --> L9
|
||||
|
||||
style L1 fill:#f9f,stroke:#333
|
||||
style L2 fill:#bbf,stroke:#333
|
||||
style L3 fill:#f9f,stroke:#333
|
||||
style L4 fill:#bfb,stroke:#333
|
||||
```
|
||||
|
||||
**Ex**:
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Autobase Core"
|
||||
CORE[System Core]
|
||||
LIN[Linearizer]
|
||||
VIEW[View]
|
||||
end
|
||||
|
||||
subgraph "Writers"
|
||||
W1[Writer A Core]
|
||||
W2[Writer B Core]
|
||||
W3[Writer C Core]
|
||||
end
|
||||
|
||||
subgraph "Apply Function"
|
||||
OPEN[Open View]
|
||||
APPLY[Apply Nodes]
|
||||
CLOSE[Close View]
|
||||
end
|
||||
|
||||
W1 --> CORE
|
||||
W2 --> CORE
|
||||
W3 --> CORE
|
||||
|
||||
CORE --> LIN
|
||||
LIN --> APPLY
|
||||
APPLY --> VIEW
|
||||
|
||||
OPEN --> VIEW
|
||||
CLOSE --> VIEW
|
||||
|
||||
style CORE fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style LIN fill:#bbf,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
## Key Features
|
||||
|
||||
### 1. Causal Ordering
|
||||
|
||||
Writers explicitly reference previous nodes, creating a causal DAG:
|
||||
|
||||
```js
|
||||
const base = new Autobase(store.get('system'))
|
||||
await base.load([writer1, writer2])
|
||||
await base.append({op: 'put', path: '/doc'})
|
||||
const db = base.db // Hyperbee view
|
||||
// Each node references its dependencies
|
||||
const node = {
|
||||
value: { action: 'create_user', user: 'alice' },
|
||||
causalLinks: [
|
||||
{ key: writerAKey, length: 5 },
|
||||
{ key: writerBKey, length: 3 }
|
||||
]
|
||||
}
|
||||
```
|
||||
|
||||
**Loaders**: external-pointer for dynamic writers.
|
||||
### 2. Linearization
|
||||
|
||||
**Perf**: Linear scan proofs, sparse.
|
||||
The linearizer analyzes causal references to produce a total order:
|
||||
|
||||
**Drive Use**: Versioned FS oplog.
|
||||
```
|
||||
Rules:
|
||||
1. Nodes never precede nodes they reference
|
||||
2. Ordering is eventually consistent
|
||||
3. Forks are resolved deterministically
|
||||
```
|
||||
|
||||
~1200 chars expanded.
|
||||
### 3. Signed Length Checkpoints
|
||||
|
||||
Indexers sign checkpoints where ordering becomes immutable:
|
||||
|
||||
```js
|
||||
// Before signedLength: ordering may change
|
||||
// After signedLength: ordering is fixed
|
||||
console.log(base.signedLength) // 150
|
||||
// Everything before block 150 is permanent
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Creating an Autobase
|
||||
|
||||
```js
|
||||
const Corestore = require('corestore')
|
||||
const Autobase = require('autobase')
|
||||
|
||||
const store = new Corestore('./storage')
|
||||
|
||||
// Create new autobase
|
||||
const base = new Autobase(store, null, {
|
||||
open: (store) => store.get('view'),
|
||||
apply: async (nodes, view, host) => {
|
||||
for (const { value } of nodes) {
|
||||
await view.append(value)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
await base.ready()
|
||||
console.log('Key:', base.key.toString('hex'))
|
||||
```
|
||||
|
||||
### Opening Existing Autobase
|
||||
|
||||
```js
|
||||
// Load existing by key
|
||||
const base = new Autobase(store, bootstrapKey, {
|
||||
open,
|
||||
apply
|
||||
})
|
||||
|
||||
await base.ready()
|
||||
```
|
||||
|
||||
### Append Operations
|
||||
|
||||
```js
|
||||
// Append as writer
|
||||
await base.append({ action: 'create', id: 1 })
|
||||
await base.append({ action: 'update', id: 1, data: 'value' })
|
||||
|
||||
// Optimistic append (for unacknowledged writers)
|
||||
await base.append({ action: 'propose', id: 2 }, { optimistic: true })
|
||||
```
|
||||
|
||||
### Update and Sync
|
||||
|
||||
```js
|
||||
// Fetch all available data
|
||||
await base.update()
|
||||
|
||||
// Check view length
|
||||
console.log('View length:', base.view.length)
|
||||
|
||||
// Read from view
|
||||
for (let i = 0; i < base.view.length; i++) {
|
||||
const entry = await base.view.get(i)
|
||||
console.log(i, entry)
|
||||
}
|
||||
```
|
||||
|
||||
### Writer Management
|
||||
|
||||
```js
|
||||
// Add writer (in apply function)
|
||||
async function apply(nodes, view, host) {
|
||||
for (const { value } of nodes) {
|
||||
if (value.addWriter) {
|
||||
await host.addWriter(value.addWriter, { indexer: true })
|
||||
}
|
||||
if (value.removeWriter) {
|
||||
await host.removeWriter(value.removeWriter)
|
||||
}
|
||||
await view.append(value)
|
||||
}
|
||||
}
|
||||
|
||||
// Check if indexer
|
||||
console.log('Is indexer:', base.isIndexer)
|
||||
|
||||
// Check if writable
|
||||
console.log('Is writable:', base.writable)
|
||||
```
|
||||
|
||||
### View Functions
|
||||
|
||||
#### Open
|
||||
|
||||
```js
|
||||
function open(store) {
|
||||
// Create your view structure
|
||||
return store.get('my-view')
|
||||
|
||||
// Or use Hyperbee for key-value
|
||||
const Hyperbee = require('hyperbee')
|
||||
return new Hyperbee(store.get('bee'), {
|
||||
keyEncoding: 'utf-8',
|
||||
valueEncoding: 'json'
|
||||
})
|
||||
}
|
||||
```
|
||||
|
||||
#### Apply
|
||||
|
||||
```js
|
||||
async function apply(nodes, view, host) {
|
||||
// nodes: Array of { value, from, ... }
|
||||
// view: Your view structure
|
||||
// host: AutobaseHostCalls instance
|
||||
|
||||
for (const node of nodes) {
|
||||
const { value } = node
|
||||
|
||||
// Handle different operations
|
||||
switch (value.type) {
|
||||
case 'add_writer':
|
||||
await host.addWriter(value.key, { indexer: value.indexer })
|
||||
break
|
||||
case 'remove_writer':
|
||||
if (host.removeable(value.key)) {
|
||||
await host.removeWriter(value.key)
|
||||
}
|
||||
break
|
||||
case 'interrupt':
|
||||
host.interrupt('Manual stop requested')
|
||||
return
|
||||
default:
|
||||
await view.append(value)
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
#### Close
|
||||
|
||||
```js
|
||||
async function close(view) {
|
||||
// Cleanup resources
|
||||
await view.close()
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Optimistic Appends
|
||||
|
||||
Allow non-writers to propose blocks:
|
||||
|
||||
```js
|
||||
const base = new Autobase(store, bootstrapKey, {
|
||||
optimistic: true,
|
||||
async apply(nodes, view, host) {
|
||||
for (const node of nodes) {
|
||||
// Verify optimistic block
|
||||
if (!verifyBlock(node.value)) continue
|
||||
|
||||
// Acknowledge valid writers
|
||||
if (isValidWriter(node.value)) {
|
||||
await host.ackWriter(node.from.key)
|
||||
}
|
||||
|
||||
await view.append(node.value)
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
// Append optimistically
|
||||
await base.append({ data: 'proposal' }, { optimistic: true })
|
||||
```
|
||||
|
||||
### Fast Forward
|
||||
|
||||
Speed up catching up to quorum:
|
||||
|
||||
```js
|
||||
const base = new Autobase(store, bootstrapKey, {
|
||||
fastForward: true, // Enable fast forward
|
||||
// ...
|
||||
})
|
||||
|
||||
// Listen for fast forward events
|
||||
base.on('fast-forward', (to, from) => {
|
||||
console.log(`Fast forwarded from ${from} to ${to}`)
|
||||
})
|
||||
```
|
||||
|
||||
### Auto Acknowledgements
|
||||
|
||||
```js
|
||||
const base = new Autobase(store, bootstrapKey, {
|
||||
ackInterval: 1000, // Auto-ack every 1 second
|
||||
// ...
|
||||
})
|
||||
|
||||
// Manual ack
|
||||
await base.ack()
|
||||
|
||||
// Background ack
|
||||
await base.ack(true)
|
||||
```
|
||||
|
||||
### Encryption
|
||||
|
||||
```js
|
||||
const base = new Autobase(store, bootstrapKey, {
|
||||
encrypt: true, // Auto-generate encryption key
|
||||
// OR
|
||||
encryptionKey: myKey, // Use specific key
|
||||
encrypted: true // Expect encrypted base
|
||||
})
|
||||
```
|
||||
|
||||
## Complete Example: Distributed Counter
|
||||
|
||||
```js
|
||||
const Corestore = require('corestore')
|
||||
const Autobase = require('autobase')
|
||||
|
||||
async function createCounter(store, key = null) {
|
||||
return new Autobase(store, key, {
|
||||
open(store) {
|
||||
return store.get('counter-view')
|
||||
},
|
||||
|
||||
async apply(nodes, view, host) {
|
||||
for (const { value } of nodes) {
|
||||
switch (value.type) {
|
||||
case 'increment':
|
||||
const current = await view.get(0) || { count: 0 }
|
||||
current.count += value.amount || 1
|
||||
await view.put(0, current)
|
||||
break
|
||||
|
||||
case 'add_writer':
|
||||
await host.addWriter(value.key, { indexer: true })
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Usage
|
||||
const store = new Corestore('./counter-storage')
|
||||
const counter = await createCounter(store)
|
||||
await counter.ready()
|
||||
|
||||
// Increment
|
||||
await counter.append({ type: 'increment', amount: 5 })
|
||||
|
||||
// Update and read
|
||||
await counter.update()
|
||||
const state = await counter.view.get(0)
|
||||
console.log('Count:', state.count)
|
||||
```
|
||||
|
||||
## Complete Example: Collaborative Todo List
|
||||
|
||||
```js
|
||||
const Corestore = require('corestore')
|
||||
const Autobase = require('autobase')
|
||||
const Hyperbee = require('hyperbee')
|
||||
const cenc = require('compact-encoding')
|
||||
|
||||
async function createTodoList(store, key = null) {
|
||||
return new Autobase(store, key, {
|
||||
open(store) {
|
||||
return new Hyperbee(store.get('todos'), {
|
||||
keyEncoding: cenc.string,
|
||||
valueEncoding: cenc.json
|
||||
})
|
||||
},
|
||||
|
||||
async apply(nodes, view, host) {
|
||||
for (const { value } of nodes) {
|
||||
switch (value.type) {
|
||||
case 'add_todo':
|
||||
await view.put(value.id, {
|
||||
text: value.text,
|
||||
done: false,
|
||||
createdAt: Date.now()
|
||||
})
|
||||
break
|
||||
|
||||
case 'complete_todo':
|
||||
const todo = await view.get(value.id)
|
||||
if (todo) {
|
||||
todo.done = true
|
||||
todo.completedAt = Date.now()
|
||||
await view.put(value.id, todo)
|
||||
}
|
||||
break
|
||||
|
||||
case 'delete_todo':
|
||||
await view.del(value.id)
|
||||
break
|
||||
|
||||
case 'add_writer':
|
||||
await host.addWriter(value.key, { indexer: true })
|
||||
break
|
||||
}
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
// Usage
|
||||
const store = new Corestore('./todo-storage')
|
||||
const todos = await createTodoList(store)
|
||||
await todos.ready()
|
||||
|
||||
// Add todo
|
||||
await todos.append({
|
||||
type: 'add_todo',
|
||||
id: 'todo-1',
|
||||
text: 'Learn Autobase'
|
||||
})
|
||||
|
||||
// Complete todo
|
||||
await todos.append({
|
||||
type: 'complete_todo',
|
||||
id: 'todo-1'
|
||||
})
|
||||
|
||||
// List all
|
||||
await todos.update()
|
||||
const stream = todos.view.createReadStream()
|
||||
for await (const { key, value } of stream) {
|
||||
console.log(`${value.done ? '[x]' : '[ ]'} ${value.text}`)
|
||||
}
|
||||
```
|
||||
|
||||
## Replication
|
||||
|
||||
```js
|
||||
const Hyperswarm = require('hyperswarm')
|
||||
|
||||
const swarm = new Hyperswarm()
|
||||
swarm.on('connection', (conn) => store.replicate(conn))
|
||||
|
||||
// Join discovery topic
|
||||
swarm.join(base.discoveryKey)
|
||||
|
||||
// Or use base.replicate()
|
||||
const stream = base.replicate(true) // isInitiator
|
||||
stream.pipe(otherStream).pipe(stream)
|
||||
```
|
||||
|
||||
## Events
|
||||
|
||||
```js
|
||||
// View updated
|
||||
base.on('update', () => {
|
||||
console.log('View updated, new length:', base.view.length)
|
||||
})
|
||||
|
||||
// Became indexer
|
||||
base.on('is-indexer', () => {
|
||||
console.log('Now an indexer!')
|
||||
})
|
||||
|
||||
// Became writer
|
||||
base.on('writable', () => {
|
||||
console.log('Now a writer!')
|
||||
})
|
||||
|
||||
// Fast forward occurred
|
||||
base.on('fast-forward', (to, from) => {
|
||||
console.log(`Fast forwarded: ${from} -> ${to}`)
|
||||
})
|
||||
|
||||
// Interrupted
|
||||
base.on('interrupt', (reason) => {
|
||||
console.log('Interrupted:', reason)
|
||||
})
|
||||
|
||||
// Warning
|
||||
base.on('warning', (warning) => {
|
||||
console.warn('Warning:', warning)
|
||||
})
|
||||
|
||||
// Error
|
||||
base.on('error', (err) => {
|
||||
console.error('Error:', err)
|
||||
})
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Deterministic Apply**: Use only view argument for state, no external variables
|
||||
2. **Idempotent Operations**: Apply should handle duplicate nodes gracefully
|
||||
3. **Validation**: Always validate values in apply before processing
|
||||
4. **Interruptions**: Use host.interrupt() for breaking changes
|
||||
5. **Checkpoints**: Monitor signedLength for durability guarantees
|
||||
|
||||
```js
|
||||
async function apply(nodes, view, host) {
|
||||
for (const node of nodes) {
|
||||
// Always validate
|
||||
if (!isValid(node.value)) {
|
||||
console.warn('Invalid node:', node)
|
||||
continue
|
||||
}
|
||||
|
||||
// Deterministic: only use view and node
|
||||
await view.append(transform(node.value))
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
|
||||
**Module Type**: Core Data Structure | **Ecosystem Role**: Multi-writer Consensus | **Dependencies**: hypercore, corestore
|
||||
|
||||
+298
@@ -0,0 +1,298 @@
|
||||
# B4A - Buffer for Array
|
||||
|
||||
## Overview
|
||||
|
||||
**B4A** (Buffer for Array) is a critical compatibility layer that bridges Node.js `Buffer` and JavaScript `Uint8Array` classes. This module is foundational to the entire Holepunch ecosystem, enabling seamless binary data operations across Node.js, browsers, React Native, and Bare runtime environments.
|
||||
|
||||
## Core Purpose
|
||||
|
||||
In the Hypercore ecosystem, binary data flows through multiple environments:
|
||||
- **Node.js**: Uses `Buffer` (subclass of `Uint8Array`)
|
||||
- **Browsers**: Only have `Uint8Array`
|
||||
- **React Native**: Limited native buffer support
|
||||
- **Bare**: Custom runtime needing cross-platform compatibility
|
||||
|
||||
B4A abstracts these differences, allowing the same code to run everywhere without environment checks.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "B4A Compatibility Layer"
|
||||
B4A[b4a module]
|
||||
API[Unified API]
|
||||
end
|
||||
|
||||
subgraph "Runtime Environments"
|
||||
NODE[Node.js Buffer]
|
||||
BROWSER[Browser Uint8Array]
|
||||
RN[React Native<br/>react-native-b4a]
|
||||
BARE[Bare Runtime]
|
||||
end
|
||||
|
||||
B4A --> API
|
||||
API --> NODE
|
||||
API --> BROWSER
|
||||
API --> RN
|
||||
API --> BARE
|
||||
|
||||
style B4A fill:#f9f,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Core Buffer Operations
|
||||
|
||||
#### `b4a.alloc(size[, fill[, encoding]])`
|
||||
Allocate a new buffer of specified size. Safe initialization (zero-filled).
|
||||
|
||||
```js
|
||||
const buf = b4a.alloc(1024) // 1KB zero-filled buffer
|
||||
const filled = b4a.alloc(256, 0xFF) // Filled with 0xFF
|
||||
const string = b4a.alloc(100, 'abc') // Filled with 'abc' repeated
|
||||
```
|
||||
|
||||
#### `b4a.allocUnsafe(size)`
|
||||
Fast allocation without zero-filling. Use with caution.
|
||||
|
||||
```js
|
||||
// Fast but potentially contains old memory data
|
||||
const fast = b4a.allocUnsafe(8192)
|
||||
```
|
||||
|
||||
#### `b4a.from(array|string|buffer|arrayBuffer)`
|
||||
Create buffer from various sources.
|
||||
|
||||
```js
|
||||
// From string
|
||||
const fromString = b4a.from('Hello', 'utf8')
|
||||
|
||||
// From array
|
||||
const fromArray = b4a.from([0x48, 0x65, 0x6C, 0x6C, 0x6F])
|
||||
|
||||
// From ArrayBuffer
|
||||
const ab = new ArrayBuffer(8)
|
||||
const fromAB = b4a.from(ab)
|
||||
|
||||
// From another buffer
|
||||
const source = b4a.from('source')
|
||||
const copy = b4a.from(source)
|
||||
```
|
||||
|
||||
### Data Operations
|
||||
|
||||
#### `b4a.concat(buffers[, totalLength])`
|
||||
Concatenate multiple buffers efficiently.
|
||||
|
||||
```js
|
||||
const parts = [
|
||||
b4a.from('Hello '),
|
||||
b4a.from('World'),
|
||||
b4a.from('!')
|
||||
]
|
||||
const combined = b4a.concat(parts)
|
||||
console.log(combined.toString()) // 'Hello World!'
|
||||
```
|
||||
|
||||
#### `b4a.copy(source, target[, targetStart[, sourceStart[, sourceEnd]]])`
|
||||
Copy data between buffers with precise control.
|
||||
|
||||
```js
|
||||
const source = b4a.from('Hello World')
|
||||
const target = b4a.alloc(5)
|
||||
b4a.copy(source, target, 0, 6, 11) // Copy 'World'
|
||||
console.log(target.toString()) // 'World'
|
||||
```
|
||||
|
||||
#### `b4a.compare(buf1, buf2)` & `b4a.equals(buf1, buf2)`
|
||||
Buffer comparison operations.
|
||||
|
||||
```js
|
||||
const a = b4a.from('abc')
|
||||
const b = b4a.from('abc')
|
||||
const c = b4a.from('def')
|
||||
|
||||
console.log(b4a.equals(a, b)) // true
|
||||
console.log(b4a.equals(a, c)) // false
|
||||
console.log(b4a.compare(a, c)) // -1 (a < c)
|
||||
```
|
||||
|
||||
### String Operations
|
||||
|
||||
#### `b4a.toString(buffer[, encoding[, start[, end]]])`
|
||||
Convert buffer to string with encoding support.
|
||||
|
||||
```js
|
||||
const buf = b4a.from('48656c6c6f', 'hex')
|
||||
console.log(b4a.toString(buf, 'utf8')) // 'Hello'
|
||||
|
||||
// Partial conversion
|
||||
const partial = b4a.toString(buf, 'utf8', 0, 2) // 'He'
|
||||
```
|
||||
|
||||
#### `b4a.byteLength(string[, encoding])`
|
||||
Get byte length without creating buffer.
|
||||
|
||||
```js
|
||||
console.log(b4a.byteLength('Hello', 'utf8')) // 5
|
||||
console.log(b4a.byteLength('Hello', 'hex')) // 2 (interpreted as hex)
|
||||
```
|
||||
|
||||
### Search Operations
|
||||
|
||||
```js
|
||||
const buf = b4a.from('Hello World Hello')
|
||||
|
||||
// Find index
|
||||
console.log(b4a.indexOf(buf, 'World')) // 6
|
||||
console.log(b4a.lastIndexOf(buf, 'Hello')) // 12
|
||||
|
||||
// Check inclusion
|
||||
console.log(b4a.includes(buf, 'World')) // true
|
||||
```
|
||||
|
||||
### Type Checking
|
||||
|
||||
#### `b4a.isBuffer(value)`
|
||||
Returns `true` for both Buffer and Uint8Array - key for cross-platform compatibility.
|
||||
|
||||
```js
|
||||
const buf = b4a.from('test')
|
||||
console.log(b4a.isBuffer(buf)) // true
|
||||
console.log(b4a.isBuffer(new Uint8Array(10))) // true
|
||||
console.log(b4a.isBuffer('string')) // false
|
||||
```
|
||||
|
||||
#### `b4a.isEncoding(encoding)`
|
||||
Check if encoding is supported.
|
||||
|
||||
```js
|
||||
console.log(b4a.isEncoding('utf8')) // true
|
||||
console.log(b4a.isEncoding('base64')) // true
|
||||
console.log(b4a.isEncoding('invalid')) // false
|
||||
```
|
||||
|
||||
### Binary Data Operations
|
||||
|
||||
#### Read/Write Methods
|
||||
|
||||
```js
|
||||
const buf = b4a.alloc(16)
|
||||
|
||||
// Write integers (Little Endian)
|
||||
b4a.writeUInt32LE(buf, 0x12345678, 0)
|
||||
b4a.writeInt32LE(buf, -1000, 4)
|
||||
b4a.writeFloatLE(buf, 3.14159, 8)
|
||||
b4a.writeDoubleLE(buf, Math.PI, 12)
|
||||
|
||||
// Read integers
|
||||
console.log(b4a.readUInt32LE(buf, 0)) // 305419896
|
||||
console.log(b4a.readInt32LE(buf, 4)) // -1000
|
||||
console.log(b4a.readFloatLE(buf, 8)) // ~3.14159
|
||||
console.log(b4a.readDoubleLE(buf, 12)) // 3.141592653589793
|
||||
```
|
||||
|
||||
#### Byte Swapping
|
||||
|
||||
```js
|
||||
const buf = b4a.from([0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08])
|
||||
|
||||
b4a.swap16(buf) // Swap every 2 bytes
|
||||
console.log(buf) // [0x02, 0x01, 0x04, 0x03, 0x06, 0x05, 0x08, 0x07]
|
||||
|
||||
b4a.swap32(buf) // Swap every 4 bytes
|
||||
b4a.swap64(buf) // Swap every 8 bytes
|
||||
```
|
||||
|
||||
### React Native Optimization
|
||||
|
||||
When `react-native-b4a` is installed, B4A automatically uses optimized native implementations:
|
||||
|
||||
```bash
|
||||
npm install react-native-b4a
|
||||
```
|
||||
|
||||
This provides significant performance improvements for buffer operations on mobile devices.
|
||||
|
||||
## Common Patterns in Hypercore Ecosystem
|
||||
|
||||
### Pattern 1: Data Encoding/Decoding
|
||||
|
||||
```js
|
||||
const b4a = require('b4a')
|
||||
const cenc = require('compact-encoding')
|
||||
|
||||
// Encode message
|
||||
const message = { type: 'hello', data: b4a.from('payload') }
|
||||
const encoded = cenc.encode(cenc.json, message)
|
||||
|
||||
// Decode message
|
||||
const decoded = cenc.decode(cenc.json, encoded)
|
||||
```
|
||||
|
||||
### Pattern 2: Hash Operations
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
const b4a = require('b4a')
|
||||
|
||||
// Create key pair
|
||||
const keyPair = crypto.keyPair()
|
||||
|
||||
// Sign data
|
||||
const data = b4a.from('message to sign')
|
||||
const signature = crypto.sign(data, keyPair.secretKey)
|
||||
|
||||
// Verify
|
||||
const valid = crypto.verify(data, signature, keyPair.publicKey)
|
||||
```
|
||||
|
||||
### Pattern 3: Streaming Data
|
||||
|
||||
```js
|
||||
const b4a = require('b4a')
|
||||
|
||||
// Accumulate chunks
|
||||
const chunks = []
|
||||
stream.on('data', chunk => {
|
||||
chunks.push(chunk)
|
||||
})
|
||||
|
||||
stream.on('end', () => {
|
||||
const full = b4a.concat(chunks)
|
||||
console.log('Total bytes:', full.length)
|
||||
})
|
||||
```
|
||||
|
||||
## Platform-Specific Behavior
|
||||
|
||||
| Operation | Node.js | Browser | React Native |
|
||||
|-----------|---------|---------|--------------|
|
||||
| `b4a.alloc()` | `Buffer.alloc()` | `new Uint8Array()` | Optimized native |
|
||||
| `b4a.from(string)` | `Buffer.from()` | TextEncoder | Optimized native |
|
||||
| `toString()` | `buf.toString()` | TextDecoder | Optimized native |
|
||||
| Type returned | `Buffer` | `Uint8Array` | `Uint8Array` |
|
||||
|
||||
## Integration with Other Modules
|
||||
|
||||
B4A is used throughout the ecosystem:
|
||||
- **hypercore**: All block data handling
|
||||
- **hyperdht**: Key and signature operations
|
||||
- **compact-encoding**: Binary encoding foundation
|
||||
- **sodium-native**: Crypto buffer operations
|
||||
- **protomux**: Message framing
|
||||
|
||||
## Performance Considerations
|
||||
|
||||
1. **Use `allocUnsafe` sparingly**: Only when performance is critical and you immediately overwrite all bytes
|
||||
2. **Pre-allocate when possible**: Avoid repeated allocations in loops
|
||||
3. **Use `concat` with totalLength**: Providing totalLength avoids intermediate allocations
|
||||
4. **React Native**: Install `react-native-b4a` for 10-100x performance improvement
|
||||
|
||||
## License
|
||||
|
||||
Apache 2.0
|
||||
|
||||
---
|
||||
|
||||
**Module Type**: Core Utility | **Ecosystem Role**: Cross-platform foundation | **Dependencies**: None
|
||||
@@ -0,0 +1,495 @@
|
||||
# bare-bundle - Application Bundle Format
|
||||
|
||||
## Overview
|
||||
|
||||
**bare-bundle** is the application packaging format for the Bare JavaScript runtime, inspired by Electron's ASAR format. It enables efficient distribution of JavaScript applications by bundling all files into a single archive with fast random access and a structured header for metadata.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Bundle Structure"
|
||||
HEADER[Header Section]
|
||||
FILES[File Data Section]
|
||||
end
|
||||
|
||||
subgraph "Header Contents"
|
||||
VERSION[Version]
|
||||
ID[Bundle ID]
|
||||
MAIN[Main Entry]
|
||||
IMPORTS[Import Maps]
|
||||
ADDONS[Addon List]
|
||||
ASSETS[Asset List]
|
||||
FILEMAP[File Map<br/>offset + length]
|
||||
end
|
||||
|
||||
subgraph "Usage Flow"
|
||||
CREATE[Create Bundle]
|
||||
DISTRIBUTE[Distribute]
|
||||
LOAD[Load at Runtime]
|
||||
EXECUTE[Execute]
|
||||
end
|
||||
|
||||
HEADER --> VERSION
|
||||
HEADER --> ID
|
||||
HEADER --> MAIN
|
||||
HEADER --> IMPORTS
|
||||
HEADER --> ADDONS
|
||||
HEADER --> ASSETS
|
||||
HEADER --> FILEMAP
|
||||
|
||||
CREATE --> HEADER
|
||||
CREATE --> FILES
|
||||
HEADER --> DISTRIBUTE
|
||||
FILES --> DISTRIBUTE
|
||||
DISTRIBUTE --> LOAD
|
||||
LOAD --> EXECUTE
|
||||
|
||||
style HEADER fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style FILEMAP fill:#bbf,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
## Bundle Format Specification
|
||||
|
||||
### File Layout
|
||||
|
||||
```
|
||||
[#!hashbang - optional]
|
||||
<header length as uint32>
|
||||
<header JSON as string>
|
||||
<file 1 data>
|
||||
<file 2 data>
|
||||
...
|
||||
<file n data>
|
||||
```
|
||||
|
||||
### Header Structure
|
||||
|
||||
```js
|
||||
{
|
||||
"version": 0, // Bundle format version
|
||||
"id": "unique-id", // Optional bundle identifier
|
||||
"main": "file:///main.js", // Entry point URL
|
||||
"imports": { // Import mappings
|
||||
"from-module": "to-module"
|
||||
},
|
||||
"resolutions": { // Pre-resolved imports
|
||||
"file:///module.js": { "#internal": "file:///internal.js" }
|
||||
},
|
||||
"addons": [ // Native addon paths
|
||||
"file:///native/addon.node"
|
||||
],
|
||||
"assets": [ // Static asset paths
|
||||
"file:///assets/logo.png"
|
||||
],
|
||||
"files": { // File offset map
|
||||
"file:///main.js": {
|
||||
"offset": 0, // Byte offset from end of header
|
||||
"length": 1024, // File size in bytes
|
||||
"mode": 0o644 // Unix file permissions
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Creating Bundles
|
||||
|
||||
```js
|
||||
const Bundle = require('bare-bundle')
|
||||
|
||||
// Create new bundle
|
||||
const bundle = new Bundle()
|
||||
|
||||
// Add files
|
||||
bundle.write('/index.js', Buffer.from('console.log("Hello")'))
|
||||
bundle.write('/lib/utils.js', Buffer.from('module.exports = {}'))
|
||||
bundle.write('/package.json', Buffer.from(JSON.stringify({
|
||||
"name": "my-app",
|
||||
"main": "./index.js"
|
||||
})))
|
||||
|
||||
// Set entry point
|
||||
bundle.main = new URL('file:///index.js')
|
||||
|
||||
// Set bundle ID
|
||||
bundle.id = 'my-app-v1.0.0'
|
||||
|
||||
// Add import mappings
|
||||
bundle.imports = {
|
||||
"#config": "file:///config.js"
|
||||
}
|
||||
|
||||
// Serialize to buffer
|
||||
const buffer = bundle.toBuffer()
|
||||
|
||||
// Save to file
|
||||
require('bare-fs').writeFileSync('app.bundle', buffer)
|
||||
```
|
||||
|
||||
### Loading Bundles
|
||||
|
||||
```js
|
||||
const Bundle = require('bare-bundle')
|
||||
const fs = require('bare-fs')
|
||||
|
||||
// Load from buffer
|
||||
const buffer = fs.readFileSync('app.bundle')
|
||||
const bundle = new Bundle(buffer)
|
||||
|
||||
// Access bundle metadata
|
||||
console.log(bundle.version) // 0
|
||||
console.log(bundle.id) // 'my-app-v1.0.0'
|
||||
console.log(bundle.main) // URL { ... }
|
||||
|
||||
// Read files
|
||||
const mainSource = bundle.read(bundle.main)
|
||||
console.log(mainSource.toString()) // 'console.log("Hello")'
|
||||
|
||||
// List all files
|
||||
for (const [url, info] of bundle.files) {
|
||||
console.log(url, info.offset, info.length, info.mode)
|
||||
}
|
||||
```
|
||||
|
||||
### Streaming Bundles
|
||||
|
||||
```js
|
||||
const Bundle = require('bare-bundle')
|
||||
|
||||
// Create bundle from stream
|
||||
const bundle = new Bundle()
|
||||
|
||||
// Stream files in
|
||||
bundle.write('/large-file.js', largeBuffer)
|
||||
|
||||
// Pipe to output
|
||||
const stream = bundle.createStream()
|
||||
stream.pipe(fs.createWriteStream('output.bundle'))
|
||||
```
|
||||
|
||||
## Bundle File Modes
|
||||
|
||||
File permissions are stored as Unix-style mode flags:
|
||||
|
||||
```js
|
||||
// Regular file
|
||||
{ mode: 0o644 } // rw-r--r--
|
||||
|
||||
// Executable
|
||||
{ mode: 0o755 } // rwxr-xr-x
|
||||
|
||||
// Read-only
|
||||
{ mode: 0o444 } // r--r--r--
|
||||
```
|
||||
|
||||
## Practical Examples
|
||||
|
||||
### Example 1: Simple Application Bundle
|
||||
|
||||
```js
|
||||
const Bundle = require('bare-bundle')
|
||||
|
||||
function createAppBundle() {
|
||||
const bundle = new Bundle()
|
||||
|
||||
// Main entry
|
||||
bundle.write('/index.js', Buffer.from(`
|
||||
const { greet } = require('./lib/greetings')
|
||||
greet('World')
|
||||
`))
|
||||
|
||||
// Library file
|
||||
bundle.write('/lib/greetings.js', Buffer.from(`
|
||||
exports.greet = function(name) {
|
||||
console.log('Hello, ' + name + '!')
|
||||
}
|
||||
`))
|
||||
|
||||
// Package metadata
|
||||
bundle.write('/package.json', Buffer.from(JSON.stringify({
|
||||
name: 'greeting-app',
|
||||
version: '1.0.0',
|
||||
main: './index.js'
|
||||
})))
|
||||
|
||||
// Set main entry
|
||||
bundle.main = new URL('file:///index.js')
|
||||
|
||||
return bundle.toBuffer()
|
||||
}
|
||||
```
|
||||
|
||||
### Example 2: Bundle with Native Addons
|
||||
|
||||
```js
|
||||
const Bundle = require('bare-bundle')
|
||||
|
||||
function createNativeBundle() {
|
||||
const bundle = new Bundle()
|
||||
|
||||
// JavaScript code
|
||||
bundle.write('/index.js', Buffer.from(`
|
||||
const addon = require.addon()
|
||||
addon.doSomething()
|
||||
`))
|
||||
|
||||
// Native addon for different platforms
|
||||
bundle.write('/prebuilds/darwin-arm64/addon.node', darwinArm64Buffer)
|
||||
bundle.write('/prebuilds/linux-x64/addon.node', linuxX64Buffer)
|
||||
bundle.write('/prebuilds/win32-x64/addon.node', win32X64Buffer)
|
||||
|
||||
// Track addons in header
|
||||
bundle.addons = [
|
||||
new URL('file:///prebuilds/darwin-arm64/addon.node'),
|
||||
new URL('file:///prebuilds/linux-x64/addon.node'),
|
||||
new URL('file:///prebuilds/win32-x64/addon.node')
|
||||
]
|
||||
|
||||
bundle.main = new URL('file:///index.js')
|
||||
|
||||
return bundle.toBuffer()
|
||||
}
|
||||
```
|
||||
|
||||
### Example 3: Bundle with Import Maps
|
||||
|
||||
```js
|
||||
const Bundle = require('bare-bundle')
|
||||
|
||||
function createMappedBundle() {
|
||||
const bundle = new Bundle()
|
||||
|
||||
// Application code
|
||||
bundle.write('/src/index.js', Buffer.from(`
|
||||
import { helper } from '#utils'
|
||||
import config from '#config'
|
||||
|
||||
helper(config.apiUrl)
|
||||
`))
|
||||
|
||||
bundle.write('/src/utils.js', Buffer.from(`
|
||||
export const helper = (url) => console.log(url)
|
||||
`))
|
||||
|
||||
bundle.write('/config/prod.js', Buffer.from(`
|
||||
export default { apiUrl: 'https://api.example.com' }
|
||||
`))
|
||||
|
||||
// Define import mappings
|
||||
bundle.imports = {
|
||||
"#utils": "file:///src/utils.js",
|
||||
"#config": "file:///config/prod.js"
|
||||
}
|
||||
|
||||
bundle.main = new URL('file:///src/index.js')
|
||||
|
||||
return bundle.toBuffer()
|
||||
}
|
||||
```
|
||||
|
||||
### Example 4: Bundle with Pre-resolved Imports
|
||||
|
||||
```js
|
||||
const Bundle = require('bare-bundle')
|
||||
|
||||
function createPreResolvedBundle() {
|
||||
const bundle = new Bundle()
|
||||
|
||||
// Main module
|
||||
bundle.write('/index.js', Buffer.from(`
|
||||
import foo from 'dependency'
|
||||
foo()
|
||||
`))
|
||||
|
||||
// Pre-resolve imports to avoid runtime resolution
|
||||
bundle.resolutions = {
|
||||
"file:///index.js": {
|
||||
"dependency": "file:///node_modules/dependency/index.js"
|
||||
}
|
||||
}
|
||||
|
||||
bundle.main = new URL('file:///index.js')
|
||||
|
||||
return bundle.toBuffer()
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with Bare Runtime
|
||||
|
||||
### Loading and Executing
|
||||
|
||||
```js
|
||||
const Bundle = require('bare-bundle')
|
||||
const Module = require('bare-module')
|
||||
|
||||
function runBundle(bundleBuffer) {
|
||||
// Parse bundle
|
||||
const bundle = new Bundle(bundleBuffer)
|
||||
|
||||
// Create custom protocol for bundle
|
||||
const bundleProtocol = new Module.Protocol({
|
||||
*resolve(specifier, parentURL) {
|
||||
// Handle bundle URLs
|
||||
if (specifier.startsWith('bundle:')) {
|
||||
yield new URL(specifier)
|
||||
}
|
||||
},
|
||||
|
||||
read(url) {
|
||||
// Read from bundle
|
||||
const path = url.pathname
|
||||
return bundle.read(new URL('file://' + path))
|
||||
},
|
||||
|
||||
exists(url) {
|
||||
const path = url.pathname
|
||||
return bundle.files.has('file://' + path)
|
||||
}
|
||||
})
|
||||
|
||||
// Load main module
|
||||
const main = Module.load(bundle.main, bundle.read(bundle.main), {
|
||||
protocol: bundleProtocol
|
||||
})
|
||||
|
||||
return main.exports
|
||||
}
|
||||
```
|
||||
|
||||
### With bare-module
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
const Bundle = require('bare-bundle')
|
||||
|
||||
// Bundle type is recognized by bare-module
|
||||
const bundle = new Bundle(bundleBuffer)
|
||||
|
||||
// Load as bundle type
|
||||
const module = Module.load(bundle.main, bundle.read(bundle.main), {
|
||||
type: Module.constants.types.BUNDLE
|
||||
})
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
| Operation | Time Complexity | Notes |
|
||||
|-----------|----------------|-------|
|
||||
| Load bundle | O(1) | Header parsed once |
|
||||
| Read file | O(1) | Direct offset lookup |
|
||||
| List files | O(n) | Iterate file map |
|
||||
| Create bundle | O(n) | Linear file addition |
|
||||
| Serialize | O(n) | Linear buffer construction |
|
||||
|
||||
### Memory Usage
|
||||
|
||||
- Header: ~100 bytes + file map
|
||||
- File map: ~50 bytes per file entry
|
||||
- Data: Original file sizes
|
||||
- Total overhead: ~5-10%
|
||||
|
||||
## Bundle Version History
|
||||
|
||||
- **Version 0**: Initial format
|
||||
- JSON header
|
||||
- File offset map
|
||||
- Import maps support
|
||||
- Resolution maps support
|
||||
|
||||
## Security Considerations
|
||||
|
||||
1. **Path Traversal**: Bundles should validate paths to prevent `../../../etc/passwd` style attacks
|
||||
2. **Executable Permissions**: Respect mode flags when extracting
|
||||
3. **Integrity**: Consider signing bundles for verification
|
||||
|
||||
```js
|
||||
// Validate paths
|
||||
function validatePath(path) {
|
||||
// Ensure path doesn't escape bundle root
|
||||
const resolved = require('bare-path').resolve('/', path)
|
||||
if (!resolved.startsWith('/')) {
|
||||
throw new Error('Invalid path: ' + path)
|
||||
}
|
||||
return resolved
|
||||
}
|
||||
```
|
||||
|
||||
## Tools and Utilities
|
||||
|
||||
### Bundle Inspector
|
||||
|
||||
```js
|
||||
function inspectBundle(buffer) {
|
||||
const bundle = new Bundle(buffer)
|
||||
|
||||
console.log('Bundle ID:', bundle.id)
|
||||
console.log('Version:', bundle.version)
|
||||
console.log('Main:', bundle.main)
|
||||
console.log('Files:')
|
||||
|
||||
for (const [url, info] of bundle.files) {
|
||||
console.log(' ', url)
|
||||
console.log(' offset:', info.offset)
|
||||
console.log(' length:', info.length)
|
||||
console.log(' mode:', info.mode.toString(8))
|
||||
}
|
||||
|
||||
console.log('Addons:', bundle.addons)
|
||||
console.log('Assets:', bundle.assets)
|
||||
console.log('Imports:', bundle.imports)
|
||||
}
|
||||
```
|
||||
|
||||
### Bundle Diff
|
||||
|
||||
```js
|
||||
function diffBundles(oldBuf, newBuf) {
|
||||
const old = new Bundle(oldBuf)
|
||||
const new_ = new Bundle(newBuf)
|
||||
|
||||
const changes = {
|
||||
added: [],
|
||||
removed: [],
|
||||
modified: []
|
||||
}
|
||||
|
||||
// Find added/modified
|
||||
for (const [url, newInfo] of new_.files) {
|
||||
if (!old.files.has(url)) {
|
||||
changes.added.push(url)
|
||||
} else {
|
||||
const oldInfo = old.files.get(url)
|
||||
if (oldInfo.length !== newInfo.length) {
|
||||
changes.modified.push(url)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Find removed
|
||||
for (const [url] of old.files) {
|
||||
if (!new_.files.has(url)) {
|
||||
changes.removed.push(url)
|
||||
}
|
||||
}
|
||||
|
||||
return changes
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Minimize Bundle Size**: Exclude unnecessary files (tests, docs, dev deps)
|
||||
2. **Use Import Maps**: Simplify module resolution
|
||||
3. **Pre-resolve Imports**: Speed up runtime loading
|
||||
4. **Version Your Bundles**: Use the `id` field for tracking
|
||||
5. **Compress Large Assets**: Consider pre-compressing large static files
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
|
||||
**Module Type**: Core Infrastructure | **Ecosystem Role**: Application Packaging | **Used By**: Bare runtime, Pear CLI
|
||||
@@ -0,0 +1,518 @@
|
||||
# bare-module - JavaScript Module System for Bare
|
||||
|
||||
## Overview
|
||||
|
||||
**bare-module** is the foundational module system that powers the Bare JavaScript runtime. It provides comprehensive support for both CommonJS (`require()`) and ES Modules (`import`), along with advanced features like conditional exports, import maps, and addon resolution. This is the core infrastructure that makes Bare a viable Node.js alternative for embedded and mobile applications.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Module System"
|
||||
RESOLVER[Module Resolution]
|
||||
LOADER[Module Loader]
|
||||
CACHE[Module Cache]
|
||||
PROTOCOL[Protocol Layer]
|
||||
end
|
||||
|
||||
subgraph "Module Types"
|
||||
CJS[CommonJS<br/>.js/.cjs]
|
||||
ESM[ES Modules<br/>.mjs/import]
|
||||
JSON[JSON Files]
|
||||
ADDON[Native Addons]
|
||||
BUNDLE[bare-bundle]
|
||||
end
|
||||
|
||||
subgraph "Package.json Features"
|
||||
EXPORTS[Conditional Exports]
|
||||
IMPORTS[Import Maps]
|
||||
TYPE[Module Type]
|
||||
ENGINES[Engine Requirements]
|
||||
end
|
||||
|
||||
RESOLVER --> LOADER
|
||||
LOADER --> CACHE
|
||||
PROTOCOL --> RESOLVER
|
||||
|
||||
LOADER --> CJS
|
||||
LOADER --> ESM
|
||||
LOADER --> JSON
|
||||
LOADER --> ADDON
|
||||
LOADER --> BUNDLE
|
||||
|
||||
RESOLVER --> EXPORTS
|
||||
RESOLVER --> IMPORTS
|
||||
RESOLVER --> TYPE
|
||||
RESOLVER --> ENGINES
|
||||
|
||||
style RESOLVER fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style LOADER fill:#bbf,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
## Module Resolution Algorithm
|
||||
|
||||
Bare-module implements a sophisticated resolution algorithm supporting:
|
||||
|
||||
### 1. Standard Resolution
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
|
||||
// Resolve relative paths
|
||||
const url = Module.resolve('./lib/utils', parentURL)
|
||||
|
||||
// Resolve package names
|
||||
const lodashURL = Module.resolve('lodash', parentURL)
|
||||
|
||||
// Resolve with conditions
|
||||
const url = Module.resolve('pkg', parentURL, {
|
||||
conditions: ['import', 'bare']
|
||||
})
|
||||
```
|
||||
|
||||
### 2. Conditional Exports
|
||||
|
||||
Packages can provide different entry points based on context:
|
||||
|
||||
```json
|
||||
{
|
||||
"name": "my-package",
|
||||
"exports": {
|
||||
".": {
|
||||
"import": "./index.mjs",
|
||||
"require": "./index.cjs",
|
||||
"bare": "./bare.js",
|
||||
"node": "./node.js",
|
||||
"default": "./fallback.js"
|
||||
},
|
||||
"./submodule": "./lib/submodule.js"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
**Condition Priority** (most specific first):
|
||||
1. `import` / `require` - Loading method
|
||||
2. `asset` - Asset loading
|
||||
3. `addon` - Native addon loading
|
||||
4. `bare` / `node` - Runtime environment
|
||||
5. `<platform>` / `<arch>` - Platform-specific (iOS, Android, etc.)
|
||||
6. `simulator` - Simulator builds
|
||||
7. `default` - Fallback (always last)
|
||||
|
||||
### 3. Import Maps
|
||||
|
||||
Private package-level import mappings:
|
||||
|
||||
```json
|
||||
{
|
||||
"imports": {
|
||||
"#config": {
|
||||
"bare": "./config.bare.js",
|
||||
"node": "./config.node.js",
|
||||
"default": "./config.js"
|
||||
},
|
||||
"#utils": "./lib/utils.js"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Usage in code:
|
||||
```js
|
||||
import config from '#config'
|
||||
import { helper } from '#utils'
|
||||
```
|
||||
|
||||
## CommonJS Module System
|
||||
|
||||
### Global Objects
|
||||
|
||||
When loading CommonJS modules, these globals are available:
|
||||
|
||||
```js
|
||||
// __dirname - Directory of current module
|
||||
console.log(__dirname) // file:///path/to/module/
|
||||
|
||||
// __filename - Full path to current module
|
||||
console.log(__filename) // file:///path/to/module/file.js
|
||||
|
||||
// require - Module loader
|
||||
const utils = require('./utils')
|
||||
|
||||
// module - Current module reference
|
||||
module.exports = { foo: 'bar' }
|
||||
|
||||
// exports - Shortcut for module.exports
|
||||
exports.baz = 123
|
||||
```
|
||||
|
||||
### require() API
|
||||
|
||||
#### Basic Usage
|
||||
|
||||
```js
|
||||
// Load core modules
|
||||
const fs = require('bare-fs')
|
||||
const path = require('bare-path')
|
||||
|
||||
// Load local modules
|
||||
const local = require('./local-module')
|
||||
|
||||
// Load npm packages
|
||||
const lodash = require('lodash')
|
||||
```
|
||||
|
||||
#### require.resolve()
|
||||
|
||||
```js
|
||||
// Get resolved path without loading
|
||||
const resolved = require.resolve('some-package')
|
||||
console.log(resolved) // file:///path/to/node_modules/some-package/index.js
|
||||
```
|
||||
|
||||
#### require.addon()
|
||||
|
||||
Load native addons with automatic platform resolution:
|
||||
|
||||
```js
|
||||
// Load addon for current platform
|
||||
const native = require.addon()
|
||||
|
||||
// Load specific addon
|
||||
const specific = require.addon('./native', parentURL)
|
||||
|
||||
// Get addon host string (platform-arch)
|
||||
console.log(require.addon.host) // 'darwin-arm64', 'linux-x64', etc.
|
||||
|
||||
// Resolve addon path
|
||||
const addonPath = require.addon.resolve('./my-addon')
|
||||
```
|
||||
|
||||
#### require.asset()
|
||||
|
||||
Load static assets:
|
||||
|
||||
```js
|
||||
const fs = require('bare-fs')
|
||||
|
||||
// Resolve asset path
|
||||
const assetPath = require.asset('./config.json')
|
||||
|
||||
// Read asset
|
||||
const config = JSON.parse(fs.readFileSync(assetPath))
|
||||
```
|
||||
|
||||
#### require.main
|
||||
|
||||
Access the entry module:
|
||||
|
||||
```js
|
||||
if (require.main === module) {
|
||||
console.log('This is the main module')
|
||||
}
|
||||
```
|
||||
|
||||
#### require.cache
|
||||
|
||||
Access the module cache:
|
||||
|
||||
```js
|
||||
// Clear specific module from cache
|
||||
delete require.cache[require.resolve('./module')]
|
||||
|
||||
// Clear entire cache (use with caution)
|
||||
Object.keys(require.cache).forEach(key => {
|
||||
delete require.cache[key]
|
||||
})
|
||||
```
|
||||
|
||||
## ES Module System
|
||||
|
||||
### Import Syntax
|
||||
|
||||
```js
|
||||
// Default import
|
||||
import fs from 'bare-fs'
|
||||
|
||||
// Named imports
|
||||
import { readFile, writeFile } from 'bare-fs'
|
||||
|
||||
// Namespace import
|
||||
import * as path from 'bare-path'
|
||||
|
||||
// Dynamic import
|
||||
const module = await import('./dynamic-module.js')
|
||||
|
||||
// Import with attributes
|
||||
import json from './config.json' with { type: 'json' }
|
||||
```
|
||||
|
||||
### import.meta
|
||||
|
||||
```js
|
||||
// Current module URL
|
||||
console.log(import.meta.url) // file:///path/to/current.js
|
||||
|
||||
// Check if main module
|
||||
if (import.meta.main) {
|
||||
console.log('Entry point')
|
||||
}
|
||||
|
||||
// Module cache
|
||||
console.log(import.meta.cache)
|
||||
|
||||
// Resolve specifiers
|
||||
const resolved = import.meta.resolve('./other.js')
|
||||
|
||||
// Addon loading
|
||||
const native = import.meta.addon()
|
||||
|
||||
// Asset loading
|
||||
const asset = import.meta.asset('./data.txt')
|
||||
```
|
||||
|
||||
## Module Types
|
||||
|
||||
The system supports multiple module types:
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
|
||||
console.log(Module.constants.types)
|
||||
// {
|
||||
// SCRIPT: 0, // CommonJS
|
||||
// MODULE: 1, // ES Module
|
||||
// JSON: 2, // JSON file
|
||||
// BUNDLE: 3, // bare-bundle
|
||||
// ADDON: 4, // Native addon
|
||||
// BINARY: 5, // Binary file
|
||||
// TEXT: 6 // Text file
|
||||
// }
|
||||
```
|
||||
|
||||
## Custom Protocols
|
||||
|
||||
Define custom module loading protocols:
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
|
||||
const customProtocol = new Module.Protocol({
|
||||
// Preprocess specifier before resolution
|
||||
preresolve(specifier, parentURL) {
|
||||
if (specifier.startsWith('custom:')) {
|
||||
return specifier.replace('custom:', '')
|
||||
}
|
||||
return specifier
|
||||
},
|
||||
|
||||
// Post-process resolved URL
|
||||
postresolve(url) {
|
||||
return new URL(url)
|
||||
},
|
||||
|
||||
// Resolve specifier to URL
|
||||
*resolve(specifier, parentURL, imports) {
|
||||
if (specifier.startsWith('hyper:')) {
|
||||
yield new URL(specifier)
|
||||
}
|
||||
},
|
||||
|
||||
// Check if URL exists
|
||||
exists(url) {
|
||||
return checkExists(url)
|
||||
},
|
||||
|
||||
// Read module source
|
||||
read(url) {
|
||||
return readFile(url)
|
||||
},
|
||||
|
||||
// Handle addon URLs
|
||||
addon(url) {
|
||||
return new URL(url.pathname + '.node')
|
||||
},
|
||||
|
||||
// Handle asset URLs
|
||||
asset(url) {
|
||||
return url
|
||||
}
|
||||
})
|
||||
|
||||
// Use custom protocol
|
||||
const module = Module.load(url, source, {
|
||||
protocol: customProtocol
|
||||
})
|
||||
```
|
||||
|
||||
## Package.json Configuration
|
||||
|
||||
### Type Field
|
||||
|
||||
Controls how `.js` files are interpreted:
|
||||
|
||||
```json
|
||||
{
|
||||
"type": "module" // .js files = ES modules
|
||||
}
|
||||
```
|
||||
|
||||
Without this field, `.js` defaults to CommonJS.
|
||||
|
||||
### Engines Field
|
||||
|
||||
Specify runtime requirements:
|
||||
|
||||
```json
|
||||
{
|
||||
"engines": {
|
||||
"bare": ">=1.0.5",
|
||||
"node": ">=18.0.0"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
Resolution fails if requirements aren't met.
|
||||
|
||||
### Exports Sugar
|
||||
|
||||
Single export shorthand:
|
||||
|
||||
```json
|
||||
{
|
||||
"exports": "./index.js"
|
||||
}
|
||||
```
|
||||
|
||||
Equivalent to:
|
||||
|
||||
```json
|
||||
{
|
||||
"exports": {
|
||||
".": "./index.js"
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Creating Custom require()
|
||||
|
||||
Useful for REPLs and isolated contexts:
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
|
||||
// Create isolated require
|
||||
const customRequire = Module.createRequire(
|
||||
new URL('file:///virtual/path/'),
|
||||
{
|
||||
type: Module.constants.types.SCRIPT,
|
||||
conditions: ['bare', 'import'],
|
||||
builtins: {
|
||||
'custom:fs': customFsImplementation
|
||||
}
|
||||
}
|
||||
)
|
||||
|
||||
// Use it
|
||||
const module = customRequire('./some-module')
|
||||
```
|
||||
|
||||
## Module States
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
|
||||
console.log(Module.constants.states)
|
||||
// {
|
||||
// EVALUATED: 1, // Module has run
|
||||
// SYNTHESIZED: 2 // Named exports detected
|
||||
// }
|
||||
```
|
||||
|
||||
## Global Cache
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
|
||||
// Access global module cache
|
||||
console.log(Module.cache)
|
||||
|
||||
// WARNING: Cache may contain modules from different bare-module versions
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
Common resolution errors:
|
||||
|
||||
```js
|
||||
try {
|
||||
const module = require('non-existent-module')
|
||||
} catch (err) {
|
||||
if (err.code === 'MODULE_NOT_FOUND') {
|
||||
console.log('Module not found')
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### With Hyperdrive
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
const Hyperdrive = require('hyperdrive')
|
||||
|
||||
// Custom protocol for hyperdrive
|
||||
const hyperProtocol = new Module.Protocol({
|
||||
async *resolve(specifier, parentURL) {
|
||||
if (specifier.startsWith('hyper:')) {
|
||||
const key = specifier.slice(6)
|
||||
yield new URL(`hyper://${key}`)
|
||||
}
|
||||
},
|
||||
|
||||
async read(url) {
|
||||
const drive = new Hyperdrive(url.hostname)
|
||||
await drive.ready()
|
||||
return drive.get(url.pathname)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### With bare-bundle
|
||||
|
||||
```js
|
||||
const Module = require('bare-module')
|
||||
const Bundle = require('bare-bundle')
|
||||
|
||||
// Load from bundle
|
||||
const bundle = new Bundle(buffer)
|
||||
const module = Module.load(bundle.main, bundle.read(bundle.main), {
|
||||
type: Module.constants.types.BUNDLE
|
||||
})
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Use `resolutions` map**: Pre-resolve imports to avoid repeated resolution
|
||||
2. **Cache warmup**: Pre-load commonly used modules
|
||||
3. **Conditional exports**: Minimize condition checks in hot paths
|
||||
4. **Lazy loading**: Use dynamic `import()` for optional dependencies
|
||||
|
||||
## Debugging
|
||||
|
||||
```js
|
||||
// Trace module resolution
|
||||
const originalResolve = Module.resolve
|
||||
Module.resolve = function(...args) {
|
||||
console.log('Resolving:', args[0])
|
||||
return originalResolve.apply(this, args)
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
|
||||
**Module Type**: Core Infrastructure | **Ecosystem Role**: Runtime Foundation | **Used By**: All Bare applications
|
||||
@@ -0,0 +1,562 @@
|
||||
# bare-process - Process Control for Bare
|
||||
|
||||
## Overview
|
||||
|
||||
**bare-process** provides Node.js-compatible process control APIs for the Bare JavaScript runtime. This module bridges the gap between Bare's lightweight architecture and the process management features developers expect from Node.js, enabling seamless portability of existing applications.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Process Control"
|
||||
EXIT[Process Exit]
|
||||
ENV[Environment Variables]
|
||||
ARGV[Command Line Args]
|
||||
STDIO[Standard I/O]
|
||||
PID[Process ID]
|
||||
CWD[Working Directory]
|
||||
end
|
||||
|
||||
subgraph "Bare Runtime"
|
||||
CORE[Core Runtime]
|
||||
GLOBAL[Global Scope]
|
||||
end
|
||||
|
||||
subgraph "Platform Abstraction"
|
||||
POSIX[POSIX Systems]
|
||||
WIN[Windows]
|
||||
MOBILE[iOS/Android]
|
||||
end
|
||||
|
||||
EXIT --> CORE
|
||||
ENV --> CORE
|
||||
ARGV --> CORE
|
||||
STDIO --> CORE
|
||||
PID --> CORE
|
||||
CWD --> CORE
|
||||
|
||||
CORE --> POSIX
|
||||
CORE --> WIN
|
||||
CORE --> MOBILE
|
||||
|
||||
GLOBAL --> EXIT
|
||||
GLOBAL --> ENV
|
||||
GLOBAL --> ARGV
|
||||
|
||||
style CORE fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style STDIO fill:#bbf,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
## Core API
|
||||
|
||||
### Process Exit
|
||||
|
||||
Control application lifecycle with exit codes:
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Exit with success (code 0)
|
||||
process.exit()
|
||||
|
||||
// Exit with specific code
|
||||
process.exit(1) // General error
|
||||
process.exit(0) // Success
|
||||
|
||||
// Exit codes following Unix conventions
|
||||
// 0 - Success
|
||||
// 1 - General error
|
||||
// 2 - Misuse of shell builtins
|
||||
// 126 - Command invoked cannot execute
|
||||
// 127 - Command not found
|
||||
// 128+n - Fatal error signal "n"
|
||||
// 130 - Script terminated by Ctrl-C
|
||||
// 255 - Exit status out of range
|
||||
```
|
||||
|
||||
### Environment Variables
|
||||
|
||||
Access and manipulate environment:
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Read environment variables
|
||||
const home = process.env.HOME
|
||||
const path = process.env.PATH
|
||||
|
||||
// Set environment variable
|
||||
process.env.MY_VAR = 'my-value'
|
||||
|
||||
// Check if variable exists
|
||||
if ('NODE_ENV' in process.env) {
|
||||
console.log('Environment:', process.env.NODE_ENV)
|
||||
}
|
||||
|
||||
// Iterate all environment variables
|
||||
for (const [key, value] of Object.entries(process.env)) {
|
||||
console.log(`${key}=${value}`)
|
||||
}
|
||||
|
||||
// Delete environment variable
|
||||
delete process.env.TEMP_VAR
|
||||
```
|
||||
|
||||
### Command Line Arguments
|
||||
|
||||
Access command-line arguments:
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// argv[0] - Bare executable path
|
||||
// argv[1] - Script being run
|
||||
// argv[2+] - User arguments
|
||||
console.log(process.argv)
|
||||
// [ '/path/to/bare', '/path/to/script.js', '--flag', 'value' ]
|
||||
|
||||
// Parse arguments
|
||||
const args = process.argv.slice(2)
|
||||
args.forEach((arg, index) => {
|
||||
console.log(`Arg ${index}: ${arg}`)
|
||||
})
|
||||
|
||||
// Example: Parse --key=value pairs
|
||||
const parsed = {}
|
||||
process.argv.slice(2).forEach(arg => {
|
||||
if (arg.startsWith('--')) {
|
||||
const [key, value] = arg.slice(2).split('=')
|
||||
parsed[key] = value || true
|
||||
}
|
||||
})
|
||||
console.log(parsed) // { flag: 'value' }
|
||||
```
|
||||
|
||||
### Process Identification
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Get process ID
|
||||
console.log('PID:', process.pid)
|
||||
|
||||
// Get parent process ID (if available)
|
||||
console.log('PPID:', process.ppid)
|
||||
|
||||
// Platform information
|
||||
console.log('Platform:', process.platform) // 'darwin', 'linux', 'win32', 'android', 'ios'
|
||||
console.log('Architecture:', process.arch) // 'arm64', 'x64', etc.
|
||||
```
|
||||
|
||||
### Working Directory
|
||||
|
||||
Manage current working directory:
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Get current working directory
|
||||
const cwd = process.cwd()
|
||||
console.log('Current directory:', cwd)
|
||||
|
||||
// Change working directory
|
||||
process.chdir('/new/path')
|
||||
console.log('New directory:', process.cwd())
|
||||
|
||||
// Common pattern: Save and restore
|
||||
try {
|
||||
const originalCwd = process.cwd()
|
||||
process.chdir('/tmp')
|
||||
// Do work in /tmp
|
||||
process.chdir(originalCwd)
|
||||
} catch (err) {
|
||||
console.error('Failed to change directory:', err)
|
||||
}
|
||||
```
|
||||
|
||||
## Global Installation
|
||||
|
||||
Make `process` available globally like Node.js:
|
||||
|
||||
```js
|
||||
// At entry point of application
|
||||
require('bare-process/global')
|
||||
|
||||
// Now process is global
|
||||
global.process.exit()
|
||||
// or simply
|
||||
process.exit()
|
||||
```
|
||||
|
||||
## Standard I/O
|
||||
|
||||
While bare-process focuses on process control, it works alongside bare-stdio for I/O:
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// These are available when bare-stdio is loaded
|
||||
console.log(process.stdin) // Standard input stream
|
||||
console.log(process.stdout) // Standard output stream
|
||||
console.log(process.stderr) // Standard error stream
|
||||
```
|
||||
|
||||
## Advanced Usage
|
||||
|
||||
### Signal Handling
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Handle process signals
|
||||
process.on('SIGINT', () => {
|
||||
console.log('Received SIGINT, cleaning up...')
|
||||
// Cleanup code here
|
||||
process.exit(0)
|
||||
})
|
||||
|
||||
process.on('SIGTERM', () => {
|
||||
console.log('Received SIGTERM, shutting down...')
|
||||
process.exit(0)
|
||||
})
|
||||
|
||||
// Custom cleanup on exit
|
||||
process.on('exit', (code) => {
|
||||
console.log(`Process exiting with code ${code}`)
|
||||
})
|
||||
```
|
||||
|
||||
### Uncaught Exception Handling
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
process.on('uncaughtException', (err) => {
|
||||
console.error('Uncaught exception:', err)
|
||||
// Log to error tracking service
|
||||
process.exit(1)
|
||||
})
|
||||
|
||||
process.on('unhandledRejection', (reason, promise) => {
|
||||
console.error('Unhandled rejection at:', promise, 'reason:', reason)
|
||||
process.exit(1)
|
||||
})
|
||||
```
|
||||
|
||||
### Memory Usage
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Get memory usage statistics
|
||||
const usage = process.memoryUsage()
|
||||
console.log('Memory usage:', usage)
|
||||
// {
|
||||
// rss: 23456789, // Resident set size
|
||||
// heapTotal: 12345678, // Total heap size
|
||||
// heapUsed: 9876543, // Used heap size
|
||||
// external: 1234567 // External memory
|
||||
// }
|
||||
|
||||
// Monitor memory
|
||||
setInterval(() => {
|
||||
const mem = process.memoryUsage()
|
||||
console.log(`Heap used: ${(mem.heapUsed / 1024 / 1024).toFixed(2)} MB`)
|
||||
}, 5000)
|
||||
```
|
||||
|
||||
### Uptime
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Get process uptime in seconds
|
||||
console.log(`Running for ${process.uptime()} seconds`)
|
||||
|
||||
// Human-readable uptime
|
||||
function formatUptime() {
|
||||
const uptime = process.uptime()
|
||||
const days = Math.floor(uptime / 86400)
|
||||
const hours = Math.floor((uptime % 86400) / 3600)
|
||||
const minutes = Math.floor((uptime % 3600) / 60)
|
||||
const seconds = Math.floor(uptime % 60)
|
||||
|
||||
return `${days}d ${hours}h ${minutes}m ${seconds}s`
|
||||
}
|
||||
|
||||
console.log(`Uptime: ${formatUptime()}`)
|
||||
```
|
||||
|
||||
## Integration Examples
|
||||
|
||||
### CLI Application Framework
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
class CLIApp {
|
||||
constructor() {
|
||||
this.commands = new Map()
|
||||
this.setupSignals()
|
||||
}
|
||||
|
||||
command(name, handler) {
|
||||
this.commands.set(name, handler)
|
||||
}
|
||||
|
||||
setupSignals() {
|
||||
process.on('SIGINT', () => this.shutdown())
|
||||
process.on('SIGTERM', () => this.shutdown())
|
||||
}
|
||||
|
||||
run() {
|
||||
const args = process.argv.slice(2)
|
||||
const [command, ...rest] = args
|
||||
|
||||
if (this.commands.has(command)) {
|
||||
try {
|
||||
this.commands.get(command)(rest)
|
||||
} catch (err) {
|
||||
console.error('Command failed:', err)
|
||||
process.exit(1)
|
||||
}
|
||||
} else {
|
||||
console.error(`Unknown command: ${command}`)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
shutdown() {
|
||||
console.log('\nShutting down gracefully...')
|
||||
// Cleanup
|
||||
process.exit(0)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const app = new CLIApp()
|
||||
|
||||
app.command('start', (args) => {
|
||||
console.log('Starting server...')
|
||||
console.log('Environment:', process.env.NODE_ENV)
|
||||
})
|
||||
|
||||
app.command('stop', () => {
|
||||
console.log('Stopping server...')
|
||||
})
|
||||
|
||||
app.run()
|
||||
```
|
||||
|
||||
### Configuration Loader
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
function loadConfig() {
|
||||
const env = process.env.NODE_ENV || 'development'
|
||||
|
||||
const configs = {
|
||||
development: {
|
||||
port: 3000,
|
||||
debug: true,
|
||||
db: 'localhost'
|
||||
},
|
||||
production: {
|
||||
port: 80,
|
||||
debug: false,
|
||||
db: process.env.DB_HOST
|
||||
}
|
||||
}
|
||||
|
||||
return configs[env] || configs.development
|
||||
}
|
||||
|
||||
// Override with CLI args
|
||||
function parseConfigOverrides(args) {
|
||||
const config = {}
|
||||
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const arg = args[i]
|
||||
if (arg.startsWith('--')) {
|
||||
const key = arg.slice(2)
|
||||
const value = args[i + 1]
|
||||
|
||||
if (value && !value.startsWith('--')) {
|
||||
config[key] = isNaN(value) ? value : Number(value)
|
||||
i++
|
||||
} else {
|
||||
config[key] = true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return config
|
||||
}
|
||||
|
||||
const config = {
|
||||
...loadConfig(),
|
||||
...parseConfigOverrides(process.argv.slice(2))
|
||||
}
|
||||
|
||||
console.log('Configuration:', config)
|
||||
```
|
||||
|
||||
### Graceful Shutdown Manager
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
class GracefulShutdown {
|
||||
constructor(timeout = 30000) {
|
||||
this.timeout = timeout
|
||||
this.handlers = []
|
||||
this.shuttingDown = false
|
||||
|
||||
process.on('SIGINT', () => this.shutdown('SIGINT'))
|
||||
process.on('SIGTERM', () => this.shutdown('SIGTERM'))
|
||||
}
|
||||
|
||||
onShutdown(handler) {
|
||||
this.handlers.push(handler)
|
||||
}
|
||||
|
||||
async shutdown(signal) {
|
||||
if (this.shuttingDown) return
|
||||
this.shuttingDown = true
|
||||
|
||||
console.log(`\nReceived ${signal}, starting graceful shutdown...`)
|
||||
|
||||
const timeout = setTimeout(() => {
|
||||
console.error('Shutdown timeout exceeded, forcing exit')
|
||||
process.exit(1)
|
||||
}, this.timeout)
|
||||
|
||||
try {
|
||||
for (const handler of this.handlers) {
|
||||
await handler()
|
||||
}
|
||||
console.log('Graceful shutdown complete')
|
||||
process.exit(0)
|
||||
} catch (err) {
|
||||
console.error('Error during shutdown:', err)
|
||||
process.exit(1)
|
||||
} finally {
|
||||
clearTimeout(timeout)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const shutdown = new GracefulShutdown(10000)
|
||||
|
||||
shutdown.onShutdown(async () => {
|
||||
console.log('Closing database connections...')
|
||||
// await db.close()
|
||||
})
|
||||
|
||||
shutdown.onShutdown(async () => {
|
||||
console.log('Stopping HTTP server...')
|
||||
// await server.close()
|
||||
})
|
||||
```
|
||||
|
||||
## Platform-Specific Behavior
|
||||
|
||||
### Environment Variable Inheritance
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Variables are inherited from parent process
|
||||
console.log('Inherited environment:')
|
||||
console.log('- HOME:', process.env.HOME)
|
||||
console.log('- USER:', process.env.USER)
|
||||
console.log('- PATH:', process.env.PATH?.slice(0, 50) + '...')
|
||||
|
||||
// Platform-specific variables
|
||||
if (process.platform === 'win32') {
|
||||
console.log('- USERPROFILE:', process.env.USERPROFILE)
|
||||
console.log('- APPDATA:', process.env.APPDATA)
|
||||
} else {
|
||||
console.log('- XDG_CONFIG_HOME:', process.env.XDG_CONFIG_HOME)
|
||||
console.log('- SHELL:', process.env.SHELL)
|
||||
}
|
||||
```
|
||||
|
||||
### Working Directory Differences
|
||||
|
||||
```js
|
||||
const process = require('bare-process')
|
||||
|
||||
// Get platform-specific separators
|
||||
const isWindows = process.platform === 'win32'
|
||||
const separator = isWindows ? '\\' : '/'
|
||||
|
||||
// Cross-platform path handling
|
||||
function resolvePath(...parts) {
|
||||
return parts.join(separator)
|
||||
}
|
||||
|
||||
// Ensure path exists before changing
|
||||
function safeChdir(path) {
|
||||
const fs = require('bare-fs')
|
||||
if (!fs.existsSync(path)) {
|
||||
throw new Error(`Directory does not exist: ${path}`)
|
||||
}
|
||||
process.chdir(path)
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Always handle errors** when changing directories
|
||||
2. **Use explicit exit codes** for different error conditions
|
||||
3. **Clean up resources** before calling `process.exit()`
|
||||
4. **Handle signals** for graceful shutdown
|
||||
5. **Validate environment variables** before using them
|
||||
|
||||
```js
|
||||
// Good practice example
|
||||
const process = require('bare-process')
|
||||
|
||||
function main() {
|
||||
// Validate required env vars
|
||||
const required = ['API_KEY', 'DATABASE_URL']
|
||||
for (const varName of required) {
|
||||
if (!process.env[varName]) {
|
||||
console.error(`Missing required environment variable: ${varName}`)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
// Setup signal handlers
|
||||
let isShuttingDown = false
|
||||
process.on('SIGINT', async () => {
|
||||
if (isShuttingDown) return
|
||||
isShuttingDown = true
|
||||
|
||||
console.log('\nShutting down...')
|
||||
// await cleanup()
|
||||
process.exit(0)
|
||||
})
|
||||
|
||||
// Run application
|
||||
try {
|
||||
// app.run()
|
||||
} catch (err) {
|
||||
console.error('Fatal error:', err)
|
||||
process.exit(1)
|
||||
}
|
||||
}
|
||||
|
||||
main()
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
|
||||
**Module Type**: Core Runtime | **Ecosystem Role**: Node.js Compatibility | **Used By**: All Bare CLI applications
|
||||
@@ -0,0 +1,704 @@
|
||||
# dht-rpc - Kademlia DHT with RPC
|
||||
|
||||
## Overview
|
||||
|
||||
**dht-rpc** provides a flexible, high-performance Distributed Hash Table (DHT) based on the Kademlia protocol, with built-in RPC capabilities. It enables peer discovery, mutable key-value storage, and custom command routing across decentralized networks. This is the foundational networking layer that powers HyperDHT and Hyperswarm.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "DHT Node"
|
||||
UDX[UDX Socket]
|
||||
ROUTING[Routing Table]
|
||||
QUERIES[Query Engine]
|
||||
RPC[RPC Handler]
|
||||
end
|
||||
|
||||
subgraph "Kademlia Protocol"
|
||||
LOOKUP[Node Lookup]
|
||||
STORE[Value Store]
|
||||
PING[Keep-alive Ping]
|
||||
DIST[Distance Metric<br/>XOR]
|
||||
end
|
||||
|
||||
subgraph "Network Operations"
|
||||
BOOTSTRAP[Bootstrap]
|
||||
QUERY[Query Routing]
|
||||
RELAY[Relay/Holepunch]
|
||||
NAT[NAT Detection]
|
||||
end
|
||||
|
||||
UDX --> ROUTING
|
||||
ROUTING --> QUERIES
|
||||
QUERIES --> RPC
|
||||
|
||||
LOOKUP --> ROUTING
|
||||
STORE --> RPC
|
||||
PING --> ROUTING
|
||||
DIST --> LOOKUP
|
||||
|
||||
BOOTSTRAP --> UDX
|
||||
QUERY --> QUERIES
|
||||
RELAY --> UDX
|
||||
NAT --> UDX
|
||||
|
||||
style ROUTING fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style RPC fill:#bbf,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Kademlia DHT
|
||||
|
||||
Kademlia organizes nodes in a binary tree where distance is measured by XOR of node IDs:
|
||||
|
||||
```
|
||||
Distance(A, B) = A.id XOR B.id
|
||||
|
||||
Node ID: 32-byte hash (usually of public key)
|
||||
Routing Table: 256 buckets (one per bit)
|
||||
Bucket Size: 20 nodes (Kademlia's "k" parameter)
|
||||
```
|
||||
|
||||
### RPC Commands
|
||||
|
||||
Custom commands are registered as numbered enums:
|
||||
|
||||
```js
|
||||
const COMMANDS = {
|
||||
PUT: 0, // Store value
|
||||
GET: 1, // Retrieve value
|
||||
PING: 2, // Keep-alive
|
||||
FIND_NODE: 3 // Lookup nodes
|
||||
}
|
||||
```
|
||||
|
||||
### Token-Based Security
|
||||
|
||||
Round-trip tokens prevent spoofing:
|
||||
|
||||
```
|
||||
1. Node A queries Node B
|
||||
2. Node B responds with token T
|
||||
3. Node A includes T in next request
|
||||
4. Node B verifies T is recent and valid
|
||||
5. Only then allows state mutation
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Creating a DHT Node
|
||||
|
||||
```js
|
||||
const DHT = require('dht-rpc')
|
||||
|
||||
// Create ephemeral node (default)
|
||||
const node = new DHT()
|
||||
|
||||
// Wait for readiness
|
||||
await node.fullyBootstrapped()
|
||||
|
||||
console.log('Node ID:', node.id?.toString('hex'))
|
||||
console.log('Public IP:', node.host)
|
||||
console.log('Public Port:', node.port)
|
||||
console.log('Firewalled:', node.firewalled)
|
||||
```
|
||||
|
||||
### Bootstrap Node
|
||||
|
||||
```js
|
||||
const DHT = require('dht-rpc')
|
||||
|
||||
// Create dedicated bootstrap node
|
||||
const bootstrap = DHT.bootstrapper(10001, '0.0.0.0')
|
||||
|
||||
await bootstrap.fullyBootstrapped()
|
||||
console.log('Bootstrap node ready on port 10001')
|
||||
|
||||
// Other nodes use this for bootstrapping
|
||||
const node = new DHT({
|
||||
bootstrap: ['127.0.0.1:10001']
|
||||
})
|
||||
```
|
||||
|
||||
### Custom RPC Commands
|
||||
|
||||
```js
|
||||
const DHT = require('dht-rpc')
|
||||
const crypto = require('crypto')
|
||||
|
||||
const VALUES = 0
|
||||
|
||||
const node = new DHT({
|
||||
bootstrap: ['localhost:10001']
|
||||
})
|
||||
|
||||
// Local storage
|
||||
const storage = new Map()
|
||||
|
||||
// Handle incoming requests
|
||||
node.on('request', (req) => {
|
||||
switch (req.command) {
|
||||
case VALUES:
|
||||
if (req.token) {
|
||||
// Commit: store the value
|
||||
const key = hash(req.value).toString('hex')
|
||||
storage.set(key, req.value)
|
||||
console.log('Stored:', key)
|
||||
return req.reply(null)
|
||||
}
|
||||
|
||||
// Query: return value if we have it
|
||||
const value = storage.get(req.target.toString('hex'))
|
||||
req.reply(value)
|
||||
break
|
||||
|
||||
default:
|
||||
req.error(DHT.ERROR_UNKNOWN_COMMAND)
|
||||
}
|
||||
})
|
||||
|
||||
function hash(value) {
|
||||
return crypto.createHash('sha256').update(value).digest()
|
||||
}
|
||||
```
|
||||
|
||||
### Querying the DHT
|
||||
|
||||
```js
|
||||
const DHT = require('dht-rpc')
|
||||
|
||||
const node = new DHT()
|
||||
|
||||
// Simple query
|
||||
const target = crypto.randomBytes(32)
|
||||
|
||||
const query = node.query({
|
||||
target,
|
||||
command: VALUES,
|
||||
value: Buffer.from('search query')
|
||||
})
|
||||
|
||||
// Process responses
|
||||
query.on('data', (data) => {
|
||||
console.log('Got response from:', data.from)
|
||||
console.log('Value:', data.value?.toString())
|
||||
})
|
||||
|
||||
// Wait for completion
|
||||
await query.finished()
|
||||
console.log('Query complete')
|
||||
console.log('Closest nodes:', query.closestNodes)
|
||||
```
|
||||
|
||||
### Committing Values
|
||||
|
||||
```js
|
||||
// Store value in DHT (requires commit)
|
||||
const value = Buffer.from('Hello, DHT!')
|
||||
const target = hash(value)
|
||||
|
||||
const query = node.query({
|
||||
target,
|
||||
command: VALUES,
|
||||
value
|
||||
}, {
|
||||
commit: true // Signal closest nodes to store
|
||||
})
|
||||
|
||||
await query.finished()
|
||||
console.log('Value stored at', target.toString('hex'))
|
||||
```
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Example 1: Distributed Key-Value Store
|
||||
|
||||
```js
|
||||
const DHT = require('dht-rpc')
|
||||
const crypto = require('crypto')
|
||||
|
||||
const PUT = 0
|
||||
const GET = 1
|
||||
|
||||
class DHTKeyValue {
|
||||
constructor(options = {}) {
|
||||
this.node = new DHT(options)
|
||||
this.storage = new Map()
|
||||
this.setupHandlers()
|
||||
}
|
||||
|
||||
setupHandlers() {
|
||||
this.node.on('request', (req) => {
|
||||
switch (req.command) {
|
||||
case PUT:
|
||||
this.handlePut(req)
|
||||
break
|
||||
case GET:
|
||||
this.handleGet(req)
|
||||
break
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
handlePut(req) {
|
||||
if (!req.token) {
|
||||
return req.error(DHT.ERROR_INVALID_TOKEN)
|
||||
}
|
||||
|
||||
const ttl = req.value.readUInt32BE(0)
|
||||
const key = req.value.slice(4, 36).toString('hex')
|
||||
const value = req.value.slice(36)
|
||||
|
||||
this.storage.set(key, {
|
||||
value,
|
||||
expires: Date.now() + ttl
|
||||
})
|
||||
|
||||
req.reply(null)
|
||||
}
|
||||
|
||||
handleGet(req) {
|
||||
const key = req.target.toString('hex')
|
||||
const entry = this.storage.get(key)
|
||||
|
||||
if (!entry || entry.expires < Date.now()) {
|
||||
this.storage.delete(key)
|
||||
return req.reply(null)
|
||||
}
|
||||
|
||||
req.reply(entry.value)
|
||||
}
|
||||
|
||||
async put(key, value, ttl = 3600000) {
|
||||
const target = Buffer.from(key, 'hex')
|
||||
const packet = Buffer.concat([
|
||||
Buffer.from([ttl >> 24, ttl >> 16, ttl >> 8, ttl]),
|
||||
target,
|
||||
Buffer.from(value)
|
||||
])
|
||||
|
||||
const query = this.node.query({
|
||||
target,
|
||||
command: PUT,
|
||||
value: packet
|
||||
}, { commit: true })
|
||||
|
||||
await query.finished()
|
||||
}
|
||||
|
||||
async get(key) {
|
||||
const target = Buffer.from(key, 'hex')
|
||||
|
||||
const query = this.node.query({
|
||||
target,
|
||||
command: GET
|
||||
})
|
||||
|
||||
for await (const data of query) {
|
||||
if (data.value) {
|
||||
return data.value.toString()
|
||||
}
|
||||
}
|
||||
|
||||
return null
|
||||
}
|
||||
|
||||
async ready() {
|
||||
await this.node.fullyBootstrapped()
|
||||
}
|
||||
|
||||
destroy() {
|
||||
return this.node.destroy()
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
async function main() {
|
||||
// Start bootstrap
|
||||
const bootstrap = DHT.bootstrapper(10001, '127.0.0.1')
|
||||
await bootstrap.fullyBootstrapped()
|
||||
|
||||
// Create nodes
|
||||
const node1 = new DHTKeyValue({
|
||||
bootstrap: ['127.0.0.1:10001']
|
||||
})
|
||||
|
||||
const node2 = new DHTKeyValue({
|
||||
bootstrap: ['127.0.0.1:10001']
|
||||
})
|
||||
|
||||
await node1.ready()
|
||||
await node2.ready()
|
||||
|
||||
// Store value
|
||||
const key = crypto.randomBytes(32).toString('hex')
|
||||
await node1.put(key, 'Hello from DHT!')
|
||||
|
||||
// Retrieve value
|
||||
const value = await node2.get(key)
|
||||
console.log('Retrieved:', value)
|
||||
|
||||
// Cleanup
|
||||
await node1.destroy()
|
||||
await node2.destroy()
|
||||
await bootstrap.destroy()
|
||||
}
|
||||
|
||||
main().catch(console.error)
|
||||
```
|
||||
|
||||
### Example 2: Peer Discovery Service
|
||||
|
||||
```js
|
||||
const DHT = require('dht-rpc')
|
||||
|
||||
const ANNOUNCE = 0
|
||||
const LOOKUP = 1
|
||||
|
||||
class PeerDiscovery {
|
||||
constructor(topic, options = {}) {
|
||||
this.topic = topic
|
||||
this.node = new DHT(options)
|
||||
this.peers = new Map()
|
||||
this.setupHandlers()
|
||||
}
|
||||
|
||||
setupHandlers() {
|
||||
this.node.on('request', (req) => {
|
||||
if (req.target.equals(this.topic)) {
|
||||
switch (req.command) {
|
||||
case ANNOUNCE:
|
||||
this.handleAnnounce(req)
|
||||
break
|
||||
case LOOKUP:
|
||||
this.handleLookup(req)
|
||||
break
|
||||
}
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
handleAnnounce(req) {
|
||||
if (!req.token) {
|
||||
return req.reply(null) // Request token first
|
||||
}
|
||||
|
||||
const peerId = req.from.host + ':' + req.from.port
|
||||
this.peers.set(peerId, {
|
||||
host: req.from.host,
|
||||
port: req.from.port,
|
||||
data: req.value,
|
||||
announced: Date.now()
|
||||
})
|
||||
|
||||
req.reply(null)
|
||||
}
|
||||
|
||||
handleLookup(req) {
|
||||
const peers = Array.from(this.peers.values())
|
||||
.filter(p => Date.now() - p.announced < 300000) // 5 min TTL
|
||||
.slice(0, 20)
|
||||
|
||||
req.reply(Buffer.from(JSON.stringify(peers)))
|
||||
}
|
||||
|
||||
async announce(host, port, metadata = {}) {
|
||||
const value = Buffer.from(JSON.stringify({
|
||||
host,
|
||||
port,
|
||||
...metadata
|
||||
}))
|
||||
|
||||
const query = this.node.query({
|
||||
target: this.topic,
|
||||
command: ANNOUNCE,
|
||||
value
|
||||
}, { commit: true })
|
||||
|
||||
await query.finished()
|
||||
}
|
||||
|
||||
async lookup() {
|
||||
const query = this.node.query({
|
||||
target: this.topic,
|
||||
command: LOOKUP
|
||||
})
|
||||
|
||||
const allPeers = new Map()
|
||||
|
||||
for await (const data of query) {
|
||||
if (data.value) {
|
||||
try {
|
||||
const peers = JSON.parse(data.value.toString())
|
||||
peers.forEach(p => {
|
||||
const id = p.host + ':' + p.port
|
||||
allPeers.set(id, p)
|
||||
})
|
||||
} catch (e) {
|
||||
// Ignore parse errors
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
return Array.from(allPeers.values())
|
||||
}
|
||||
|
||||
async ready() {
|
||||
await this.node.fullyBootstrapped()
|
||||
}
|
||||
|
||||
destroy() {
|
||||
return this.node.destroy()
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
async function main() {
|
||||
const topic = crypto.randomBytes(32)
|
||||
|
||||
const discovery = new PeerDiscovery(topic, {
|
||||
bootstrap: ['bootstrap.hyperdht.org:49737']
|
||||
})
|
||||
|
||||
await discovery.ready()
|
||||
|
||||
// Announce ourselves
|
||||
await discovery.announce(
|
||||
discovery.node.host,
|
||||
discovery.node.port,
|
||||
{ service: 'file-share', version: '1.0' }
|
||||
)
|
||||
|
||||
// Find peers
|
||||
const peers = await discovery.lookup()
|
||||
console.log('Found peers:', peers)
|
||||
|
||||
await discovery.destroy()
|
||||
}
|
||||
```
|
||||
|
||||
### Example 3: NAT Detection and Holepunch
|
||||
|
||||
```js
|
||||
const DHT = require('dht-rpc')
|
||||
|
||||
async function analyzeNat() {
|
||||
const node = new DHT()
|
||||
|
||||
await new Promise((resolve) => {
|
||||
node.once('persistent', resolve)
|
||||
})
|
||||
|
||||
console.log('NAT Analysis:')
|
||||
console.log(' Firewalled:', node.firewalled)
|
||||
console.log(' Randomized:', node.randomized)
|
||||
console.log(' Public IP:', node.host)
|
||||
console.log(' Public Port:', node.port)
|
||||
console.log(' Local Port:', node.address().port)
|
||||
|
||||
if (node.firewalled) {
|
||||
console.log(' → Node is behind NAT, will need relays')
|
||||
} else if (node.randomized) {
|
||||
console.log(' → Port is randomized, consistent connections difficult')
|
||||
} else {
|
||||
console.log(' → Node is directly accessible!')
|
||||
}
|
||||
|
||||
await node.destroy()
|
||||
}
|
||||
|
||||
// Test latency to specific node
|
||||
async function testLatency(node, target) {
|
||||
const times = []
|
||||
|
||||
for (let i = 0; i < 5; i++) {
|
||||
const start = Date.now()
|
||||
try {
|
||||
await node.ping(target)
|
||||
times.push(Date.now() - start)
|
||||
} catch (e) {
|
||||
times.push(null)
|
||||
}
|
||||
await new Promise(r => setTimeout(r, 100))
|
||||
}
|
||||
|
||||
const valid = times.filter(t => t !== null)
|
||||
const avg = valid.reduce((a, b) => a + b, 0) / valid.length
|
||||
|
||||
console.log(`Latency to ${target.host}:${target.port}:`)
|
||||
console.log(` Average: ${avg.toFixed(2)}ms`)
|
||||
console.log(` Min: ${Math.min(...valid)}ms`)
|
||||
console.log(` Max: ${Math.max(...valid)}ms`)
|
||||
}
|
||||
```
|
||||
|
||||
## Advanced Features
|
||||
|
||||
### Adaptive Mode
|
||||
|
||||
Nodes automatically determine if they should be persistent:
|
||||
|
||||
```js
|
||||
const node = new DHT({
|
||||
// Default: adaptive mode
|
||||
// Node starts ephemeral, may become persistent
|
||||
})
|
||||
|
||||
// Force persistent (for testing)
|
||||
const persistent = new DHT({
|
||||
ephemeral: false
|
||||
})
|
||||
|
||||
// Events
|
||||
node.on('persistent', () => {
|
||||
console.log('Node is now persistent in routing tables')
|
||||
})
|
||||
|
||||
node.on('ephemeral', () => {
|
||||
console.log('Node is ephemeral (not in routing tables)')
|
||||
})
|
||||
```
|
||||
|
||||
### Suspension and Resume
|
||||
|
||||
Handle mobile app backgrounding:
|
||||
|
||||
```js
|
||||
// App going to background
|
||||
await node.suspend()
|
||||
|
||||
// App resuming
|
||||
await node.resume()
|
||||
|
||||
// Handle sleep detection
|
||||
node.on('wake-up', () => {
|
||||
console.log('System woke from sleep')
|
||||
})
|
||||
```
|
||||
|
||||
### Custom Query Logic
|
||||
|
||||
```js
|
||||
const query = node.query({
|
||||
target,
|
||||
command: CUSTOM_CMD,
|
||||
value: payload
|
||||
}, {
|
||||
// Custom commit logic
|
||||
async commit(reply, dht, query) {
|
||||
// Called for each closest node response
|
||||
console.log('Committing to:', reply.from)
|
||||
|
||||
// Send authenticated request
|
||||
await dht.request({
|
||||
token: reply.token,
|
||||
target,
|
||||
command: CUSTOM_CMD,
|
||||
value: signedPayload
|
||||
}, reply.from)
|
||||
},
|
||||
|
||||
// Transform responses
|
||||
map(reply) {
|
||||
return {
|
||||
node: reply.from,
|
||||
value: reply.value,
|
||||
distance: xorDistance(target, reply.from.id)
|
||||
}
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Network Events
|
||||
|
||||
```js
|
||||
node.on('bootstrap', () => {
|
||||
console.log('Connected to bootstrap nodes')
|
||||
})
|
||||
|
||||
node.on('listening', () => {
|
||||
console.log('UDP socket ready')
|
||||
})
|
||||
|
||||
node.on('ready', () => {
|
||||
console.log('Fully bootstrapped')
|
||||
})
|
||||
|
||||
node.on('persistent', () => {
|
||||
console.log('Now persistent')
|
||||
})
|
||||
|
||||
node.on('wake-up', () => {
|
||||
console.log('Woke from sleep')
|
||||
})
|
||||
|
||||
node.on('network-change', (interfaces) => {
|
||||
console.log('Network changed:', interfaces)
|
||||
})
|
||||
|
||||
node.on('nat-update', (host, port) => {
|
||||
console.log(`NAT mapping: ${host}:${port}`)
|
||||
})
|
||||
|
||||
node.on('close', () => {
|
||||
console.log('Node destroyed')
|
||||
})
|
||||
```
|
||||
|
||||
## Error Codes
|
||||
|
||||
```js
|
||||
DHT.OK = 0 // Success
|
||||
DHT.ERROR_UNKNOWN_COMMAND = 1 // Unknown RPC command
|
||||
DHT.ERROR_INVALID_TOKEN = 2 // Invalid/Expired token
|
||||
// Application errors should start at 16
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
| Metric | Value | Notes |
|
||||
|--------|-------|-------|
|
||||
| Lookup time | <1s | To closest nodes |
|
||||
| Routing table | 256 buckets | 20 nodes per bucket |
|
||||
| Concurrent queries | Unlimited | Async iterators |
|
||||
| Token TTL | ~5 min | Prevents replay |
|
||||
| Bootstrap time | ~2-5s | Depends on network |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use tokens for mutations**: Always require tokens for state changes
|
||||
2. **Validate data**: Don't trust incoming requests blindly
|
||||
3. **Handle timeouts**: Network is unreliable
|
||||
4. **Implement TTL**: Clean up stale data
|
||||
5. **Monitor NAT status**: Adapt behavior based on connectivity
|
||||
|
||||
```js
|
||||
// Good practice: Always validate
|
||||
node.on('request', (req) => {
|
||||
if (!isValidCommand(req.command)) {
|
||||
return req.error(DHT.ERROR_UNKNOWN_COMMAND)
|
||||
}
|
||||
|
||||
if (isMutation(req.command) && !req.token) {
|
||||
return req.error(DHT.ERROR_INVALID_TOKEN)
|
||||
}
|
||||
|
||||
// Validate payload size
|
||||
if (req.value && req.value.length > MAX_SIZE) {
|
||||
return req.error(16) // Custom: Payload too large
|
||||
}
|
||||
|
||||
// Process request
|
||||
handleRequest(req)
|
||||
})
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
---
|
||||
|
||||
**Module Type**: Core Networking | **Ecosystem Role**: DHT Foundation | **Used By**: hyperdht, hyperswarm, all peer discovery
|
||||
@@ -0,0 +1,552 @@
|
||||
# hypercore-crypto - Cryptographic Primitives
|
||||
|
||||
## Overview
|
||||
|
||||
**hypercore-crypto** provides the fundamental cryptographic primitives used throughout the Hypercore ecosystem. Built on battle-tested Ed25519 signatures and BLAKE2b hashing, it powers the security guarantees of append-only logs, peer authentication, and data integrity across all Hypercore-based applications.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "Cryptographic Primitives"
|
||||
KEYS[Ed25519 Key Pairs]
|
||||
SIGN[Signing]
|
||||
VERIFY[Verification]
|
||||
HASH[BLAKE2b Hashing]
|
||||
RANDOM[Secure Random]
|
||||
end
|
||||
|
||||
subgraph "Hypercore Operations"
|
||||
TREE[Merkle Tree]
|
||||
DISCOVERY[Discovery Keys]
|
||||
NAMESPACE[Namespaces]
|
||||
PROOFS[Proof Verification]
|
||||
end
|
||||
|
||||
subgraph "Security Guarantees"
|
||||
AUTH[Writer Authentication]
|
||||
INTEGRITY[Data Integrity]
|
||||
PRIVACY[Privacy Preserving]
|
||||
end
|
||||
|
||||
KEYS --> SIGN
|
||||
KEYS --> VERIFY
|
||||
SIGN --> PROOFS
|
||||
VERIFY --> PROOFS
|
||||
HASH --> TREE
|
||||
HASH --> DISCOVERY
|
||||
|
||||
TREE --> AUTH
|
||||
TREE --> INTEGRITY
|
||||
DISCOVERY --> PRIVACY
|
||||
NAMESPACE --> AUTH
|
||||
|
||||
style KEYS fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style HASH fill:#bbf,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
## Core API
|
||||
|
||||
### Key Pairs
|
||||
|
||||
Generate Ed25519 key pairs for writer identification:
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
// Generate new key pair
|
||||
const keyPair = crypto.keyPair()
|
||||
|
||||
console.log('Public Key:', keyPair.publicKey.toString('hex'))
|
||||
// 64 character hex string
|
||||
|
||||
console.log('Secret Key:', keyPair.secretKey.toString('hex'))
|
||||
// 128 character hex string (includes public key)
|
||||
|
||||
// Key pair structure
|
||||
// {
|
||||
// publicKey: <Buffer 32 bytes>,
|
||||
// secretKey: <Buffer 64 bytes>
|
||||
// }
|
||||
```
|
||||
|
||||
### Signing and Verification
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
const b4a = require('b4a')
|
||||
|
||||
const keyPair = crypto.keyPair()
|
||||
|
||||
// Message to sign
|
||||
const message = b4a.from('Hello, Hypercore!')
|
||||
|
||||
// Sign message
|
||||
const signature = crypto.sign(message, keyPair.secretKey)
|
||||
console.log('Signature:', signature.toString('hex'))
|
||||
// 128 character hex string (Ed25519 signature)
|
||||
|
||||
// Verify signature
|
||||
const isValid = crypto.verify(message, signature, keyPair.publicKey)
|
||||
console.log('Valid:', isValid) // true
|
||||
|
||||
// Tampered message fails verification
|
||||
const tampered = b4a.from('Goodbye, Hypercore!')
|
||||
const isTamperedValid = crypto.verify(tampered, signature, keyPair.publicKey)
|
||||
console.log('Tampered valid:', isTamperedValid) // false
|
||||
```
|
||||
|
||||
### Merkle Tree Operations
|
||||
|
||||
#### Data Hashing
|
||||
|
||||
Hash leaf nodes (data blocks):
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
// Hash a data block
|
||||
const data = Buffer.from('Block content')
|
||||
const leafHash = crypto.data(data)
|
||||
|
||||
console.log('Leaf hash:', leafHash.toString('hex'))
|
||||
// 64 character BLAKE2b hash
|
||||
```
|
||||
|
||||
#### Parent Hashing
|
||||
|
||||
Combine child nodes into parent hashes:
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
// Two child nodes
|
||||
const left = {
|
||||
index: 0,
|
||||
hash: crypto.data(Buffer.from('Left block')),
|
||||
size: 10
|
||||
}
|
||||
|
||||
const right = {
|
||||
index: 1,
|
||||
hash: crypto.data(Buffer.from('Right block')),
|
||||
size: 11
|
||||
}
|
||||
|
||||
// Compute parent hash
|
||||
const parentHash = crypto.parent(left, right)
|
||||
console.log('Parent hash:', parentHash.toString('hex'))
|
||||
```
|
||||
|
||||
#### Tree Root Hashing
|
||||
|
||||
Hash the merkle root from tree peaks:
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
// Tree peaks (for a tree with 5 nodes)
|
||||
const peaks = [
|
||||
{ index: 4, hash: peak1Hash, size: size1 },
|
||||
{ index: 2, hash: peak2Hash, size: size2 }
|
||||
]
|
||||
|
||||
// Compute root hash
|
||||
const rootHash = crypto.tree(peaks)
|
||||
console.log('Root hash:', rootHash.toString('hex'))
|
||||
```
|
||||
|
||||
### Discovery Keys
|
||||
|
||||
Generate privacy-preserving discovery keys:
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
const keyPair = crypto.keyPair()
|
||||
|
||||
// Generate discovery key from public key
|
||||
const discoveryKey = crypto.discoveryKey(keyPair.publicKey)
|
||||
|
||||
console.log('Public Key:', keyPair.publicKey.toString('hex'))
|
||||
console.log('Discovery Key:', discoveryKey.toString('hex'))
|
||||
|
||||
// Discovery key is derived but doesn't reveal public key
|
||||
// Used for DHT topics without exposing the actual feed key
|
||||
```
|
||||
|
||||
### Namespaces
|
||||
|
||||
Create namespaced capabilities:
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
// Create namespace from public name
|
||||
const namespaces = crypto.namespace('hypercore', 3)
|
||||
|
||||
console.log('Namespaces:')
|
||||
namespaces.forEach((ns, i) => {
|
||||
console.log(` ${i}:`, ns.toString('hex'))
|
||||
})
|
||||
|
||||
// Use for algorithm-specific operations
|
||||
// Prevents cross-protocol attacks
|
||||
```
|
||||
|
||||
### Random Bytes
|
||||
|
||||
Generate cryptographically secure random data:
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
// Generate 32 random bytes
|
||||
const random = crypto.randomBytes(32)
|
||||
console.log('Random:', random.toString('hex'))
|
||||
|
||||
// Use for nonces, salts, keys
|
||||
const nonce = crypto.randomBytes(24)
|
||||
const salt = crypto.randomBytes(16)
|
||||
```
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Example 1: Verifiable Log
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
const b4a = require('b4a')
|
||||
|
||||
class VerifiableLog {
|
||||
constructor() {
|
||||
this.blocks = []
|
||||
this.hashes = []
|
||||
this.keyPair = crypto.keyPair()
|
||||
}
|
||||
|
||||
append(data) {
|
||||
const index = this.blocks.length
|
||||
|
||||
// Hash the data
|
||||
const hash = crypto.data(b4a.from(data))
|
||||
|
||||
// Sign the hash
|
||||
const signature = crypto.sign(hash, this.keyPair.secretKey)
|
||||
|
||||
this.blocks.push({
|
||||
index,
|
||||
data,
|
||||
hash,
|
||||
signature
|
||||
})
|
||||
|
||||
this.hashes.push(hash)
|
||||
|
||||
return { index, hash, signature }
|
||||
}
|
||||
|
||||
verify(index) {
|
||||
const block = this.blocks[index]
|
||||
if (!block) return false
|
||||
|
||||
// Verify signature
|
||||
return crypto.verify(block.hash, block.signature, this.keyPair.publicKey)
|
||||
}
|
||||
|
||||
getPublicKey() {
|
||||
return this.keyPair.publicKey
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const log = new VerifiableLog()
|
||||
|
||||
log.append('First block')
|
||||
log.append('Second block')
|
||||
log.append('Third block')
|
||||
|
||||
console.log('Block 0 valid:', log.verify(0)) // true
|
||||
console.log('Block 1 valid:', log.verify(1)) // true
|
||||
console.log('Block 2 valid:', log.verify(2)) // true
|
||||
```
|
||||
|
||||
### Example 2: Merkle Tree Builder
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
const b4a = require('b4a')
|
||||
|
||||
class MerkleTree {
|
||||
constructor() {
|
||||
this.leaves = []
|
||||
this.layers = []
|
||||
}
|
||||
|
||||
push(data) {
|
||||
const hash = crypto.data(b4a.from(data))
|
||||
this.leaves.push({
|
||||
index: this.leaves.length,
|
||||
hash,
|
||||
size: data.length
|
||||
})
|
||||
this.rebuild()
|
||||
}
|
||||
|
||||
rebuild() {
|
||||
this.layers = [this.leaves]
|
||||
|
||||
while (this.layers[0].length > 1) {
|
||||
const layer = this.layers[0]
|
||||
const nextLayer = []
|
||||
|
||||
for (let i = 0; i < layer.length; i += 2) {
|
||||
if (i + 1 < layer.length) {
|
||||
// Combine pair
|
||||
nextLayer.push({
|
||||
index: layer[i].index,
|
||||
hash: crypto.parent(layer[i], layer[i + 1]),
|
||||
size: layer[i].size + layer[i + 1].size
|
||||
})
|
||||
} else {
|
||||
// Promote single
|
||||
nextLayer.push(layer[i])
|
||||
}
|
||||
}
|
||||
|
||||
this.layers.unshift(nextLayer)
|
||||
}
|
||||
}
|
||||
|
||||
root() {
|
||||
if (this.layers.length === 0) return null
|
||||
return this.layers[0][0]
|
||||
}
|
||||
|
||||
proof(index) {
|
||||
// Generate inclusion proof for leaf at index
|
||||
const proof = []
|
||||
let idx = index
|
||||
|
||||
for (let i = this.layers.length - 1; i > 0; i--) {
|
||||
const layer = this.layers[i]
|
||||
const isRight = idx % 2 === 1
|
||||
const siblingIdx = isRight ? idx - 1 : idx + 1
|
||||
|
||||
if (siblingIdx < layer.length) {
|
||||
proof.push({
|
||||
position: isRight ? 'left' : 'right',
|
||||
hash: layer[siblingIdx].hash
|
||||
})
|
||||
}
|
||||
|
||||
idx = Math.floor(idx / 2)
|
||||
}
|
||||
|
||||
return proof
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const tree = new MerkleTree()
|
||||
|
||||
tree.push('Block 0')
|
||||
tree.push('Block 1')
|
||||
tree.push('Block 2')
|
||||
tree.push('Block 3')
|
||||
|
||||
console.log('Root:', tree.root().hash.toString('hex'))
|
||||
console.log('Proof for Block 1:', tree.proof(1))
|
||||
```
|
||||
|
||||
### Example 3: Secure Channel Setup
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
const b4a = require('b4a')
|
||||
|
||||
class SecureChannel {
|
||||
constructor() {
|
||||
this.keyPair = crypto.keyPair()
|
||||
this.sessionKey = crypto.randomBytes(32)
|
||||
}
|
||||
|
||||
getPublicKey() {
|
||||
return this.keyPair.publicKey
|
||||
}
|
||||
|
||||
getDiscoveryKey() {
|
||||
return crypto.discoveryKey(this.keyPair.publicKey)
|
||||
}
|
||||
|
||||
signHandshake() {
|
||||
const message = b4a.concat([
|
||||
this.keyPair.publicKey,
|
||||
this.sessionKey
|
||||
])
|
||||
return crypto.sign(message, this.keyPair.secretKey)
|
||||
}
|
||||
|
||||
verifyHandshake(publicKey, sessionKey, signature) {
|
||||
const message = b4a.concat([publicKey, sessionKey])
|
||||
return crypto.verify(message, signature, publicKey)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const alice = new SecureChannel()
|
||||
const bob = new SecureChannel()
|
||||
|
||||
console.log('Alice discovery key:', alice.getDiscoveryKey().toString('hex'))
|
||||
console.log('Bob discovery key:', bob.getDiscoveryKey().toString('hex'))
|
||||
|
||||
// Alice signs handshake for Bob
|
||||
const aliceSignature = alice.signHandshake()
|
||||
const isValid = bob.verifyHandshake(
|
||||
alice.getPublicKey(),
|
||||
alice.sessionKey,
|
||||
aliceSignature
|
||||
)
|
||||
|
||||
console.log('Handshake valid:', isValid)
|
||||
```
|
||||
|
||||
## Integration with Hypercore
|
||||
|
||||
### Block Signing
|
||||
|
||||
```js
|
||||
const Hypercore = require('hypercore')
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
// Hypercore internally uses hypercore-crypto for:
|
||||
// - Generating writer key pairs
|
||||
// - Signing each block
|
||||
// - Verifying blocks on read
|
||||
// - Building merkle trees
|
||||
|
||||
const core = new Hypercore('./storage', {
|
||||
// Uses crypto.keyPair() internally
|
||||
})
|
||||
|
||||
await core.ready()
|
||||
|
||||
// Each append signs the block
|
||||
await core.append('Data')
|
||||
// Internally: crypto.sign(hash, secretKey)
|
||||
|
||||
// Each get verifies the signature
|
||||
const block = await core.get(0)
|
||||
// Internally: crypto.verify(hash, signature, publicKey)
|
||||
```
|
||||
|
||||
### Proof Verification
|
||||
|
||||
```js
|
||||
const Hypercore = require('hypercore')
|
||||
|
||||
const core = new Hypercore('./storage')
|
||||
await core.ready()
|
||||
|
||||
// Create proof
|
||||
const proof = await core.proof({
|
||||
block: { index: 5, nodes: 2 }
|
||||
})
|
||||
|
||||
// Proof contains:
|
||||
// - Block signature
|
||||
// - Merkle tree nodes
|
||||
// - Root signature
|
||||
|
||||
// Verify proof
|
||||
const verified = await core.verify(proof, {
|
||||
publicKey: core.key
|
||||
})
|
||||
|
||||
console.log('Proof verified:', verified)
|
||||
```
|
||||
|
||||
## Security Considerations
|
||||
|
||||
### Key Management
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
// NEVER log or transmit secret keys
|
||||
const keyPair = crypto.keyPair()
|
||||
|
||||
// Safe to share
|
||||
console.log('Public Key:', keyPair.publicKey.toString('hex'))
|
||||
|
||||
// NEVER do this in production
|
||||
// console.log('Secret Key:', keyPair.secretKey.toString('hex'))
|
||||
|
||||
// Store secret keys securely
|
||||
// - Hardware security modules (HSM)
|
||||
// - Encrypted key stores
|
||||
// - OS keychain
|
||||
```
|
||||
|
||||
### Replay Protection
|
||||
|
||||
```js
|
||||
const crypto = require('hypercore-crypto')
|
||||
const b4a = require('b4a')
|
||||
|
||||
// Include sequence numbers in signed messages
|
||||
function createSignedMessage(data, seq, keyPair) {
|
||||
const message = b4a.concat([
|
||||
b4a.from([seq]),
|
||||
b4a.from(data)
|
||||
])
|
||||
|
||||
return {
|
||||
seq,
|
||||
data,
|
||||
signature: crypto.sign(message, keyPair.secretKey)
|
||||
}
|
||||
}
|
||||
|
||||
function verifySignedMessage(msg, expectedSeq, publicKey) {
|
||||
const message = b4a.concat([
|
||||
b4a.from([msg.seq]),
|
||||
b4a.from(msg.data)
|
||||
])
|
||||
|
||||
// Check sequence first
|
||||
if (msg.seq !== expectedSeq) {
|
||||
return false
|
||||
}
|
||||
|
||||
return crypto.verify(message, msg.signature, publicKey)
|
||||
}
|
||||
```
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
| Operation | Time | Notes |
|
||||
|-----------|------|-------|
|
||||
| keyPair() | ~1ms | Ed25519 key generation |
|
||||
| sign() | ~0.1ms | Ed25519 signing |
|
||||
| verify() | ~0.3ms | Ed25519 verification |
|
||||
| data() | ~0.01ms | BLAKE2b hashing |
|
||||
| parent() | ~0.01ms | BLAKE2b hashing |
|
||||
| tree() | ~0.01ms | BLAKE2b hashing |
|
||||
| randomBytes() | ~0.01ms | OS random source |
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Reuse key pairs**: Generate once, store securely
|
||||
2. **Verify early**: Check signatures before processing
|
||||
3. **Use discovery keys**: Don't expose public keys unnecessarily
|
||||
4. **Namespace operations**: Prevent cross-protocol confusion
|
||||
5. **Random nonces**: Never reuse nonces for encryption
|
||||
|
||||
## License
|
||||
|
||||
MIT
|
||||
|
||||
---
|
||||
|
||||
**Module Type**: Core Cryptography | **Ecosystem Role**: Security Foundation | **Used By**: hypercore, hyperdht, hyperswarm, all P2P protocols
|
||||
@@ -0,0 +1,640 @@
|
||||
# protomux-rpc - Multiplexed RPC Protocol
|
||||
|
||||
## Overview
|
||||
|
||||
**protomux-rpc** provides a simple yet powerful RPC (Remote Procedure Call) system built on top of Protomux channels. It enables structured request-response communication over framed streams, making it ideal for building distributed applications, microservices, and peer-to-peer protocols in the Hypercore ecosystem.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "RPC Layer"
|
||||
CLIENT[RPC Client]
|
||||
SERVER[RPC Server]
|
||||
METHODS[Method Registry]
|
||||
end
|
||||
|
||||
subgraph "Protomux Layer"
|
||||
MUX[Protomux Muxer]
|
||||
CHANNELS[Channels]
|
||||
FRAMING[Message Framing]
|
||||
end
|
||||
|
||||
subgraph "Transport"
|
||||
STREAM[Framed Stream]
|
||||
HYPERSWARM[Hyperswarm]
|
||||
DIRECT[Direct Connection]
|
||||
end
|
||||
|
||||
CLIENT --> MUX
|
||||
SERVER --> MUX
|
||||
METHODS --> SERVER
|
||||
|
||||
MUX --> CHANNELS
|
||||
CHANNELS --> FRAMING
|
||||
|
||||
FRAMING --> STREAM
|
||||
STREAM --> HYPERSWARM
|
||||
STREAM --> DIRECT
|
||||
|
||||
style MUX fill:#f9f,stroke:#333,stroke-width:2px
|
||||
style RPC fill:#bbf,stroke:#333,stroke-width:2px
|
||||
```
|
||||
|
||||
## Core Concepts
|
||||
|
||||
### Request-Response Model
|
||||
|
||||
```
|
||||
Client Server
|
||||
| |
|
||||
|--- Request (method, payload) ----------->|
|
||||
| |
|
||||
| |--> Handler
|
||||
| |<-- Response
|
||||
| |
|
||||
|<-- Response (result or error) -----------|
|
||||
```
|
||||
|
||||
### Compact Encoding
|
||||
|
||||
Messages use compact-encoding for efficient binary serialization:
|
||||
|
||||
```js
|
||||
// Request format
|
||||
{
|
||||
id: uint, // Request ID for matching response
|
||||
method: string, // Method name
|
||||
value: raw // Binary payload
|
||||
}
|
||||
|
||||
// Response format
|
||||
{
|
||||
flags: bitfield, // Error indicators
|
||||
id: uint, // Matching request ID
|
||||
value: raw // Response data
|
||||
}
|
||||
```
|
||||
|
||||
### Channel Multiplexing
|
||||
|
||||
Multiple RPC channels can share one transport stream via Protomux:
|
||||
|
||||
```
|
||||
Stream
|
||||
├── Channel 1: RPC (methods A, B, C)
|
||||
├── Channel 2: RPC (methods X, Y, Z)
|
||||
└── Channel 3: Raw data
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Creating RPC Connection
|
||||
|
||||
```js
|
||||
const ProtomuxRPC = require('protomux-rpc')
|
||||
|
||||
// From framed stream
|
||||
const rpc = new ProtomuxRPC(stream)
|
||||
|
||||
// From existing protomux instance
|
||||
const rpc = new ProtomuxRPC(mux, {
|
||||
protocol: 'my-protocol'
|
||||
})
|
||||
|
||||
// Wait for ready
|
||||
await rpc.fullyOpened()
|
||||
```
|
||||
|
||||
### Server-Side: Responding to Requests
|
||||
|
||||
```js
|
||||
const rpc = new ProtomuxRPC(stream)
|
||||
|
||||
// Register method handler
|
||||
rpc.respond('echo', (req) => {
|
||||
return req // Echo back the request
|
||||
})
|
||||
|
||||
// With encoding
|
||||
rpc.respond('add', {
|
||||
requestEncoding: cenc.uint,
|
||||
responseEncoding: cenc.uint
|
||||
}, (a) => {
|
||||
return a + 1
|
||||
})
|
||||
|
||||
// Async handler
|
||||
rpc.respond('fetch', async (req) => {
|
||||
const data = await database.get(req.toString())
|
||||
return data
|
||||
})
|
||||
|
||||
// Remove handler
|
||||
rpc.unrespond('echo')
|
||||
```
|
||||
|
||||
### Client-Side: Making Requests
|
||||
|
||||
```js
|
||||
const rpc = new ProtomuxRPC(stream)
|
||||
|
||||
// Simple request
|
||||
const result = await rpc.request('echo', Buffer.from('hello'))
|
||||
console.log(result.toString()) // 'hello'
|
||||
|
||||
// With encoding
|
||||
const cenc = require('compact-encoding')
|
||||
const sum = await rpc.request('add', 5, {
|
||||
requestEncoding: cenc.uint,
|
||||
responseEncoding: cenc.uint
|
||||
})
|
||||
console.log(sum) // 6
|
||||
|
||||
// With timeout
|
||||
const result = await rpc.request('slow', data, {
|
||||
timeout: 5000 // 5 seconds
|
||||
})
|
||||
|
||||
// Fire-and-forget event
|
||||
rpc.event('log', Buffer.from('message'))
|
||||
```
|
||||
|
||||
### Connection Management
|
||||
|
||||
```js
|
||||
// Check state
|
||||
console.log('Opened:', rpc.opened)
|
||||
console.log('Closed:', rpc.closed)
|
||||
|
||||
// Graceful close
|
||||
await rpc.end()
|
||||
|
||||
// Force close
|
||||
rpc.destroy()
|
||||
|
||||
// With error
|
||||
rpc.destroy(new Error('Connection failed'))
|
||||
```
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Example 1: Calculator Service
|
||||
|
||||
```js
|
||||
const ProtomuxRPC = require('protomux-rpc')
|
||||
const cenc = require('compact-encoding')
|
||||
|
||||
// Request/response encoding
|
||||
const CalcRequest = {
|
||||
preencode(state, m) {
|
||||
cenc.string.preencode(state, m.op)
|
||||
cenc.float.preencode(state, m.a)
|
||||
cenc.float.preencode(state, m.b)
|
||||
},
|
||||
encode(state, m) {
|
||||
cenc.string.encode(state, m.op)
|
||||
cenc.float.encode(state, m.a)
|
||||
cenc.float.encode(state, m.b)
|
||||
},
|
||||
decode(state) {
|
||||
return {
|
||||
op: cenc.string.decode(state),
|
||||
a: cenc.float.decode(state),
|
||||
b: cenc.float.decode(state)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Server
|
||||
function createCalculatorServer(stream) {
|
||||
const rpc = new ProtomuxRPC(stream)
|
||||
|
||||
rpc.respond('calc', {
|
||||
requestEncoding: CalcRequest,
|
||||
responseEncoding: cenc.float
|
||||
}, (req) => {
|
||||
switch (req.op) {
|
||||
case 'add': return req.a + req.b
|
||||
case 'sub': return req.a - req.b
|
||||
case 'mul': return req.a * req.b
|
||||
case 'div':
|
||||
if (req.b === 0) throw new Error('Division by zero')
|
||||
return req.a / req.b
|
||||
default: throw new Error('Unknown operation')
|
||||
}
|
||||
})
|
||||
|
||||
return rpc
|
||||
}
|
||||
|
||||
// Client
|
||||
async function calculatorClient(stream) {
|
||||
const rpc = new ProtomuxRPC(stream)
|
||||
await rpc.fullyOpened()
|
||||
|
||||
return {
|
||||
async calculate(op, a, b) {
|
||||
return rpc.request('calc', { op, a, b }, {
|
||||
requestEncoding: CalcRequest,
|
||||
responseEncoding: cenc.float
|
||||
})
|
||||
},
|
||||
close: () => rpc.end()
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
async function main() {
|
||||
// Create streams (e.g., from hyperswarm)
|
||||
const { stream1, stream2 } = createTestStreams()
|
||||
|
||||
// Server
|
||||
const server = createCalculatorServer(stream1)
|
||||
|
||||
// Client
|
||||
const client = await calculatorClient(stream2)
|
||||
|
||||
// Calculate
|
||||
console.log('2 + 3 =', await client.calculate('add', 2, 3))
|
||||
console.log('10 - 4 =', await client.calculate('sub', 10, 4))
|
||||
console.log('5 * 6 =', await client.calculate('mul', 5, 6))
|
||||
console.log('15 / 3 =', await client.calculate('div', 15, 3))
|
||||
|
||||
// Cleanup
|
||||
await client.close()
|
||||
}
|
||||
```
|
||||
|
||||
### Example 2: Database Proxy
|
||||
|
||||
```js
|
||||
const ProtomuxRPC = require('protomux-rpc')
|
||||
const cenc = require('compact-encoding')
|
||||
|
||||
class DatabaseRPC {
|
||||
constructor(db) {
|
||||
this.db = db
|
||||
this.methods = new Map()
|
||||
this.setupMethods()
|
||||
}
|
||||
|
||||
setupMethods() {
|
||||
// GET method
|
||||
this.methods.set('get', {
|
||||
requestEncoding: cenc.string,
|
||||
responseEncoding: cenc.raw,
|
||||
handler: async (key) => {
|
||||
const value = await this.db.get(key)
|
||||
return value ? Buffer.from(value) : null
|
||||
}
|
||||
})
|
||||
|
||||
// PUT method
|
||||
this.methods.set('put', {
|
||||
requestEncoding: {
|
||||
preencode(state, m) {
|
||||
cenc.string.preencode(state, m.key)
|
||||
cenc.raw.preencode(state, m.value)
|
||||
},
|
||||
encode(state, m) {
|
||||
cenc.string.encode(state, m.key)
|
||||
cenc.raw.encode(state, m.value)
|
||||
},
|
||||
decode(state) {
|
||||
return {
|
||||
key: cenc.string.decode(state),
|
||||
value: cenc.raw.decode(state)
|
||||
}
|
||||
}
|
||||
},
|
||||
responseEncoding: cenc.bool,
|
||||
handler: async ({ key, value }) => {
|
||||
await this.db.put(key, value)
|
||||
return true
|
||||
}
|
||||
})
|
||||
|
||||
// DELETE method
|
||||
this.methods.set('del', {
|
||||
requestEncoding: cenc.string,
|
||||
responseEncoding: cenc.bool,
|
||||
handler: async (key) => {
|
||||
await this.db.del(key)
|
||||
return true
|
||||
}
|
||||
})
|
||||
|
||||
// LIST method
|
||||
this.methods.set('list', {
|
||||
requestEncoding: cenc.string, // prefix
|
||||
responseEncoding: cenc.array(cenc.string),
|
||||
handler: async (prefix) => {
|
||||
const keys = await this.db.list(prefix)
|
||||
return keys
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
attach(stream) {
|
||||
const rpc = new ProtomuxRPC(stream)
|
||||
|
||||
for (const [name, { requestEncoding, responseEncoding, handler }] of this.methods) {
|
||||
rpc.respond(name, { requestEncoding, responseEncoding }, handler)
|
||||
}
|
||||
|
||||
return rpc
|
||||
}
|
||||
|
||||
async connect(stream) {
|
||||
const rpc = new ProtomuxRPC(stream)
|
||||
await rpc.fullyOpened()
|
||||
|
||||
return {
|
||||
get: (key) => rpc.request('get', key, this.methods.get('get')),
|
||||
put: (key, value) => rpc.request('put', { key, value }, this.methods.get('put')),
|
||||
del: (key) => rpc.request('del', key, this.methods.get('del')),
|
||||
list: (prefix) => rpc.request('list', prefix, this.methods.get('list')),
|
||||
close: () => rpc.end()
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
async function main() {
|
||||
const { stream1, stream2 } = createTestStreams()
|
||||
|
||||
// Simulated database
|
||||
const db = new Map()
|
||||
|
||||
// Server
|
||||
const dbRPC = new DatabaseRPC(db)
|
||||
const server = dbRPC.attach(stream1)
|
||||
|
||||
// Client
|
||||
const client = await dbRPC.connect(stream2)
|
||||
|
||||
// Operations
|
||||
await client.put('user:1', Buffer.from('Alice'))
|
||||
await client.put('user:2', Buffer.from('Bob'))
|
||||
|
||||
const user1 = await client.get('user:1')
|
||||
console.log('User 1:', user1.toString())
|
||||
|
||||
const users = await client.list('user:')
|
||||
console.log('Users:', users)
|
||||
|
||||
await client.close()
|
||||
}
|
||||
```
|
||||
|
||||
### Example 3: Event Streaming
|
||||
|
||||
```js
|
||||
const ProtomuxRPC = require('protomux-rpc')
|
||||
|
||||
class EventStreamer {
|
||||
constructor() {
|
||||
this.listeners = new Map()
|
||||
}
|
||||
|
||||
attach(stream) {
|
||||
const rpc = new ProtomuxRPC(stream)
|
||||
|
||||
// Subscribe to events
|
||||
rpc.respond('subscribe', (topic) => {
|
||||
if (!this.listeners.has(topic)) {
|
||||
this.listeners.set(topic, new Set())
|
||||
}
|
||||
this.listeners.get(topic).add(rpc)
|
||||
return { success: true }
|
||||
})
|
||||
|
||||
// Unsubscribe
|
||||
rpc.respond('unsubscribe', (topic) => {
|
||||
const topicListeners = this.listeners.get(topic)
|
||||
if (topicListeners) {
|
||||
topicListeners.delete(rpc)
|
||||
}
|
||||
return { success: true }
|
||||
})
|
||||
|
||||
rpc.on('close', () => {
|
||||
// Cleanup subscriptions
|
||||
for (const [topic, listeners] of this.listeners) {
|
||||
listeners.delete(rpc)
|
||||
}
|
||||
})
|
||||
|
||||
return rpc
|
||||
}
|
||||
|
||||
async emit(topic, event) {
|
||||
const listeners = this.listeners.get(topic)
|
||||
if (!listeners) return
|
||||
|
||||
const promises = []
|
||||
for (const rpc of listeners) {
|
||||
// Use event() for fire-and-forget
|
||||
promises.push(
|
||||
rpc.event('event', Buffer.from(JSON.stringify({ topic, event })))
|
||||
.catch(() => {}) // Ignore errors from disconnected clients
|
||||
)
|
||||
}
|
||||
|
||||
await Promise.all(promises)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
async function main() {
|
||||
const { stream1, stream2 } = createTestStreams()
|
||||
|
||||
const streamer = new EventStreamer()
|
||||
const server = streamer.attach(stream1)
|
||||
|
||||
// Client
|
||||
const client = new ProtomuxRPC(stream2)
|
||||
await client.fullyOpened()
|
||||
|
||||
// Subscribe
|
||||
await client.request('subscribe', Buffer.from('updates'))
|
||||
|
||||
// Listen for events
|
||||
client.respond('event', (data) => {
|
||||
const event = JSON.parse(data.toString())
|
||||
console.log('Received:', event)
|
||||
})
|
||||
|
||||
// Server emits
|
||||
setInterval(() => {
|
||||
streamer.emit('updates', { time: Date.now() })
|
||||
}, 1000)
|
||||
}
|
||||
```
|
||||
|
||||
## Integration with Hyperswarm
|
||||
|
||||
```js
|
||||
const Hyperswarm = require('hyperswarm')
|
||||
const ProtomuxRPC = require('protomux-rpc')
|
||||
const crypto = require('hypercore-crypto')
|
||||
|
||||
const swarm = new Hyperswarm()
|
||||
const topic = crypto.randomBytes(32)
|
||||
|
||||
// Server
|
||||
swarm.on('connection', (conn) => {
|
||||
const rpc = new ProtomuxRPC(conn)
|
||||
|
||||
rpc.respond('greet', (name) => {
|
||||
return Buffer.from(`Hello, ${name.toString()}!`)
|
||||
})
|
||||
})
|
||||
|
||||
swarm.join(topic, { server: true })
|
||||
|
||||
// Client
|
||||
const clientSwarm = new Hyperswarm()
|
||||
|
||||
clientSwarm.on('connection', async (conn) => {
|
||||
const rpc = new ProtomuxRPC(conn)
|
||||
await rpc.fullyOpened()
|
||||
|
||||
const greeting = await rpc.request('greet', Buffer.from('World'))
|
||||
console.log(greeting.toString())
|
||||
|
||||
await rpc.end()
|
||||
clientSwarm.destroy()
|
||||
})
|
||||
|
||||
clientSwarm.join(topic, { client: true })
|
||||
```
|
||||
|
||||
## Error Handling
|
||||
|
||||
```js
|
||||
// Server throws error
|
||||
rpc.respond('risky', () => {
|
||||
throw new Error('Something went wrong')
|
||||
})
|
||||
|
||||
// Client receives error
|
||||
try {
|
||||
await rpc.request('risky', data)
|
||||
} catch (err) {
|
||||
console.error('RPC Error:', err.message)
|
||||
console.error('Error Code:', err.code)
|
||||
}
|
||||
|
||||
// Custom error codes
|
||||
rpc.respond('auth', (token) => {
|
||||
if (!isValid(token)) {
|
||||
const err = new Error('Invalid token')
|
||||
err.code = 'AUTH_FAILED'
|
||||
throw err
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
## Performance Optimization
|
||||
|
||||
```js
|
||||
// Batch requests
|
||||
async function batchRequests(rpc, items) {
|
||||
const promises = items.map(item =>
|
||||
rpc.request('process', item)
|
||||
)
|
||||
return Promise.all(promises)
|
||||
}
|
||||
|
||||
// Cork/uncork for batching
|
||||
rpc.cork()
|
||||
for (let i = 0; i < 100; i++) {
|
||||
rpc.event('log', Buffer.from(`Event ${i}`))
|
||||
}
|
||||
rpc.uncork()
|
||||
|
||||
// Use appropriate encodings
|
||||
const cenc = require('compact-encoding')
|
||||
|
||||
// Bad: JSON strings
|
||||
rpc.request('data', Buffer.from(JSON.stringify(bigObject)))
|
||||
|
||||
// Good: Compact encoding
|
||||
rpc.request('data', bigObject, {
|
||||
requestEncoding: MyEncoding
|
||||
})
|
||||
```
|
||||
|
||||
## Protocol Details
|
||||
|
||||
### Message Types
|
||||
|
||||
```js
|
||||
// Request (type 0)
|
||||
{
|
||||
id: uint, // Request identifier
|
||||
method: string, // Method name
|
||||
value: raw // Payload
|
||||
}
|
||||
|
||||
// Response (type 1)
|
||||
{
|
||||
flags: bitfield(4), // error, code, cause, context
|
||||
id: uint, // Request ID
|
||||
// If error flag set:
|
||||
error: string, // Error message
|
||||
// If code flag set:
|
||||
code: string, // Error code
|
||||
// If cause flag set:
|
||||
cause: string, // Cause message
|
||||
causeCode: string, // Cause code
|
||||
// If context flag set:
|
||||
context: string, // Additional context
|
||||
// If no error:
|
||||
value: raw // Response payload
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
1. **Use typed encodings**: Define schemas with compact-encoding
|
||||
2. **Handle timeouts**: Network is unreliable
|
||||
3. **Graceful degradation**: Handle missing methods
|
||||
4. **Resource cleanup**: Remove handlers on close
|
||||
5. **Error context**: Provide meaningful error messages
|
||||
|
||||
```js
|
||||
// Good practice example
|
||||
class RPCService {
|
||||
constructor() {
|
||||
this.handlers = new Map()
|
||||
}
|
||||
|
||||
register(rpc) {
|
||||
for (const [name, handler] of this.handlers) {
|
||||
rpc.respond(name, handler)
|
||||
}
|
||||
|
||||
rpc.on('close', () => {
|
||||
// Cleanup when connection closes
|
||||
for (const name of this.handlers.keys()) {
|
||||
rpc.unrespond(name)
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
addMethod(name, handler) {
|
||||
this.handlers.set(name, handler)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
|
||||
**Module Type**: Core Networking | **Ecosystem Role**: RPC Foundation | **Used By**: hrpc, hyperdht, hyperswarm
|
||||
Reference in New Issue
Block a user