This commit is contained in:
Raven Scott
2026-02-19 06:28:01 -05:00
parent 57f593b44d
commit 074aa1831f
32 changed files with 5014 additions and 249 deletions
+547 -18
View File
@@ -1,33 +1,562 @@
# Autobase - Multi-Writer Virtual Core
# Autobase - Multiwriter Data Structure
## Deep Dive
## Overview
Autobase aggregates Hypercores from writers into virtual append-only log. CRDT-like for collab.
**Autobase** is a revolutionary multiwriter data structure that combines multiple Hypercore writer feeds into a single, linearized, eventually consistent view. Using the event sourcing pattern, Autobase enables distributed systems to merge inputs from multiple writers while maintaining causal consistency.
**Bootstrap**: load writers (keys), index tx proofs.
## Core Concepts
## Flow
### The Problem
Traditional databases have a single writer (leader). When that writer fails, the system stops or requires complex failover. Autobase solves this by:
- Allowing multiple writers to append to their own feeds
- Linearizing all writes into a consistent order
- Supporting dynamic writer membership
- Providing checkpointing for fast recovery
### Event Sourcing Pattern
```mermaid
graph TD
Writers[W1/W2 Append Tx] --> Proofs[Signed Headers]
Proofs --> VirtualCore[Merge Sorted]
VirtualCore --> Hyperbee[Index/DB]
graph TB
subgraph "Writer A"
A1[Block 0]
A2[Block 1]
A3[Block 2]
end
subgraph "Writer B"
B1[Block 0]
B2[Block 1]
end
subgraph "Writer C"
C1[Block 0]
C2[Block 1]
C3[Block 2]
C4[Block 3]
end
subgraph "Linearized View"
L1[A:0]
L2[B:0]
L3[A:1]
L4[C:0]
L5[B:1]
L6[A:2]
L7[C:1]
L8[C:2]
L9[C:3]
end
A1 --> L1
B1 --> L2
A2 --> L3
C1 --> L4
B2 --> L5
A3 --> L6
C2 --> L7
C3 --> L8
C4 --> L9
style L1 fill:#f9f,stroke:#333
style L2 fill:#bbf,stroke:#333
style L3 fill:#f9f,stroke:#333
style L4 fill:#bfb,stroke:#333
```
**Ex**:
## Architecture
```mermaid
graph TB
subgraph "Autobase Core"
CORE[System Core]
LIN[Linearizer]
VIEW[View]
end
subgraph "Writers"
W1[Writer A Core]
W2[Writer B Core]
W3[Writer C Core]
end
subgraph "Apply Function"
OPEN[Open View]
APPLY[Apply Nodes]
CLOSE[Close View]
end
W1 --> CORE
W2 --> CORE
W3 --> CORE
CORE --> LIN
LIN --> APPLY
APPLY --> VIEW
OPEN --> VIEW
CLOSE --> VIEW
style CORE fill:#f9f,stroke:#333,stroke-width:2px
style LIN fill:#bbf,stroke:#333,stroke-width:2px
```
## Key Features
### 1. Causal Ordering
Writers explicitly reference previous nodes, creating a causal DAG:
```js
const base = new Autobase(store.get('system'))
await base.load([writer1, writer2])
await base.append({op: 'put', path: '/doc'})
const db = base.db // Hyperbee view
// Each node references its dependencies
const node = {
value: { action: 'create_user', user: 'alice' },
causalLinks: [
{ key: writerAKey, length: 5 },
{ key: writerBKey, length: 3 }
]
}
```
**Loaders**: external-pointer for dynamic writers.
### 2. Linearization
**Perf**: Linear scan proofs, sparse.
The linearizer analyzes causal references to produce a total order:
**Drive Use**: Versioned FS oplog.
```
Rules:
1. Nodes never precede nodes they reference
2. Ordering is eventually consistent
3. Forks are resolved deterministically
```
~1200 chars expanded.
### 3. Signed Length Checkpoints
Indexers sign checkpoints where ordering becomes immutable:
```js
// Before signedLength: ordering may change
// After signedLength: ordering is fixed
console.log(base.signedLength) // 150
// Everything before block 150 is permanent
```
## API Reference
### Creating an Autobase
```js
const Corestore = require('corestore')
const Autobase = require('autobase')
const store = new Corestore('./storage')
// Create new autobase
const base = new Autobase(store, null, {
open: (store) => store.get('view'),
apply: async (nodes, view, host) => {
for (const { value } of nodes) {
await view.append(value)
}
}
})
await base.ready()
console.log('Key:', base.key.toString('hex'))
```
### Opening Existing Autobase
```js
// Load existing by key
const base = new Autobase(store, bootstrapKey, {
open,
apply
})
await base.ready()
```
### Append Operations
```js
// Append as writer
await base.append({ action: 'create', id: 1 })
await base.append({ action: 'update', id: 1, data: 'value' })
// Optimistic append (for unacknowledged writers)
await base.append({ action: 'propose', id: 2 }, { optimistic: true })
```
### Update and Sync
```js
// Fetch all available data
await base.update()
// Check view length
console.log('View length:', base.view.length)
// Read from view
for (let i = 0; i < base.view.length; i++) {
const entry = await base.view.get(i)
console.log(i, entry)
}
```
### Writer Management
```js
// Add writer (in apply function)
async function apply(nodes, view, host) {
for (const { value } of nodes) {
if (value.addWriter) {
await host.addWriter(value.addWriter, { indexer: true })
}
if (value.removeWriter) {
await host.removeWriter(value.removeWriter)
}
await view.append(value)
}
}
// Check if indexer
console.log('Is indexer:', base.isIndexer)
// Check if writable
console.log('Is writable:', base.writable)
```
### View Functions
#### Open
```js
function open(store) {
// Create your view structure
return store.get('my-view')
// Or use Hyperbee for key-value
const Hyperbee = require('hyperbee')
return new Hyperbee(store.get('bee'), {
keyEncoding: 'utf-8',
valueEncoding: 'json'
})
}
```
#### Apply
```js
async function apply(nodes, view, host) {
// nodes: Array of { value, from, ... }
// view: Your view structure
// host: AutobaseHostCalls instance
for (const node of nodes) {
const { value } = node
// Handle different operations
switch (value.type) {
case 'add_writer':
await host.addWriter(value.key, { indexer: value.indexer })
break
case 'remove_writer':
if (host.removeable(value.key)) {
await host.removeWriter(value.key)
}
break
case 'interrupt':
host.interrupt('Manual stop requested')
return
default:
await view.append(value)
}
}
}
```
#### Close
```js
async function close(view) {
// Cleanup resources
await view.close()
}
```
## Advanced Features
### Optimistic Appends
Allow non-writers to propose blocks:
```js
const base = new Autobase(store, bootstrapKey, {
optimistic: true,
async apply(nodes, view, host) {
for (const node of nodes) {
// Verify optimistic block
if (!verifyBlock(node.value)) continue
// Acknowledge valid writers
if (isValidWriter(node.value)) {
await host.ackWriter(node.from.key)
}
await view.append(node.value)
}
}
})
// Append optimistically
await base.append({ data: 'proposal' }, { optimistic: true })
```
### Fast Forward
Speed up catching up to quorum:
```js
const base = new Autobase(store, bootstrapKey, {
fastForward: true, // Enable fast forward
// ...
})
// Listen for fast forward events
base.on('fast-forward', (to, from) => {
console.log(`Fast forwarded from ${from} to ${to}`)
})
```
### Auto Acknowledgements
```js
const base = new Autobase(store, bootstrapKey, {
ackInterval: 1000, // Auto-ack every 1 second
// ...
})
// Manual ack
await base.ack()
// Background ack
await base.ack(true)
```
### Encryption
```js
const base = new Autobase(store, bootstrapKey, {
encrypt: true, // Auto-generate encryption key
// OR
encryptionKey: myKey, // Use specific key
encrypted: true // Expect encrypted base
})
```
## Complete Example: Distributed Counter
```js
const Corestore = require('corestore')
const Autobase = require('autobase')
async function createCounter(store, key = null) {
return new Autobase(store, key, {
open(store) {
return store.get('counter-view')
},
async apply(nodes, view, host) {
for (const { value } of nodes) {
switch (value.type) {
case 'increment':
const current = await view.get(0) || { count: 0 }
current.count += value.amount || 1
await view.put(0, current)
break
case 'add_writer':
await host.addWriter(value.key, { indexer: true })
break
}
}
}
})
}
// Usage
const store = new Corestore('./counter-storage')
const counter = await createCounter(store)
await counter.ready()
// Increment
await counter.append({ type: 'increment', amount: 5 })
// Update and read
await counter.update()
const state = await counter.view.get(0)
console.log('Count:', state.count)
```
## Complete Example: Collaborative Todo List
```js
const Corestore = require('corestore')
const Autobase = require('autobase')
const Hyperbee = require('hyperbee')
const cenc = require('compact-encoding')
async function createTodoList(store, key = null) {
return new Autobase(store, key, {
open(store) {
return new Hyperbee(store.get('todos'), {
keyEncoding: cenc.string,
valueEncoding: cenc.json
})
},
async apply(nodes, view, host) {
for (const { value } of nodes) {
switch (value.type) {
case 'add_todo':
await view.put(value.id, {
text: value.text,
done: false,
createdAt: Date.now()
})
break
case 'complete_todo':
const todo = await view.get(value.id)
if (todo) {
todo.done = true
todo.completedAt = Date.now()
await view.put(value.id, todo)
}
break
case 'delete_todo':
await view.del(value.id)
break
case 'add_writer':
await host.addWriter(value.key, { indexer: true })
break
}
}
}
})
}
// Usage
const store = new Corestore('./todo-storage')
const todos = await createTodoList(store)
await todos.ready()
// Add todo
await todos.append({
type: 'add_todo',
id: 'todo-1',
text: 'Learn Autobase'
})
// Complete todo
await todos.append({
type: 'complete_todo',
id: 'todo-1'
})
// List all
await todos.update()
const stream = todos.view.createReadStream()
for await (const { key, value } of stream) {
console.log(`${value.done ? '[x]' : '[ ]'} ${value.text}`)
}
```
## Replication
```js
const Hyperswarm = require('hyperswarm')
const swarm = new Hyperswarm()
swarm.on('connection', (conn) => store.replicate(conn))
// Join discovery topic
swarm.join(base.discoveryKey)
// Or use base.replicate()
const stream = base.replicate(true) // isInitiator
stream.pipe(otherStream).pipe(stream)
```
## Events
```js
// View updated
base.on('update', () => {
console.log('View updated, new length:', base.view.length)
})
// Became indexer
base.on('is-indexer', () => {
console.log('Now an indexer!')
})
// Became writer
base.on('writable', () => {
console.log('Now a writer!')
})
// Fast forward occurred
base.on('fast-forward', (to, from) => {
console.log(`Fast forwarded: ${from} -> ${to}`)
})
// Interrupted
base.on('interrupt', (reason) => {
console.log('Interrupted:', reason)
})
// Warning
base.on('warning', (warning) => {
console.warn('Warning:', warning)
})
// Error
base.on('error', (err) => {
console.error('Error:', err)
})
```
## Best Practices
1. **Deterministic Apply**: Use only view argument for state, no external variables
2. **Idempotent Operations**: Apply should handle duplicate nodes gracefully
3. **Validation**: Always validate values in apply before processing
4. **Interruptions**: Use host.interrupt() for breaking changes
5. **Checkpoints**: Monitor signedLength for durability guarantees
```js
async function apply(nodes, view, host) {
for (const node of nodes) {
// Always validate
if (!isValid(node.value)) {
console.warn('Invalid node:', node)
continue
}
// Deterministic: only use view and node
await view.append(transform(node.value))
}
}
```
## License
Apache-2.0
---
**Module Type**: Core Data Structure | **Ecosystem Role**: Multi-writer Consensus | **Dependencies**: hypercore, corestore
+298
View File
@@ -0,0 +1,298 @@
# B4A - Buffer for Array
## Overview
**B4A** (Buffer for Array) is a critical compatibility layer that bridges Node.js `Buffer` and JavaScript `Uint8Array` classes. This module is foundational to the entire Holepunch ecosystem, enabling seamless binary data operations across Node.js, browsers, React Native, and Bare runtime environments.
## Core Purpose
In the Hypercore ecosystem, binary data flows through multiple environments:
- **Node.js**: Uses `Buffer` (subclass of `Uint8Array`)
- **Browsers**: Only have `Uint8Array`
- **React Native**: Limited native buffer support
- **Bare**: Custom runtime needing cross-platform compatibility
B4A abstracts these differences, allowing the same code to run everywhere without environment checks.
## Architecture
```mermaid
graph TB
subgraph "B4A Compatibility Layer"
B4A[b4a module]
API[Unified API]
end
subgraph "Runtime Environments"
NODE[Node.js Buffer]
BROWSER[Browser Uint8Array]
RN[React Native<br/>react-native-b4a]
BARE[Bare Runtime]
end
B4A --> API
API --> NODE
API --> BROWSER
API --> RN
API --> BARE
style B4A fill:#f9f,stroke:#333,stroke-width:2px
```
## API Reference
### Core Buffer Operations
#### `b4a.alloc(size[, fill[, encoding]])`
Allocate a new buffer of specified size. Safe initialization (zero-filled).
```js
const buf = b4a.alloc(1024) // 1KB zero-filled buffer
const filled = b4a.alloc(256, 0xFF) // Filled with 0xFF
const string = b4a.alloc(100, 'abc') // Filled with 'abc' repeated
```
#### `b4a.allocUnsafe(size)`
Fast allocation without zero-filling. Use with caution.
```js
// Fast but potentially contains old memory data
const fast = b4a.allocUnsafe(8192)
```
#### `b4a.from(array|string|buffer|arrayBuffer)`
Create buffer from various sources.
```js
// From string
const fromString = b4a.from('Hello', 'utf8')
// From array
const fromArray = b4a.from([0x48, 0x65, 0x6C, 0x6C, 0x6F])
// From ArrayBuffer
const ab = new ArrayBuffer(8)
const fromAB = b4a.from(ab)
// From another buffer
const source = b4a.from('source')
const copy = b4a.from(source)
```
### Data Operations
#### `b4a.concat(buffers[, totalLength])`
Concatenate multiple buffers efficiently.
```js
const parts = [
b4a.from('Hello '),
b4a.from('World'),
b4a.from('!')
]
const combined = b4a.concat(parts)
console.log(combined.toString()) // 'Hello World!'
```
#### `b4a.copy(source, target[, targetStart[, sourceStart[, sourceEnd]]])`
Copy data between buffers with precise control.
```js
const source = b4a.from('Hello World')
const target = b4a.alloc(5)
b4a.copy(source, target, 0, 6, 11) // Copy 'World'
console.log(target.toString()) // 'World'
```
#### `b4a.compare(buf1, buf2)` & `b4a.equals(buf1, buf2)`
Buffer comparison operations.
```js
const a = b4a.from('abc')
const b = b4a.from('abc')
const c = b4a.from('def')
console.log(b4a.equals(a, b)) // true
console.log(b4a.equals(a, c)) // false
console.log(b4a.compare(a, c)) // -1 (a < c)
```
### String Operations
#### `b4a.toString(buffer[, encoding[, start[, end]]])`
Convert buffer to string with encoding support.
```js
const buf = b4a.from('48656c6c6f', 'hex')
console.log(b4a.toString(buf, 'utf8')) // 'Hello'
// Partial conversion
const partial = b4a.toString(buf, 'utf8', 0, 2) // 'He'
```
#### `b4a.byteLength(string[, encoding])`
Get byte length without creating buffer.
```js
console.log(b4a.byteLength('Hello', 'utf8')) // 5
console.log(b4a.byteLength('Hello', 'hex')) // 2 (interpreted as hex)
```
### Search Operations
```js
const buf = b4a.from('Hello World Hello')
// Find index
console.log(b4a.indexOf(buf, 'World')) // 6
console.log(b4a.lastIndexOf(buf, 'Hello')) // 12
// Check inclusion
console.log(b4a.includes(buf, 'World')) // true
```
### Type Checking
#### `b4a.isBuffer(value)`
Returns `true` for both Buffer and Uint8Array - key for cross-platform compatibility.
```js
const buf = b4a.from('test')
console.log(b4a.isBuffer(buf)) // true
console.log(b4a.isBuffer(new Uint8Array(10))) // true
console.log(b4a.isBuffer('string')) // false
```
#### `b4a.isEncoding(encoding)`
Check if encoding is supported.
```js
console.log(b4a.isEncoding('utf8')) // true
console.log(b4a.isEncoding('base64')) // true
console.log(b4a.isEncoding('invalid')) // false
```
### Binary Data Operations
#### Read/Write Methods
```js
const buf = b4a.alloc(16)
// Write integers (Little Endian)
b4a.writeUInt32LE(buf, 0x12345678, 0)
b4a.writeInt32LE(buf, -1000, 4)
b4a.writeFloatLE(buf, 3.14159, 8)
b4a.writeDoubleLE(buf, Math.PI, 12)
// Read integers
console.log(b4a.readUInt32LE(buf, 0)) // 305419896
console.log(b4a.readInt32LE(buf, 4)) // -1000
console.log(b4a.readFloatLE(buf, 8)) // ~3.14159
console.log(b4a.readDoubleLE(buf, 12)) // 3.141592653589793
```
#### Byte Swapping
```js
const buf = b4a.from([0x01, 0x02, 0x03, 0x04, 0x05, 0x06, 0x07, 0x08])
b4a.swap16(buf) // Swap every 2 bytes
console.log(buf) // [0x02, 0x01, 0x04, 0x03, 0x06, 0x05, 0x08, 0x07]
b4a.swap32(buf) // Swap every 4 bytes
b4a.swap64(buf) // Swap every 8 bytes
```
### React Native Optimization
When `react-native-b4a` is installed, B4A automatically uses optimized native implementations:
```bash
npm install react-native-b4a
```
This provides significant performance improvements for buffer operations on mobile devices.
## Common Patterns in Hypercore Ecosystem
### Pattern 1: Data Encoding/Decoding
```js
const b4a = require('b4a')
const cenc = require('compact-encoding')
// Encode message
const message = { type: 'hello', data: b4a.from('payload') }
const encoded = cenc.encode(cenc.json, message)
// Decode message
const decoded = cenc.decode(cenc.json, encoded)
```
### Pattern 2: Hash Operations
```js
const crypto = require('hypercore-crypto')
const b4a = require('b4a')
// Create key pair
const keyPair = crypto.keyPair()
// Sign data
const data = b4a.from('message to sign')
const signature = crypto.sign(data, keyPair.secretKey)
// Verify
const valid = crypto.verify(data, signature, keyPair.publicKey)
```
### Pattern 3: Streaming Data
```js
const b4a = require('b4a')
// Accumulate chunks
const chunks = []
stream.on('data', chunk => {
chunks.push(chunk)
})
stream.on('end', () => {
const full = b4a.concat(chunks)
console.log('Total bytes:', full.length)
})
```
## Platform-Specific Behavior
| Operation | Node.js | Browser | React Native |
|-----------|---------|---------|--------------|
| `b4a.alloc()` | `Buffer.alloc()` | `new Uint8Array()` | Optimized native |
| `b4a.from(string)` | `Buffer.from()` | TextEncoder | Optimized native |
| `toString()` | `buf.toString()` | TextDecoder | Optimized native |
| Type returned | `Buffer` | `Uint8Array` | `Uint8Array` |
## Integration with Other Modules
B4A is used throughout the ecosystem:
- **hypercore**: All block data handling
- **hyperdht**: Key and signature operations
- **compact-encoding**: Binary encoding foundation
- **sodium-native**: Crypto buffer operations
- **protomux**: Message framing
## Performance Considerations
1. **Use `allocUnsafe` sparingly**: Only when performance is critical and you immediately overwrite all bytes
2. **Pre-allocate when possible**: Avoid repeated allocations in loops
3. **Use `concat` with totalLength**: Providing totalLength avoids intermediate allocations
4. **React Native**: Install `react-native-b4a` for 10-100x performance improvement
## License
Apache 2.0
---
**Module Type**: Core Utility | **Ecosystem Role**: Cross-platform foundation | **Dependencies**: None
+495
View File
@@ -0,0 +1,495 @@
# bare-bundle - Application Bundle Format
## Overview
**bare-bundle** is the application packaging format for the Bare JavaScript runtime, inspired by Electron's ASAR format. It enables efficient distribution of JavaScript applications by bundling all files into a single archive with fast random access and a structured header for metadata.
## Architecture
```mermaid
graph TB
subgraph "Bundle Structure"
HEADER[Header Section]
FILES[File Data Section]
end
subgraph "Header Contents"
VERSION[Version]
ID[Bundle ID]
MAIN[Main Entry]
IMPORTS[Import Maps]
ADDONS[Addon List]
ASSETS[Asset List]
FILEMAP[File Map<br/>offset + length]
end
subgraph "Usage Flow"
CREATE[Create Bundle]
DISTRIBUTE[Distribute]
LOAD[Load at Runtime]
EXECUTE[Execute]
end
HEADER --> VERSION
HEADER --> ID
HEADER --> MAIN
HEADER --> IMPORTS
HEADER --> ADDONS
HEADER --> ASSETS
HEADER --> FILEMAP
CREATE --> HEADER
CREATE --> FILES
HEADER --> DISTRIBUTE
FILES --> DISTRIBUTE
DISTRIBUTE --> LOAD
LOAD --> EXECUTE
style HEADER fill:#f9f,stroke:#333,stroke-width:2px
style FILEMAP fill:#bbf,stroke:#333,stroke-width:2px
```
## Bundle Format Specification
### File Layout
```
[#!hashbang - optional]
<header length as uint32>
<header JSON as string>
<file 1 data>
<file 2 data>
...
<file n data>
```
### Header Structure
```js
{
"version": 0, // Bundle format version
"id": "unique-id", // Optional bundle identifier
"main": "file:///main.js", // Entry point URL
"imports": { // Import mappings
"from-module": "to-module"
},
"resolutions": { // Pre-resolved imports
"file:///module.js": { "#internal": "file:///internal.js" }
},
"addons": [ // Native addon paths
"file:///native/addon.node"
],
"assets": [ // Static asset paths
"file:///assets/logo.png"
],
"files": { // File offset map
"file:///main.js": {
"offset": 0, // Byte offset from end of header
"length": 1024, // File size in bytes
"mode": 0o644 // Unix file permissions
}
}
}
```
## API Reference
### Creating Bundles
```js
const Bundle = require('bare-bundle')
// Create new bundle
const bundle = new Bundle()
// Add files
bundle.write('/index.js', Buffer.from('console.log("Hello")'))
bundle.write('/lib/utils.js', Buffer.from('module.exports = {}'))
bundle.write('/package.json', Buffer.from(JSON.stringify({
"name": "my-app",
"main": "./index.js"
})))
// Set entry point
bundle.main = new URL('file:///index.js')
// Set bundle ID
bundle.id = 'my-app-v1.0.0'
// Add import mappings
bundle.imports = {
"#config": "file:///config.js"
}
// Serialize to buffer
const buffer = bundle.toBuffer()
// Save to file
require('bare-fs').writeFileSync('app.bundle', buffer)
```
### Loading Bundles
```js
const Bundle = require('bare-bundle')
const fs = require('bare-fs')
// Load from buffer
const buffer = fs.readFileSync('app.bundle')
const bundle = new Bundle(buffer)
// Access bundle metadata
console.log(bundle.version) // 0
console.log(bundle.id) // 'my-app-v1.0.0'
console.log(bundle.main) // URL { ... }
// Read files
const mainSource = bundle.read(bundle.main)
console.log(mainSource.toString()) // 'console.log("Hello")'
// List all files
for (const [url, info] of bundle.files) {
console.log(url, info.offset, info.length, info.mode)
}
```
### Streaming Bundles
```js
const Bundle = require('bare-bundle')
// Create bundle from stream
const bundle = new Bundle()
// Stream files in
bundle.write('/large-file.js', largeBuffer)
// Pipe to output
const stream = bundle.createStream()
stream.pipe(fs.createWriteStream('output.bundle'))
```
## Bundle File Modes
File permissions are stored as Unix-style mode flags:
```js
// Regular file
{ mode: 0o644 } // rw-r--r--
// Executable
{ mode: 0o755 } // rwxr-xr-x
// Read-only
{ mode: 0o444 } // r--r--r--
```
## Practical Examples
### Example 1: Simple Application Bundle
```js
const Bundle = require('bare-bundle')
function createAppBundle() {
const bundle = new Bundle()
// Main entry
bundle.write('/index.js', Buffer.from(`
const { greet } = require('./lib/greetings')
greet('World')
`))
// Library file
bundle.write('/lib/greetings.js', Buffer.from(`
exports.greet = function(name) {
console.log('Hello, ' + name + '!')
}
`))
// Package metadata
bundle.write('/package.json', Buffer.from(JSON.stringify({
name: 'greeting-app',
version: '1.0.0',
main: './index.js'
})))
// Set main entry
bundle.main = new URL('file:///index.js')
return bundle.toBuffer()
}
```
### Example 2: Bundle with Native Addons
```js
const Bundle = require('bare-bundle')
function createNativeBundle() {
const bundle = new Bundle()
// JavaScript code
bundle.write('/index.js', Buffer.from(`
const addon = require.addon()
addon.doSomething()
`))
// Native addon for different platforms
bundle.write('/prebuilds/darwin-arm64/addon.node', darwinArm64Buffer)
bundle.write('/prebuilds/linux-x64/addon.node', linuxX64Buffer)
bundle.write('/prebuilds/win32-x64/addon.node', win32X64Buffer)
// Track addons in header
bundle.addons = [
new URL('file:///prebuilds/darwin-arm64/addon.node'),
new URL('file:///prebuilds/linux-x64/addon.node'),
new URL('file:///prebuilds/win32-x64/addon.node')
]
bundle.main = new URL('file:///index.js')
return bundle.toBuffer()
}
```
### Example 3: Bundle with Import Maps
```js
const Bundle = require('bare-bundle')
function createMappedBundle() {
const bundle = new Bundle()
// Application code
bundle.write('/src/index.js', Buffer.from(`
import { helper } from '#utils'
import config from '#config'
helper(config.apiUrl)
`))
bundle.write('/src/utils.js', Buffer.from(`
export const helper = (url) => console.log(url)
`))
bundle.write('/config/prod.js', Buffer.from(`
export default { apiUrl: 'https://api.example.com' }
`))
// Define import mappings
bundle.imports = {
"#utils": "file:///src/utils.js",
"#config": "file:///config/prod.js"
}
bundle.main = new URL('file:///src/index.js')
return bundle.toBuffer()
}
```
### Example 4: Bundle with Pre-resolved Imports
```js
const Bundle = require('bare-bundle')
function createPreResolvedBundle() {
const bundle = new Bundle()
// Main module
bundle.write('/index.js', Buffer.from(`
import foo from 'dependency'
foo()
`))
// Pre-resolve imports to avoid runtime resolution
bundle.resolutions = {
"file:///index.js": {
"dependency": "file:///node_modules/dependency/index.js"
}
}
bundle.main = new URL('file:///index.js')
return bundle.toBuffer()
}
```
## Integration with Bare Runtime
### Loading and Executing
```js
const Bundle = require('bare-bundle')
const Module = require('bare-module')
function runBundle(bundleBuffer) {
// Parse bundle
const bundle = new Bundle(bundleBuffer)
// Create custom protocol for bundle
const bundleProtocol = new Module.Protocol({
*resolve(specifier, parentURL) {
// Handle bundle URLs
if (specifier.startsWith('bundle:')) {
yield new URL(specifier)
}
},
read(url) {
// Read from bundle
const path = url.pathname
return bundle.read(new URL('file://' + path))
},
exists(url) {
const path = url.pathname
return bundle.files.has('file://' + path)
}
})
// Load main module
const main = Module.load(bundle.main, bundle.read(bundle.main), {
protocol: bundleProtocol
})
return main.exports
}
```
### With bare-module
```js
const Module = require('bare-module')
const Bundle = require('bare-bundle')
// Bundle type is recognized by bare-module
const bundle = new Bundle(bundleBuffer)
// Load as bundle type
const module = Module.load(bundle.main, bundle.read(bundle.main), {
type: Module.constants.types.BUNDLE
})
```
## Performance Characteristics
| Operation | Time Complexity | Notes |
|-----------|----------------|-------|
| Load bundle | O(1) | Header parsed once |
| Read file | O(1) | Direct offset lookup |
| List files | O(n) | Iterate file map |
| Create bundle | O(n) | Linear file addition |
| Serialize | O(n) | Linear buffer construction |
### Memory Usage
- Header: ~100 bytes + file map
- File map: ~50 bytes per file entry
- Data: Original file sizes
- Total overhead: ~5-10%
## Bundle Version History
- **Version 0**: Initial format
- JSON header
- File offset map
- Import maps support
- Resolution maps support
## Security Considerations
1. **Path Traversal**: Bundles should validate paths to prevent `../../../etc/passwd` style attacks
2. **Executable Permissions**: Respect mode flags when extracting
3. **Integrity**: Consider signing bundles for verification
```js
// Validate paths
function validatePath(path) {
// Ensure path doesn't escape bundle root
const resolved = require('bare-path').resolve('/', path)
if (!resolved.startsWith('/')) {
throw new Error('Invalid path: ' + path)
}
return resolved
}
```
## Tools and Utilities
### Bundle Inspector
```js
function inspectBundle(buffer) {
const bundle = new Bundle(buffer)
console.log('Bundle ID:', bundle.id)
console.log('Version:', bundle.version)
console.log('Main:', bundle.main)
console.log('Files:')
for (const [url, info] of bundle.files) {
console.log(' ', url)
console.log(' offset:', info.offset)
console.log(' length:', info.length)
console.log(' mode:', info.mode.toString(8))
}
console.log('Addons:', bundle.addons)
console.log('Assets:', bundle.assets)
console.log('Imports:', bundle.imports)
}
```
### Bundle Diff
```js
function diffBundles(oldBuf, newBuf) {
const old = new Bundle(oldBuf)
const new_ = new Bundle(newBuf)
const changes = {
added: [],
removed: [],
modified: []
}
// Find added/modified
for (const [url, newInfo] of new_.files) {
if (!old.files.has(url)) {
changes.added.push(url)
} else {
const oldInfo = old.files.get(url)
if (oldInfo.length !== newInfo.length) {
changes.modified.push(url)
}
}
}
// Find removed
for (const [url] of old.files) {
if (!new_.files.has(url)) {
changes.removed.push(url)
}
}
return changes
}
```
## Best Practices
1. **Minimize Bundle Size**: Exclude unnecessary files (tests, docs, dev deps)
2. **Use Import Maps**: Simplify module resolution
3. **Pre-resolve Imports**: Speed up runtime loading
4. **Version Your Bundles**: Use the `id` field for tracking
5. **Compress Large Assets**: Consider pre-compressing large static files
## License
Apache-2.0
---
**Module Type**: Core Infrastructure | **Ecosystem Role**: Application Packaging | **Used By**: Bare runtime, Pear CLI
+518
View File
@@ -0,0 +1,518 @@
# bare-module - JavaScript Module System for Bare
## Overview
**bare-module** is the foundational module system that powers the Bare JavaScript runtime. It provides comprehensive support for both CommonJS (`require()`) and ES Modules (`import`), along with advanced features like conditional exports, import maps, and addon resolution. This is the core infrastructure that makes Bare a viable Node.js alternative for embedded and mobile applications.
## Architecture
```mermaid
graph TB
subgraph "Module System"
RESOLVER[Module Resolution]
LOADER[Module Loader]
CACHE[Module Cache]
PROTOCOL[Protocol Layer]
end
subgraph "Module Types"
CJS[CommonJS<br/>.js/.cjs]
ESM[ES Modules<br/>.mjs/import]
JSON[JSON Files]
ADDON[Native Addons]
BUNDLE[bare-bundle]
end
subgraph "Package.json Features"
EXPORTS[Conditional Exports]
IMPORTS[Import Maps]
TYPE[Module Type]
ENGINES[Engine Requirements]
end
RESOLVER --> LOADER
LOADER --> CACHE
PROTOCOL --> RESOLVER
LOADER --> CJS
LOADER --> ESM
LOADER --> JSON
LOADER --> ADDON
LOADER --> BUNDLE
RESOLVER --> EXPORTS
RESOLVER --> IMPORTS
RESOLVER --> TYPE
RESOLVER --> ENGINES
style RESOLVER fill:#f9f,stroke:#333,stroke-width:2px
style LOADER fill:#bbf,stroke:#333,stroke-width:2px
```
## Module Resolution Algorithm
Bare-module implements a sophisticated resolution algorithm supporting:
### 1. Standard Resolution
```js
const Module = require('bare-module')
// Resolve relative paths
const url = Module.resolve('./lib/utils', parentURL)
// Resolve package names
const lodashURL = Module.resolve('lodash', parentURL)
// Resolve with conditions
const url = Module.resolve('pkg', parentURL, {
conditions: ['import', 'bare']
})
```
### 2. Conditional Exports
Packages can provide different entry points based on context:
```json
{
"name": "my-package",
"exports": {
".": {
"import": "./index.mjs",
"require": "./index.cjs",
"bare": "./bare.js",
"node": "./node.js",
"default": "./fallback.js"
},
"./submodule": "./lib/submodule.js"
}
}
```
**Condition Priority** (most specific first):
1. `import` / `require` - Loading method
2. `asset` - Asset loading
3. `addon` - Native addon loading
4. `bare` / `node` - Runtime environment
5. `<platform>` / `<arch>` - Platform-specific (iOS, Android, etc.)
6. `simulator` - Simulator builds
7. `default` - Fallback (always last)
### 3. Import Maps
Private package-level import mappings:
```json
{
"imports": {
"#config": {
"bare": "./config.bare.js",
"node": "./config.node.js",
"default": "./config.js"
},
"#utils": "./lib/utils.js"
}
}
```
Usage in code:
```js
import config from '#config'
import { helper } from '#utils'
```
## CommonJS Module System
### Global Objects
When loading CommonJS modules, these globals are available:
```js
// __dirname - Directory of current module
console.log(__dirname) // file:///path/to/module/
// __filename - Full path to current module
console.log(__filename) // file:///path/to/module/file.js
// require - Module loader
const utils = require('./utils')
// module - Current module reference
module.exports = { foo: 'bar' }
// exports - Shortcut for module.exports
exports.baz = 123
```
### require() API
#### Basic Usage
```js
// Load core modules
const fs = require('bare-fs')
const path = require('bare-path')
// Load local modules
const local = require('./local-module')
// Load npm packages
const lodash = require('lodash')
```
#### require.resolve()
```js
// Get resolved path without loading
const resolved = require.resolve('some-package')
console.log(resolved) // file:///path/to/node_modules/some-package/index.js
```
#### require.addon()
Load native addons with automatic platform resolution:
```js
// Load addon for current platform
const native = require.addon()
// Load specific addon
const specific = require.addon('./native', parentURL)
// Get addon host string (platform-arch)
console.log(require.addon.host) // 'darwin-arm64', 'linux-x64', etc.
// Resolve addon path
const addonPath = require.addon.resolve('./my-addon')
```
#### require.asset()
Load static assets:
```js
const fs = require('bare-fs')
// Resolve asset path
const assetPath = require.asset('./config.json')
// Read asset
const config = JSON.parse(fs.readFileSync(assetPath))
```
#### require.main
Access the entry module:
```js
if (require.main === module) {
console.log('This is the main module')
}
```
#### require.cache
Access the module cache:
```js
// Clear specific module from cache
delete require.cache[require.resolve('./module')]
// Clear entire cache (use with caution)
Object.keys(require.cache).forEach(key => {
delete require.cache[key]
})
```
## ES Module System
### Import Syntax
```js
// Default import
import fs from 'bare-fs'
// Named imports
import { readFile, writeFile } from 'bare-fs'
// Namespace import
import * as path from 'bare-path'
// Dynamic import
const module = await import('./dynamic-module.js')
// Import with attributes
import json from './config.json' with { type: 'json' }
```
### import.meta
```js
// Current module URL
console.log(import.meta.url) // file:///path/to/current.js
// Check if main module
if (import.meta.main) {
console.log('Entry point')
}
// Module cache
console.log(import.meta.cache)
// Resolve specifiers
const resolved = import.meta.resolve('./other.js')
// Addon loading
const native = import.meta.addon()
// Asset loading
const asset = import.meta.asset('./data.txt')
```
## Module Types
The system supports multiple module types:
```js
const Module = require('bare-module')
console.log(Module.constants.types)
// {
// SCRIPT: 0, // CommonJS
// MODULE: 1, // ES Module
// JSON: 2, // JSON file
// BUNDLE: 3, // bare-bundle
// ADDON: 4, // Native addon
// BINARY: 5, // Binary file
// TEXT: 6 // Text file
// }
```
## Custom Protocols
Define custom module loading protocols:
```js
const Module = require('bare-module')
const customProtocol = new Module.Protocol({
// Preprocess specifier before resolution
preresolve(specifier, parentURL) {
if (specifier.startsWith('custom:')) {
return specifier.replace('custom:', '')
}
return specifier
},
// Post-process resolved URL
postresolve(url) {
return new URL(url)
},
// Resolve specifier to URL
*resolve(specifier, parentURL, imports) {
if (specifier.startsWith('hyper:')) {
yield new URL(specifier)
}
},
// Check if URL exists
exists(url) {
return checkExists(url)
},
// Read module source
read(url) {
return readFile(url)
},
// Handle addon URLs
addon(url) {
return new URL(url.pathname + '.node')
},
// Handle asset URLs
asset(url) {
return url
}
})
// Use custom protocol
const module = Module.load(url, source, {
protocol: customProtocol
})
```
## Package.json Configuration
### Type Field
Controls how `.js` files are interpreted:
```json
{
"type": "module" // .js files = ES modules
}
```
Without this field, `.js` defaults to CommonJS.
### Engines Field
Specify runtime requirements:
```json
{
"engines": {
"bare": ">=1.0.5",
"node": ">=18.0.0"
}
}
```
Resolution fails if requirements aren't met.
### Exports Sugar
Single export shorthand:
```json
{
"exports": "./index.js"
}
```
Equivalent to:
```json
{
"exports": {
".": "./index.js"
}
}
```
## Creating Custom require()
Useful for REPLs and isolated contexts:
```js
const Module = require('bare-module')
// Create isolated require
const customRequire = Module.createRequire(
new URL('file:///virtual/path/'),
{
type: Module.constants.types.SCRIPT,
conditions: ['bare', 'import'],
builtins: {
'custom:fs': customFsImplementation
}
}
)
// Use it
const module = customRequire('./some-module')
```
## Module States
```js
const Module = require('bare-module')
console.log(Module.constants.states)
// {
// EVALUATED: 1, // Module has run
// SYNTHESIZED: 2 // Named exports detected
// }
```
## Global Cache
```js
const Module = require('bare-module')
// Access global module cache
console.log(Module.cache)
// WARNING: Cache may contain modules from different bare-module versions
```
## Error Handling
Common resolution errors:
```js
try {
const module = require('non-existent-module')
} catch (err) {
if (err.code === 'MODULE_NOT_FOUND') {
console.log('Module not found')
}
}
```
## Integration Examples
### With Hyperdrive
```js
const Module = require('bare-module')
const Hyperdrive = require('hyperdrive')
// Custom protocol for hyperdrive
const hyperProtocol = new Module.Protocol({
async *resolve(specifier, parentURL) {
if (specifier.startsWith('hyper:')) {
const key = specifier.slice(6)
yield new URL(`hyper://${key}`)
}
},
async read(url) {
const drive = new Hyperdrive(url.hostname)
await drive.ready()
return drive.get(url.pathname)
}
})
```
### With bare-bundle
```js
const Module = require('bare-module')
const Bundle = require('bare-bundle')
// Load from bundle
const bundle = new Bundle(buffer)
const module = Module.load(bundle.main, bundle.read(bundle.main), {
type: Module.constants.types.BUNDLE
})
```
## Performance Tips
1. **Use `resolutions` map**: Pre-resolve imports to avoid repeated resolution
2. **Cache warmup**: Pre-load commonly used modules
3. **Conditional exports**: Minimize condition checks in hot paths
4. **Lazy loading**: Use dynamic `import()` for optional dependencies
## Debugging
```js
// Trace module resolution
const originalResolve = Module.resolve
Module.resolve = function(...args) {
console.log('Resolving:', args[0])
return originalResolve.apply(this, args)
}
```
## License
Apache-2.0
---
**Module Type**: Core Infrastructure | **Ecosystem Role**: Runtime Foundation | **Used By**: All Bare applications
+562
View File
@@ -0,0 +1,562 @@
# bare-process - Process Control for Bare
## Overview
**bare-process** provides Node.js-compatible process control APIs for the Bare JavaScript runtime. This module bridges the gap between Bare's lightweight architecture and the process management features developers expect from Node.js, enabling seamless portability of existing applications.
## Architecture
```mermaid
graph TB
subgraph "Process Control"
EXIT[Process Exit]
ENV[Environment Variables]
ARGV[Command Line Args]
STDIO[Standard I/O]
PID[Process ID]
CWD[Working Directory]
end
subgraph "Bare Runtime"
CORE[Core Runtime]
GLOBAL[Global Scope]
end
subgraph "Platform Abstraction"
POSIX[POSIX Systems]
WIN[Windows]
MOBILE[iOS/Android]
end
EXIT --> CORE
ENV --> CORE
ARGV --> CORE
STDIO --> CORE
PID --> CORE
CWD --> CORE
CORE --> POSIX
CORE --> WIN
CORE --> MOBILE
GLOBAL --> EXIT
GLOBAL --> ENV
GLOBAL --> ARGV
style CORE fill:#f9f,stroke:#333,stroke-width:2px
style STDIO fill:#bbf,stroke:#333,stroke-width:2px
```
## Core API
### Process Exit
Control application lifecycle with exit codes:
```js
const process = require('bare-process')
// Exit with success (code 0)
process.exit()
// Exit with specific code
process.exit(1) // General error
process.exit(0) // Success
// Exit codes following Unix conventions
// 0 - Success
// 1 - General error
// 2 - Misuse of shell builtins
// 126 - Command invoked cannot execute
// 127 - Command not found
// 128+n - Fatal error signal "n"
// 130 - Script terminated by Ctrl-C
// 255 - Exit status out of range
```
### Environment Variables
Access and manipulate environment:
```js
const process = require('bare-process')
// Read environment variables
const home = process.env.HOME
const path = process.env.PATH
// Set environment variable
process.env.MY_VAR = 'my-value'
// Check if variable exists
if ('NODE_ENV' in process.env) {
console.log('Environment:', process.env.NODE_ENV)
}
// Iterate all environment variables
for (const [key, value] of Object.entries(process.env)) {
console.log(`${key}=${value}`)
}
// Delete environment variable
delete process.env.TEMP_VAR
```
### Command Line Arguments
Access command-line arguments:
```js
const process = require('bare-process')
// argv[0] - Bare executable path
// argv[1] - Script being run
// argv[2+] - User arguments
console.log(process.argv)
// [ '/path/to/bare', '/path/to/script.js', '--flag', 'value' ]
// Parse arguments
const args = process.argv.slice(2)
args.forEach((arg, index) => {
console.log(`Arg ${index}: ${arg}`)
})
// Example: Parse --key=value pairs
const parsed = {}
process.argv.slice(2).forEach(arg => {
if (arg.startsWith('--')) {
const [key, value] = arg.slice(2).split('=')
parsed[key] = value || true
}
})
console.log(parsed) // { flag: 'value' }
```
### Process Identification
```js
const process = require('bare-process')
// Get process ID
console.log('PID:', process.pid)
// Get parent process ID (if available)
console.log('PPID:', process.ppid)
// Platform information
console.log('Platform:', process.platform) // 'darwin', 'linux', 'win32', 'android', 'ios'
console.log('Architecture:', process.arch) // 'arm64', 'x64', etc.
```
### Working Directory
Manage current working directory:
```js
const process = require('bare-process')
// Get current working directory
const cwd = process.cwd()
console.log('Current directory:', cwd)
// Change working directory
process.chdir('/new/path')
console.log('New directory:', process.cwd())
// Common pattern: Save and restore
try {
const originalCwd = process.cwd()
process.chdir('/tmp')
// Do work in /tmp
process.chdir(originalCwd)
} catch (err) {
console.error('Failed to change directory:', err)
}
```
## Global Installation
Make `process` available globally like Node.js:
```js
// At entry point of application
require('bare-process/global')
// Now process is global
global.process.exit()
// or simply
process.exit()
```
## Standard I/O
While bare-process focuses on process control, it works alongside bare-stdio for I/O:
```js
const process = require('bare-process')
// These are available when bare-stdio is loaded
console.log(process.stdin) // Standard input stream
console.log(process.stdout) // Standard output stream
console.log(process.stderr) // Standard error stream
```
## Advanced Usage
### Signal Handling
```js
const process = require('bare-process')
// Handle process signals
process.on('SIGINT', () => {
console.log('Received SIGINT, cleaning up...')
// Cleanup code here
process.exit(0)
})
process.on('SIGTERM', () => {
console.log('Received SIGTERM, shutting down...')
process.exit(0)
})
// Custom cleanup on exit
process.on('exit', (code) => {
console.log(`Process exiting with code ${code}`)
})
```
### Uncaught Exception Handling
```js
const process = require('bare-process')
process.on('uncaughtException', (err) => {
console.error('Uncaught exception:', err)
// Log to error tracking service
process.exit(1)
})
process.on('unhandledRejection', (reason, promise) => {
console.error('Unhandled rejection at:', promise, 'reason:', reason)
process.exit(1)
})
```
### Memory Usage
```js
const process = require('bare-process')
// Get memory usage statistics
const usage = process.memoryUsage()
console.log('Memory usage:', usage)
// {
// rss: 23456789, // Resident set size
// heapTotal: 12345678, // Total heap size
// heapUsed: 9876543, // Used heap size
// external: 1234567 // External memory
// }
// Monitor memory
setInterval(() => {
const mem = process.memoryUsage()
console.log(`Heap used: ${(mem.heapUsed / 1024 / 1024).toFixed(2)} MB`)
}, 5000)
```
### Uptime
```js
const process = require('bare-process')
// Get process uptime in seconds
console.log(`Running for ${process.uptime()} seconds`)
// Human-readable uptime
function formatUptime() {
const uptime = process.uptime()
const days = Math.floor(uptime / 86400)
const hours = Math.floor((uptime % 86400) / 3600)
const minutes = Math.floor((uptime % 3600) / 60)
const seconds = Math.floor(uptime % 60)
return `${days}d ${hours}h ${minutes}m ${seconds}s`
}
console.log(`Uptime: ${formatUptime()}`)
```
## Integration Examples
### CLI Application Framework
```js
const process = require('bare-process')
class CLIApp {
constructor() {
this.commands = new Map()
this.setupSignals()
}
command(name, handler) {
this.commands.set(name, handler)
}
setupSignals() {
process.on('SIGINT', () => this.shutdown())
process.on('SIGTERM', () => this.shutdown())
}
run() {
const args = process.argv.slice(2)
const [command, ...rest] = args
if (this.commands.has(command)) {
try {
this.commands.get(command)(rest)
} catch (err) {
console.error('Command failed:', err)
process.exit(1)
}
} else {
console.error(`Unknown command: ${command}`)
process.exit(1)
}
}
shutdown() {
console.log('\nShutting down gracefully...')
// Cleanup
process.exit(0)
}
}
// Usage
const app = new CLIApp()
app.command('start', (args) => {
console.log('Starting server...')
console.log('Environment:', process.env.NODE_ENV)
})
app.command('stop', () => {
console.log('Stopping server...')
})
app.run()
```
### Configuration Loader
```js
const process = require('bare-process')
function loadConfig() {
const env = process.env.NODE_ENV || 'development'
const configs = {
development: {
port: 3000,
debug: true,
db: 'localhost'
},
production: {
port: 80,
debug: false,
db: process.env.DB_HOST
}
}
return configs[env] || configs.development
}
// Override with CLI args
function parseConfigOverrides(args) {
const config = {}
for (let i = 0; i < args.length; i++) {
const arg = args[i]
if (arg.startsWith('--')) {
const key = arg.slice(2)
const value = args[i + 1]
if (value && !value.startsWith('--')) {
config[key] = isNaN(value) ? value : Number(value)
i++
} else {
config[key] = true
}
}
}
return config
}
const config = {
...loadConfig(),
...parseConfigOverrides(process.argv.slice(2))
}
console.log('Configuration:', config)
```
### Graceful Shutdown Manager
```js
const process = require('bare-process')
class GracefulShutdown {
constructor(timeout = 30000) {
this.timeout = timeout
this.handlers = []
this.shuttingDown = false
process.on('SIGINT', () => this.shutdown('SIGINT'))
process.on('SIGTERM', () => this.shutdown('SIGTERM'))
}
onShutdown(handler) {
this.handlers.push(handler)
}
async shutdown(signal) {
if (this.shuttingDown) return
this.shuttingDown = true
console.log(`\nReceived ${signal}, starting graceful shutdown...`)
const timeout = setTimeout(() => {
console.error('Shutdown timeout exceeded, forcing exit')
process.exit(1)
}, this.timeout)
try {
for (const handler of this.handlers) {
await handler()
}
console.log('Graceful shutdown complete')
process.exit(0)
} catch (err) {
console.error('Error during shutdown:', err)
process.exit(1)
} finally {
clearTimeout(timeout)
}
}
}
// Usage
const shutdown = new GracefulShutdown(10000)
shutdown.onShutdown(async () => {
console.log('Closing database connections...')
// await db.close()
})
shutdown.onShutdown(async () => {
console.log('Stopping HTTP server...')
// await server.close()
})
```
## Platform-Specific Behavior
### Environment Variable Inheritance
```js
const process = require('bare-process')
// Variables are inherited from parent process
console.log('Inherited environment:')
console.log('- HOME:', process.env.HOME)
console.log('- USER:', process.env.USER)
console.log('- PATH:', process.env.PATH?.slice(0, 50) + '...')
// Platform-specific variables
if (process.platform === 'win32') {
console.log('- USERPROFILE:', process.env.USERPROFILE)
console.log('- APPDATA:', process.env.APPDATA)
} else {
console.log('- XDG_CONFIG_HOME:', process.env.XDG_CONFIG_HOME)
console.log('- SHELL:', process.env.SHELL)
}
```
### Working Directory Differences
```js
const process = require('bare-process')
// Get platform-specific separators
const isWindows = process.platform === 'win32'
const separator = isWindows ? '\\' : '/'
// Cross-platform path handling
function resolvePath(...parts) {
return parts.join(separator)
}
// Ensure path exists before changing
function safeChdir(path) {
const fs = require('bare-fs')
if (!fs.existsSync(path)) {
throw new Error(`Directory does not exist: ${path}`)
}
process.chdir(path)
}
```
## Best Practices
1. **Always handle errors** when changing directories
2. **Use explicit exit codes** for different error conditions
3. **Clean up resources** before calling `process.exit()`
4. **Handle signals** for graceful shutdown
5. **Validate environment variables** before using them
```js
// Good practice example
const process = require('bare-process')
function main() {
// Validate required env vars
const required = ['API_KEY', 'DATABASE_URL']
for (const varName of required) {
if (!process.env[varName]) {
console.error(`Missing required environment variable: ${varName}`)
process.exit(1)
}
}
// Setup signal handlers
let isShuttingDown = false
process.on('SIGINT', async () => {
if (isShuttingDown) return
isShuttingDown = true
console.log('\nShutting down...')
// await cleanup()
process.exit(0)
})
// Run application
try {
// app.run()
} catch (err) {
console.error('Fatal error:', err)
process.exit(1)
}
}
main()
```
## License
Apache-2.0
---
**Module Type**: Core Runtime | **Ecosystem Role**: Node.js Compatibility | **Used By**: All Bare CLI applications
+704
View File
@@ -0,0 +1,704 @@
# dht-rpc - Kademlia DHT with RPC
## Overview
**dht-rpc** provides a flexible, high-performance Distributed Hash Table (DHT) based on the Kademlia protocol, with built-in RPC capabilities. It enables peer discovery, mutable key-value storage, and custom command routing across decentralized networks. This is the foundational networking layer that powers HyperDHT and Hyperswarm.
## Architecture
```mermaid
graph TB
subgraph "DHT Node"
UDX[UDX Socket]
ROUTING[Routing Table]
QUERIES[Query Engine]
RPC[RPC Handler]
end
subgraph "Kademlia Protocol"
LOOKUP[Node Lookup]
STORE[Value Store]
PING[Keep-alive Ping]
DIST[Distance Metric<br/>XOR]
end
subgraph "Network Operations"
BOOTSTRAP[Bootstrap]
QUERY[Query Routing]
RELAY[Relay/Holepunch]
NAT[NAT Detection]
end
UDX --> ROUTING
ROUTING --> QUERIES
QUERIES --> RPC
LOOKUP --> ROUTING
STORE --> RPC
PING --> ROUTING
DIST --> LOOKUP
BOOTSTRAP --> UDX
QUERY --> QUERIES
RELAY --> UDX
NAT --> UDX
style ROUTING fill:#f9f,stroke:#333,stroke-width:2px
style RPC fill:#bbf,stroke:#333,stroke-width:2px
```
## Core Concepts
### Kademlia DHT
Kademlia organizes nodes in a binary tree where distance is measured by XOR of node IDs:
```
Distance(A, B) = A.id XOR B.id
Node ID: 32-byte hash (usually of public key)
Routing Table: 256 buckets (one per bit)
Bucket Size: 20 nodes (Kademlia's "k" parameter)
```
### RPC Commands
Custom commands are registered as numbered enums:
```js
const COMMANDS = {
PUT: 0, // Store value
GET: 1, // Retrieve value
PING: 2, // Keep-alive
FIND_NODE: 3 // Lookup nodes
}
```
### Token-Based Security
Round-trip tokens prevent spoofing:
```
1. Node A queries Node B
2. Node B responds with token T
3. Node A includes T in next request
4. Node B verifies T is recent and valid
5. Only then allows state mutation
```
## API Reference
### Creating a DHT Node
```js
const DHT = require('dht-rpc')
// Create ephemeral node (default)
const node = new DHT()
// Wait for readiness
await node.fullyBootstrapped()
console.log('Node ID:', node.id?.toString('hex'))
console.log('Public IP:', node.host)
console.log('Public Port:', node.port)
console.log('Firewalled:', node.firewalled)
```
### Bootstrap Node
```js
const DHT = require('dht-rpc')
// Create dedicated bootstrap node
const bootstrap = DHT.bootstrapper(10001, '0.0.0.0')
await bootstrap.fullyBootstrapped()
console.log('Bootstrap node ready on port 10001')
// Other nodes use this for bootstrapping
const node = new DHT({
bootstrap: ['127.0.0.1:10001']
})
```
### Custom RPC Commands
```js
const DHT = require('dht-rpc')
const crypto = require('crypto')
const VALUES = 0
const node = new DHT({
bootstrap: ['localhost:10001']
})
// Local storage
const storage = new Map()
// Handle incoming requests
node.on('request', (req) => {
switch (req.command) {
case VALUES:
if (req.token) {
// Commit: store the value
const key = hash(req.value).toString('hex')
storage.set(key, req.value)
console.log('Stored:', key)
return req.reply(null)
}
// Query: return value if we have it
const value = storage.get(req.target.toString('hex'))
req.reply(value)
break
default:
req.error(DHT.ERROR_UNKNOWN_COMMAND)
}
})
function hash(value) {
return crypto.createHash('sha256').update(value).digest()
}
```
### Querying the DHT
```js
const DHT = require('dht-rpc')
const node = new DHT()
// Simple query
const target = crypto.randomBytes(32)
const query = node.query({
target,
command: VALUES,
value: Buffer.from('search query')
})
// Process responses
query.on('data', (data) => {
console.log('Got response from:', data.from)
console.log('Value:', data.value?.toString())
})
// Wait for completion
await query.finished()
console.log('Query complete')
console.log('Closest nodes:', query.closestNodes)
```
### Committing Values
```js
// Store value in DHT (requires commit)
const value = Buffer.from('Hello, DHT!')
const target = hash(value)
const query = node.query({
target,
command: VALUES,
value
}, {
commit: true // Signal closest nodes to store
})
await query.finished()
console.log('Value stored at', target.toString('hex'))
```
## Complete Examples
### Example 1: Distributed Key-Value Store
```js
const DHT = require('dht-rpc')
const crypto = require('crypto')
const PUT = 0
const GET = 1
class DHTKeyValue {
constructor(options = {}) {
this.node = new DHT(options)
this.storage = new Map()
this.setupHandlers()
}
setupHandlers() {
this.node.on('request', (req) => {
switch (req.command) {
case PUT:
this.handlePut(req)
break
case GET:
this.handleGet(req)
break
}
})
}
handlePut(req) {
if (!req.token) {
return req.error(DHT.ERROR_INVALID_TOKEN)
}
const ttl = req.value.readUInt32BE(0)
const key = req.value.slice(4, 36).toString('hex')
const value = req.value.slice(36)
this.storage.set(key, {
value,
expires: Date.now() + ttl
})
req.reply(null)
}
handleGet(req) {
const key = req.target.toString('hex')
const entry = this.storage.get(key)
if (!entry || entry.expires < Date.now()) {
this.storage.delete(key)
return req.reply(null)
}
req.reply(entry.value)
}
async put(key, value, ttl = 3600000) {
const target = Buffer.from(key, 'hex')
const packet = Buffer.concat([
Buffer.from([ttl >> 24, ttl >> 16, ttl >> 8, ttl]),
target,
Buffer.from(value)
])
const query = this.node.query({
target,
command: PUT,
value: packet
}, { commit: true })
await query.finished()
}
async get(key) {
const target = Buffer.from(key, 'hex')
const query = this.node.query({
target,
command: GET
})
for await (const data of query) {
if (data.value) {
return data.value.toString()
}
}
return null
}
async ready() {
await this.node.fullyBootstrapped()
}
destroy() {
return this.node.destroy()
}
}
// Usage
async function main() {
// Start bootstrap
const bootstrap = DHT.bootstrapper(10001, '127.0.0.1')
await bootstrap.fullyBootstrapped()
// Create nodes
const node1 = new DHTKeyValue({
bootstrap: ['127.0.0.1:10001']
})
const node2 = new DHTKeyValue({
bootstrap: ['127.0.0.1:10001']
})
await node1.ready()
await node2.ready()
// Store value
const key = crypto.randomBytes(32).toString('hex')
await node1.put(key, 'Hello from DHT!')
// Retrieve value
const value = await node2.get(key)
console.log('Retrieved:', value)
// Cleanup
await node1.destroy()
await node2.destroy()
await bootstrap.destroy()
}
main().catch(console.error)
```
### Example 2: Peer Discovery Service
```js
const DHT = require('dht-rpc')
const ANNOUNCE = 0
const LOOKUP = 1
class PeerDiscovery {
constructor(topic, options = {}) {
this.topic = topic
this.node = new DHT(options)
this.peers = new Map()
this.setupHandlers()
}
setupHandlers() {
this.node.on('request', (req) => {
if (req.target.equals(this.topic)) {
switch (req.command) {
case ANNOUNCE:
this.handleAnnounce(req)
break
case LOOKUP:
this.handleLookup(req)
break
}
}
})
}
handleAnnounce(req) {
if (!req.token) {
return req.reply(null) // Request token first
}
const peerId = req.from.host + ':' + req.from.port
this.peers.set(peerId, {
host: req.from.host,
port: req.from.port,
data: req.value,
announced: Date.now()
})
req.reply(null)
}
handleLookup(req) {
const peers = Array.from(this.peers.values())
.filter(p => Date.now() - p.announced < 300000) // 5 min TTL
.slice(0, 20)
req.reply(Buffer.from(JSON.stringify(peers)))
}
async announce(host, port, metadata = {}) {
const value = Buffer.from(JSON.stringify({
host,
port,
...metadata
}))
const query = this.node.query({
target: this.topic,
command: ANNOUNCE,
value
}, { commit: true })
await query.finished()
}
async lookup() {
const query = this.node.query({
target: this.topic,
command: LOOKUP
})
const allPeers = new Map()
for await (const data of query) {
if (data.value) {
try {
const peers = JSON.parse(data.value.toString())
peers.forEach(p => {
const id = p.host + ':' + p.port
allPeers.set(id, p)
})
} catch (e) {
// Ignore parse errors
}
}
}
return Array.from(allPeers.values())
}
async ready() {
await this.node.fullyBootstrapped()
}
destroy() {
return this.node.destroy()
}
}
// Usage
async function main() {
const topic = crypto.randomBytes(32)
const discovery = new PeerDiscovery(topic, {
bootstrap: ['bootstrap.hyperdht.org:49737']
})
await discovery.ready()
// Announce ourselves
await discovery.announce(
discovery.node.host,
discovery.node.port,
{ service: 'file-share', version: '1.0' }
)
// Find peers
const peers = await discovery.lookup()
console.log('Found peers:', peers)
await discovery.destroy()
}
```
### Example 3: NAT Detection and Holepunch
```js
const DHT = require('dht-rpc')
async function analyzeNat() {
const node = new DHT()
await new Promise((resolve) => {
node.once('persistent', resolve)
})
console.log('NAT Analysis:')
console.log(' Firewalled:', node.firewalled)
console.log(' Randomized:', node.randomized)
console.log(' Public IP:', node.host)
console.log(' Public Port:', node.port)
console.log(' Local Port:', node.address().port)
if (node.firewalled) {
console.log(' → Node is behind NAT, will need relays')
} else if (node.randomized) {
console.log(' → Port is randomized, consistent connections difficult')
} else {
console.log(' → Node is directly accessible!')
}
await node.destroy()
}
// Test latency to specific node
async function testLatency(node, target) {
const times = []
for (let i = 0; i < 5; i++) {
const start = Date.now()
try {
await node.ping(target)
times.push(Date.now() - start)
} catch (e) {
times.push(null)
}
await new Promise(r => setTimeout(r, 100))
}
const valid = times.filter(t => t !== null)
const avg = valid.reduce((a, b) => a + b, 0) / valid.length
console.log(`Latency to ${target.host}:${target.port}:`)
console.log(` Average: ${avg.toFixed(2)}ms`)
console.log(` Min: ${Math.min(...valid)}ms`)
console.log(` Max: ${Math.max(...valid)}ms`)
}
```
## Advanced Features
### Adaptive Mode
Nodes automatically determine if they should be persistent:
```js
const node = new DHT({
// Default: adaptive mode
// Node starts ephemeral, may become persistent
})
// Force persistent (for testing)
const persistent = new DHT({
ephemeral: false
})
// Events
node.on('persistent', () => {
console.log('Node is now persistent in routing tables')
})
node.on('ephemeral', () => {
console.log('Node is ephemeral (not in routing tables)')
})
```
### Suspension and Resume
Handle mobile app backgrounding:
```js
// App going to background
await node.suspend()
// App resuming
await node.resume()
// Handle sleep detection
node.on('wake-up', () => {
console.log('System woke from sleep')
})
```
### Custom Query Logic
```js
const query = node.query({
target,
command: CUSTOM_CMD,
value: payload
}, {
// Custom commit logic
async commit(reply, dht, query) {
// Called for each closest node response
console.log('Committing to:', reply.from)
// Send authenticated request
await dht.request({
token: reply.token,
target,
command: CUSTOM_CMD,
value: signedPayload
}, reply.from)
},
// Transform responses
map(reply) {
return {
node: reply.from,
value: reply.value,
distance: xorDistance(target, reply.from.id)
}
}
})
```
## Network Events
```js
node.on('bootstrap', () => {
console.log('Connected to bootstrap nodes')
})
node.on('listening', () => {
console.log('UDP socket ready')
})
node.on('ready', () => {
console.log('Fully bootstrapped')
})
node.on('persistent', () => {
console.log('Now persistent')
})
node.on('wake-up', () => {
console.log('Woke from sleep')
})
node.on('network-change', (interfaces) => {
console.log('Network changed:', interfaces)
})
node.on('nat-update', (host, port) => {
console.log(`NAT mapping: ${host}:${port}`)
})
node.on('close', () => {
console.log('Node destroyed')
})
```
## Error Codes
```js
DHT.OK = 0 // Success
DHT.ERROR_UNKNOWN_COMMAND = 1 // Unknown RPC command
DHT.ERROR_INVALID_TOKEN = 2 // Invalid/Expired token
// Application errors should start at 16
```
## Performance Characteristics
| Metric | Value | Notes |
|--------|-------|-------|
| Lookup time | <1s | To closest nodes |
| Routing table | 256 buckets | 20 nodes per bucket |
| Concurrent queries | Unlimited | Async iterators |
| Token TTL | ~5 min | Prevents replay |
| Bootstrap time | ~2-5s | Depends on network |
## Best Practices
1. **Use tokens for mutations**: Always require tokens for state changes
2. **Validate data**: Don't trust incoming requests blindly
3. **Handle timeouts**: Network is unreliable
4. **Implement TTL**: Clean up stale data
5. **Monitor NAT status**: Adapt behavior based on connectivity
```js
// Good practice: Always validate
node.on('request', (req) => {
if (!isValidCommand(req.command)) {
return req.error(DHT.ERROR_UNKNOWN_COMMAND)
}
if (isMutation(req.command) && !req.token) {
return req.error(DHT.ERROR_INVALID_TOKEN)
}
// Validate payload size
if (req.value && req.value.length > MAX_SIZE) {
return req.error(16) // Custom: Payload too large
}
// Process request
handleRequest(req)
})
```
## License
MIT
---
**Module Type**: Core Networking | **Ecosystem Role**: DHT Foundation | **Used By**: hyperdht, hyperswarm, all peer discovery
+552
View File
@@ -0,0 +1,552 @@
# hypercore-crypto - Cryptographic Primitives
## Overview
**hypercore-crypto** provides the fundamental cryptographic primitives used throughout the Hypercore ecosystem. Built on battle-tested Ed25519 signatures and BLAKE2b hashing, it powers the security guarantees of append-only logs, peer authentication, and data integrity across all Hypercore-based applications.
## Architecture
```mermaid
graph TB
subgraph "Cryptographic Primitives"
KEYS[Ed25519 Key Pairs]
SIGN[Signing]
VERIFY[Verification]
HASH[BLAKE2b Hashing]
RANDOM[Secure Random]
end
subgraph "Hypercore Operations"
TREE[Merkle Tree]
DISCOVERY[Discovery Keys]
NAMESPACE[Namespaces]
PROOFS[Proof Verification]
end
subgraph "Security Guarantees"
AUTH[Writer Authentication]
INTEGRITY[Data Integrity]
PRIVACY[Privacy Preserving]
end
KEYS --> SIGN
KEYS --> VERIFY
SIGN --> PROOFS
VERIFY --> PROOFS
HASH --> TREE
HASH --> DISCOVERY
TREE --> AUTH
TREE --> INTEGRITY
DISCOVERY --> PRIVACY
NAMESPACE --> AUTH
style KEYS fill:#f9f,stroke:#333,stroke-width:2px
style HASH fill:#bbf,stroke:#333,stroke-width:2px
```
## Core API
### Key Pairs
Generate Ed25519 key pairs for writer identification:
```js
const crypto = require('hypercore-crypto')
// Generate new key pair
const keyPair = crypto.keyPair()
console.log('Public Key:', keyPair.publicKey.toString('hex'))
// 64 character hex string
console.log('Secret Key:', keyPair.secretKey.toString('hex'))
// 128 character hex string (includes public key)
// Key pair structure
// {
// publicKey: <Buffer 32 bytes>,
// secretKey: <Buffer 64 bytes>
// }
```
### Signing and Verification
```js
const crypto = require('hypercore-crypto')
const b4a = require('b4a')
const keyPair = crypto.keyPair()
// Message to sign
const message = b4a.from('Hello, Hypercore!')
// Sign message
const signature = crypto.sign(message, keyPair.secretKey)
console.log('Signature:', signature.toString('hex'))
// 128 character hex string (Ed25519 signature)
// Verify signature
const isValid = crypto.verify(message, signature, keyPair.publicKey)
console.log('Valid:', isValid) // true
// Tampered message fails verification
const tampered = b4a.from('Goodbye, Hypercore!')
const isTamperedValid = crypto.verify(tampered, signature, keyPair.publicKey)
console.log('Tampered valid:', isTamperedValid) // false
```
### Merkle Tree Operations
#### Data Hashing
Hash leaf nodes (data blocks):
```js
const crypto = require('hypercore-crypto')
// Hash a data block
const data = Buffer.from('Block content')
const leafHash = crypto.data(data)
console.log('Leaf hash:', leafHash.toString('hex'))
// 64 character BLAKE2b hash
```
#### Parent Hashing
Combine child nodes into parent hashes:
```js
const crypto = require('hypercore-crypto')
// Two child nodes
const left = {
index: 0,
hash: crypto.data(Buffer.from('Left block')),
size: 10
}
const right = {
index: 1,
hash: crypto.data(Buffer.from('Right block')),
size: 11
}
// Compute parent hash
const parentHash = crypto.parent(left, right)
console.log('Parent hash:', parentHash.toString('hex'))
```
#### Tree Root Hashing
Hash the merkle root from tree peaks:
```js
const crypto = require('hypercore-crypto')
// Tree peaks (for a tree with 5 nodes)
const peaks = [
{ index: 4, hash: peak1Hash, size: size1 },
{ index: 2, hash: peak2Hash, size: size2 }
]
// Compute root hash
const rootHash = crypto.tree(peaks)
console.log('Root hash:', rootHash.toString('hex'))
```
### Discovery Keys
Generate privacy-preserving discovery keys:
```js
const crypto = require('hypercore-crypto')
const keyPair = crypto.keyPair()
// Generate discovery key from public key
const discoveryKey = crypto.discoveryKey(keyPair.publicKey)
console.log('Public Key:', keyPair.publicKey.toString('hex'))
console.log('Discovery Key:', discoveryKey.toString('hex'))
// Discovery key is derived but doesn't reveal public key
// Used for DHT topics without exposing the actual feed key
```
### Namespaces
Create namespaced capabilities:
```js
const crypto = require('hypercore-crypto')
// Create namespace from public name
const namespaces = crypto.namespace('hypercore', 3)
console.log('Namespaces:')
namespaces.forEach((ns, i) => {
console.log(` ${i}:`, ns.toString('hex'))
})
// Use for algorithm-specific operations
// Prevents cross-protocol attacks
```
### Random Bytes
Generate cryptographically secure random data:
```js
const crypto = require('hypercore-crypto')
// Generate 32 random bytes
const random = crypto.randomBytes(32)
console.log('Random:', random.toString('hex'))
// Use for nonces, salts, keys
const nonce = crypto.randomBytes(24)
const salt = crypto.randomBytes(16)
```
## Complete Examples
### Example 1: Verifiable Log
```js
const crypto = require('hypercore-crypto')
const b4a = require('b4a')
class VerifiableLog {
constructor() {
this.blocks = []
this.hashes = []
this.keyPair = crypto.keyPair()
}
append(data) {
const index = this.blocks.length
// Hash the data
const hash = crypto.data(b4a.from(data))
// Sign the hash
const signature = crypto.sign(hash, this.keyPair.secretKey)
this.blocks.push({
index,
data,
hash,
signature
})
this.hashes.push(hash)
return { index, hash, signature }
}
verify(index) {
const block = this.blocks[index]
if (!block) return false
// Verify signature
return crypto.verify(block.hash, block.signature, this.keyPair.publicKey)
}
getPublicKey() {
return this.keyPair.publicKey
}
}
// Usage
const log = new VerifiableLog()
log.append('First block')
log.append('Second block')
log.append('Third block')
console.log('Block 0 valid:', log.verify(0)) // true
console.log('Block 1 valid:', log.verify(1)) // true
console.log('Block 2 valid:', log.verify(2)) // true
```
### Example 2: Merkle Tree Builder
```js
const crypto = require('hypercore-crypto')
const b4a = require('b4a')
class MerkleTree {
constructor() {
this.leaves = []
this.layers = []
}
push(data) {
const hash = crypto.data(b4a.from(data))
this.leaves.push({
index: this.leaves.length,
hash,
size: data.length
})
this.rebuild()
}
rebuild() {
this.layers = [this.leaves]
while (this.layers[0].length > 1) {
const layer = this.layers[0]
const nextLayer = []
for (let i = 0; i < layer.length; i += 2) {
if (i + 1 < layer.length) {
// Combine pair
nextLayer.push({
index: layer[i].index,
hash: crypto.parent(layer[i], layer[i + 1]),
size: layer[i].size + layer[i + 1].size
})
} else {
// Promote single
nextLayer.push(layer[i])
}
}
this.layers.unshift(nextLayer)
}
}
root() {
if (this.layers.length === 0) return null
return this.layers[0][0]
}
proof(index) {
// Generate inclusion proof for leaf at index
const proof = []
let idx = index
for (let i = this.layers.length - 1; i > 0; i--) {
const layer = this.layers[i]
const isRight = idx % 2 === 1
const siblingIdx = isRight ? idx - 1 : idx + 1
if (siblingIdx < layer.length) {
proof.push({
position: isRight ? 'left' : 'right',
hash: layer[siblingIdx].hash
})
}
idx = Math.floor(idx / 2)
}
return proof
}
}
// Usage
const tree = new MerkleTree()
tree.push('Block 0')
tree.push('Block 1')
tree.push('Block 2')
tree.push('Block 3')
console.log('Root:', tree.root().hash.toString('hex'))
console.log('Proof for Block 1:', tree.proof(1))
```
### Example 3: Secure Channel Setup
```js
const crypto = require('hypercore-crypto')
const b4a = require('b4a')
class SecureChannel {
constructor() {
this.keyPair = crypto.keyPair()
this.sessionKey = crypto.randomBytes(32)
}
getPublicKey() {
return this.keyPair.publicKey
}
getDiscoveryKey() {
return crypto.discoveryKey(this.keyPair.publicKey)
}
signHandshake() {
const message = b4a.concat([
this.keyPair.publicKey,
this.sessionKey
])
return crypto.sign(message, this.keyPair.secretKey)
}
verifyHandshake(publicKey, sessionKey, signature) {
const message = b4a.concat([publicKey, sessionKey])
return crypto.verify(message, signature, publicKey)
}
}
// Usage
const alice = new SecureChannel()
const bob = new SecureChannel()
console.log('Alice discovery key:', alice.getDiscoveryKey().toString('hex'))
console.log('Bob discovery key:', bob.getDiscoveryKey().toString('hex'))
// Alice signs handshake for Bob
const aliceSignature = alice.signHandshake()
const isValid = bob.verifyHandshake(
alice.getPublicKey(),
alice.sessionKey,
aliceSignature
)
console.log('Handshake valid:', isValid)
```
## Integration with Hypercore
### Block Signing
```js
const Hypercore = require('hypercore')
const crypto = require('hypercore-crypto')
// Hypercore internally uses hypercore-crypto for:
// - Generating writer key pairs
// - Signing each block
// - Verifying blocks on read
// - Building merkle trees
const core = new Hypercore('./storage', {
// Uses crypto.keyPair() internally
})
await core.ready()
// Each append signs the block
await core.append('Data')
// Internally: crypto.sign(hash, secretKey)
// Each get verifies the signature
const block = await core.get(0)
// Internally: crypto.verify(hash, signature, publicKey)
```
### Proof Verification
```js
const Hypercore = require('hypercore')
const core = new Hypercore('./storage')
await core.ready()
// Create proof
const proof = await core.proof({
block: { index: 5, nodes: 2 }
})
// Proof contains:
// - Block signature
// - Merkle tree nodes
// - Root signature
// Verify proof
const verified = await core.verify(proof, {
publicKey: core.key
})
console.log('Proof verified:', verified)
```
## Security Considerations
### Key Management
```js
const crypto = require('hypercore-crypto')
// NEVER log or transmit secret keys
const keyPair = crypto.keyPair()
// Safe to share
console.log('Public Key:', keyPair.publicKey.toString('hex'))
// NEVER do this in production
// console.log('Secret Key:', keyPair.secretKey.toString('hex'))
// Store secret keys securely
// - Hardware security modules (HSM)
// - Encrypted key stores
// - OS keychain
```
### Replay Protection
```js
const crypto = require('hypercore-crypto')
const b4a = require('b4a')
// Include sequence numbers in signed messages
function createSignedMessage(data, seq, keyPair) {
const message = b4a.concat([
b4a.from([seq]),
b4a.from(data)
])
return {
seq,
data,
signature: crypto.sign(message, keyPair.secretKey)
}
}
function verifySignedMessage(msg, expectedSeq, publicKey) {
const message = b4a.concat([
b4a.from([msg.seq]),
b4a.from(msg.data)
])
// Check sequence first
if (msg.seq !== expectedSeq) {
return false
}
return crypto.verify(message, msg.signature, publicKey)
}
```
## Performance Characteristics
| Operation | Time | Notes |
|-----------|------|-------|
| keyPair() | ~1ms | Ed25519 key generation |
| sign() | ~0.1ms | Ed25519 signing |
| verify() | ~0.3ms | Ed25519 verification |
| data() | ~0.01ms | BLAKE2b hashing |
| parent() | ~0.01ms | BLAKE2b hashing |
| tree() | ~0.01ms | BLAKE2b hashing |
| randomBytes() | ~0.01ms | OS random source |
## Best Practices
1. **Reuse key pairs**: Generate once, store securely
2. **Verify early**: Check signatures before processing
3. **Use discovery keys**: Don't expose public keys unnecessarily
4. **Namespace operations**: Prevent cross-protocol confusion
5. **Random nonces**: Never reuse nonces for encryption
## License
MIT
---
**Module Type**: Core Cryptography | **Ecosystem Role**: Security Foundation | **Used By**: hypercore, hyperdht, hyperswarm, all P2P protocols
+640
View File
@@ -0,0 +1,640 @@
# protomux-rpc - Multiplexed RPC Protocol
## Overview
**protomux-rpc** provides a simple yet powerful RPC (Remote Procedure Call) system built on top of Protomux channels. It enables structured request-response communication over framed streams, making it ideal for building distributed applications, microservices, and peer-to-peer protocols in the Hypercore ecosystem.
## Architecture
```mermaid
graph TB
subgraph "RPC Layer"
CLIENT[RPC Client]
SERVER[RPC Server]
METHODS[Method Registry]
end
subgraph "Protomux Layer"
MUX[Protomux Muxer]
CHANNELS[Channels]
FRAMING[Message Framing]
end
subgraph "Transport"
STREAM[Framed Stream]
HYPERSWARM[Hyperswarm]
DIRECT[Direct Connection]
end
CLIENT --> MUX
SERVER --> MUX
METHODS --> SERVER
MUX --> CHANNELS
CHANNELS --> FRAMING
FRAMING --> STREAM
STREAM --> HYPERSWARM
STREAM --> DIRECT
style MUX fill:#f9f,stroke:#333,stroke-width:2px
style RPC fill:#bbf,stroke:#333,stroke-width:2px
```
## Core Concepts
### Request-Response Model
```
Client Server
| |
|--- Request (method, payload) ----------->|
| |
| |--> Handler
| |<-- Response
| |
|<-- Response (result or error) -----------|
```
### Compact Encoding
Messages use compact-encoding for efficient binary serialization:
```js
// Request format
{
id: uint, // Request ID for matching response
method: string, // Method name
value: raw // Binary payload
}
// Response format
{
flags: bitfield, // Error indicators
id: uint, // Matching request ID
value: raw // Response data
}
```
### Channel Multiplexing
Multiple RPC channels can share one transport stream via Protomux:
```
Stream
├── Channel 1: RPC (methods A, B, C)
├── Channel 2: RPC (methods X, Y, Z)
└── Channel 3: Raw data
```
## API Reference
### Creating RPC Connection
```js
const ProtomuxRPC = require('protomux-rpc')
// From framed stream
const rpc = new ProtomuxRPC(stream)
// From existing protomux instance
const rpc = new ProtomuxRPC(mux, {
protocol: 'my-protocol'
})
// Wait for ready
await rpc.fullyOpened()
```
### Server-Side: Responding to Requests
```js
const rpc = new ProtomuxRPC(stream)
// Register method handler
rpc.respond('echo', (req) => {
return req // Echo back the request
})
// With encoding
rpc.respond('add', {
requestEncoding: cenc.uint,
responseEncoding: cenc.uint
}, (a) => {
return a + 1
})
// Async handler
rpc.respond('fetch', async (req) => {
const data = await database.get(req.toString())
return data
})
// Remove handler
rpc.unrespond('echo')
```
### Client-Side: Making Requests
```js
const rpc = new ProtomuxRPC(stream)
// Simple request
const result = await rpc.request('echo', Buffer.from('hello'))
console.log(result.toString()) // 'hello'
// With encoding
const cenc = require('compact-encoding')
const sum = await rpc.request('add', 5, {
requestEncoding: cenc.uint,
responseEncoding: cenc.uint
})
console.log(sum) // 6
// With timeout
const result = await rpc.request('slow', data, {
timeout: 5000 // 5 seconds
})
// Fire-and-forget event
rpc.event('log', Buffer.from('message'))
```
### Connection Management
```js
// Check state
console.log('Opened:', rpc.opened)
console.log('Closed:', rpc.closed)
// Graceful close
await rpc.end()
// Force close
rpc.destroy()
// With error
rpc.destroy(new Error('Connection failed'))
```
## Complete Examples
### Example 1: Calculator Service
```js
const ProtomuxRPC = require('protomux-rpc')
const cenc = require('compact-encoding')
// Request/response encoding
const CalcRequest = {
preencode(state, m) {
cenc.string.preencode(state, m.op)
cenc.float.preencode(state, m.a)
cenc.float.preencode(state, m.b)
},
encode(state, m) {
cenc.string.encode(state, m.op)
cenc.float.encode(state, m.a)
cenc.float.encode(state, m.b)
},
decode(state) {
return {
op: cenc.string.decode(state),
a: cenc.float.decode(state),
b: cenc.float.decode(state)
}
}
}
// Server
function createCalculatorServer(stream) {
const rpc = new ProtomuxRPC(stream)
rpc.respond('calc', {
requestEncoding: CalcRequest,
responseEncoding: cenc.float
}, (req) => {
switch (req.op) {
case 'add': return req.a + req.b
case 'sub': return req.a - req.b
case 'mul': return req.a * req.b
case 'div':
if (req.b === 0) throw new Error('Division by zero')
return req.a / req.b
default: throw new Error('Unknown operation')
}
})
return rpc
}
// Client
async function calculatorClient(stream) {
const rpc = new ProtomuxRPC(stream)
await rpc.fullyOpened()
return {
async calculate(op, a, b) {
return rpc.request('calc', { op, a, b }, {
requestEncoding: CalcRequest,
responseEncoding: cenc.float
})
},
close: () => rpc.end()
}
}
// Usage
async function main() {
// Create streams (e.g., from hyperswarm)
const { stream1, stream2 } = createTestStreams()
// Server
const server = createCalculatorServer(stream1)
// Client
const client = await calculatorClient(stream2)
// Calculate
console.log('2 + 3 =', await client.calculate('add', 2, 3))
console.log('10 - 4 =', await client.calculate('sub', 10, 4))
console.log('5 * 6 =', await client.calculate('mul', 5, 6))
console.log('15 / 3 =', await client.calculate('div', 15, 3))
// Cleanup
await client.close()
}
```
### Example 2: Database Proxy
```js
const ProtomuxRPC = require('protomux-rpc')
const cenc = require('compact-encoding')
class DatabaseRPC {
constructor(db) {
this.db = db
this.methods = new Map()
this.setupMethods()
}
setupMethods() {
// GET method
this.methods.set('get', {
requestEncoding: cenc.string,
responseEncoding: cenc.raw,
handler: async (key) => {
const value = await this.db.get(key)
return value ? Buffer.from(value) : null
}
})
// PUT method
this.methods.set('put', {
requestEncoding: {
preencode(state, m) {
cenc.string.preencode(state, m.key)
cenc.raw.preencode(state, m.value)
},
encode(state, m) {
cenc.string.encode(state, m.key)
cenc.raw.encode(state, m.value)
},
decode(state) {
return {
key: cenc.string.decode(state),
value: cenc.raw.decode(state)
}
}
},
responseEncoding: cenc.bool,
handler: async ({ key, value }) => {
await this.db.put(key, value)
return true
}
})
// DELETE method
this.methods.set('del', {
requestEncoding: cenc.string,
responseEncoding: cenc.bool,
handler: async (key) => {
await this.db.del(key)
return true
}
})
// LIST method
this.methods.set('list', {
requestEncoding: cenc.string, // prefix
responseEncoding: cenc.array(cenc.string),
handler: async (prefix) => {
const keys = await this.db.list(prefix)
return keys
}
})
}
attach(stream) {
const rpc = new ProtomuxRPC(stream)
for (const [name, { requestEncoding, responseEncoding, handler }] of this.methods) {
rpc.respond(name, { requestEncoding, responseEncoding }, handler)
}
return rpc
}
async connect(stream) {
const rpc = new ProtomuxRPC(stream)
await rpc.fullyOpened()
return {
get: (key) => rpc.request('get', key, this.methods.get('get')),
put: (key, value) => rpc.request('put', { key, value }, this.methods.get('put')),
del: (key) => rpc.request('del', key, this.methods.get('del')),
list: (prefix) => rpc.request('list', prefix, this.methods.get('list')),
close: () => rpc.end()
}
}
}
// Usage
async function main() {
const { stream1, stream2 } = createTestStreams()
// Simulated database
const db = new Map()
// Server
const dbRPC = new DatabaseRPC(db)
const server = dbRPC.attach(stream1)
// Client
const client = await dbRPC.connect(stream2)
// Operations
await client.put('user:1', Buffer.from('Alice'))
await client.put('user:2', Buffer.from('Bob'))
const user1 = await client.get('user:1')
console.log('User 1:', user1.toString())
const users = await client.list('user:')
console.log('Users:', users)
await client.close()
}
```
### Example 3: Event Streaming
```js
const ProtomuxRPC = require('protomux-rpc')
class EventStreamer {
constructor() {
this.listeners = new Map()
}
attach(stream) {
const rpc = new ProtomuxRPC(stream)
// Subscribe to events
rpc.respond('subscribe', (topic) => {
if (!this.listeners.has(topic)) {
this.listeners.set(topic, new Set())
}
this.listeners.get(topic).add(rpc)
return { success: true }
})
// Unsubscribe
rpc.respond('unsubscribe', (topic) => {
const topicListeners = this.listeners.get(topic)
if (topicListeners) {
topicListeners.delete(rpc)
}
return { success: true }
})
rpc.on('close', () => {
// Cleanup subscriptions
for (const [topic, listeners] of this.listeners) {
listeners.delete(rpc)
}
})
return rpc
}
async emit(topic, event) {
const listeners = this.listeners.get(topic)
if (!listeners) return
const promises = []
for (const rpc of listeners) {
// Use event() for fire-and-forget
promises.push(
rpc.event('event', Buffer.from(JSON.stringify({ topic, event })))
.catch(() => {}) // Ignore errors from disconnected clients
)
}
await Promise.all(promises)
}
}
// Usage
async function main() {
const { stream1, stream2 } = createTestStreams()
const streamer = new EventStreamer()
const server = streamer.attach(stream1)
// Client
const client = new ProtomuxRPC(stream2)
await client.fullyOpened()
// Subscribe
await client.request('subscribe', Buffer.from('updates'))
// Listen for events
client.respond('event', (data) => {
const event = JSON.parse(data.toString())
console.log('Received:', event)
})
// Server emits
setInterval(() => {
streamer.emit('updates', { time: Date.now() })
}, 1000)
}
```
## Integration with Hyperswarm
```js
const Hyperswarm = require('hyperswarm')
const ProtomuxRPC = require('protomux-rpc')
const crypto = require('hypercore-crypto')
const swarm = new Hyperswarm()
const topic = crypto.randomBytes(32)
// Server
swarm.on('connection', (conn) => {
const rpc = new ProtomuxRPC(conn)
rpc.respond('greet', (name) => {
return Buffer.from(`Hello, ${name.toString()}!`)
})
})
swarm.join(topic, { server: true })
// Client
const clientSwarm = new Hyperswarm()
clientSwarm.on('connection', async (conn) => {
const rpc = new ProtomuxRPC(conn)
await rpc.fullyOpened()
const greeting = await rpc.request('greet', Buffer.from('World'))
console.log(greeting.toString())
await rpc.end()
clientSwarm.destroy()
})
clientSwarm.join(topic, { client: true })
```
## Error Handling
```js
// Server throws error
rpc.respond('risky', () => {
throw new Error('Something went wrong')
})
// Client receives error
try {
await rpc.request('risky', data)
} catch (err) {
console.error('RPC Error:', err.message)
console.error('Error Code:', err.code)
}
// Custom error codes
rpc.respond('auth', (token) => {
if (!isValid(token)) {
const err = new Error('Invalid token')
err.code = 'AUTH_FAILED'
throw err
}
})
```
## Performance Optimization
```js
// Batch requests
async function batchRequests(rpc, items) {
const promises = items.map(item =>
rpc.request('process', item)
)
return Promise.all(promises)
}
// Cork/uncork for batching
rpc.cork()
for (let i = 0; i < 100; i++) {
rpc.event('log', Buffer.from(`Event ${i}`))
}
rpc.uncork()
// Use appropriate encodings
const cenc = require('compact-encoding')
// Bad: JSON strings
rpc.request('data', Buffer.from(JSON.stringify(bigObject)))
// Good: Compact encoding
rpc.request('data', bigObject, {
requestEncoding: MyEncoding
})
```
## Protocol Details
### Message Types
```js
// Request (type 0)
{
id: uint, // Request identifier
method: string, // Method name
value: raw // Payload
}
// Response (type 1)
{
flags: bitfield(4), // error, code, cause, context
id: uint, // Request ID
// If error flag set:
error: string, // Error message
// If code flag set:
code: string, // Error code
// If cause flag set:
cause: string, // Cause message
causeCode: string, // Cause code
// If context flag set:
context: string, // Additional context
// If no error:
value: raw // Response payload
}
```
## Best Practices
1. **Use typed encodings**: Define schemas with compact-encoding
2. **Handle timeouts**: Network is unreliable
3. **Graceful degradation**: Handle missing methods
4. **Resource cleanup**: Remove handlers on close
5. **Error context**: Provide meaningful error messages
```js
// Good practice example
class RPCService {
constructor() {
this.handlers = new Map()
}
register(rpc) {
for (const [name, handler] of this.handlers) {
rpc.respond(name, handler)
}
rpc.on('close', () => {
// Cleanup when connection closes
for (const name of this.handlers.keys()) {
rpc.unrespond(name)
}
})
}
addMethod(name, handler) {
this.handlers.set(name, handler)
}
}
```
## License
Apache-2.0
---
**Module Type**: Core Networking | **Ecosystem Role**: RPC Foundation | **Used By**: hrpc, hyperdht, hyperswarm