This commit is contained in:
Raven Scott
2026-02-19 07:18:29 -05:00
parent cdd9720ab2
commit b367aa3643
18 changed files with 6109 additions and 25 deletions
+378
View File
@@ -0,0 +1,378 @@
# autobase-discovery-cli - Service Discovery for Autobase
## Overview
autobase-discovery-cli is a command-line interface for autobase-discovery, providing tools to run discovery servers and query service information from Autobase databases. It enables decentralized service discovery by leveraging the Autobase multi-writer architecture.
### Key Features
- **Server mode**: Run a discovery service that indexes Autobase entries
- **Client queries**: List services by name from the discovery database
- **Secure RPC**: Public-key authenticated remote procedure calls
- **Real-time updates**: Database key stabilizes as indexers process entries
- **JSON logging**: Structured logs compatible with pino ecosystem
### Use Cases
- **Service registry**: Discover microservices in distributed systems
- **Peer discovery**: Find nodes in P2P networks
- **Dynamic configuration**: Query runtime service endpoints
- **Multi-tenant discovery**: Isolate services by namespace
## Architecture
```mermaid
graph TB
subgraph "Discovery Server"
CLI[autodiscovery CLI]
RPC[RPC Server<br/>Auth-required]
INDEXER[Autobase Indexer]
DB[Discovery Database]
end
subgraph "Clients"
CLIENT[autodiscovery-client]
QUERY[List Services Query]
end
subgraph "Autobase Network"
AB[Autobase Writers]
INPUT[Service Entries]
end
AB -->|Replicate| INDEXER
INPUT --> AB
INDEXER --> DB
DB --> CLI
CLI --> RPC
CLIENT --> QUERY
QUERY -->|RPC| RPC
```
## Installation
```bash
npm install -g autobase-discovery-cli
```
## Quick Start
### Run Discovery Server
```bash
# Generate a keypair for secure RPC
# (Use hypercore-sign or similar to generate keys)
autodiscovery run <rpc-allowed-public-key> | pino-pretty
```
The server will output:
- RPC server's public key
- Discovery database key (updates as indexers process)
### Query Services (Client)
```bash
# List all instances of a service
autodiscovery-client list <database-key> <service-name>
```
## CLI Reference
### Server Commands
#### `autodiscovery run <rpc-allowed-public-key>`
Start the discovery server.
**Parameters:**
- `rpc-allowed-public-key`: Public key authorized to make RPC calls
**Output:**
- RPC server public key
- Database key (stabilizes after first indexer entry)
**Logging:**
```bash
# JSON format (default)
autodiscovery run <pubkey>
# Human-readable
autodiscovery run <pubkey> | pino-pretty
```
**Note:** The database key updates as new indexers are processed. Add at least one entry to stabilize the initial database key.
### Client Commands
#### `autodiscovery-client list <database-key> <service-name>`
List all discovered instances of a service.
**Parameters:**
- `database-key`: The discovery database key
- `service-name`: Name of the service to query
**Output:**
```
service-instance-1 192.168.1.100:8080
service-instance-2 192.168.1.101:8080
```
#### `autodiscovery-client --help`
Show all available commands and options.
## Complete Examples
### Example 1: Service Discovery Setup
```bash
# 1. Generate RPC keypair (signer)
hypercore-sign generate-keys
# Note the public key: ocmjxpzg...
# 2. Start discovery server
autodiscovery run ocmjxpzghcx5gbhkky7qubn5pr4fpcxwr5mu4hjw43dqs3qhid3y | pino-pretty
# Output:
# Server public key: abc123...
# Database key: def456...
# Waiting for indexers...
# 3. Add a service entry to Autobase (via your app)
# (See autobase-discovery module for API)
# 4. Query for services
autodiscovery-client list def456... api-gateway
# Output:
# api-gateway-prod-1 10.0.1.10:3000
# api-gateway-prod-2 10.0.1.11:3000
```
### Example 2: Multi-Service Discovery
```bash
# Terminal 1: Run discovery server
export RPC_PUBKEY="ocmjxpzghcx5gbhkky7qubn5pr4fpcxwr5mu4hjw43dqs3qhid3y"
autodiscovery run $RPC_PUBKEY 2>&1 | tee discovery.log | pino-pretty
# Extract database key from logs (after first entry)
DB_KEY=$(grep "Database key" discovery.log | tail -1 | awk '{print $3}')
# Terminal 2: Query different services
autodiscovery-client list $DB_KEY user-service
autodiscovery-client list $DB_KEY payment-service
autodiscovery-client list $DB_KEY notification-service
```
### Example 3: Integration Script
```bash
#!/bin/bash
# start-discovery.sh
RPC_KEY="${RPC_KEY:-$(cat ~/.discovery-rpc-key.pub)}"
LOG_FILE="/var/log/discovery.log"
# Start server with log rotation
autodiscovery run "$RPC_KEY" >> "$LOG_FILE" 2>&1 &
PID=$!
# Wait for database key
for i in {1..30}; do
DB_KEY=$(grep -o 'Database key: [a-f0-9]*' "$LOG_FILE" | tail -1 | awk '{print $3}')
if [ -n "$DB_KEY" ]; then
echo "Discovery server ready"
echo "Database key: $DB_KEY"
echo $DB_KEY > /run/discovery-db-key
break
fi
sleep 1
done
echo $PID > /run/discovery.pid
```
### Example 4: Client Wrapper
```js
#!/usr/bin/env node
const { execSync } = require('child_process')
class DiscoveryClient {
constructor(dbKey) {
this.dbKey = dbKey
}
list(serviceName) {
try {
const output = execSync(
`autodiscovery-client list ${this.dbKey} ${serviceName}`,
{ encoding: 'utf8' }
)
return output.trim().split('\n').map(line => {
const [name, address] = line.trim().split(/\s+/)
return { name, address }
})
} catch (err) {
console.error('Query failed:', err.message)
return []
}
}
getServiceUrl(serviceName, index = 0) {
const instances = this.list(serviceName)
if (instances.length === 0) {
throw new Error(`No instances found for service: ${serviceName}`)
}
return instances[index % instances.length].address
}
}
// Usage
const client = new DiscoveryClient(process.env.DISCOVERY_DB_KEY)
const userServiceUrl = client.getServiceUrl('user-service')
console.log(`User service: http://${userServiceUrl}`)
const allGateways = client.list('api-gateway')
console.log(`Found ${allGateways.length} API gateways`)
```
## Security
### RPC Authentication
The discovery server requires a public key for RPC authentication:
```bash
# Generate keypair
hypercore-sign generate-keys
# Use public key when starting server
autodiscovery run <public-key>
# Only clients with matching secret key can query
```
### Network Security
- Run behind firewall for internal networks
- Use TLS for public deployments
- Rotate RPC keys periodically
## Integration with Autobase
### Service Entry Format
Entries written to Autobase for discovery:
```js
{
type: 'service-announcement',
name: 'user-service',
instance: 'user-service-prod-1',
address: '10.0.1.10:3000',
metadata: {
version: '2.1.0',
region: 'us-east-1',
health: 'healthy'
},
timestamp: Date.now()
}
```
### Indexer Configuration
The discovery server indexes these entries:
```js
// In your Autobase setup
const Autobase = require('autobase')
const DiscoveryIndexer = require('autobase-discovery')
const base = new Autobase(store, key, {
async apply(nodes, view, base) {
for (const node of nodes) {
if (node.value.type === 'service-announcement') {
await view.insert(node.value)
}
}
}
})
```
## Best Practices
### Database Key Management
```bash
# Wait for key stabilization
start_server() {
autodiscovery run $RPC_KEY &
SERVER_PID=$!
# Wait for database key
while [ -z "$DB_KEY" ]; do
sleep 1
DB_KEY=$(get_db_key_from_logs)
done
# Store for clients
echo $DB_KEY > /etc/discovery/db-key
}
```
### Monitoring
```bash
# Check server health
check_discovery() {
if pgrep -f "autodiscovery run" > /dev/null; then
echo "Server running"
# Test query
DB_KEY=$(cat /etc/discovery/db-key)
autodiscovery-client list $DB_KEY health-check > /dev/null && \
echo "Query OK" || echo "Query failed"
else
echo "Server not running"
fi
}
```
### Log Analysis
```bash
# Parse discovery events
cat /var/log/discovery.log | pino-pretty | grep -E "(announce|unannounce|query)"
# Monitor for errors
tail -f /var/log/discovery.log | jq 'select(.level >= 50)'
```
## Troubleshooting
### Database Key Keeps Changing
**Cause:** Not enough entries processed by indexers
**Solution:** Add at least one entry to stabilize the key
### RPC Authentication Fails
**Cause:** Wrong public key or key mismatch
**Solution:** Verify keys match between server and client
### No Services Found
**Cause:** Services not announcing or wrong database key
**Solution:** Check Autobase entries and database key
## License
Apache-2.0
---
**Module Type**: CLI Tool | **Ecosystem Role**: Service Discovery | **Dependencies**: autobase-discovery, Autobase
+139
View File
@@ -0,0 +1,139 @@
# bare-addon - Native Addon Template
## Overview
**bare-addon** is a template repository for creating Bare native addons. It provides the build infrastructure for compiling C/C++ native bindings for use with the Bare JavaScript runtime.
## Quick Start
### Creating a New Addon
1. Use this repository as a template on GitHub
2. Clone your new repository
3. Implement your native code in `binding.c`
4. Build and publish
## Building
### Install Build Tool
```bash
npm i -g bare-make
```
### Generate Build System
```bash
# Generate (run once per checkout)
bare-make generate
# Debug build
bare-make generate --debug
# Regenerate (after toolchain updates)
bare-make generate --no-cache
```
### Compile
```bash
# Build bindings
bare-make build
# Install to prebuilds/
bare-make install
# Link for development (faster iteration)
bare-make install --link
```
## Project Structure
```
my-addon/
├── binding.c # Native bindings
├── CMakeLists.txt # Build configuration
├── package.json # Package manifest
├── prebuilds/ # Compiled binaries
│ ├── darwin-arm64/
│ ├── linux-x64/
│ └── win32-x64/
└── .github/
└── workflows/
└── prebuild.yml # CI/CD workflow
```
## Adding Dependencies
### External Native Libraries
Use `cmake-fetch` to include native libraries:
```bash
npm i -D cmake-fetch
```
```cmake
# CMakeLists.txt
find_package(cmake-fetch REQUIRED PATHS node_modules/cmake-fetch)
fetch_package("github:holepunchto/liburl")
target_link_libraries(
${bare_addon}
PUBLIC
url
)
```
## Publishing
### Version Bump
```bash
npm version <increment>
git push
git push --tags
```
### Automated Prebuilds
```bash
# Trigger CI workflow
gh workflow run prebuild --ref <version>
# Watch progress
gh run watch
# Download prebuilds
gh run download --name prebuilds --dir prebuilds
```
### Manual Publish
```bash
# Verify prebuilds exist
ls prebuilds/
# Publish to npm
npm publish
```
## Troubleshooting
### Cache Issues
Check cache state:
```bash
bare --print 'Bare.Addon.cache'
```
Clear by bumping version in `package.json`.
## License
Apache-2.0
---
**Module Type**: Template | **Ecosystem Role**: Native Addon Development | **Build Tool**: bare-make
+19
View File
@@ -0,0 +1,19 @@
# bare-boot - Boot Drive Loader
## Overview
**bare-boot** boots drives in the Bare runtime. It provides the mechanism for loading and initializing Bare applications from various drive sources.
## Usage
```js
const boot = require('bare-boot')
```
## License
Apache-2.0
---
**Module Type**: Core Runtime | **Ecosystem Role**: Application Bootstrapping | **Used With**: Bare
+199
View File
@@ -0,0 +1,199 @@
# bare-build - Application Builder for Bare
## Overview
**bare-build** packages JavaScript applications as native application bundles or standalone executables for desktop and mobile platforms. It supports building for Linux, macOS, Windows, Android, and iOS with platform-specific packaging formats.
## Supported Platforms & Formats
| Platform | Unpackaged | Packaged | Standalone |
|----------|-----------|----------|------------|
| Linux | .AppDir | .AppImage | ELF executable |
| Android | .apk | .aab | ELF executable |
| macOS | .app | .pkg | Mach-O executable |
| iOS | .app | .pkg | Mach-O executable |
| Windows | Directory | .msix | PE executable |
## Quick Start
### CLI Usage
```bash
# Build for current platform
bare-build app.js
# Build for multiple platforms
bare-build \
--host darwin-arm64 \
--host darwin-x64 \
--host linux-x64 \
--icon icon.icns \
--identifier com.example.App \
app.js
# Build standalone executable
bare-build --standalone app.js
# Build with packaging
bare-build --package app.js
```
### Programmatic Usage
```js
const build = require('bare-build')
for await (const resource of build('/path/to/app.js', {
base: '/path/to/',
hosts: ['darwin-arm64', 'darwin-x64'],
icon: 'icon.icns',
identifier: 'com.example.App',
standalone: true
})) {
console.log('Built:', resource)
}
```
## Configuration Options
```js
{
// Application metadata
name: 'MyApp',
version: '1.0.0',
author: 'Author Name',
description: 'Application description',
// Application identity
identifier: 'com.example.app',
icon: 'path/to/icon',
// Build configuration
base: '.', // Base path
hosts: ['darwin-arm64'], // Target platforms
out: './dist', // Output directory
// Runtime options
runtime: 'bare-gtk/runtime', // Alternative runtime
// Build modes
standalone: false, // Standalone executable
package: false, // Package for distribution
sign: false, // Sign the application
// macOS signing
identity: 'Apple Development',
entitlements: 'path/to/entitlements',
hardenedRuntime: false,
// Windows signing
subject: 'CN=Publisher',
thumbprint: 'cert-thumbprint',
// Linux signing
key: 'gpg-key-id',
// Android signing
keystore: 'path/to/keystore',
keystorePassword: 'password'
}
```
## Runtimes
### Portable Runtimes (Default)
Suitable for CLI applications, runs the Bare I/O event loop only.
### Native Runtimes (GUI Applications)
For native GUI apps requiring system event loop integration:
| Platform | Runtime | CLI Flag |
|----------|---------|----------|
| Linux | bare-gtk | `--runtime bare-gtk/runtime` |
| Android | bare-ndk | `--runtime bare-ndk/runtime` |
| macOS | bare-app-kit | `--runtime bare-app-kit/runtime` |
| iOS | bare-ui-kit | `--runtime bare-ui-kit/runtime` |
| Windows | bare-win-ui | `--runtime bare-win-ui/runtime` |
## CLI Flags
```bash
bare-build [flags] <entry>
--name, -n Application name
--author Author name
--description Application description
--icon, -i Application icon
--identifier Unique identifier (e.g., com.example.app)
--manifest Platform-specific manifest
--resources Additional resources
--base Base path (default: .)
--host Target host (can specify multiple)
--out, -o Output directory
--runtime Runtime specifier
--standalone Build standalone executable
--package Package for distribution
--sign Sign the application
--identity macOS signing identity
--entitlements macOS entitlements file
--hardened-runtime Enable macOS hardened runtime
--subject Windows signing subject
--thumbprint Windows certificate thumbprint
--key Linux GPG signing key
--keystore Android keystore path
--keystore-password Android keystore password
```
## Examples
### Build Cross-Platform CLI Tool
```bash
bare-build \
--name "MyCLI" \
--identifier com.example.mycli \
--host darwin-arm64 \
--host darwin-x64 \
--host linux-x64 \
--host win32-x64 \
--standalone \
cli.js
```
### Build macOS GUI App
```bash
bare-build \
--name "MyApp" \
--identifier com.example.myapp \
--icon assets/icon.icns \
--runtime bare-app-kit/runtime \
--host darwin-arm64 \
--host darwin-x64 \
--package \
--sign \
--identity "Developer ID Application" \
app.js
```
### Build Android App
```bash
bare-build \
--name "MyAndroidApp" \
--identifier com.example.android \
--icon assets/icon.png \
--runtime bare-ndk/runtime \
--host android-arm64 \
--keystore release.keystore \
--keystore-password $KEYSTORE_PASSWORD \
app.js
```
## License
Apache-2.0
---
**Module Type**: Build Tool | **Ecosystem Role**: Application Packaging | **Used By**: Bare CLI
+558
View File
@@ -0,0 +1,558 @@
# bare-delta - Binary Delta Compression for Bare
## Overview
bare-delta provides binary patch creation and application for the Bare runtime. It implements an enhanced version of Fossil SCM's delta compression algorithm with SIMD acceleration and zstd compression support, enabling efficient binary diffing for incremental updates.
### Key Features
- **Binary patching**: Create minimal patches between file versions
- **Fossil SCM algorithm**: Proven delta compression with enhancements
- **SIMD acceleration**: Hardware-accelerated hash computations
- **zstd compression**: Optional patch compression for smaller sizes
- **Sync/Async APIs**: Both blocking and non-blocking interfaces
- **Batch operations**: Apply multiple patches sequentially
### Use Cases
- **Application updates**: Incremental binary patches for software distribution
- **Version control**: Efficient storage of file revisions
- **Backup systems**: Store only changed portions of files
- **Sync protocols**: Minimize data transfer for updates
- **Delta compression**: Reduce bandwidth and storage requirements
## Architecture
```mermaid
graph LR
subgraph "Patch Creation"
ORIG[Original Data]
MOD[Modified Data]
DELTA[Delta Algorithm<br/>Fossil SCM + SIMD]
PATCH[Binary Patch<br/>Optional zstd]
end
subgraph "Patch Application"
ORIG2[Original Data]
PATCH2[Patch]
APPLY[Apply Algorithm]
RESULT[Reconstructed Data]
end
ORIG --> DELTA
MOD --> DELTA
DELTA --> PATCH
ORIG2 --> APPLY
PATCH2 --> APPLY
APPLY --> RESULT
```
### Algorithm Enhancements
1. **SIMD Optimization**: Parallel hash computation using modern CPU instructions
2. **Compact Encoding**: Replaced Fossil's base-64 integers with compact-encoding
3. **zstd Integration**: Optional compression for patches
4. **Streaming Support**: Memory-efficient processing of large files
## Installation
```bash
npm install bare-delta
```
## Quick Start
### Create and Apply a Patch
```js
const { create, apply } = require('bare-delta')
const original = Buffer.from('Hello world!')
const modified = Buffer.from('Hello Bare world!')
// Create patch
const patch = await create(original, modified)
console.log(`Patch size: ${patch.length} bytes`)
// Apply patch
const result = await apply(original, patch)
console.log(result.toString()) // 'Hello Bare world!'
```
### Synchronous API
```js
const { createSync, applySync } = require('bare-delta')
const patch = createSync(original, modified)
const result = applySync(original, patch)
```
### Batch Patch Application
```js
const { applyBatch } = require('bare-delta')
// Apply multiple patches in sequence
const patches = [patch1, patch2, patch3]
const finalResult = await applyBatch(original, patches)
```
## API Reference
### Async API
#### `create(original, modified[, options])`
Creates a binary patch between two buffers.
**Parameters:**
- `original` (Buffer | Uint8Array): Original data
- `modified` (Buffer | Uint8Array): Modified data
- `options` (object, optional):
- `hashWindowSize` (number): Hash window size, must be power of 2 (default: 16)
- `searchDepth` (number): Maximum search depth for matches (default: 250)
- `compressed` (boolean): Whether to compress the patch with zstd (default: false)
**Returns:** Promise<Buffer> - The binary patch
#### `apply(original, patch)`
Applies a binary patch to reconstruct modified data. Auto-detects compression.
**Parameters:**
- `original` (Buffer | Uint8Array): Original data
- `patch` (Buffer | Uint8Array): Patch from `create()`
**Returns:** Promise<Buffer> - Reconstructed data
#### `applyBatch(original, patches)`
Applies multiple patches sequentially.
**Parameters:**
- `original` (Buffer | Uint8Array): Original data
- `patches` (Array<Buffer | Uint8Array>): Patches to apply in order
**Returns:** Promise<Buffer> - Final reconstructed data
### Sync API
#### `createSync(original, modified[, options])`
Synchronous version of `create()`.
**Returns:** Buffer
#### `applySync(original, patch)`
Synchronous version of `apply()`.
**Returns:** Buffer
#### `applyBatchSync(original, patches)`
Synchronous version of `applyBatch()`.
**Returns:** Buffer
## Complete Examples
### Example 1: File Versioning System
```js
const { createSync, applySync } = require('bare-delta')
const fs = require('bare-fs')
class VersionedFile {
constructor(basePath) {
this.basePath = basePath
this.versions = []
}
saveVersion(data) {
if (this.versions.length === 0) {
// First version: store full file
fs.writeFileSync(`${this.basePath}.v0`, data)
this.versions.push({ type: 'full', size: data.length })
} else {
// Subsequent versions: store delta
const prevVersion = this.versions.length - 1
const prevData = this.loadVersion(prevVersion)
const patch = createSync(prevData, data, { compressed: true })
fs.writeFileSync(`${this.basePath}.v${this.versions.length}.patch`, patch)
this.versions.push({
type: 'delta',
size: patch.length,
parent: prevVersion
})
}
return this.versions.length - 1
}
loadVersion(versionNum) {
if (versionNum === 0) {
return fs.readFileSync(`${this.basePath}.v0`)
}
// Reconstruct by applying deltas
let result = this.loadVersion(0)
for (let i = 1; i <= versionNum; i++) {
const patch = fs.readFileSync(`${this.basePath}.v${i}.patch`)
result = applySync(result, patch)
}
return result
}
getStorageStats() {
const fullSize = this.versions[0]?.size || 0
const deltaSizes = this.versions
.filter(v => v.type === 'delta')
.reduce((sum, v) => sum + v.size, 0)
return {
versions: this.versions.length,
fullStorage: fullSize + deltaSizes,
spaceSaved: fullSize * (this.versions.length - 1) - deltaSizes
}
}
}
// Usage
const file = new VersionedFile('/tmp/myfile')
file.saveVersion(Buffer.from('Version 1 content'))
file.saveVersion(Buffer.from('Version 2 with some changes'))
file.saveVersion(Buffer.from('Version 3 more updates here'))
console.log(file.getStorageStats())
```
### Example 2: Incremental Backup
```js
const { create, apply } = require('bare-delta')
const fs = require('bare-fs')
const path = require('bare-path')
class IncrementalBackup {
constructor(backupDir) {
this.backupDir = backupDir
this.manifest = this.loadManifest()
}
loadManifest() {
try {
return JSON.parse(fs.readFileSync(
path.join(this.backupDir, 'manifest.json'),
'utf8'
))
} catch {
return { snapshots: [] }
}
}
async createSnapshot(sourceDir, snapshotName) {
const snapshotDir = path.join(this.backupDir, snapshotName)
fs.mkdirSync(snapshotDir, { recursive: true })
const files = fs.readdirSync(sourceDir)
const lastSnapshot = this.manifest.snapshots[this.manifest.snapshots.length - 1]
for (const file of files) {
const sourcePath = path.join(sourceDir, file)
const currentData = fs.readFileSync(sourcePath)
if (lastSnapshot) {
const prevPath = path.join(this.backupDir, lastSnapshot.name, file)
if (fs.existsSync(prevPath)) {
const prevData = fs.readFileSync(prevPath)
const patch = await create(prevData, currentData, { compressed: true })
if (patch.length < currentData.length * 0.5) {
// Store delta if it's smaller than 50% of file
fs.writeFileSync(
path.join(snapshotDir, `${file}.delta`),
patch
)
continue
}
}
}
// Store full file
fs.writeFileSync(path.join(snapshotDir, file), currentData)
}
this.manifest.snapshots.push({
name: snapshotName,
timestamp: Date.now()
})
fs.writeFileSync(
path.join(this.backupDir, 'manifest.json'),
JSON.stringify(this.manifest, null, 2)
)
return snapshotName
}
async restore(snapshotName, targetDir) {
const snapshotIndex = this.manifest.snapshots.findIndex(
s => s.name === snapshotName
)
if (snapshotIndex === -1) {
throw new Error(`Snapshot ${snapshotName} not found`)
}
fs.mkdirSync(targetDir, { recursive: true })
// Reconstruct by applying all deltas up to target
const files = new Map()
for (let i = 0; i <= snapshotIndex; i++) {
const snapshot = this.manifest.snapshots[i]
const snapshotDir = path.join(this.backupDir, snapshot.name)
for (const entry of fs.readdirSync(snapshotDir)) {
if (entry.endsWith('.delta')) {
const filename = entry.slice(0, -6)
const prevData = files.get(filename) || Buffer.alloc(0)
const patch = fs.readFileSync(path.join(snapshotDir, entry))
files.set(filename, await apply(prevData, patch))
} else {
files.set(entry, fs.readFileSync(path.join(snapshotDir, entry)))
}
}
}
// Write restored files
for (const [filename, data] of files) {
fs.writeFileSync(path.join(targetDir, filename), data)
}
}
}
// Usage
const backup = new IncrementalBackup('/backups/myapp')
await backup.createSnapshot('/app/data', 'snapshot-1')
await backup.createSnapshot('/app/data', 'snapshot-2')
await backup.restore('snapshot-2', '/restore/target')
```
### Example 3: Network-Sync Protocol
```js
const { create, applyBatch } = require('bare-delta')
class DeltaSyncProtocol {
constructor() {
this.localVersions = new Map() // file -> { hash, data }
}
async generateSyncRequest(files) {
return files.map(file => ({
path: file,
hash: this.localVersions.get(file)?.hash || null
}))
}
async processSyncResponse(requests, remoteFiles) {
const patches = []
for (const req of requests) {
const remote = remoteFiles.find(f => f.path === req.path)
if (!remote) {
patches.push({ path: req.path, action: 'delete' })
} else if (req.hash !== remote.hash) {
const local = this.localVersions.get(req.path)
if (local) {
// Create delta
const patch = await create(local.data, remote.data, {
compressed: true
})
patches.push({
path: req.path,
action: 'patch',
patch: patch,
originalHash: req.hash,
newHash: remote.hash
})
} else {
// New file
patches.push({
path: req.path,
action: 'full',
data: remote.data,
hash: remote.hash
})
}
}
}
return patches
}
applyPatches(patches) {
for (const p of patches) {
switch (p.action) {
case 'delete':
this.localVersions.delete(p.path)
break
case 'full':
this.localVersions.set(p.path, {
hash: p.hash,
data: p.data
})
break
case 'patch':
const local = this.localVersions.get(p.path)
if (local && this.hash(local.data) === p.originalHash) {
const newData = await apply(local.data, p.patch)
this.localVersions.set(p.path, {
hash: p.newHash,
data: newData
})
}
break
}
}
}
hash(data) {
// Simple hash for demo
return require('bare-crypto').createHash('sha256')
.update(data)
.digest('hex')
}
}
```
## Configuration Options
### Tuning Parameters
```js
// Fast but less optimal patches
const fast = await create(original, modified, {
hashWindowSize: 32,
searchDepth: 100,
compressed: false
})
// Slow but optimal patches
const optimal = await create(original, modified, {
hashWindowSize: 8,
searchDepth: 1000,
compressed: true
})
// Balanced (defaults)
const balanced = await create(original, modified)
```
### Parameter Guide
| Parameter | Effect | Range |
|-----------|--------|-------|
| `hashWindowSize` | Larger = faster, less precise | 8-64 (power of 2) |
| `searchDepth` | Higher = better matches, slower | 100-1000 |
| `compressed` | zstd compression | true/false |
## Performance Characteristics
- **Speed**: 50-200 MB/s for patch creation (depends on options)
- **Memory**: O(n) where n is original file size
- **Patch size**: Typically 5-20% of changed data
- **Compression**: zstd adds 10-30% CPU but 20-50% size reduction
## Best Practices
### When to Use Deltas
```js
// Good: Large files with small changes
const largeFile = fs.readFileSync('database.db')
const modified = applyChanges(largeFile)
const patch = createSync(largeFile, modified)
// Patch will be small
// Bad: Completely different data
const unrelated = Buffer.from('totally different content')
const patch = createSync(largeFile, unrelated)
// Patch may be larger than modified file!
```
### Error Handling
```js
const { create, apply } = require('bare-delta')
try {
const patch = await create(original, modified)
if (patch.length >= modified.length) {
console.warn('Delta larger than full file, storing full instead')
return modified
}
return patch
} catch (err) {
console.error('Delta creation failed:', err)
// Fallback to full file
return modified
}
```
### Streaming Large Files
```js
const { create } = require('bare-delta')
// For very large files, process in chunks
async function* createDeltaStream(originalPath, modifiedPath, chunkSize = 10 * 1024 * 1024) {
const original = fs.openSync(originalPath, 'r')
const modified = fs.openSync(modifiedPath, 'r')
let offset = 0
while (true) {
const origChunk = Buffer.alloc(chunkSize)
const modChunk = Buffer.alloc(chunkSize)
const origRead = fs.readSync(original, origChunk, 0, chunkSize, offset)
const modRead = fs.readSync(modified, modChunk, 0, chunkSize, offset)
if (origRead === 0 && modRead === 0) break
const patch = await create(
origChunk.slice(0, origRead),
modChunk.slice(0, modRead)
)
yield { offset, patch }
offset += chunkSize
}
fs.closeSync(original)
fs.closeSync(modified)
}
```
## License
Apache-2.0
This project incorporates code from Fossil SCM's delta compression algorithm, licensed under BSD-2-Clause.
---
**Module Type**: Algorithm/Utility | **Ecosystem Role**: Data Compression | **Dependencies**: compact-encoding, zstd (optional)
+19
View File
@@ -0,0 +1,19 @@
# bare-dev - Development Tooling for Bare
## Overview
**bare-dev** provides development tooling for the Bare JavaScript runtime. It includes utilities for debugging, testing, and developing Bare applications.
## Usage
```js
const dev = require('bare-dev')
```
## License
Apache-2.0
---
**Module Type**: Dev Tool | **Ecosystem Role**: Development Utilities | **Part Of**: Bare Ecosystem
+541
View File
@@ -0,0 +1,541 @@
# bare-encoding - Text Encoding for Bare
## Overview
bare-encoding provides WHATWG-compliant text encoding interfaces for the Bare runtime. It implements the standard `TextEncoder` and `TextDecoder` APIs, enabling consistent string-to-binary and binary-to-string conversions across Bare applications.
### Key Features
- **WHATWG compliant**: Standard TextEncoder/TextDecoder APIs
- **UTF-8 support**: Full UTF-8 encoding and decoding
- **Streaming support**: Process data incrementally
- **Bare integration**: Optimized for Bare runtime environment
- **Lightweight**: Minimal overhead implementation
### Use Cases
- **Text processing**: Convert between strings and binary data
- **Network protocols**: Encode/decode text-based protocols
- **File I/O**: Read and write text files
- **JSON processing**: Handle text encodings in data interchange
- **Internationalization**: Support for multilingual text
## Installation
```bash
npm install bare-encoding
```
## Quick Start
### Basic Encoding
```js
const { TextEncoder, TextDecoder } = require('bare-encoding')
// Encode string to Uint8Array
const encoder = new TextEncoder()
const encoded = encoder.encode('Hello, 世界!')
console.log(encoded) // Uint8Array [72, 101, 108, 108, 111, ...]
// Decode Uint8Array to string
const decoder = new TextDecoder()
const decoded = decoder.decode(encoded)
console.log(decoded) // 'Hello, 世界!'
```
### Working with Buffers
```js
const { TextEncoder, TextDecoder } = require('bare-encoding')
// Encode to Buffer-compatible format
const text = 'Bare runtime is fast!'
const encoder = new TextEncoder()
const uint8Array = encoder.encode(text)
// Convert to Buffer if needed
const buffer = Buffer.from(uint8Array)
// Decode back
const decoder = new TextDecoder()
const result = decoder.decode(buffer)
console.log(result) // 'Bare runtime is fast!'
```
## API Reference
### TextEncoder
Standard WHATWG TextEncoder implementation.
#### `new TextEncoder()`
Create a new TextEncoder instance.
#### `encoder.encode(text)`
Encode a string to UTF-8 bytes.
**Parameters:**
- `text` (string): String to encode
**Returns:** Uint8Array - Encoded bytes
**Example:**
```js
const encoder = new TextEncoder()
const bytes = encoder.encode('Hello, World!')
// Uint8Array(13) [72, 101, 108, 108, 111, 44, 32, 87, 111, 114, 108, 100, 33]
```
#### `encoder.encodeInto(text, destination)`
Encode a string directly into a pre-allocated buffer.
**Parameters:**
- `text` (string): String to encode
- `destination` (Uint8Array): Target buffer
**Returns:** object with `read` and `written` properties
**Example:**
```js
const encoder = new TextEncoder()
const buffer = new Uint8Array(100)
const result = encoder.encodeInto('Hello', buffer)
console.log(result) // { read: 5, written: 5 }
```
### TextDecoder
Standard WHATWG TextDecoder implementation.
#### `new TextDecoder([label[, options]])`
Create a new TextDecoder instance.
**Parameters:**
- `label` (string, optional): Encoding label (default: 'utf-8')
- `options` (object, optional):
- `fatal` (boolean): Throw on invalid data (default: false)
- `ignoreBOM` (boolean): Ignore byte order marker (default: false)
#### `decoder.decode([input[, options]])`
Decode binary data to a string.
**Parameters:**
- `input` (BufferSource, optional): Data to decode
- `options` (object, optional):
- `stream` (boolean): Process in streaming mode (default: false)
**Returns:** string - Decoded text
**Example:**
```js
const decoder = new TextDecoder()
const bytes = new Uint8Array([72, 101, 108, 108, 111])
const text = decoder.decode(bytes)
console.log(text) // 'Hello'
```
## Complete Examples
### Example 1: Network Protocol Handler
```js
const { TextEncoder, TextDecoder } = require('bare-encoding')
class TextProtocol {
constructor() {
this.encoder = new TextEncoder()
this.decoder = new TextDecoder()
this.buffer = new Uint8Array(0)
}
encodeMessage(message) {
const json = JSON.stringify(message)
const payload = this.encoder.encode(json)
// Add length header
const length = new Uint8Array(4)
new DataView(length.buffer).setUint32(0, payload.length, false)
const result = new Uint8Array(length.length + payload.length)
result.set(length, 0)
result.set(payload, 4)
return result
}
decodeMessages(data) {
// Append to buffer
const newBuffer = new Uint8Array(this.buffer.length + data.length)
newBuffer.set(this.buffer, 0)
newBuffer.set(data, this.buffer.length)
this.buffer = newBuffer
const messages = []
let offset = 0
while (offset + 4 <= this.buffer.length) {
const length = new DataView(this.buffer.buffer).getUint32(offset, false)
if (offset + 4 + length > this.buffer.length) {
break // Incomplete message
}
const payload = this.buffer.slice(offset + 4, offset + 4 + length)
const text = this.decoder.decode(payload)
messages.push(JSON.parse(text))
offset += 4 + length
}
// Keep remaining incomplete data
this.buffer = this.buffer.slice(offset)
return messages
}
}
// Usage
const protocol = new TextProtocol()
const message1 = { type: 'hello', data: 'world' }
const message2 = { type: 'ping', timestamp: Date.now() }
const encoded = protocol.encodeMessage(message1)
const more = protocol.encodeMessage(message2)
const combined = new Uint8Array(encoded.length + more.length)
combined.set(encoded, 0)
combined.set(more, encoded.length)
const decoded = protocol.decodeMessages(combined)
console.log(decoded) // [ { type: 'hello', ... }, { type: 'ping', ... } ]
```
### Example 2: File Reader/Writer
```js
const { TextEncoder, TextDecoder } = require('bare-encoding')
const fs = require('bare-fs')
class TextFile {
constructor(path) {
this.path = path
this.encoder = new TextEncoder()
this.decoder = new TextDecoder()
}
write(text, encoding = 'utf-8') {
const data = this.encoder.encode(text)
fs.writeFileSync(this.path, Buffer.from(data))
}
read() {
const buffer = fs.readFileSync(this.path)
return this.decoder.decode(buffer)
}
append(text) {
const data = this.encoder.encode(text)
const existing = this.exists() ? fs.readFileSync(this.path) : Buffer.alloc(0)
const combined = Buffer.concat([existing, Buffer.from(data)])
fs.writeFileSync(this.path, combined)
}
exists() {
try {
fs.accessSync(this.path)
return true
} catch {
return false
}
}
readLines() {
const text = this.read()
return text.split('\n')
}
writeLines(lines) {
const text = lines.join('\n')
this.write(text)
}
}
// Usage
const file = new TextFile('/tmp/myfile.txt')
file.write('Hello, World!\nThis is line 2.')
console.log(file.read())
file.append('\nAppended line.')
console.log(file.readLines())
```
### Example 3: Config File Handler
```js
const { TextEncoder, TextDecoder } = require('bare-encoding')
const fs = require('bare-fs')
class ConfigFile {
constructor(path) {
this.path = path
this.encoder = new TextEncoder()
this.decoder = new TextDecoder()
this.cache = null
}
load() {
try {
const data = fs.readFileSync(this.path)
const text = this.decoder.decode(data)
this.cache = JSON.parse(text)
return this.cache
} catch (err) {
if (err.code === 'ENOENT') {
this.cache = {}
return this.cache
}
throw err
}
}
save(config) {
const text = JSON.stringify(config, null, 2)
const data = this.encoder.encode(text)
fs.writeFileSync(this.path, Buffer.from(data))
this.cache = config
}
get(key, defaultValue = undefined) {
if (!this.cache) this.load()
return this.cache[key] !== undefined ? this.cache[key] : defaultValue
}
set(key, value) {
if (!this.cache) this.load()
this.cache[key] = value
this.save(this.cache)
}
delete(key) {
if (!this.cache) this.load()
delete this.cache[key]
this.save(this.cache)
}
}
// Usage
const config = new ConfigFile('/tmp/app-config.json')
config.set('theme', 'dark')
config.set('notifications', true)
config.set('language', 'en')
console.log(config.get('theme')) // 'dark'
console.log(config.get('volume', 100)) // 100 (default)
const all = config.load()
console.log(all)
```
### Example 4: Streaming Text Processor
```js
const { TextDecoder } = require('bare-encoding')
const stream = require('bare-stream')
class StreamingTextProcessor extends stream.Transform {
constructor(options = {}) {
super(options)
this.decoder = new TextDecoder('utf-8', { fatal: false })
this.buffer = ''
this.delimiter = options.delimiter || '\n'
}
_transform(chunk, encoding, callback) {
// Decode binary chunk to text
const text = this.decoder.decode(chunk, { stream: true })
this.buffer += text
// Process complete lines
let delimiterIndex
while ((delimiterIndex = this.buffer.indexOf(this.delimiter)) !== -1) {
const line = this.buffer.slice(0, delimiterIndex)
this.buffer = this.buffer.slice(delimiterIndex + 1)
// Process line
const processed = this.processLine(line)
this.push(processed)
}
callback()
}
_flush(callback) {
// Process any remaining data
if (this.buffer.length > 0) {
const processed = this.processLine(this.buffer)
this.push(processed)
}
callback()
}
processLine(line) {
// Override in subclass for custom processing
return line.toUpperCase() + this.delimiter
}
}
// Usage
const processor = new StreamingTextProcessor()
processor.on('data', (chunk) => {
console.log('Processed:', chunk.toString())
})
// Simulate incoming data
const encoder = new (require('bare-encoding').TextEncoder)()
processor.write(encoder.encode('hello world\n'))
processor.write(encoder.encode('line two\n'))
processor.write(encoder.encode('partial'))
processor.end(encoder.encode(' completion\n'))
```
## Integration with Other Modules
### With bare-fetch
```js
const fetch = require('bare-fetch')
const { TextEncoder, TextDecoder } = require('bare-encoding')
async function fetchText(url) {
const response = await fetch(url)
const bytes = await response.bytes()
const decoder = new TextDecoder()
return decoder.decode(bytes)
}
async function postText(url, text) {
const encoder = new TextEncoder()
const body = encoder.encode(text)
return fetch(url, {
method: 'POST',
body,
headers: {
'Content-Type': 'text/plain'
}
})
}
```
### With bare-fs
```js
const fs = require('bare-fs')
const { TextEncoder, TextDecoder } = require('bare-encoding')
// Read text file
function readTextFile(path) {
const buffer = fs.readFileSync(path)
const decoder = new TextDecoder()
return decoder.decode(buffer)
}
// Write text file
function writeTextFile(path, text) {
const encoder = new TextEncoder()
const data = encoder.encode(text)
fs.writeFileSync(path, Buffer.from(data))
}
```
## Best Practices
### Reuse Encoder/Decoder Instances
```js
const { TextEncoder, TextDecoder } = require('bare-encoding')
// Good: Reuse instances
const encoder = new TextEncoder()
const decoder = new TextDecoder()
for (const text of messages) {
const bytes = encoder.encode(text) // Fast, reused instance
}
// Less optimal: Create new instances
for (const text of messages) {
const bytes = new TextEncoder().encode(text) // Slower, creates objects
}
```
### Handle Encoding Errors
```js
const { TextDecoder } = require('bare-encoding')
// Strict mode: throws on invalid data
const strict = new TextDecoder('utf-8', { fatal: true })
try {
const text = strict.decode(invalidData)
} catch (err) {
console.error('Invalid UTF-8 data:', err)
}
// Lenient mode: replaces invalid sequences
const lenient = new TextDecoder('utf-8', { fatal: false })
const text = lenient.decode(invalidData) // Uses replacement character
```
### Streaming Large Files
```js
const { TextDecoder } = require('bare-encoding')
const fs = require('bare-fs')
// Process large files in chunks
function* readTextChunks(path, chunkSize = 64 * 1024) {
const decoder = new TextDecoder('utf-8', { stream: true })
const fd = fs.openSync(path, 'r')
const buffer = Buffer.alloc(chunkSize)
try {
while (true) {
const bytesRead = fs.readSync(fd, buffer, 0, chunkSize, null)
if (bytesRead === 0) break
const chunk = buffer.slice(0, bytesRead)
const text = decoder.decode(chunk, { stream: true })
yield text
}
// Finalize
yield decoder.decode()
} finally {
fs.closeSync(fd)
}
}
// Usage
for (const chunk of readTextChunks('/tmp/bigfile.txt')) {
processChunk(chunk)
}
```
## License
Apache-2.0
---
**Module Type**: Runtime/Standard | **Ecosystem Role**: Core API | **Dependencies**: None (built-in standards)
+538
View File
@@ -0,0 +1,538 @@
# bare-env - Environment Variables for Bare
## Overview
bare-env provides environment variable support for the Bare runtime, allowing JavaScript applications to access and manage process environment variables in a way that's compatible with both Bare and Node.js environments.
### Key Features
- **Standard interface**: Works like Node.js `process.env`
- **Cross-platform**: Consistent behavior across operating systems
- **Bare optimized**: Designed for lightweight Bare runtime
- **Mutable**: Read and modify environment variables
- **Object-like API**: Familiar key-value access patterns
### Use Cases
- **Configuration**: Load settings from environment variables
- **Feature flags**: Enable/disable features via env vars
- **Secrets management**: Access API keys and credentials
- **Path management**: Work with PATH and other system variables
- **Debugging**: Enable debug modes and logging levels
## Installation
```bash
npm install bare-env
```
## Quick Start
### Basic Usage
```js
const env = require('bare-env')
// Read environment variables
console.log(env.PATH)
console.log(env.HOME)
console.log(env.NODE_ENV)
// Check if variable exists
if (env.API_KEY) {
console.log('API key is set')
}
```
### Setting Variables
```js
const env = require('bare-env')
// Set new variables
env.DEBUG = 'true'
env.MY_APP_CONFIG = '/etc/myapp'
// Variables are now available
console.log(env.DEBUG) // 'true'
```
### Type Conversion
```js
const env = require('bare-env')
// Environment variables are always strings
const port = parseInt(env.PORT, 10) || 3000
const debug = env.DEBUG === 'true'
const features = env.FEATURES ? env.FEATURES.split(',') : []
console.log({ port, debug, features })
```
## API Reference
### Module Interface
The bare-env module exports an object that behaves like `process.env`.
#### Property Access
```js
const env = require('bare-env')
// Get variable
const value = env.VARIABLE_NAME
// Set variable
env.VARIABLE_NAME = 'value'
// Check existence
const exists = 'VARIABLE_NAME' in env
// Delete variable
delete env.VARIABLE_NAME
```
### Common Environment Variables
| Variable | Description | Example |
|----------|-------------|---------|
| `PATH` | Executable search path | `/usr/local/bin:/usr/bin` |
| `HOME` | User home directory | `/home/user` |
| `USER` | Current username | `john` |
| `PWD` | Current working directory | `/home/user/project` |
| `NODE_ENV` | Application environment | `production`, `development` |
## Complete Examples
### Example 1: Configuration Loader
```js
const env = require('bare-env')
class Config {
constructor() {
this.values = {}
this.load()
}
load() {
// Required variables
this.require('APP_NAME')
this.require('APP_VERSION')
// Optional variables with defaults
this.values.port = this.getInt('PORT', 3000)
this.values.host = env.HOST || '0.0.0.0'
this.values.debug = this.getBool('DEBUG', false)
this.values.logLevel = env.LOG_LEVEL || 'info'
// Array from comma-separated string
this.values.corsOrigins = this.getArray('CORS_ORIGINS', [])
// JSON configuration
this.values.features = this.getJson('FEATURES', {})
}
require(name) {
if (!env[name]) {
throw new Error(`Required environment variable ${name} is not set`)
}
this.values[name.toLowerCase()] = env[name]
}
get(name, defaultValue = undefined) {
return env[name] !== undefined ? env[name] : defaultValue
}
getInt(name, defaultValue = 0) {
const value = env[name]
if (value === undefined) return defaultValue
const parsed = parseInt(value, 10)
if (isNaN(parsed)) {
throw new Error(`Environment variable ${name} must be a valid integer`)
}
return parsed
}
getBool(name, defaultValue = false) {
const value = env[name]
if (value === undefined) return defaultValue
return value.toLowerCase() === 'true' || value === '1'
}
getArray(name, defaultValue = []) {
const value = env[name]
if (value === undefined) return defaultValue
return value.split(',').map(s => s.trim()).filter(Boolean)
}
getJson(name, defaultValue = null) {
const value = env[name]
if (value === undefined) return defaultValue
try {
return JSON.parse(value)
} catch (err) {
throw new Error(`Environment variable ${name} must be valid JSON: ${err.message}`)
}
}
get all() {
return { ...this.values }
}
}
// Usage
const config = new Config()
// Set env vars for demo
env.APP_NAME = 'MyApp'
env.APP_VERSION = '1.0.0'
env.PORT = '8080'
env.DEBUG = 'true'
env.FEATURES = '{"auth": true, "cache": false}'
config.load()
console.log(config.all)
```
### Example 2: Feature Flags
```js
const env = require('bare-env')
class FeatureFlags {
constructor() {
this.flags = new Map()
this.load()
}
load() {
// Load all FEATURE_ prefixed variables
for (const [key, value] of Object.entries(env)) {
if (key.startsWith('FEATURE_')) {
const flagName = key.slice(8).toLowerCase()
this.flags.set(flagName, value === 'true' || value === '1')
}
}
}
isEnabled(flag) {
return this.flags.get(flag) || false
}
enable(flag) {
this.flags.set(flag, true)
env[`FEATURE_${flag.toUpperCase()}`] = 'true'
}
disable(flag) {
this.flags.set(flag, false)
env[`FEATURE_${flag.toUpperCase()}`] = 'false'
}
toggle(flag) {
const current = this.isEnabled(flag)
if (current) {
this.disable(flag)
} else {
this.enable(flag)
}
return !current
}
list() {
return Object.fromEntries(this.flags)
}
}
// Usage
const features = new FeatureFlags()
// Set flags for demo
env.FEATURE_NEW_UI = 'true'
env.FEATURE_BETA_API = 'false'
env.FEATURE_ANALYTICS = 'true'
features.load()
console.log('New UI enabled:', features.isEnabled('new_ui'))
console.log('All flags:', features.list())
features.enable('dark_mode')
console.log('Dark mode:', features.isEnabled('dark_mode'))
```
### Example 3: Secret Manager
```js
const env = require('bare-env')
class SecretManager {
constructor() {
this.secrets = new Map()
this.prefix = 'SECRET_'
}
load() {
for (const [key, value] of Object.entries(env)) {
if (key.startsWith(this.prefix)) {
const name = key.slice(this.prefix.length)
this.secrets.set(name, value)
}
}
}
get(name) {
const value = this.secrets.get(name)
if (!value) {
throw new Error(`Secret ${name} not found`)
}
return value
}
getOrNull(name) {
return this.secrets.get(name) || null
}
set(name, value) {
this.secrets.set(name, value)
env[`${this.prefix}${name}`] = value
}
has(name) {
return this.secrets.has(name)
}
require(...names) {
for (const name of names) {
if (!this.has(name)) {
throw new Error(`Required secret ${name} is not set`)
}
}
}
mask(name) {
const value = this.getOrNull(name)
if (!value) return null
if (value.length <= 8) return '****'
return value.slice(0, 4) + '****' + value.slice(-4)
}
listNames() {
return Array.from(this.secrets.keys())
}
}
// Usage
const secrets = new SecretManager()
// Set secrets for demo
env.SECRET_API_KEY = 'sk-1234567890abcdef'
env.SECRET_DATABASE_URL = 'postgres://user:pass@localhost/db'
env.SECRET_JWT_SECRET = 'my-super-secret-jwt-key'
secrets.load()
console.log('API Key:', secrets.mask('API_KEY'))
console.log('Available secrets:', secrets.listNames())
try {
secrets.require('API_KEY', 'DATABASE_URL')
console.log('All required secrets present')
} catch (err) {
console.error(err.message)
}
```
### Example 4: Path Utilities
```js
const env = require('bare-env')
const path = require('bare-path')
class PathManager {
constructor() {
this.pathSeparator = process.platform === 'win32' ? ';' : ':'
}
getPath() {
return (env.PATH || '').split(this.pathSeparator).filter(Boolean)
}
addToPath(newPath, prepend = true) {
const paths = this.getPath()
// Remove if already exists
const index = paths.indexOf(newPath)
if (index > -1) {
paths.splice(index, 1)
}
// Add to beginning or end
if (prepend) {
paths.unshift(newPath)
} else {
paths.push(newPath)
}
env.PATH = paths.join(this.pathSeparator)
}
removeFromPath(targetPath) {
const paths = this.getPath().filter(p => p !== targetPath)
env.PATH = paths.join(this.pathSeparator)
}
findInPath(command) {
const paths = this.getPath()
const extensions = process.platform === 'win32'
? (env.PATHEXT || '.EXE').split(';')
: ['']
for (const dir of paths) {
for (const ext of extensions) {
const fullPath = path.join(dir, command + ext.toLowerCase())
try {
const fs = require('bare-fs')
fs.accessSync(fullPath, fs.constants.X_OK)
return fullPath
} catch {
continue
}
}
}
return null
}
getHome() {
return env.HOME || env.USERPROFILE || '/tmp'
}
expandPath(filepath) {
if (filepath.startsWith('~/')) {
return path.join(this.getHome(), filepath.slice(2))
}
return filepath
}
}
// Usage
const paths = new PathManager()
console.log('Current PATH entries:', paths.getPath().length)
console.log('Home directory:', paths.getHome())
console.log('Expanded path:', paths.expandPath('~/documents'))
paths.addToPath('/usr/local/bin')
console.log('Added to PATH:', paths.getPath()[0])
```
## Integration with Other Modules
### With bare-fetch
```js
const fetch = require('bare-fetch')
const env = require('bare-env')
async function apiRequest(endpoint, options = {}) {
const baseUrl = env.API_BASE_URL || 'https://api.example.com'
const apiKey = env.API_KEY
const response = await fetch(`${baseUrl}${endpoint}`, {
...options,
headers: {
'Authorization': `Bearer ${apiKey}`,
'Content-Type': 'application/json',
...options.headers
}
})
return response
}
```
### With bare-fs
```js
const fs = require('bare-fs')
const env = require('bare-env')
function getDataDir() {
const home = env.HOME || '/tmp'
const dataDir = env.XDG_DATA_HOME || `${home}/.local/share`
return dataDir
}
function ensureDataDir() {
const dir = getDataDir()
if (!fs.existsSync(dir)) {
fs.mkdirSync(dir, { recursive: true })
}
return dir
}
```
## Best Practices
### Validate Required Variables
```js
const env = require('bare-env')
function validateEnv() {
const required = ['DATABASE_URL', 'JWT_SECRET', 'API_KEY']
const missing = required.filter(key => !env[key])
if (missing.length > 0) {
console.error('Missing required environment variables:', missing.join(', '))
process.exit(1)
}
}
// Run at startup
validateEnv()
```
### Use Strong Typing
```js
const env = require('bare-env')
const config = {
port: parseInt(env.PORT, 10) || 3000,
debug: env.DEBUG === 'true',
workers: parseInt(env.WORKERS, 10) || require('bare-os').availableParallelism(),
timeout: parseInt(env.TIMEOUT, 10) || 30000
}
```
### Security Considerations
```js
const env = require('bare-env')
// Never log secrets
function safeLogConfig() {
const safeEnv = { ...env }
// Mask sensitive values
for (const key of Object.keys(safeEnv)) {
if (key.includes('SECRET') || key.includes('KEY') || key.includes('PASSWORD')) {
safeEnv[key] = '***'
}
}
console.log('Environment:', safeEnv)
}
```
## License
Apache-2.0
---
**Module Type**: Runtime/Standard | **Ecosystem Role**: Core API | **Dependencies**: None
+626 -12
View File
@@ -1,26 +1,640 @@
# bare-events v2.8.2 - EventEmitter # bare-events - Event Emitters for Bare
## Overview ## Overview
**Stable** WHATWG/Node EventEmitter. on/emit/once/off. AbortSignal support. bare-events provides EventEmitter functionality for the Bare runtime, implementing the standard Node.js EventEmitter API. It enables publish-subscribe patterns, allowing objects to emit named events and listeners to respond asynchronously.
### Key Features
- **Node.js compatible**: Drop-in replacement for Node.js events module
- **Standard API**: `.on()`, `.emit()`, `.once()`, `.off()`, etc.
- **Error handling**: Special handling for 'error' events
- **Memory leak detection**: Warnings for too many listeners
- **Performance optimized**: Efficient event dispatch
- **AbortSignal support**: Integration with cancellation signals
### Use Cases
- **Asynchronous communication**: Decoupled component interactions
- **State changes**: Notify listeners of data updates
- **Progress tracking**: Emit progress during long operations
- **Plugin systems**: Allow extensions to hook into events
- **Real-time updates**: Stream data to multiple consumers
## Installation
```bash
npm install bare-events
```
## Quick Start
### Basic Usage
```js ```js
const EventEmitter = require('bare-events') const EventEmitter = require('bare-events')
const ee = new EventEmitter() const emitter = new EventEmitter()
ee.on('event', data => console.log(data))
ee.emit('event', 'hello')
ee.prependListener('pre', () => {}) // Order // Listen for events
ee.off('event', handler) emitter.on('hello', (data) => {
console.log('Received:', data)
})
// Emit events
emitter.emit('hello', 'world')
// Output: Received: world
``` ```
**Global**: ./global patches global.EventTarget? ### Once Listeners
**Async**: await ee.once('ready') ```js
const EventEmitter = require('bare-events')
**Deps**: None core, opt bare-abort-controller. const emitter = new EventEmitter()
**Use**: TCP servers, drive watches, P2P signals. // Listen only once
emitter.once('ready', () => {
console.log('Ready! (only fires once)')
})
**Source**: github/holepunchto/bare-events emitter.emit('ready') // Fires
emitter.emit('ready') // Ignored
```
### Async/Await with Once
```js
const EventEmitter = require('bare-events')
const emitter = new EventEmitter()
// Wait for event with async/await
async function waitForReady() {
await emitter.once('ready')
console.log('Now ready!')
}
waitForReady()
setTimeout(() => emitter.emit('ready'), 1000)
```
### Removing Listeners
```js
const EventEmitter = require('bare-events')
const emitter = new EventEmitter()
function handler(data) {
console.log(data)
}
emitter.on('message', handler)
emitter.emit('message', 'hello') // Fires
// Remove listener
emitter.off('message', handler)
// or: emitter.removeListener('message', handler)
emitter.emit('message', 'world') // Ignored
```
## API Reference
### EventEmitter
Main class for event handling.
#### `new EventEmitter([options])`
Create a new EventEmitter instance.
**Parameters:**
- `options` (object, optional):
- `captureRejections` (boolean): Capture promise rejections (default: false)
#### `emitter.on(eventName, listener)`
Add a listener for an event.
**Parameters:**
- `eventName` (string | symbol): Event name
- `listener` (function): Callback function
**Returns:** EventEmitter (for chaining)
#### `emitter.once(eventName, listener)`
Add a one-time listener.
**Parameters:**
- `eventName` (string | symbol): Event name
- `listener` (function): Callback function
**Returns:** EventEmitter (for chaining)
#### `emitter.off(eventName, listener)` / `emitter.removeListener(eventName, listener)`
Remove a specific listener.
**Parameters:**
- `eventName` (string | symbol): Event name
- `listener` (function): Listener to remove
**Returns:** EventEmitter (for chaining)
#### `emitter.removeAllListeners([eventName])`
Remove all listeners, or all listeners for a specific event.
**Parameters:**
- `eventName` (string | symbol, optional): Specific event to clear
**Returns:** EventEmitter (for chaining)
#### `emitter.emit(eventName[, ...args])`
Emit an event with arguments.
**Parameters:**
- `eventName` (string | symbol): Event to emit
- `...args`: Arguments passed to listeners
**Returns:** boolean - Whether any listeners were called
#### `emitter.listenerCount(eventName)`
Get the number of listeners for an event.
**Parameters:**
- `eventName` (string | symbol): Event name
**Returns:** number - Count of listeners
#### `emitter.eventNames()`
Get an array of all event names with listeners.
**Returns:** Array<string | symbol>
#### `emitter.prependListener(eventName, listener)`
Add listener to the beginning of the listeners array.
#### `emitter.prependOnceListener(eventName, listener)`
Add one-time listener to the beginning.
### Class Properties
#### `EventEmitter.defaultMaxListeners`
Default maximum number of listeners per event before warnings. Default: 10.
#### `emitter.setMaxListeners(n)`
Set the maximum listeners for this instance.
#### `emitter.getMaxListeners()`
Get the current maximum listeners setting.
## Complete Examples
### Example 1: Data Store with Events
```js
const EventEmitter = require('bare-events')
class ObservableStore extends EventEmitter {
constructor() {
super()
this.data = new Map()
}
set(key, value) {
const oldValue = this.data.get(key)
this.data.set(key, value)
this.emit('change', { key, oldValue, newValue: value })
this.emit(`change:${key}`, { oldValue, newValue: value })
if (oldValue === undefined) {
this.emit('add', { key, value })
} else {
this.emit('update', { key, oldValue, newValue: value })
}
}
get(key) {
return this.data.get(key)
}
delete(key) {
const value = this.data.get(key)
if (this.data.delete(key)) {
this.emit('delete', { key, value })
this.emit('change', { key, oldValue: value, newValue: undefined })
return true
}
return false
}
has(key) {
return this.data.has(key)
}
clear() {
const keys = Array.from(this.data.keys())
this.data.clear()
this.emit('clear', { keys })
}
keys() {
return Array.from(this.data.keys())
}
values() {
return Array.from(this.data.values())
}
entries() {
return Array.from(this.data.entries())
}
}
// Usage
const store = new ObservableStore()
store.on('add', ({ key, value }) => {
console.log(`Added: ${key} = ${value}`)
})
store.on('update', ({ key, oldValue, newValue }) => {
console.log(`Updated: ${key} from ${oldValue} to ${newValue}`)
})
store.on('change:user', ({ newValue }) => {
console.log(`User changed to: ${newValue}`)
})
store.set('user', 'alice') // Triggers 'add' and 'change'
store.set('user', 'bob') // Triggers 'update' and 'change:user'
store.set('role', 'admin') // Triggers 'add'
```
### Example 2: Job Queue
```js
const EventEmitter = require('bare-events')
class JobQueue extends EventEmitter {
constructor(concurrency = 3) {
super()
this.concurrency = concurrency
this.queue = []
this.running = new Set()
this.processed = 0
this.failed = 0
this.lastId = 0
}
add(job, priority = 0) {
const jobWrapper = {
id: ++this.lastId,
job,
priority,
added: Date.now()
}
// Insert by priority (higher first)
const index = this.queue.findIndex(j => j.priority < priority)
if (index === -1) {
this.queue.push(jobWrapper)
} else {
this.queue.splice(index, 0, jobWrapper)
}
this.emit('job:added', jobWrapper)
this.process()
return jobWrapper.id
}
async process() {
if (this.running.size >= this.concurrency || this.queue.length === 0) {
return
}
const jobWrapper = this.queue.shift()
this.running.add(jobWrapper.id)
this.emit('job:started', jobWrapper)
try {
const result = await Promise.resolve(jobWrapper.job())
this.processed++
this.emit('job:completed', jobWrapper, result)
} catch (err) {
this.failed++
this.emit('job:failed', jobWrapper, err)
} finally {
this.running.delete(jobWrapper.id)
this.process()
}
if (this.queue.length === 0 && this.running.size === 0) {
this.emit('drain')
}
}
get stats() {
return {
pending: this.queue.length,
running: this.running.size,
processed: this.processed,
failed: this.failed
}
}
clear() {
this.queue = []
this.emit('cleared')
}
}
// Usage
const queue = new JobQueue(2)
queue.on('job:added', (job) => {
console.log(`Job ${job.id} added with priority ${job.priority}`)
})
queue.on('job:completed', (job, result) => {
console.log(`Job ${job.id} completed:`, result)
})
queue.on('drain', () => {
console.log('All jobs completed!')
})
// Add some jobs
queue.add(() => Promise.resolve('data-1'), 5)
queue.add(() => Promise.resolve('data-2'), 1)
queue.add(() => Promise.resolve('data-3'), 10)
```
### Example 3: Connection Manager
```js
const EventEmitter = require('bare-events')
class ConnectionManager extends EventEmitter {
constructor(options = {}) {
super()
this.connections = new Map()
this.maxConnections = options.maxConnections || 100
this.reconnectDelay = options.reconnectDelay || 5000
this.attempts = new Map()
}
add(id, connection) {
if (this.connections.size >= this.maxConnections) {
this.emit('error', new Error('Max connections reached'))
return false
}
this.connections.set(id, connection)
this.attempts.set(id, 0)
connection.on('close', () => {
this.handleDisconnect(id)
})
connection.on('error', (err) => {
this.emit('connection:error', id, err)
})
this.emit('connection', id, connection)
return true
}
remove(id) {
const connection = this.connections.get(id)
if (connection) {
connection.close()
this.connections.delete(id)
this.attempts.delete(id)
this.emit('disconnection', id)
return true
}
return false
}
handleDisconnect(id) {
this.connections.delete(id)
this.emit('disconnection', id)
const attempts = this.attempts.get(id) || 0
if (attempts < 3) {
this.attempts.set(id, attempts + 1)
this.emit('reconnecting', id, attempts + 1)
setTimeout(() => {
this.emit('reconnect', id)
}, this.reconnectDelay * (attempts + 1))
} else {
this.attempts.delete(id)
this.emit('reconnect:failed', id)
}
}
broadcast(event, data) {
for (const [id, connection] of this.connections) {
try {
connection.send(event, data)
} catch (err) {
this.emit('broadcast:error', id, err)
}
}
}
get(id) {
return this.connections.get(id)
}
has(id) {
return this.connections.has(id)
}
get count() {
return this.connections.size
}
get ids() {
return Array.from(this.connections.keys())
}
closeAll() {
for (const [id, connection] of this.connections) {
connection.close()
}
this.connections.clear()
this.attempts.clear()
this.emit('close:all')
}
}
// Usage
const manager = new ConnectionManager({ maxConnections: 50 })
manager.on('connection', (id, conn) => {
console.log(`Client ${id} connected`)
})
manager.on('disconnection', (id) => {
console.log(`Client ${id} disconnected`)
})
manager.on('reconnecting', (id, attempt) => {
console.log(`Reconnecting ${id}, attempt ${attempt}`)
})
```
## Best Practices
### Memory Management
```js
const EventEmitter = require('bare-events')
class SafeEmitter extends EventEmitter {
constructor() {
super()
// Limit listeners to prevent memory leaks
this.setMaxListeners(20)
}
dispose() {
// Always clean up listeners
this.removeAllListeners()
}
}
// Usage with cleanup
const emitter = new SafeEmitter()
function onData(data) {
console.log(data)
}
emitter.on('data', onData)
// Later, when done
emitter.off('data', onData)
// Or
emitter.dispose()
```
### Error Handling
```js
const EventEmitter = require('bare-events')
const emitter = new EventEmitter()
// Always handle error events
emitter.on('error', (err) => {
console.error('Emitter error:', err)
})
// Or use captureRejections
const safeEmitter = new EventEmitter({ captureRejections: true })
safeEmitter.on('async', async (data) => {
// If this throws, it will emit 'error'
await processAsync(data)
})
```
### Event Naming
```js
const EventEmitter = require('bare-events')
class GoodNaming extends EventEmitter {
// Use namespacing for related events
start() {
this.emit('process:start')
// ... do work
this.emit('process:progress', 50)
// ... more work
this.emit('process:complete')
}
// Use past tense for completed actions
save(data) {
this.emit('saving', data) // About to save
// ... save
this.emit('saved', data) // Saved successfully
}
}
```
## Integration with Other Modules
### With bare-stream
```js
const EventEmitter = require('bare-events')
const { Readable } = require('bare-stream')
class EventStream extends Readable {
constructor(emitter, event) {
super()
this.emitter = emitter
this.event = event
emitter.on(event, (data) => {
this.push(data)
})
emitter.once('end', () => {
this.push(null)
})
}
_read() {
// Data pushed from event handler
}
}
```
### With AbortSignal
```js
const EventEmitter = require('bare-events')
const AbortController = require('bare-abort-controller')
const controller = new AbortController()
const signal = controller.signal
const emitter = new EventEmitter()
// Listen until aborted
emitter.on('data', handler)
signal.addEventListener('abort', () => {
emitter.off('data', handler)
})
// Later
controller.abort()
```
## License
Apache-2.0
---
**Module Type**: Runtime/Standard | **Ecosystem Role**: Core API | **Dependencies**: None
+721
View File
@@ -0,0 +1,721 @@
# bare-fetch - HTTP Client for Bare
## Overview
bare-fetch provides a WHATWG-compliant Fetch API implementation for the Bare runtime. It enables HTTP/HTTPS requests with a modern Promise-based interface, supporting streaming responses, custom headers, and various authentication methods.
### Key Features
- **WHATWG compliant**: Standard fetch() API as in browsers
- **Promise-based**: Modern async/await support
- **Streaming**: Access response body as a stream
- **Flexible**: Custom headers, methods, and body types
- **Redirects**: Automatic redirect following with tracking
- **Bare optimized**: Native integration with Bare runtime
### Use Cases
- **API clients**: Communicate with REST and GraphQL APIs
- **Data fetching**: Download files and resources
- **Webhooks**: Send and receive HTTP callbacks
- **Service mesh**: Inter-service communication
- **Proxying**: Forward requests to other services
## Installation
```bash
npm install bare-fetch
```
## Quick Start
### Basic GET Request
```js
const fetch = require('bare-fetch')
const response = await fetch('https://api.example.com/data')
const data = await response.json()
console.log(data)
```
### POST with JSON Body
```js
const fetch = require('bare-fetch')
const response = await fetch('https://api.example.com/users', {
method: 'POST',
headers: {
'Content-Type': 'application/json'
},
body: JSON.stringify({
name: 'John Doe',
email: '[email protected]'
})
})
const user = await response.json()
```
### Error Handling
```js
const fetch = require('bare-fetch')
try {
const response = await fetch('https://api.example.com/data')
if (!response.ok) {
throw new Error(`HTTP ${response.status}: ${response.statusText}`)
}
const data = await response.json()
console.log(data)
} catch (err) {
console.error('Request failed:', err.message)
}
```
## API Reference
### fetch(url[, options])
Perform an HTTP request.
**Parameters:**
- `url` (string | URL): Request URL
- `options` (object, optional):
- `method` (string): HTTP method (default: 'GET')
- `headers` (object | Headers): Request headers
- `body` (string | Buffer | Uint8Array | ReadableStream): Request body
- `redirect` (string): 'follow', 'error', or 'manual' (default: 'follow')
**Returns:** Promise<Response>
### Response Object
#### Properties
- `status` (number): HTTP status code (e.g., 200, 404)
- `statusText` (string): HTTP status text (e.g., 'OK', 'Not Found')
- `ok` (boolean): True if status is 200-299
- `redirected` (boolean): Whether request was redirected
- `headers` (Headers): Response headers
- `body` (ReadableStream | null): Response body stream
- `bodyUsed` (boolean): Whether body has been consumed
#### Methods
##### `response.text()`
Read response as UTF-8 text.
**Returns:** Promise<string>
##### `response.json()`
Parse response as JSON.
**Returns:** Promise<object>
##### `response.arrayBuffer()`
Read response as ArrayBuffer.
**Returns:** Promise<ArrayBuffer>
##### `response.bytes()`
Read response as Uint8Array.
**Returns:** Promise<Uint8Array>
##### `response.buffer()`
Read response as Buffer.
**Returns:** Promise<Buffer>
## Complete Examples
### Example 1: REST API Client
```js
const fetch = require('bare-fetch')
class RESTClient {
constructor(baseURL, options = {}) {
this.baseURL = baseURL
this.defaultHeaders = {
'Content-Type': 'application/json',
...options.headers
}
this.authToken = options.authToken
}
async request(path, options = {}) {
const url = `${this.baseURL}${path}`
const headers = {
...this.defaultHeaders,
...options.headers
}
if (this.authToken) {
headers['Authorization'] = `Bearer ${this.authToken}`
}
const response = await fetch(url, {
...options,
headers
})
if (!response.ok) {
const error = new Error(`HTTP ${response.status}: ${response.statusText}`)
error.status = response.status
error.response = response
throw error
}
return response
}
async get(path, options = {}) {
const response = await this.request(path, { ...options, method: 'GET' })
return response.json()
}
async post(path, body, options = {}) {
const response = await this.request(path, {
...options,
method: 'POST',
body: JSON.stringify(body)
})
return response.json()
}
async put(path, body, options = {}) {
const response = await this.request(path, {
...options,
method: 'PUT',
body: JSON.stringify(body)
})
return response.json()
}
async delete(path, options = {}) {
const response = await this.request(path, { ...options, method: 'DELETE' })
if (response.status === 204) {
return null
}
return response.json()
}
async patch(path, body, options = {}) {
const response = await this.request(path, {
...options,
method: 'PATCH',
body: JSON.stringify(body)
})
return response.json()
}
}
// Usage
const api = new RESTClient('https://api.example.com/v1', {
authToken: 'my-secret-token'
})
// GET users
const users = await api.get('/users')
console.log(users)
// POST new user
const newUser = await api.post('/users', {
name: 'Alice',
email: '[email protected]'
})
// PUT update
const updated = await api.put('/users/123', {
name: 'Alice Smith'
})
// DELETE
await api.delete('/users/123')
```
### Example 2: File Downloader with Progress
```js
const fetch = require('bare-fetch')
const fs = require('bare-fs')
class FileDownloader {
constructor() {
this.downloads = new Map()
}
async download(url, destPath, options = {}) {
const response = await fetch(url)
if (!response.ok) {
throw new Error(`Download failed: ${response.status}`)
}
const totalSize = parseInt(response.headers.get('content-length'), 10) || 0
let downloaded = 0
const writeStream = fs.createWriteStream(destPath)
const reader = response.body.getReader()
const download = {
url,
destPath,
totalSize,
downloaded: 0,
startTime: Date.now(),
status: 'downloading'
}
this.downloads.set(url, download)
try {
while (true) {
const { done, value } = await reader.read()
if (done) break
writeStream.write(Buffer.from(value))
downloaded += value.length
download.downloaded = downloaded
if (options.onProgress) {
const progress = totalSize ? (downloaded / totalSize) * 100 : 0
const speed = downloaded / ((Date.now() - download.startTime) / 1000)
options.onProgress({
url,
downloaded,
totalSize,
progress,
speed
})
}
}
writeStream.end()
download.status = 'completed'
return {
path: destPath,
size: downloaded,
duration: Date.now() - download.startTime
}
} catch (err) {
download.status = 'failed'
download.error = err.message
throw err
} finally {
if (options.keepTracking) {
// Keep in map for status checks
} else {
this.downloads.delete(url)
}
}
}
getStatus(url) {
return this.downloads.get(url)
}
getAllStatuses() {
return Array.from(this.downloads.entries()).map(([url, info]) => ({
url,
...info
}))
}
}
// Usage
const downloader = new FileDownloader()
await downloader.download(
'https://example.com/large-file.zip',
'/tmp/download.zip',
{
onProgress: ({ progress, speed }) => {
console.log(`Progress: ${progress.toFixed(1)}% (${(speed / 1024).toFixed(1)} KB/s)`)
}
}
)
```
### Example 3: Webhook Handler
```js
const fetch = require('bare-fetch')
class WebhookClient {
constructor(options = {}) {
this.timeout = options.timeout || 30000
this.retries = options.retries || 3
this.retryDelay = options.retryDelay || 1000
this.signatureSecret = options.signatureSecret
}
async send(url, payload, options = {}) {
const headers = {
'Content-Type': 'application/json',
'User-Agent': 'WebhookClient/1.0',
...options.headers
}
// Add signature if secret is configured
if (this.signatureSecret) {
const crypto = require('bare-crypto')
const signature = crypto
.createHmac('sha256', this.signatureSecret)
.update(JSON.stringify(payload))
.digest('hex')
headers['X-Webhook-Signature'] = `sha256=${signature}`
}
const body = JSON.stringify(payload)
let lastError
for (let attempt = 1; attempt <= this.retries; attempt++) {
try {
const controller = new AbortController()
const timeoutId = setTimeout(() => controller.abort(), this.timeout)
const response = await fetch(url, {
method: 'POST',
headers,
body,
signal: controller.signal
})
clearTimeout(timeoutId)
if (response.ok) {
return {
success: true,
status: response.status,
attempt
}
}
lastError = new Error(`HTTP ${response.status}: ${response.statusText}`)
// Don't retry 4xx errors (client errors)
if (response.status >= 400 && response.status < 500) {
throw lastError
}
} catch (err) {
lastError = err
if (attempt === this.retries) {
throw new Error(`Failed after ${this.retries} attempts: ${err.message}`)
}
// Wait before retry
await this.sleep(this.retryDelay * attempt)
}
}
throw lastError
}
sleep(ms) {
return new Promise(resolve => setTimeout(resolve, ms))
}
async broadcast(urls, payload) {
const results = await Promise.allSettled(
urls.map(url => this.send(url, payload))
)
return results.map((result, index) => ({
url: urls[index],
success: result.status === 'fulfilled',
result: result.status === 'fulfilled' ? result.value : result.reason
}))
}
}
// Usage
const webhook = new WebhookClient({
signatureSecret: 'my-secret',
retries: 3,
timeout: 10000
})
// Send to single endpoint
await webhook.send('https://partner.com/webhook', {
event: 'order.created',
orderId: '12345',
timestamp: Date.now()
})
// Broadcast to multiple
const results = await webhook.broadcast([
'https://service1.com/webhook',
'https://service2.com/webhook',
'https://service3.com/webhook'
], {
event: 'user.signup',
userId: 'user-123'
})
console.log(results)
```
### Example 4: Health Check Monitor
```js
const fetch = require('bare-fetch')
class HealthMonitor {
constructor(options = {}) {
this.interval = options.interval || 60000
this.timeout = options.timeout || 5000
this.services = new Map()
this.statuses = new Map()
}
addService(name, url, options = {}) {
this.services.set(name, {
url,
method: options.method || 'GET',
expectedStatus: options.expectedStatus || 200,
headers: options.headers || {}
})
}
removeService(name) {
this.services.delete(name)
this.statuses.delete(name)
}
async checkService(name) {
const service = this.services.get(name)
if (!service) {
throw new Error(`Unknown service: ${name}`)
}
const startTime = Date.now()
try {
const controller = new AbortController()
const timeoutId = setTimeout(() => controller.abort(), this.timeout)
const response = await fetch(service.url, {
method: service.method,
headers: service.headers,
signal: controller.signal
})
clearTimeout(timeoutId)
const latency = Date.now() - startTime
const healthy = response.status === service.expectedStatus
const status = {
name,
healthy,
status: response.status,
latency,
timestamp: Date.now()
}
this.statuses.set(name, status)
return status
} catch (err) {
const status = {
name,
healthy: false,
error: err.message,
latency: Date.now() - startTime,
timestamp: Date.now()
}
this.statuses.set(name, status)
return status
}
}
async checkAll() {
const results = await Promise.all(
Array.from(this.services.keys()).map(name => this.checkService(name))
)
return results
}
start() {
this.stop()
this.intervalId = setInterval(() => {
this.checkAll().catch(console.error)
}, this.interval)
}
stop() {
if (this.intervalId) {
clearInterval(this.intervalId)
this.intervalId = null
}
}
getStatus(name) {
return this.statuses.get(name)
}
getAllStatuses() {
return Array.from(this.statuses.entries()).map(([name, status]) => ({
name,
...status
}))
}
getHealthyServices() {
return this.getAllStatuses().filter(s => s.healthy)
}
getUnhealthyServices() {
return this.getAllStatuses().filter(s => !s.healthy)
}
}
// Usage
const monitor = new HealthMonitor({
interval: 30000, // Check every 30 seconds
timeout: 5000 // 5 second timeout
})
monitor.addService('api', 'https://api.example.com/health')
monitor.addService('database', 'https://db.example.com/status')
monitor.addService('cache', 'https://cache.example.com/ping')
monitor.start()
// Check status
setInterval(() => {
const unhealthy = monitor.getUnhealthyServices()
if (unhealthy.length > 0) {
console.error('Unhealthy services:', unhealthy)
}
}, 60000)
```
## Best Practices
### Always Check Response Status
```js
const fetch = require('bare-fetch')
// Bad: Assumes success
const response = await fetch(url)
const data = await response.json() // May throw on error
// Good: Check status
const response = await fetch(url)
if (!response.ok) {
throw new Error(`Request failed: ${response.status}`)
}
const data = await response.json()
```
### Set Timeouts
```js
const fetch = require('bare-fetch')
const controller = new AbortController()
const timeoutId = setTimeout(() => controller.abort(), 10000)
try {
const response = await fetch(url, { signal: controller.signal })
clearTimeout(timeoutId)
// ... process response
} catch (err) {
if (err.name === 'AbortError') {
console.error('Request timed out')
}
throw err
}
```
### Handle Redirects Appropriately
```js
const fetch = require('bare-fetch')
// Follow redirects (default)
const response = await fetch(url, { redirect: 'follow' })
console.log('Redirected:', response.redirected)
// Don't follow redirects
const response = await fetch(url, { redirect: 'manual' })
if (response.status >= 300 && response.status < 400) {
const location = response.headers.get('location')
console.log('Redirect to:', location)
}
```
## Integration with Other Modules
### With bare-env
```js
const fetch = require('bare-fetch')
const env = require('bare-env')
const apiClient = {
async request(path, options = {}) {
const baseURL = env.API_BASE_URL || 'https://api.example.com'
const apiKey = env.API_KEY
return fetch(`${baseURL}${path}`, {
...options,
headers: {
'Authorization': `Bearer ${apiKey}`,
...options.headers
}
})
}
}
```
### With bare-stream
```js
const fetch = require('bare-fetch')
const { PassThrough } = require('bare-stream')
async function streamToFile(url, filePath) {
const response = await fetch(url)
const fileStream = require('bare-fs').createWriteStream(filePath)
// Pipe response to file
const stream = response.body
stream.pipe(fileStream)
return new Promise((resolve, reject) => {
fileStream.on('finish', resolve)
fileStream.on('error', reject)
})
}
```
## License
Apache-2.0
---
**Module Type**: Runtime/Standard | **Ecosystem Role**: Network API | **Dependencies**: None
+65
View File
@@ -0,0 +1,65 @@
# bare-timers - Native Timers
## Overview
**bare-timers** provides native timer implementations for JavaScript in the Bare runtime. It offers setTimeout, setInterval, and related functions optimized for performance.
## Usage
```js
const {
setTimeout,
clearTimeout,
setInterval,
clearInterval,
setImmediate,
clearImmediate
} = require('bare-timers')
// Timeout
const timeout = setTimeout(() => {
console.log('Delayed execution')
}, 1000)
clearTimeout(timeout)
// Interval
const interval = setInterval(() => {
console.log('Recurring execution')
}, 1000)
clearInterval(interval)
// Immediate
setImmediate(() => {
console.log('Next tick')
})
```
## API
### setTimeout(callback, delay, ...args)
Schedule a one-time callback after delay milliseconds.
### clearTimeout(timeout)
Cancel a scheduled timeout.
### setInterval(callback, delay, ...args)
Schedule recurring callbacks every delay milliseconds.
### clearInterval(interval)
Cancel a scheduled interval.
### setImmediate(callback, ...args)
Schedule callback for immediate execution after I/O events.
### clearImmediate(immediate)
Cancel a scheduled immediate.
## License
Apache-2.0
---
**Module Type**: Core Runtime | **Ecosystem Role**: Timer Implementation | **Part Of**: Bare Runtime
+153
View File
@@ -0,0 +1,153 @@
# boot-drive - Drive Application Loader
## Overview
**boot-drive** runs applications from Hyperdrive or Localdrive. It provides a bootloader that prepares and executes code from drives, supporting both dynamic execution and bundling for deployment.
## Usage
### Basic Usage
```js
const Boot = require('boot-drive')
// Create bootloader
const boot = new Boot(drive)
// Prepare the drive
await boot.warmup()
// Start the application
const exported = boot.start()
console.log(exported)
```
### Bundle for Distribution
```js
// Create standalone bundle
const source = boot.stringify()
// Save to file or eval directly
fs.writeFileSync('bundle.js', source)
// or
eval(source)
```
## API Reference
### new Boot(drive, [options])
Creates a bootloader instance.
```js
const boot = new Boot(drive, {
entrypoint: 'index.js', // Main file to run
cwd: '.', // Working directory for prebuilds
absolutePrebuilds: false, // Use absolute paths for prebuilds
cache: {}, // Shared require.cache
dependencies: new Map(), // Shared linker dependencies
additionalBuiltins: [], // Additional builtin modules
builtinsMap: { // Core module mappings
fs: 'bare-fs',
path: 'bare-path'
},
sourceOverwrites: {}, // Override source files
host: 'darwin-arm64', // Target platform
platform: process.platform, // Legacy
arch: process.arch // Legacy
})
```
### boot.warmup([entrypoint])
Prepares the drive for execution.
- Resolves entrypoint from `package.json` main or defaults to `index.js`
- Links dependencies
- Loads prebuilds
### boot.start([entrypoint])
Runs the drive application.
- Forces `absolutePrebuilds: true`
- Returns the module exports
### boot.stringify([entrypoint])
Bundles the drive into a string.
- Includes all dependencies
- Embeds source code
- Can be saved to file or evaluated
## Complete Example
```js
const Boot = require('boot-drive')
const Hyperdrive = require('hyperdrive')
const Corestore = require('corestore')
async function runApp(driveKey) {
const store = new Corestore('./storage')
const drive = new Hyperdrive(store, driveKey)
await drive.ready()
const boot = new Boot(drive, {
entrypoint: 'app.js',
builtinsMap: {
fs: 'bare-fs',
path: 'bare-path',
os: 'bare-os'
}
})
await boot.warmup()
const app = boot.start()
if (app.run) {
await app.run()
}
return app
}
// Or bundle for offline use
async function bundleApp(driveKey) {
const store = new Corestore('./storage')
const drive = new Hyperdrive(store, driveKey)
await drive.ready()
const boot = new Boot(drive)
await boot.warmup()
const bundle = boot.stringify()
fs.writeFileSync('app-bundle.js', bundle)
return bundle
}
```
## Prebuilds
Native modules are loaded from `prebuilds/`:
```
prebuilds/
├── darwin-arm64/
│ └── native-addon.node
├── linux-x64/
│ └── native-addon.node
└── win32-x64/
└── native-addon.node
```
## License
Apache-2.0
---
**Module Type**: Runtime | **Ecosystem Role**: Drive Application Loader | **Works With**: Hyperdrive, Localdrive
+62
View File
@@ -0,0 +1,62 @@
# cmake-fetch - CMake Package Manager
## Overview
**cmake-fetch** is a minimal package manager for CMake based on `FetchContent`. It simplifies fetching external dependencies in CMake-based projects.
## Installation
```bash
npm i cmake-fetch
```
## CMake Integration
```cmake
find_package(cmake-fetch REQUIRED PATHS node_modules/cmake-fetch)
```
## API
### parse_fetch_specifier(specifier target args)
Parse a package specifier into components.
### fetch_package(specifier [options])
Fetch a package from various sources.
```cmake
fetch_package(<specifier>
[SOURCE_SUBDIR <path>]
[SOURCE_DIR <var>]
[BINARY_DIR <var>]
[PATCHES <path...>]
)
```
## Specifier Format
```
github:user/repo@version
gitlab:user/repo@version
url:https://example.com/lib.zip
```
## Example
```cmake
find_package(cmake-fetch REQUIRED PATHS node_modules/cmake-fetch)
fetch_package("github:holepunchto/liburl")
target_link_libraries(mytarget PUBLIC url)
```
## License
Apache-2.0
---
**Module Type**: Build Tool | **Ecosystem Role**: CMake Package Management | **Build System**: CMake
+427 -11
View File
@@ -1,20 +1,436 @@
# hrpc - Holepunch RPC # hrpc - Holepunch RPC Framework
## Overview ## Overview
Protomux-based RPC over swarm conns. Modular/service. hrpc is an append-only API definition and code generation framework for building type-safe RPC systems over Hypercore streams. It provides a schema-first approach to defining request/response interfaces with optional streaming support, automatically generating consistent RPC code that can be shared across services.
**Ex**: ### Key Features
```js - **Schema-first API definition**: Define request/response interfaces using Hyperschema
const hrpc = require('hrpc') - **Auto-generated RPC code**: Consistent, type-safe code generation to disk
const server = hrpc.createServer() - **Streaming support**: Handle both simple requests and complex duplex streams
server.register('echo', {echo(req) { return req }}) - **Send-only commands**: Fire-and-forget messaging patterns
server.pipe(conn).pipe(server) - **Schema consistency**: Enforces version consistency to prevent breaking changes
- **Protomux integration**: Built on top of Protomux for multiplexed streams
### Ecosystem Role
hrpc serves as the RPC foundation for the Holepunch ecosystem, providing standardized communication patterns for distributed applications. It's used by higher-level frameworks like Pear and Hypercore services to define consistent APIs.
## Architecture
```mermaid
graph TB
subgraph "Definition Phase"
HS[Hyperschema<br/>Type Definitions]
HRPC[HRPC Builder<br/>API Definitions]
end
subgraph "Generation Phase"
GEN[Code Generator<br/>Creates RPC Classes]
DISK[Generated Files<br/>spec/hrpc/]
end
subgraph "Runtime Phase"
CLIENT[RPC Client<br/>Method Calls]
SERVER[RPC Server<br/>Request Handlers]
STREAM[Stream Transport<br/>Protomux/Hypercore]
end
HS --> HRPC
HRPC --> GEN
GEN --> DISK
DISK --> CLIENT
DISK --> SERVER
CLIENT --> STREAM
SERVER --> STREAM
STREAM -.->|duplex| CLIENT
STREAM -.->|responses| SERVER
``` ```
**Modular**: proto defs, pools. ## Installation
**Stack**: hyperswarm → protomux → hrpc. ```bash
npm install hrpc
```
**Source**: github/holepunchto/hrpc ## Quick Start
### 1. Define Your Schema
```js
const HRPCBuilder = require('hrpc')
const Hyperschema = require('hyperschema')
const path = require('path')
const SCHEMA_DIR = path.join(__dirname, 'spec', 'hyperschema')
const HRPC_DIR = path.join(__dirname, 'spec', 'hrpc')
// Define your types using Hyperschema
const schema = Hyperschema.from(SCHEMA_DIR)
const schemaNs = schema.namespace('example')
schemaNs.register({
name: 'echo-request',
fields: [{ name: 'message', type: 'string' }]
})
schemaNs.register({
name: 'echo-response',
fields: [{ name: 'echo', type: 'string' }]
})
Hyperschema.toDisk(schema)
// Define your RPC interface
const builder = HRPCBuilder.from(SCHEMA_DIR, HRPC_DIR)
const ns = builder.namespace('example')
ns.register({
name: 'echo',
request: { name: '@example/echo-request', stream: false },
response: { name: '@example/echo-response', stream: false }
})
HRPCBuilder.toDisk(builder)
```
### 2. Use the Generated RPC
```js
const { PassThrough } = require('bare-stream')
const HRPC = require('./spec/hrpc') // Auto-generated
const stream = new PassThrough()
const rpc = new HRPC(stream)
// Server-side handler
rpc.onEcho((data) => {
return { echo: `Echo: ${data.message}` }
})
// Client-side call
const res = await rpc.echo({ message: 'Hello, World!' })
console.log(res) // => { echo: 'Echo: Hello, World!' }
```
## API Reference
### HRPCBuilder
The main class for defining RPC interfaces.
#### `HRPCBuilder.from(schemaDir, hrpcDir)`
Load or create a new HRPC builder from disk.
**Parameters:**
- `schemaDir` (string): Path to Hyperschema definitions
- `hrpcDir` (string): Path to store HRPC definitions
**Returns:** `HRPCBuilder` instance
#### `builder.namespace(name)`
Get or create a namespace for organizing commands.
**Parameters:**
- `name` (string): Namespace identifier
**Returns:** `Namespace` instance
#### `HRPCBuilder.toDisk(builder)`
Persist the builder configuration to disk.
**Parameters:**
- `builder` (HRPCBuilder): Builder instance to save
### Namespace
Container for related RPC commands.
#### `ns.register(options)`
Register a new RPC command.
**Parameters:**
- `options.name` (string): Command name
- `options.request` (object): Request type configuration
- `name` (string): Full type name (e.g., '@namespace/type-name')
- `stream` (boolean): Whether request is streaming
- `send` (boolean): For send-only commands
- `options.response` (object): Response type configuration (optional for send-only)
- `name` (string): Full type name
- `stream` (boolean): Whether response is streaming
### Generated RPC Class
Auto-generated class for runtime RPC operations.
#### `new HRPC(stream)`
Create RPC instance over a duplex stream.
**Parameters:**
- `stream` (Duplex): Any duplex stream (Protomux, bare-stream, etc.)
#### `rpc.onCommand(handler)`
Register server-side handler for a command.
**Parameters:**
- `handler` (function): Handler function
- For simple: `(data) => response`
- For streaming: `(stream) => void`
#### `rpc.command(data)`
Execute client-side command.
**Parameters:**
- `data` (object): Request data
**Returns:** Promise resolving to response (or duplex stream for streaming)
## Complete Examples
### Example 1: Simple Request-Response
```js
const HRPCBuilder = require('hrpc')
const Hyperschema = require('hyperschema')
const path = require('path')
// Define schema
const schema = Hyperschema.from('./spec/hyperschema')
const ns = schema.namespace('calculator')
ns.register({
name: 'add-request',
fields: [
{ name: 'a', type: 'uint' },
{ name: 'b', type: 'uint' }
]
})
ns.register({
name: 'add-response',
fields: [{ name: 'result', type: 'uint' }]
})
Hyperschema.toDisk(schema)
// Define RPC
const builder = HRPCBuilder.from('./spec/hyperschema', './spec/hrpc')
const rpcNs = builder.namespace('calculator')
rpcNs.register({
name: 'add',
request: { name: '@calculator/add-request', stream: false },
response: { name: '@calculator/add-response', stream: false }
})
HRPCBuilder.toDisk(builder)
```
```js
// Server
const HRPC = require('./spec/hrpc')
const rpc = new HRPC(stream)
rpc.onAdd(({ a, b }) => {
return { result: a + b }
})
// Client
const HRPC = require('./spec/hrpc')
const rpc = new HRPC(stream)
const { result } = await rpc.add({ a: 5, b: 3 })
console.log(result) // 8
```
### Example 2: Streaming Commands
```js
// Define duplex streaming command
rpcNs.register({
name: 'chat',
request: { name: '@chat/message', stream: true },
response: { name: '@chat/message', stream: true }
})
// Server-side duplex handler
rpc.onChat((stream) => {
stream.on('data', (msg) => {
console.log(`Received: ${msg.text}`)
stream.write({ text: `Server: Got "${msg.text}"` })
})
})
// Client-side duplex usage
const chat = rpc.chat()
chat.write({ text: 'Hello!' })
chat.on('data', (msg) => {
console.log(msg.text)
})
```
### Example 3: Send-Only Commands
```js
// Define send-only command (fire and forget)
rpcNs.register({
name: 'log',
request: { name: '@logging/log-entry', send: true }
})
// Client just sends, no waiting for response
rpc.log({ level: 'info', message: 'Something happened' })
// Server handles without returning
rpc.onLog((entry) => {
console.log(`[${entry.level}] ${entry.message}`)
})
```
### Example 4: Multi-Service Architecture
```js
// users.hrpc
const usersNs = builder.namespace('users')
usersNs.register({
name: 'getUser',
request: { name: '@users/get-request', stream: false },
response: { name: '@users/user', stream: false }
})
// posts.hrpc
const postsNs = builder.namespace('posts')
postsNs.register({
name: 'createPost',
request: { name: '@posts/create-request', stream: false },
response: { name: '@posts/post', stream: false }
})
// Usage
const UsersRPC = require('./spec/users')
const PostsRPC = require('./spec/posts')
const userRpc = new UsersRPC(userStream)
const postRpc = new PostsRPC(postStream)
const user = await userRpc.getUser({ id: 123 })
const post = await postRpc.createPost({
userId: user.id,
content: 'Hello!'
})
```
## Best Practices
### Schema Organization
```
project/
├── spec/
│ ├── hyperschema/ # Type definitions
│ │ ├── users.json
│ │ └── posts.json
│ └── hrpc/ # RPC definitions
│ ├── users.json
│ └── posts.json
└── src/
├── rpc/
│ └── index.js # Generated RPC imports
```
### Naming Conventions
- Use descriptive command names: `getUser`, not `get`
- Namespace by domain: `@users/`, `@posts/`, `@auth/`
- Suffix request/response types: `*-request`, `*-response`
- Keep namespaces globally unique to avoid collisions
### Error Handling
```js
rpc.onCommand((data) => {
try {
const result = processData(data)
return { success: true, data: result }
} catch (err) {
return { success: false, error: err.message }
}
})
```
### Schema Versioning
hrpc enforces consistency by throwing when schemas change:
```js
// This will throw if already registered with different config
ns.register({
name: 'command',
request: { name: '@example/request', stream: true } // Changed!
})
// Error: Schema already registered with different configuration
```
To evolve APIs:
1. Create new commands with version suffixes: `getUserV2`
2. Or use new namespaces: `@users-v2/`
3. Maintain backward compatibility during transitions
## Performance Considerations
- **Code generation**: Run at build time, not runtime
- **Stream reuse**: Keep RPC connections alive for multiple calls
- **Batching**: For high-frequency operations, consider streaming
- **Binary encoding**: Leverages Hyperschema's efficient binary format
## Security Considerations
- Validate all input data on server-side
- Use Protomux encryption for sensitive data
- Consider authentication at the stream level before RPC
- Be cautious with streaming commands - implement rate limiting
## Integration with Other Modules
### With Hyperswarm
```js
const Hyperswarm = require('hyperswarm')
const HRPC = require('./spec/hrpc')
const swarm = new Hyperswarm()
swarm.on('connection', (conn) => {
const rpc = new HRPC(conn)
rpc.onGetData((req) => {
return fetchFromHypercore(req.key)
})
})
```
### With Protomux
```js
const Protomux = require('protomux')
const HRPC = require('./spec/hrpc')
const mux = new Protomux(stream)
const rpc = new HRPC(mux.createChannel({ protocol: 'my-rpc' }))
```
## License
Apache-2.0
---
**Module Type**: RPC Framework | **Ecosystem Role**: Communication Infrastructure | **Dependencies**: Hyperschema, Protomux
File diff suppressed because one or more lines are too long
+184
View File
@@ -0,0 +1,184 @@
# hypercore-encryption - Dynamic Encryption Provider
## Overview
**hypercore-encryption** provides dynamic encryption for Hypercore, allowing blocks to be encrypted with different keys over time. This enables advanced use cases like key rotation and per-block encryption schemes.
## Usage
### Basic Setup
```js
const HypercoreEncryption = require('hypercore-encryption')
const Hypercore = require('hypercore')
// Key provider function
const getEncryptionKey = async (id) => {
// Fetch key for encryption ID
return {
id, // Encryption scheme ID
encryptionKey // The encryption key
}
}
// Create encryption provider
const encryption = new HypercoreEncryption(getEncryptionKey)
// Create encrypted core
const core = new Hypercore(storage, {
encryption: encryption.createEncryptionProvider({
transform(ctx, entropy, compat) {
return {
block: deriveBlockKey(entropy),
hash: deriveHashKey(entropy)
}
}
})
})
await core.ready()
await core.append('encrypted data')
```
## API Reference
### new HypercoreEncryption(getEncryptionKey)
Creates an encryption provider.
```js
const encryption = new HypercoreEncryption(async (id) => {
// id is the encryption scheme ID
// if id === -1, return the latest key
return {
id: encryptionId,
encryptionKey: keyBuffer
}
})
```
### encryption.createEncryptionProvider(options)
Creates a provider for use with Hypercore.
```js
const provider = encryption.createEncryptionProvider({
transform(ctx, entropy, compat) {
// ctx: { id, index }
// entropy: Random bytes for key derivation
// compat: True if compat key expected
return {
block: blockEncryptionKey,
hash: hashKey, // Optional for compat
blinding: blindingKey // Required for compat
}
},
compat(ctx, index) {
// Return true if compat encryption needed
return index < legacyCutoff
}
})
```
### encryption.clear()
Clears cached encryption keys.
```js
encryption.clear()
```
### encryption.get(id)
Fetches encryption key by ID.
```js
const { id, encryptionKey } = await encryption.get(5)
// Pass -1 to get latest key
const latest = await encryption.get(-1)
```
## Key Derivation
### Block Key
Used for encrypting block content.
### Hash Key
Used for tree node hashing.
### Blinding Key
Required for legacy compat mode.
## Complete Example
```js
const HypercoreEncryption = require('hypercore-encryption')
const Hypercore = require('hypercore')
const crypto = require('crypto')
class KeyManager {
constructor() {
this.keys = new Map()
this.currentId = 0
}
generateKey() {
const id = ++this.currentId
const key = crypto.randomBytes(32)
this.keys.set(id, key)
return { id, key }
}
async getKey(id) {
if (id === -1) {
// Return latest
const latest = Array.from(this.keys.entries()).pop()
if (!latest) throw new Error('No keys available')
return { id: latest[0], encryptionKey: latest[1] }
}
const key = this.keys.get(id)
if (!key) throw new Error(`Key ${id} not found`)
return { id, encryptionKey: key }
}
}
// Setup
const keyManager = new KeyManager()
keyManager.generateKey() // Initial key
const encryption = new HypercoreEncryption(
(id) => keyManager.getKey(id)
)
const core = new Hypercore('./storage', {
encryption: encryption.createEncryptionProvider({
transform(ctx, entropy, compat) {
const blockKey = crypto.hkdfSync('sha256', entropy,
Buffer.from('block'), '', 32)
const hashKey = crypto.hkdfSync('sha256', entropy,
Buffer.from('hash'), '', 32)
return { block: blockKey, hash: hashKey }
}
})
})
// Rotate key
keyManager.generateKey()
encryption.clear()
// New blocks use new key
await core.append('data with new key')
```
## License
Apache-2.0
---
**Module Type**: Encryption | **Ecosystem Role**: Dynamic Encryption | **Works With**: Hypercore
+593 -1
View File
@@ -1 +1,593 @@
# Rabin Native\n\nlibrabin bindings for JS (content deduplication?).\n\nUpdated Feb 19, 2026.\n\nRepo: [github.com/holepunchto/rabin-native](https://github.com/holepunchto/rabin-native) # rabin-native - Content-Defined Chunking
## Overview
rabin-native provides JavaScript bindings for the Rabin fingerprinting algorithm, enabling content-defined chunking of data streams. This algorithm is essential for deduplication systems, as it identifies chunk boundaries based on data content rather than fixed positions, allowing identical chunks to be detected even when shifted within a file.
### Key Features
- **Content-defined chunking**: Chunk boundaries determined by data patterns
- **Variable chunk sizes**: Configurable min/max chunk sizes
- **Streaming API**: Process data incrementally
- **High performance**: Native C++ implementation via librabin
- **Deduplication support**: Identical content produces identical chunks
### Use Cases
- **Distributed storage**: Efficient synchronization with rsync-like algorithms
- **Version control**: Detect moved or shifted content
- **Backup systems**: Deduplicate data across versions
- **P2P file sharing**: Resumable downloads and efficient seeding
- **Hypercore**: Used internally for block deduplication
## Architecture
```mermaid
graph LR
subgraph "Input Stream"
DATA[Raw Data<br/>Files/Streams]
end
subgraph "Rabin Chunking"
CHUNKER[Chunker<br/>Sliding Window]
RABIN[Rabin Fingerprint<br/>Polynomial Hash]
BOUNDARY[Boundary Detection<br/>Pattern Match]
end
subgraph "Output"
CHUNKS[Variable Chunks<br/>Content-Defined]
META[Chunk Metadata<br/>Offset + Length]
end
DATA --> CHUNKER
CHUNKER --> RABIN
RABIN --> BOUNDARY
BOUNDARY --> CHUNKS
BOUNDARY --> META
```
### How Rabin Chunking Works
1. **Sliding Window**: A fixed-size window slides over the data
2. **Fingerprint Calculation**: Rabin polynomial hash computed for each window
3. **Boundary Detection**: When fingerprint matches a pattern (e.g., low bits = 0), a boundary is declared
4. **Chunk Extraction**: Data between boundaries forms a chunk
5. **Variable Size**: Chunks average to target size but vary based on content
## Installation
```bash
npm install rabin-native
```
## Quick Start
### Basic Chunking
```js
const rabin = require('rabin-native')
const chunker = new rabin.Chunker()
const chunks = []
// Push data incrementally
for (const chunk of chunker.push(data)) {
chunks.push(chunk)
}
// Get final chunk
const lastChunk = chunker.end()
if (lastChunk) chunks.push(lastChunk)
console.log(`Split into ${chunks.length} chunks`)
```
### Chunking a File
```js
const fs = require('fs')
const rabin = require('rabin-native')
const chunker = new rabin.Chunker()
const stream = fs.createReadStream('large-file.bin')
const chunks = []
stream.on('data', (data) => {
for (const chunk of chunker.push(data)) {
chunks.push(chunk)
console.log(`Chunk: offset=${chunk.offset}, length=${chunk.length}`)
}
})
stream.on('end', () => {
const lastChunk = chunker.end()
if (lastChunk) chunks.push(lastChunk)
console.log(`Total chunks: ${chunks.length}`)
})
```
## API Reference
### Chunker
Main class for Rabin chunking operations.
#### `new rabin.Chunker([options])`
Create a new Rabin chunker instance.
**Parameters:**
- `options` (object, optional): Configuration options
- `minSize` (number): Minimum chunk size in bytes (default: 512 KiB)
- `maxSize` (number): Maximum chunk size in bytes (default: 8 MiB)
**Returns:** `Chunker` instance
#### `chunker.push(data)`
Push data into the chunker.
**Parameters:**
- `data` (Buffer): Data to process
**Returns:** Iterator yielding chunk objects
Each chunk object:
```js
{
length: number, // Size of chunk in bytes
offset: number // Offset within the stream
}
```
#### `chunker.end()`
Finalize the chunker and return any trailing data.
**Returns:** Chunk object or `null` if no trailing data
## Complete Examples
### Example 1: Basic File Deduplication
```js
const fs = require('fs')
const crypto = require('crypto')
const rabin = require('rabin-native')
async function deduplicateFile(filepath) {
const chunker = new rabin.Chunker({
minSize: 64 * 1024, // 64 KiB minimum
maxSize: 1024 * 1024 // 1 MiB maximum
})
const chunks = []
const chunkHashes = new Map()
const stream = fs.createReadStream(filepath)
return new Promise((resolve, reject) => {
stream.on('data', (data) => {
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
if (!chunkHashes.has(hash)) {
chunkHashes.set(hash, chunkData)
}
chunks.push({ hash, ...chunk })
}
})
stream.on('end', () => {
const lastChunk = chunker.end()
if (lastChunk) {
// Process final chunk
chunks.push(lastChunk)
}
resolve({
totalChunks: chunks.length,
uniqueChunks: chunkHashes.size,
deduplicationRatio: chunks.length / chunkHashes.size,
chunks
})
})
stream.on('error', reject)
})
}
// Usage
deduplicateFile('./myfile.dat').then(result => {
console.log(`Deduplication ratio: ${result.deduplicationRatio.toFixed(2)}x`)
})
```
### Example 2: Resumable File Upload
```js
const rabin = require('rabin-native')
const fs = require('fs')
class ResumableUploader {
constructor(chunkSize = { min: 256 * 1024, max: 2 * 1024 * 1024 }) {
this.chunker = new rabin.Chunker(chunkSize)
this.uploadedChunks = new Set()
}
async uploadFile(filepath, uploadChunk) {
const stream = fs.createReadStream(filepath)
const pendingChunks = []
return new Promise((resolve, reject) => {
stream.on('data', (data) => {
for (const chunk of this.chunker.push(data)) {
const chunkId = `${chunk.offset}-${chunk.length}`
if (!this.uploadedChunks.has(chunkId)) {
pendingChunks.push(uploadChunk(chunkId, chunk, data))
}
}
})
stream.on('end', async () => {
const lastChunk = this.chunker.end()
if (lastChunk) {
const chunkId = `final-${lastChunk.length}`
pendingChunks.push(uploadChunk(chunkId, lastChunk))
}
await Promise.all(pendingChunks)
resolve()
})
stream.on('error', reject)
})
}
markUploaded(chunkId) {
this.uploadedChunks.add(chunkId)
}
}
// Usage
const uploader = new ResumableUploader()
// Simulate server that tracks uploaded chunks
const serverChunks = new Set()
uploader.uploadFile('./large-file.zip', async (id, meta, buffer) => {
if (!serverChunks.has(id)) {
console.log(`Uploading chunk ${id} (${meta.length} bytes)`)
// await uploadToServer(id, buffer)
serverChunks.add(id)
uploader.markUploaded(id)
} else {
console.log(`Skipping already uploaded chunk ${id}`)
}
})
```
### Example 3: Content-Addressed Storage
```js
const rabin = require('rabin-native')
const crypto = require('crypto')
class ContentAddressedStore {
constructor() {
this.chunks = new Map()
}
async store(data) {
const chunker = new rabin.Chunker({
minSize: 32 * 1024,
maxSize: 256 * 1024
})
const chunkIds = []
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
const id = crypto.createHash('sha256').update(chunkData).digest('hex')
if (!this.chunks.has(id)) {
this.chunks.set(id, chunkData)
}
chunkIds.push(id)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
const id = crypto.createHash('sha256').update(lastData).digest('hex')
if (!this.chunks.has(id)) {
this.chunks.set(id, lastData)
}
chunkIds.push(id)
}
return {
rootHash: crypto.createHash('sha256').update(data).digest('hex'),
chunks: chunkIds
}
}
retrieve(chunkIds) {
const chunks = chunkIds.map(id => this.chunks.get(id))
return Buffer.concat(chunks)
}
}
// Usage
const store = new ContentAddressedStore()
const data = Buffer.alloc(1024 * 1024)
data.fill('A') // 1MB of data
const { rootHash, chunks } = await store.store(data)
console.log(`Stored as ${chunks.length} chunks`)
// Store similar data - only new chunks are added
const data2 = Buffer.concat([data, Buffer.from('B')])
const result2 = await store.store(data2)
console.log(`Similar data: ${result2.chunks.length} chunks (many reused)`)
```
### Example 4: Delta Sync Algorithm
```js
const rabin = require('rabin-native')
const crypto = require('crypto')
class DeltaSync {
constructor() {
this.chunkIndex = new Map() // hash -> [files]
}
indexFile(filepath, data) {
const chunker = new rabin.Chunker()
const chunks = []
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
if (!this.chunkIndex.has(hash)) {
this.chunkIndex.set(hash, [])
}
this.chunkIndex.get(hash).push({ filepath, offset: chunk.offset })
chunks.push(hash)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
const hash = crypto.createHash('sha256').update(lastData).digest('hex')
if (!this.chunkIndex.has(hash)) {
this.chunkIndex.set(hash, [])
}
this.chunkIndex.get(hash).push({
filepath,
offset: data.length - lastChunk.length
})
chunks.push(hash)
}
return chunks
}
computeDelta(oldData, newData) {
const oldChunks = this.getChunkHashes(oldData)
const newChunks = this.getChunkHashes(newData)
const unchanged = []
const changed = []
for (let i = 0; i < newChunks.length; i++) {
if (oldChunks.includes(newChunks[i])) {
unchanged.push({ index: i, hash: newChunks[i] })
} else {
changed.push({ index: i, hash: newChunks[i] })
}
}
return { unchanged, changed }
}
getChunkHashes(data) {
const chunker = new rabin.Chunker()
const hashes = []
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
hashes.push(hash)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
const hash = crypto.createHash('sha256').update(lastData).digest('hex')
hashes.push(hash)
}
return hashes
}
}
// Usage
const sync = new DeltaSync()
const v1 = Buffer.from('Hello World! This is version 1.')
const v2 = Buffer.from('Hello World! This is version 2 with changes.')
sync.indexFile('doc.txt', v1)
const delta = sync.computeDelta(v1, v2)
console.log(`Unchanged chunks: ${delta.unchanged.length}`)
console.log(`Changed chunks: ${delta.changed.length}`)
```
## Configuration Options
### Choosing Chunk Sizes
```js
// Small chunks - better deduplication, more overhead
const small = new rabin.Chunker({
minSize: 16 * 1024, // 16 KiB
maxSize: 128 * 1024 // 128 KiB
})
// Medium chunks - balanced (default)
const medium = new rabin.Chunker()
// min: 512 KiB, max: 8 MiB
// Large chunks - less overhead, less granular
const large = new rabin.Chunker({
minSize: 2 * 1024 * 1024, // 2 MiB
maxSize: 16 * 1024 * 1024 // 16 MiB
})
```
### Trade-offs
| Size | Deduplication | Overhead | Use Case |
|------|--------------|----------|----------|
| Small | Excellent | High | Source code, small files |
| Medium | Good | Medium | General purpose |
| Large | Moderate | Low | Large media files |
## Performance Characteristics
- **Throughput**: Processes data at ~100-500 MB/s depending on hardware
- **Memory**: O(1) - processes streaming data without buffering entire file
- **CPU**: Single-threaded, can be parallelized across multiple files
- **Chunk variance**: Typically ±25% around average of (min + max) / 2
## Integration with Other Modules
### With Hypercore
```js
const Hypercore = require('hypercore')
const rabin = require('rabin-native')
const core = new Hypercore('./my-core')
function appendWithChunking(data) {
const chunker = new rabin.Chunker()
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
core.append(chunkData)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
core.append(lastData)
}
}
```
### With Hyperdrive
```js
const Hyperdrive = require('hyperdrive')
const rabin = require('rabin-native')
const drive = new Hyperdrive('./my-drive')
async function putFileChunked(path, data) {
const chunker = new rabin.Chunker()
const chunks = []
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
chunks.push(chunkData)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
chunks.push(lastData)
}
// Store chunk index and chunks
await drive.put(path, Buffer.concat(chunks))
await drive.put(`${path}.chunks`, JSON.stringify(chunks.map(c => c.length)))
}
```
## Best Practices
### Consistent Configuration
```js
// Use same settings across your application
const CHUNK_CONFIG = {
minSize: 256 * 1024,
maxSize: 2 * 1024 * 1024
}
// Reuse configuration everywhere
const chunker = new rabin.Chunker(CHUNK_CONFIG)
```
### Error Handling
```js
try {
const chunker = new rabin.Chunker()
for (const chunk of chunker.push(largeBuffer)) {
if (chunk.length > MAX_SAFE_SIZE) {
throw new Error('Chunk too large')
}
processChunk(chunk)
}
} catch (err) {
console.error('Chunking failed:', err)
}
```
### Resource Management
```js
// For processing many files
async function* chunkFiles(filePaths) {
for (const path of filePaths) {
const chunker = new rabin.Chunker()
const data = await fs.promises.readFile(path)
const chunks = []
for (const chunk of chunker.push(data)) {
chunks.push(chunk)
}
const lastChunk = chunker.end()
if (lastChunk) chunks.push(lastChunk)
yield { path, chunks }
}
}
```
## License
Apache-2.0
---
**Module Type**: Algorithm/Utility | **Ecosystem Role**: Data Processing | **Dependencies**: librabin (native)
+351
View File
@@ -0,0 +1,351 @@
# simple-seeder - Dead Simple Hypercore Seeder
## Overview
simple-seeder is a lightweight CLI tool designed for effortless seeding of Hypercore-based data structures. It provides a zero-configuration approach to keeping Hypercores, Hyperbees, and Hyperdrives available on the network by participating in the Hyperswarm DHT as a persistent peer.
### Key Features
- **Zero-configuration seeding**: Start seeding with a single command
- **Multiple resource types**: Support for cores, bees, drives, and seeders
- **Dynamic list management**: Real-time Hyperbee-based seed list
- **Multiple input methods**: Command-line args, file input, or live list
- **Lightweight**: Minimal resource footprint for always-on seeding
### Use Cases
- Content distribution nodes
- Backup and archival services
- Development testing environments
- Community-driven data persistence
- Load balancing across multiple seeders
## Architecture
```mermaid
graph TB
subgraph "Input Methods"
CLI[CLI Arguments<br/>--core, --bee, --drive]
FILE[Seed File<br/>seeds.txt]
LIST[Hyperbee List<br/>Live Updates]
end
subgraph "Seeding Core"
PARSER[Seed Parser]
STORE[Corestore]
DISCO[Hyperswarm<br/>DHT Discovery]
end
subgraph "Network"
DHT[HyperDHT]
PEERS[Connected Peers]
SEEDERS[Seeder Peers<br/>Enhanced Availability]
end
CLI --> PARSER
FILE --> PARSER
LIST --> PARSER
PARSER --> STORE
STORE --> DISCO
DISCO --> DHT
DHT --> PEERS
DHT -.->|announce| SEEDERS
```
## Installation
```bash
npm install -g simple-seeder
```
## Quick Start
### Seed from Command Line
```bash
# Seed a single core
simple-seeder -c <hypercore-key>
# Seed multiple resources
simple-seeder -c <core-key> -b <bee-key> -d <drive-key>
# Seed with enhanced seeder support
simple-seeder -c <core-key> -s <seeder-key>
```
### Seed from File
Create `seeds.txt`:
```
core <hypercore-key>
bee <hyperbee-key>
drive <hyperdrive-key>
seeders <seeder-announcement-key>
```
Run:
```bash
simple-seeder --file ./seeds.txt
```
### Manage Live Seed List
```bash
# Open management menu
simple-seeder --menu
# Get your list key and share it
# Others can seed your entire list:
simple-seeder <list-key>
```
## CLI Reference
### Options
| Flag | Long | Description | Example |
|------|------|-------------|---------|
| `-c` | `--core` | Hypercore key to seed | `-c a1b2c3...` |
| `-b` | `--bee` | Hyperbee key to seed | `-b d4e5f6...` |
| `-d` | `--drive` | Hyperdrive key to seed | `-d g7h8i9...` |
| `-s` | `--seeders` | Seeder announcement key | `-s j0k1l2...` |
| `-f` | `--file` | Path to seeds file | `--file ./seeds.txt` |
| `-m` | `--menu` | Open management UI | `--menu` |
### Input Methods (Choose One)
**Important**: You can only use one input method per process.
#### Method 1: Command-Line Arguments
```bash
simple-seeder -c <key1> -c <key2> -b <bee-key>
```
#### Method 2: Seed File
```bash
simple-seeder --file ./seeds.txt
```
#### Method 3: Live List
```bash
# Create or connect to a list
simple-seeder --menu
# Then seed the list
simple-seeder <list-key>
```
## Complete Examples
### Example 1: Basic Core Seeding
```bash
# Start seeding a Hypercore
simple-seeder -c ocmjxpzghcx5gbhkky7qubn5pr4fpcxwr5mu4hjw43dqs3qhid3y
# Output shows connection status
# Press Ctrl+C to stop
```
### Example 2: Multi-Resource Seed File
`seeds.txt`:
```
# Core containing application logs
core a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0
# Bee containing user data
bee b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0a1
# Drive containing website files
drive c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0a1b2
# Enable seeder announcements
seeders d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0a1b2c3
```
```bash
simple-seeder --file ./seeds.txt
```
### Example 3: Dynamic List Management
```bash
# Start the management interface
simple-seeder --menu
# Menu shows:
# - List key (share this with others)
# - Currently seeded items
# - Options to add/remove seeds
# On another machine, seed the entire list
simple-seeder ocmjxpzghcx5gbhkky7qubn5pr4fpcxwr5mu4hjw43dqs3qhid3y
# Any changes to the list are automatically replicated!
```
### Example 4: Seeder-Enhanced Distribution
```bash
# Seed core with seeder support for better availability
simple-seeder -c <core-key> -s <seeder-key>
# Multiple seeders can use the same seeder key
# to coordinate and provide redundant access
```
## Seed File Format
```
# Lines starting with # are comments
# Resource type followed by key
core <64-character-hex-key>
bee <64-character-hex-key>
drive <64-character-hex-key>
seeders <64-character-hex-key>
# Multiple of same type allowed
core <key-1>
core <key-2>
bee <key-3>
```
## Best Practices
### Resource Availability
```bash
# Use seeders for critical content
simple-seeder -c <important-core> -s <seeder-key>
# Run multiple instances for redundancy
# Instance 1
simple-seeder --file ./seeds-primary.txt
# Instance 2 (different machine)
simple-seeder --file ./seeds-backup.txt
```
### Performance Tuning
```bash
# For high-traffic resources, seed from multiple locations
# Server 1 (US)
simple-seeder -c <key> -s <seeder-key>
# Server 2 (EU)
simple-seeder -c <key> -s <seeder-key>
# Server 3 (Asia)
simple-seeder -c <key> -s <seeder-key>
```
### Monitoring
```bash
# Run with logging
DEBUG=simple-seeder simple-seeder -c <key>
# Check connections
# (Use the menu interface for real-time stats)
simple-seeder --menu
```
## Integration Patterns
### With Docker
```dockerfile
FROM node:18-alpine
RUN npm install -g simple-seeder
COPY seeds.txt /app/seeds.txt
CMD ["simple-seeder", "--file", "/app/seeds.txt"]
```
```bash
docker build -t my-seeder .
docker run -d my-seeder
```
### With Systemd
`/etc/systemd/system/simple-seeder.service`:
```ini
[Unit]
Description=Simple Seeder
After=network.target
[Service]
Type=simple
ExecStart=/usr/bin/simple-seeder --file /etc/seeder/seeds.txt
Restart=always
User=seeder
[Install]
WantedBy=multi-user.target
```
### Programmatic Usage
While simple-seeder is primarily a CLI tool, you can use Corestore and Hyperswarm directly:
```js
const Corestore = require('corestore')
const Hyperswarm = require('hyperswarm')
const store = new Corestore('./seeder-storage')
const swarm = new Hyperswarm()
async function seed(key) {
const core = store.get(Buffer.from(key, 'hex'))
await core.ready()
swarm.join(core.discoveryKey)
swarm.on('connection', (conn) => core.replicate(conn))
console.log(`Seeding: ${key}`)
console.log(`Peers: ${core.peers.length}`)
}
// Seed from your own list
const seeds = ['key1', 'key2', 'key3']
seeds.forEach(seed)
```
## Security Considerations
- **Public data only**: simple-seeder is for public/semi-public content
- **No encryption**: Seeded cores are available to anyone with the key
- **Resource limits**: Monitor disk usage for large drives
- **Network exposure**: Opens connections on Hyperswarm DHT
## Troubleshooting
### No peers connecting
```bash
# Check your network allows DHT traffic
# Verify the key is correct
simple-seeder -c <key> --verbose
```
### High memory usage
```bash
# Seed fewer resources per instance
# Split into multiple seed files
```
### Storage issues
```bash
# simple-seeder stores data in current directory
# Ensure adequate disk space
du -sh ./
```
## License
Apache-2.0
---
**Module Type**: CLI Tool | **Ecosystem Role**: Network Infrastructure | **Dependencies**: Corestore, Hyperswarm