This commit is contained in:
Raven Scott
2026-02-19 17:56:13 -05:00
parent 56feb7a3fb
commit c5ed79276c
554 changed files with 2 additions and 2 deletions
+118
View File
@@ -0,0 +1,118 @@
# appling-native
Bindings to `libappling` for Bare. Provides app descriptors, platform resolution, invite parsing, and install locks.
## Install
```bash
npm i appling-native
```
## Overview
- Manage Pear app descriptors and launches.
- Parse `pear://` links and inspect ids/data.
- Resolve and lock platform installations.
## Architecture
```mermaid
flowchart LR
A[App/Link/Platform] --> B[libappling binding]
B --> C[OS/Platform install]
```
## API
#### `const appling = require('appling-native')`
### App
#### `const app = new appling.App(id[, path])`
Properties: `app.id`, `app.path`.
#### `app.open([argument])`
### Link
#### `const link = appling.parse(input[, encoding])`
Properties: `link.id`, `link.data`.
### Platform
#### `const platform = await appling.resolve([directory])`
Properties: `platform.path`, `platform.key`.
#### `platform.ready(link)`
#### `platform.preflight(link[, callback])`
#### `platform.launch(app[, link])`
### Lock
#### `const lock = await appling.lock([directory])`
Properties: `lock.dir`.
#### `lock.unlock()` / `lock[Symbol.dispose]()`
## Examples
### 1) Open an app
```js
const appling = require('appling-native')
const app = new appling.App('keet')
app.open('pear://keet/invite')
```
### 2) Resolve platform
```js
const platform = await appling.resolve()
console.log(platform.path)
```
### 3) Ready check + launch
```js
if (platform.ready('pear://keet')) {
platform.launch('keet')
}
```
### 4) Lock install dir
```js
using lock = await appling.lock()
console.log(lock.dir)
```
## Best Practices
- Use `ready()` before `launch()` to avoid missing installs.
- Always release locks on shutdown.
## Performance Notes
- Most operations are lightweight; preflight may involve I/O.
## Security Considerations
- Treat `pear://` link data as untrusted input.
## Integration Notes
- Works alongside Pear runtime installers and app launch flows.
## Error Handling
- `resolve()` throws when no platform installation is found.
## License
Apache-2.0
+94
View File
@@ -0,0 +1,94 @@
# bitarray-native
Native bindings to libbitarray for fast sparse bit arrays.
## Install
```bash
npm i bitarray-native
```
## Overview
- Sparse bit storage with paging.
- Supports bit operations, ranges, and counts.
- Native performance for large bitfields.
## Architecture
```mermaid
flowchart LR
A[Bitarray] --> B[libbitarray binding]
B --> C[pages / sparse storage]
```
## API
#### `const Bitarray = require('bitarray-native')`
Methods:
- `destroy()`
- `page(index, bitfield)`
- `insert(bitfield[, start])`, `clear(bitfield[, start])`
- `get(bit)`, `set(bit[, value])`, `unset(bit)`
- `setBatch(bits[, value])`, `unsetBatch(bits)`
- `fill(value[, start[, end]])`
- `findFirst`, `firstSet`, `firstUnset`, `findLast`, `lastSet`, `lastUnset`
- `count`, `countSet`, `countUnset`
#### `Bitarray.constants.BYTES_PER_PAGE`
## Examples
### 1) Basic usage
```js
const Bitarray = require('bitarray-native')
const b = new Bitarray()
b.set(1234, true)
console.log(b.get(1234))
```
### 2) Range fill
```js
b.fill(true, 0, 1024)
```
### 3) Count bits
```js
const n = b.countSet(0, 2048)
```
### 4) Cleanup
```js
b.destroy()
```
## Best Practices
- Call `destroy()` to release native resources.
- Use `page()` for bulk page updates.
## Performance Notes
- Bulk operations are much faster than per-bit updates.
## Security Considerations
- Validate indexes and bitfield sizes to avoid RangeErrors.
## Integration Notes
- Used by `bitarray-universal` as the native backend.
## Error Handling
- Throws on invalid indices or bitfield sizes.
## License
Apache-2.0
+68
View File
@@ -0,0 +1,68 @@
# crc-native
Native CRC32 bindings for JavaScript (Node/Bare).
## Install
```bash
npm i crc-native
```
## Overview
- Computes CRC32 of a buffer using native bindings.
## Architecture
```mermaid
flowchart LR
A[Buffer] --> B[crc32]
B --> C[uint32]
```
## API
#### `const { crc32 } = require('crc-native')`
#### `crc32(buffer)`
Returns CRC32 as a number.
## Examples
### 1) Compute CRC
```js
const { crc32 } = require('crc-native')
const sum = crc32(Buffer.from('hello'))
```
### 2) Compare checksums
```js
if (crc32(bufA) !== crc32(bufB)) throw new Error('mismatch')
```
## Best Practices
- Use for integrity checks, not cryptographic security.
## Performance Notes
- Native binding is fast; good for large buffers.
## Security Considerations
- CRC32 is not secure against adversarial collisions.
## Integration Notes
- Suitable for file integrity and protocol checksums.
## Error Handling
- Throws if buffer type is invalid.
## License
Apache-2.0
+86
View File
@@ -0,0 +1,86 @@
# fx-native
JavaScript bindings for `libfx`, providing a lightweight native UI runtime.
## Install
```bash
npm install fx-native
```
## Overview
- App lifecycle and window management.
- Views, text, images, and web views.
- Runs on main thread with worker messaging.
## Architecture
```mermaid
flowchart LR
A[App] --> B[Window]
B --> C[View/Text/Image/WebView]
```
## API
#### `const app = App.shared()` / `app.run()` / `app.broadcast(buffer)` / `app.destroy()`
#### `const window = new Window(x, y, width, height, options)`
#### `const view = new View(x, y, width, height)`
#### `const text = new Text(x, y, width, height)`
#### `const image = new Image(x, y, width, height)`
#### `const webView = new WebView(x, y, width, height)`
## Examples
### 1) Create a window
```js
const { App, Window } = require('fx-native')
const app = App.shared()
const win = new Window(0, 0, 800, 600)
app.run()
```
### 2) Add a view
```js
const view = new View(0, 0, 800, 600)
win.appendChild(view)
```
### 3) Load a web page
```js
const web = new WebView(0, 0, 800, 600)
web.loadURL('https://example.com')
```
## Best Practices
- Create windows/views on the main thread only.
## Performance Notes
- Keep UI updates batched to reduce native calls.
## Security Considerations
- Validate URLs loaded in web views.
## Integration Notes
- Backed by `libfx`.
## Error Handling
- Errors surface via exceptions or event handlers.
## License
Apache-2.0
+87
View File
@@ -0,0 +1,87 @@
# quickbit-native
Native bindings for fast bitfield operations.
## Install
```sh
npm i quickbit-native
```
## Overview
`quickbit-native` provides high-performance bit operations and indexes for efficient scanning of bitfields.
## API
### Bit operations
- `get(field, bit)`
- `set(field, bit[, value])`
- `fill(field, value[, start[, end]])`
- `clear(field, chunk)`
- `findFirst(field, value[, position])`
- `findLast(field, value[, position])`
### Index
- `Index.from(fieldOrChunks)`
- `index.update(bit)`
- `index.skipFirst(value[, position])`
- `index.skipLast(value[, position])`
## Examples
### Set and get bits
```js
const quickbit = require('quickbit-native')
const field = Buffer.alloc(8)
quickbit.set(field, 3, 1)
console.log(quickbit.get(field, 3))
```
### Find first set bit
```js
const quickbit = require('quickbit-native')
const field = Buffer.alloc(8)
quickbit.set(field, 10, 1)
const pos = quickbit.findFirst(field, 1)
```
### Use an index
```js
const quickbit = require('quickbit-native')
const index = quickbit.Index.from(field)
index.update(10)
```
## Best Practices
- Pre-allocate buffers for predictable memory use.
- Use `Index` for repeated scans over large bitfields.
## Performance
- Native SIMD operations provide fast scans and updates.
## Security
- Validate bit positions to avoid out-of-range writes.
## Error Handling
- Ensure buffers are the expected size.
## Integration
- Used by Hypercore and block indexes.
## License
Apache-2.0
+593
View File
@@ -0,0 +1,593 @@
# rabin-native - Content-Defined Chunking
## Overview
rabin-native provides JavaScript bindings for the Rabin fingerprinting algorithm, enabling content-defined chunking of data streams. This algorithm is essential for deduplication systems, as it identifies chunk boundaries based on data content rather than fixed positions, allowing identical chunks to be detected even when shifted within a file.
### Key Features
- **Content-defined chunking**: Chunk boundaries determined by data patterns
- **Variable chunk sizes**: Configurable min/max chunk sizes
- **Streaming API**: Process data incrementally
- **High performance**: Native C++ implementation via librabin
- **Deduplication support**: Identical content produces identical chunks
### Use Cases
- **Distributed storage**: Efficient synchronization with rsync-like algorithms
- **Version control**: Detect moved or shifted content
- **Backup systems**: Deduplicate data across versions
- **P2P file sharing**: Resumable downloads and efficient seeding
- **Hypercore**: Used internally for block deduplication
## Architecture
```mermaid
graph LR
subgraph "Input Stream"
DATA[Raw Data<br/>Files/Streams]
end
subgraph "Rabin Chunking"
CHUNKER[Chunker<br/>Sliding Window]
RABIN[Rabin Fingerprint<br/>Polynomial Hash]
BOUNDARY[Boundary Detection<br/>Pattern Match]
end
subgraph "Output"
CHUNKS[Variable Chunks<br/>Content-Defined]
META[Chunk Metadata<br/>Offset + Length]
end
DATA --> CHUNKER
CHUNKER --> RABIN
RABIN --> BOUNDARY
BOUNDARY --> CHUNKS
BOUNDARY --> META
```
### How Rabin Chunking Works
1. **Sliding Window**: A fixed-size window slides over the data
2. **Fingerprint Calculation**: Rabin polynomial hash computed for each window
3. **Boundary Detection**: When fingerprint matches a pattern (e.g., low bits = 0), a boundary is declared
4. **Chunk Extraction**: Data between boundaries forms a chunk
5. **Variable Size**: Chunks average to target size but vary based on content
## Installation
```bash
npm install rabin-native
```
## Quick Start
### Basic Chunking
```js
const rabin = require('rabin-native')
const chunker = new rabin.Chunker()
const chunks = []
// Push data incrementally
for (const chunk of chunker.push(data)) {
chunks.push(chunk)
}
// Get final chunk
const lastChunk = chunker.end()
if (lastChunk) chunks.push(lastChunk)
console.log(`Split into ${chunks.length} chunks`)
```
### Chunking a File
```js
const fs = require('fs')
const rabin = require('rabin-native')
const chunker = new rabin.Chunker()
const stream = fs.createReadStream('large-file.bin')
const chunks = []
stream.on('data', (data) => {
for (const chunk of chunker.push(data)) {
chunks.push(chunk)
console.log(`Chunk: offset=${chunk.offset}, length=${chunk.length}`)
}
})
stream.on('end', () => {
const lastChunk = chunker.end()
if (lastChunk) chunks.push(lastChunk)
console.log(`Total chunks: ${chunks.length}`)
})
```
## API Reference
### Chunker
Main class for Rabin chunking operations.
#### `new rabin.Chunker([options])`
Create a new Rabin chunker instance.
**Parameters:**
- `options` (object, optional): Configuration options
- `minSize` (number): Minimum chunk size in bytes (default: 512 KiB)
- `maxSize` (number): Maximum chunk size in bytes (default: 8 MiB)
**Returns:** `Chunker` instance
#### `chunker.push(data)`
Push data into the chunker.
**Parameters:**
- `data` (Buffer): Data to process
**Returns:** Iterator yielding chunk objects
Each chunk object:
```js
{
length: number, // Size of chunk in bytes
offset: number // Offset within the stream
}
```
#### `chunker.end()`
Finalize the chunker and return any trailing data.
**Returns:** Chunk object or `null` if no trailing data
## Complete Examples
### Example 1: Basic File Deduplication
```js
const fs = require('fs')
const crypto = require('crypto')
const rabin = require('rabin-native')
async function deduplicateFile(filepath) {
const chunker = new rabin.Chunker({
minSize: 64 * 1024, // 64 KiB minimum
maxSize: 1024 * 1024 // 1 MiB maximum
})
const chunks = []
const chunkHashes = new Map()
const stream = fs.createReadStream(filepath)
return new Promise((resolve, reject) => {
stream.on('data', (data) => {
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
if (!chunkHashes.has(hash)) {
chunkHashes.set(hash, chunkData)
}
chunks.push({ hash, ...chunk })
}
})
stream.on('end', () => {
const lastChunk = chunker.end()
if (lastChunk) {
// Process final chunk
chunks.push(lastChunk)
}
resolve({
totalChunks: chunks.length,
uniqueChunks: chunkHashes.size,
deduplicationRatio: chunks.length / chunkHashes.size,
chunks
})
})
stream.on('error', reject)
})
}
// Usage
deduplicateFile('./myfile.dat').then(result => {
console.log(`Deduplication ratio: ${result.deduplicationRatio.toFixed(2)}x`)
})
```
### Example 2: Resumable File Upload
```js
const rabin = require('rabin-native')
const fs = require('fs')
class ResumableUploader {
constructor(chunkSize = { min: 256 * 1024, max: 2 * 1024 * 1024 }) {
this.chunker = new rabin.Chunker(chunkSize)
this.uploadedChunks = new Set()
}
async uploadFile(filepath, uploadChunk) {
const stream = fs.createReadStream(filepath)
const pendingChunks = []
return new Promise((resolve, reject) => {
stream.on('data', (data) => {
for (const chunk of this.chunker.push(data)) {
const chunkId = `${chunk.offset}-${chunk.length}`
if (!this.uploadedChunks.has(chunkId)) {
pendingChunks.push(uploadChunk(chunkId, chunk, data))
}
}
})
stream.on('end', async () => {
const lastChunk = this.chunker.end()
if (lastChunk) {
const chunkId = `final-${lastChunk.length}`
pendingChunks.push(uploadChunk(chunkId, lastChunk))
}
await Promise.all(pendingChunks)
resolve()
})
stream.on('error', reject)
})
}
markUploaded(chunkId) {
this.uploadedChunks.add(chunkId)
}
}
// Usage
const uploader = new ResumableUploader()
// Simulate server that tracks uploaded chunks
const serverChunks = new Set()
uploader.uploadFile('./large-file.zip', async (id, meta, buffer) => {
if (!serverChunks.has(id)) {
console.log(`Uploading chunk ${id} (${meta.length} bytes)`)
// await uploadToServer(id, buffer)
serverChunks.add(id)
uploader.markUploaded(id)
} else {
console.log(`Skipping already uploaded chunk ${id}`)
}
})
```
### Example 3: Content-Addressed Storage
```js
const rabin = require('rabin-native')
const crypto = require('crypto')
class ContentAddressedStore {
constructor() {
this.chunks = new Map()
}
async store(data) {
const chunker = new rabin.Chunker({
minSize: 32 * 1024,
maxSize: 256 * 1024
})
const chunkIds = []
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
const id = crypto.createHash('sha256').update(chunkData).digest('hex')
if (!this.chunks.has(id)) {
this.chunks.set(id, chunkData)
}
chunkIds.push(id)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
const id = crypto.createHash('sha256').update(lastData).digest('hex')
if (!this.chunks.has(id)) {
this.chunks.set(id, lastData)
}
chunkIds.push(id)
}
return {
rootHash: crypto.createHash('sha256').update(data).digest('hex'),
chunks: chunkIds
}
}
retrieve(chunkIds) {
const chunks = chunkIds.map(id => this.chunks.get(id))
return Buffer.concat(chunks)
}
}
// Usage
const store = new ContentAddressedStore()
const data = Buffer.alloc(1024 * 1024)
data.fill('A') // 1MB of data
const { rootHash, chunks } = await store.store(data)
console.log(`Stored as ${chunks.length} chunks`)
// Store similar data - only new chunks are added
const data2 = Buffer.concat([data, Buffer.from('B')])
const result2 = await store.store(data2)
console.log(`Similar data: ${result2.chunks.length} chunks (many reused)`)
```
### Example 4: Delta Sync Algorithm
```js
const rabin = require('rabin-native')
const crypto = require('crypto')
class DeltaSync {
constructor() {
this.chunkIndex = new Map() // hash -> [files]
}
indexFile(filepath, data) {
const chunker = new rabin.Chunker()
const chunks = []
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
if (!this.chunkIndex.has(hash)) {
this.chunkIndex.set(hash, [])
}
this.chunkIndex.get(hash).push({ filepath, offset: chunk.offset })
chunks.push(hash)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
const hash = crypto.createHash('sha256').update(lastData).digest('hex')
if (!this.chunkIndex.has(hash)) {
this.chunkIndex.set(hash, [])
}
this.chunkIndex.get(hash).push({
filepath,
offset: data.length - lastChunk.length
})
chunks.push(hash)
}
return chunks
}
computeDelta(oldData, newData) {
const oldChunks = this.getChunkHashes(oldData)
const newChunks = this.getChunkHashes(newData)
const unchanged = []
const changed = []
for (let i = 0; i < newChunks.length; i++) {
if (oldChunks.includes(newChunks[i])) {
unchanged.push({ index: i, hash: newChunks[i] })
} else {
changed.push({ index: i, hash: newChunks[i] })
}
}
return { unchanged, changed }
}
getChunkHashes(data) {
const chunker = new rabin.Chunker()
const hashes = []
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
hashes.push(hash)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
const hash = crypto.createHash('sha256').update(lastData).digest('hex')
hashes.push(hash)
}
return hashes
}
}
// Usage
const sync = new DeltaSync()
const v1 = Buffer.from('Hello World! This is version 1.')
const v2 = Buffer.from('Hello World! This is version 2 with changes.')
sync.indexFile('doc.txt', v1)
const delta = sync.computeDelta(v1, v2)
console.log(`Unchanged chunks: ${delta.unchanged.length}`)
console.log(`Changed chunks: ${delta.changed.length}`)
```
## Configuration Options
### Choosing Chunk Sizes
```js
// Small chunks - better deduplication, more overhead
const small = new rabin.Chunker({
minSize: 16 * 1024, // 16 KiB
maxSize: 128 * 1024 // 128 KiB
})
// Medium chunks - balanced (default)
const medium = new rabin.Chunker()
// min: 512 KiB, max: 8 MiB
// Large chunks - less overhead, less granular
const large = new rabin.Chunker({
minSize: 2 * 1024 * 1024, // 2 MiB
maxSize: 16 * 1024 * 1024 // 16 MiB
})
```
### Trade-offs
| Size | Deduplication | Overhead | Use Case |
|------|--------------|----------|----------|
| Small | Excellent | High | Source code, small files |
| Medium | Good | Medium | General purpose |
| Large | Moderate | Low | Large media files |
## Performance Characteristics
- **Throughput**: Processes data at ~100-500 MB/s depending on hardware
- **Memory**: O(1) - processes streaming data without buffering entire file
- **CPU**: Single-threaded, can be parallelized across multiple files
- **Chunk variance**: Typically ±25% around average of (min + max) / 2
## Integration with Other Modules
### With Hypercore
```js
const Hypercore = require('hypercore')
const rabin = require('rabin-native')
const core = new Hypercore('./my-core')
function appendWithChunking(data) {
const chunker = new rabin.Chunker()
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
core.append(chunkData)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
core.append(lastData)
}
}
```
### With Hyperdrive
```js
const Hyperdrive = require('hyperdrive')
const rabin = require('rabin-native')
const drive = new Hyperdrive('./my-drive')
async function putFileChunked(path, data) {
const chunker = new rabin.Chunker()
const chunks = []
for (const chunk of chunker.push(data)) {
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
chunks.push(chunkData)
}
const lastChunk = chunker.end()
if (lastChunk) {
const lastData = data.slice(-lastChunk.length)
chunks.push(lastData)
}
// Store chunk index and chunks
await drive.put(path, Buffer.concat(chunks))
await drive.put(`${path}.chunks`, JSON.stringify(chunks.map(c => c.length)))
}
```
## Best Practices
### Consistent Configuration
```js
// Use same settings across your application
const CHUNK_CONFIG = {
minSize: 256 * 1024,
maxSize: 2 * 1024 * 1024
}
// Reuse configuration everywhere
const chunker = new rabin.Chunker(CHUNK_CONFIG)
```
### Error Handling
```js
try {
const chunker = new rabin.Chunker()
for (const chunk of chunker.push(largeBuffer)) {
if (chunk.length > MAX_SAFE_SIZE) {
throw new Error('Chunk too large')
}
processChunk(chunk)
}
} catch (err) {
console.error('Chunking failed:', err)
}
```
### Resource Management
```js
// For processing many files
async function* chunkFiles(filePaths) {
for (const path of filePaths) {
const chunker = new rabin.Chunker()
const data = await fs.promises.readFile(path)
const chunks = []
for (const chunk of chunker.push(data)) {
chunks.push(chunk)
}
const lastChunk = chunker.end()
if (lastChunk) chunks.push(lastChunk)
yield { path, chunks }
}
}
```
## License
Apache-2.0
---
**Module Type**: Algorithm/Utility | **Ecosystem Role**: Data Processing | **Dependencies**: librabin (native)
+546
View File
@@ -0,0 +1,546 @@
# rocksdb-native - RocksDB Bindings for JavaScript
## Overview
rocksdb-native provides JavaScript bindings for RocksDB, Facebook's high-performance embedded database. It offers a fast key-value store with advanced features like column families, snapshots, and atomic transactions.
### Key Features
- **Native bindings**: Direct access to RocksDB C++ library
- **Batch operations**: Atomic read/write batches
- **Column families**: Organized data storage
- **Snapshots**: Point-in-time database views
- **Async API**: Non-blocking operations
### Use Cases
- **High-performance storage**: Low-latency key-value operations
- **Large datasets**: Efficient handling of terabyte-scale data
- **Indexed storage**: Fast lookups and range scans
- **Blockchain indexing**: Store and query blockchain data
- **Hypercore storage**: Backend for Hypercore and Hyperbee
## Architecture
```mermaid
graph TB
subgraph "JavaScript"
APP[Application]
API[rocksdb-native API]
end
subgraph "Native Layer"
BIND[RocksDB Bindings]
NAPI[Node-API]
end
subgraph "Storage Engine"
ROCKS[RocksDB C++]
MEM[MemTable]
SST[SSTables]
WAL[Write-Ahead Log]
end
APP --> API
API --> BIND
BIND --> NAPI
NAPI --> ROCKS
ROCKS --> MEM
ROCKS --> SST
ROCKS --> WAL
```
## Installation
```bash
npm install rocksdb-native
```
**Note:** On Linux, `libatomic` must be installed:
```bash
# Ubuntu/Debian
sudo apt-get install libatomic1
# CentOS/RHEL/Fedora
sudo yum install libatomic
```
## Quick Start
```js
const RocksDB = require('rocksdb-native')
// Open database
const db = new RocksDB('./example.db')
// Write data
const w = db.write()
w.put('hello', 'world')
await w.flush()
// Read data
const r = db.read()
const p = r.get('hello')
r.flush()
console.log(await p) // 'world'
```
## API Reference
### RocksDB
Main database class.
#### `new RocksDB(path[, options])`
Open or create a database.
**Parameters:**
- `path` (string): Database directory path
- `options` (object, optional):
- `createIfMissing` (boolean): Create if doesn't exist (default: true)
- `errorIfExists` (boolean): Error if exists (default: false)
**Example:**
```js
const db = new RocksDB('./mydb', {
createIfMissing: true
})
```
### Write Batch
#### `db.write()`
Create a write batch.
**Returns:** WriteBatch
#### `write.put(key, value)`
Add a put operation to the batch.
**Parameters:**
- `key` (string | Buffer): Key
- `value` (string | Buffer): Value
**Returns:** WriteBatch (for chaining)
#### `write.delete(key)`
Add a delete operation to the batch.
**Parameters:**
- `key` (string | Buffer): Key to delete
**Returns:** WriteBatch (for chaining)
#### `await write.flush()`
Execute all operations in the batch.
**Returns:** Promise<void>
### Read Snapshot
#### `db.read()`
Create a read snapshot.
**Returns:** ReadSnapshot
#### `snapshot.get(key)`
Get a value from the snapshot.
**Parameters:**
- `key` (string | Buffer): Key to look up
**Returns:** Promise<Buffer | null>
#### `snapshot.flush()`
Finalize the snapshot (required after gets).
**Returns:** void
## Complete Examples
### Example 1: Key-Value Store
```js
const RocksDB = require('rocksdb-native')
class KeyValueStore {
constructor(dbPath) {
this.db = new RocksDB(dbPath)
}
async set(key, value) {
const w = this.db.write()
w.put(key, value)
await w.flush()
}
async get(key) {
const r = this.db.read()
const p = r.get(key)
r.flush()
return await p
}
async delete(key) {
const w = this.db.write()
w.delete(key)
await w.flush()
}
async batch(operations) {
const w = this.db.write()
for (const op of operations) {
if (op.type === 'put') {
w.put(op.key, op.value)
} else if (op.type === 'del') {
w.delete(op.key)
}
}
await w.flush()
}
async close() {
// Cleanup if needed
}
}
// Usage
const store = new KeyValueStore('./kv-store')
await store.set('user:1', JSON.stringify({ name: 'Alice', age: 30 }))
await store.set('user:2', JSON.stringify({ name: 'Bob', age: 25 }))
const user1 = JSON.parse(await store.get('user:1'))
console.log(user1) // { name: 'Alice', age: 30 }
await store.close()
```
### Example 2: Counter Store
```js
const RocksDB = require('rocksdb-native')
class CounterStore {
constructor(dbPath) {
this.db = new RocksDB(dbPath)
}
async increment(key, amount = 1) {
const r = this.db.read()
const p = r.get(key)
r.flush()
const current = parseInt(await p || '0', 10)
const next = current + amount
const w = this.db.write()
w.put(key, String(next))
await w.flush()
return next
}
async decrement(key, amount = 1) {
return this.increment(key, -amount)
}
async get(key) {
const r = this.db.read()
const p = r.get(key)
r.flush()
const val = await p
return val ? parseInt(val, 10) : 0
}
async reset(key) {
const w = this.db.write()
w.put(key, '0')
await w.flush()
}
}
// Usage
const counters = new CounterStore('./counters')
await counters.increment('visits') // 1
await counters.increment('visits') // 2
await counters.increment('visits', 5) // 7
const visits = await counters.get('visits')
console.log('Total visits:', visits)
```
### Example 3: Time-Series Store
```js
const RocksDB = require('rocksdb-native')
class TimeSeriesStore {
constructor(dbPath) {
this.db = new RocksDB(dbPath)
}
key(metric, timestamp) {
return `${metric}:${timestamp.toString().padStart(15, '0')}`
}
async write(metric, value, timestamp = Date.now()) {
const w = this.db.write()
w.put(
this.key(metric, timestamp),
JSON.stringify({ value, timestamp })
)
await w.flush()
}
async readRange(metric, start, end) {
const results = []
const startKey = this.key(metric, start)
const endKey = this.key(metric, end)
// Note: rocksdb-native doesn't expose iterators directly
// In production, you'd use column families or prefix iteration
// This is a simplified example
return results
}
async getLatest(metric) {
// Simplified - would use reverse iteration in production
const r = this.db.read()
const p = r.get(this.key(metric, Date.now()))
r.flush()
const data = await p
return data ? JSON.parse(data) : null
}
}
// Usage
const ts = new TimeSeriesStore('./timeseries')
// Write metrics
await ts.write('cpu.usage', 45.2)
await ts.write('cpu.usage', 52.1)
await ts.write('memory.usage', 78.5)
const latest = await ts.getLatest('cpu.usage')
console.log('Latest CPU:', latest)
```
### Example 4: Document Store
```js
const RocksDB = require('rocksdb-native')
class DocumentStore {
constructor(dbPath) {
this.db = new RocksDB(dbPath)
}
async insert(collection, id, doc) {
const key = `${collection}:${id}`
const w = this.db.write()
w.put(key, JSON.stringify(doc))
await w.flush()
return id
}
async findById(collection, id) {
const key = `${collection}:${id}`
const r = this.db.read()
const p = r.get(key)
r.flush()
const data = await p
return data ? JSON.parse(data) : null
}
async update(collection, id, updates) {
const existing = await this.findById(collection, id)
if (!existing) return null
const updated = { ...existing, ...updates }
await this.insert(collection, id, updated)
return updated
}
async delete(collection, id) {
const key = `${collection}:${id}`
const w = this.db.write()
w.delete(key)
await w.flush()
}
async batchInsert(collection, docs) {
const w = this.db.write()
const ids = []
for (const doc of docs) {
const id = this.generateId()
const key = `${collection}:${id}`
w.put(key, JSON.stringify(doc))
ids.push(id)
}
await w.flush()
return ids
}
generateId() {
return Date.now().toString(36) + Math.random().toString(36).substr(2)
}
}
// Usage
const docs = new DocumentStore('./documents')
// Insert
const userId = await docs.insert('users', null, {
name: 'Alice',
email: '[email protected]',
createdAt: Date.now()
})
// Find
const user = await docs.findById('users', userId)
console.log('User:', user)
// Update
await docs.update('users', userId, { lastLogin: Date.now() })
// Batch insert
const posts = [
{ title: 'Post 1', content: 'Content 1' },
{ title: 'Post 2', content: 'Content 2' }
]
const postIds = await docs.batchInsert('posts', posts)
console.log('Created posts:', postIds)
```
## Best Practices
### Batch Operations
```js
// Good: Batch multiple operations
const w = db.write()
for (const item of items) {
w.put(item.key, item.value)
}
await w.flush()
// Less efficient: Individual writes
for (const item of items) {
const w = db.write()
w.put(item.key, item.value)
await w.flush()
}
```
### Read Snapshots
```js
// Always call flush() after reads
const r = db.read()
const p1 = r.get('key1')
const p2 = r.get('key2')
r.flush() // Required!
const [val1, val2] = await Promise.all([p1, p2])
```
### Error Handling
```js
try {
const w = db.write()
w.put('key', 'value')
await w.flush()
} catch (err) {
if (err.message.includes('IO error')) {
console.error('Disk error:', err)
} else {
console.error('Database error:', err)
}
}
```
### Resource Cleanup
```js
// Ensure proper cleanup
process.on('SIGINT', async () => {
// RocksDB handles cleanup on process exit
// But explicit close is better for tests
process.exit(0)
})
```
## Integration with Other Modules
### With Hypercore
```js
const RocksDB = require('rocksdb-native')
const Hypercore = require('hypercore')
const db = new RocksDB('./core-storage')
const core = new Hypercore(storage)
// Custom storage using RocksDB
function createRocksStorage(db) {
return {
read: async (offset, size) => {
const r = db.read()
const p = r.get(`block:${offset}`)
r.flush()
return await p
},
write: async (offset, data) => {
const w = db.write()
w.put(`block:${offset}`, data)
await w.flush()
}
}
}
```
### With Hyperbee
```js
const RocksDB = require('rocksdb-native')
const Hyperbee = require('hyperbee')
const db = new RocksDB('./bee-storage')
// Hyperbee uses RocksDB as underlying storage
```
## Performance Tips
1. **Use batches** for multiple writes
2. **Compact keys** - shorter keys use less memory
3. **Async operations** - don't block the event loop
4. **Proper flush** - always flush read snapshots
5. **Monitor space** - RocksDB compacts automatically but uses disk space
## License
Apache-2.0
---
**Module Type**: Storage | **Ecosystem Role**: Database Engine | **Dependencies**: librocksdb
+65
View File
@@ -0,0 +1,65 @@
# simdle-native
Native SIMD bit operations bindings.
## Install
```sh
npm i simdle-native
```
## Overview
Provides fast SIMD helpers for bitwise operations on buffers.
## API
### Functions
- `allo`, `allz`, `and`, `clear`, `clo`, `clz`, `cnt`, `cto`, `ctz`, `not`, `or`, `sum`, `xor`
## Examples
### Count set bits
```js
const simdle = require('simdle-native')
const count = simdle.cnt(Buffer.from([0xff, 0x00]))
```
### AND two buffers
```js
const out = simdle.and(a, b)
```
### Clear bits
```js
simdle.clear(buf, 0, 8)
```
## Best Practices
- Ensure buffer sizes match for binary ops.
## Performance
- SIMD acceleration makes bit ops fast on supported CPUs.
## Security
- Validate inputs to avoid out-of-range access.
## Error Handling
- Functions may throw on invalid buffer sizes.
## Integration
- Used in low-level indexing and bitmap logic.
## License
Apache-2.0
+21
View File
@@ -0,0 +1,21 @@
# sodium-native - Crypto Bindings
## Overview
High-perf libsodium addon. Ed25519/XChaCha20Poly/blake2b.
**Ex**:
```js
const sodium = require('sodium-native')
sodium.crypto_sign_keypair(pub, priv)
sodium.crypto_sign_detached(sig, msg, priv)
```
**Perf**: SIMD opt, faster than universal.
**Hyper**: Core signing, noise.
**Prebuilds**: Multi-arch.
**Source**: github/holepunchto/sodium-native
+73
View File
@@ -0,0 +1,73 @@
# sqlite3-native
Async SQLite3 bindings with VFS support.
## Install
```sh
npm i sqlite3-native
```
## Overview
Provides a native SQLite3 driver with async APIs and optional VFS integrations.
## API
### `new SQLite3()`
### `db.exec(sql)`
- **returns** query results
## Examples
### Create a table and insert data
```js
const SQLite3 = require('sqlite3-native')
const db = new SQLite3()
await db.exec('create table test (id integer, name text)')
await db.exec("insert into test values (1, 'hello')")
```
### Query rows
```js
const rows = await db.exec('select * from test')
```
### Use in a service
```js
async function init() {
const db = new SQLite3()
await db.exec('pragma journal_mode=wal')
return db
}
```
## Best Practices
- Use WAL mode for concurrent reads/writes.
## Performance
- Performance depends on disk I/O and query complexity.
## Security
- Use parameterized queries to avoid injection.
## Error Handling
- Catch errors from `exec` and handle transactions.
## Integration
- Suitable for local persistence in Pear apps.
## License
Apache-2.0
+70
View File
@@ -0,0 +1,70 @@
# tt-native
PTY bindings for Node.js via libtt.
## Install
```sh
npm i tt-native
```
## Overview
`tt-native` provides a pseudo-terminal interface. `spawn` returns a duplex stream representing the PTY.
## API
### `spawn(file[, args][, options])`
- **options**: `width`, `height`, `env`, `cwd`
- Returns `pty` stream with `pid`, `width`, `height`
- `pty.resize(width, height)`
- `pty.kill([signal])`
- `pty.on('exit', fn)`
## Examples
### Spawn a shell
```js
const { spawn } = require('tt-native')
const pty = spawn('bash', [], { width: 80, height: 24 })
pty.pipe(process.stdout)
```
### Resize PTY
```js
pty.resize(100, 30)
```
### Kill process
```js
pty.kill('SIGTERM')
```
## Best Practices
- Handle `exit` to cleanup resources.
## Performance
- PTY throughput depends on OS and process.
## Security
- Avoid spawning untrusted commands.
## Error Handling
- Handle spawn failures and permission issues.
## Integration
- Useful for terminal UIs and process control.
## License
Apache-2.0
+627
View File
@@ -0,0 +1,627 @@
# udx-native - Reliable UDP Streams
## Overview
udx-native provides reliable, multiplexed, and congestion-controlled streams over UDP. It's designed specifically for peer-to-peer networking with no handshakes, no encryption, and minimal overhead - just fast, composable streams and messages.
### Key Features
- **Reliable streams**: TCP-like reliability over UDP
- **Multiplexing**: Multiple streams per socket
- **Congestion control**: Automatic bandwidth adaptation
- **Zero handshakes**: Direct stream creation
- **Message support**: Datagram-style messaging
- **P2P optimized**: Built for peer-to-peer scenarios
### Use Cases
- **P2P networking**: Direct peer connections
- **Media streaming**: Real-time audio/video
- **Game networking**: Low-latency game state sync
- **File transfer**: Fast bulk data transfer
- **Hole punching**: NAT traversal
## Installation
```bash
npm install udx-native
```
## Quick Start
### Basic Message Example
```js
const UDX = require('udx-native')
const u = new UDX()
const a = u.createSocket()
const b = u.createSocket()
b.on('message', (message) => {
console.log('received', message.toString())
a.close()
b.close()
})
b.bind(0)
a.send(Buffer.from('hello'), b.address().port)
```
### Stream Example
```js
const UDX = require('udx-native')
const u = new UDX()
const socket1 = u.createSocket()
const socket2 = u.createSocket()
socket1.bind()
socket2.bind()
const stream1 = u.createStream(1)
const stream2 = u.createStream(2)
stream1.connect(socket1, stream2.id, socket2.address().port, '127.0.0.1')
stream2.connect(socket2, stream1.id, socket1.address().port, '127.0.0.1')
stream1.write(Buffer.from('hello'))
stream1.end()
stream2.on('data', (data) => {
console.log(data.toString())
})
stream2.on('end', () => {
stream2.end()
})
stream1.on('close', () => {
socket1.close()
})
stream2.on('close', () => {
socket2.close()
})
```
## API Reference
### UDX
Main class for UDX operations.
#### `new UDX()`
Create a new UDX instance.
#### `UDX.isIPv4(host)`
Returns `true` if host is an IPv4 address.
#### `UDX.isIPv6(host)`
Returns `true` if host is an IPv6 address.
#### `UDX.isIP(host)`
Returns address family (`4` or `6`), or `0` if invalid.
### Sockets
#### `const socket = udx.createSocket([options])`
Create a UDP socket.
**Options:**
- `ipv6Only` (boolean): IPv6 only mode (default: false)
- `reuseAddress` (boolean): Allow address reuse (default: false)
#### `socket.bind([port], [host])`
Bind socket to address.
**Parameters:**
- `port` (number): Port number (default: 0)
- `host` (string): Host address (default: binds to `::` then `0.0.0.0`)
#### `socket.address()`
Get socket address info.
**Returns:** `{ host, family, port }`
#### `await socket.send(buffer, port, [host], [ttl])`
Send a message.
#### `socket.trySend(buffer, port, [host], [ttl])`
Send without awaiting.
#### `await socket.close()`
Close the socket.
### Streams
#### `const stream = udx.createStream(id, [options])`
Create a new stream.
**Parameters:**
- `id` (number): Stream identifier
- `options` (object):
- `firewall` (function): Firewall function
- `framed` (boolean): Framed mode
- `seq` (number): Initial sequence number
#### `stream.connect(socket, remoteId, port, [host], [options])`
Connect stream to remote endpoint.
#### `stream.write(buffer)`
Write data to stream.
#### `stream.end()`
End the stream.
#### `await stream.flush()`
Wait for writes to be acknowledged.
### Network Interfaces
#### `const interfaces = udx.networkInterfaces()`
Get network interface list.
#### `const watcher = udx.watchNetworkInterfaces([onchange])`
Watch for interface changes.
## Complete Examples
### Example 1: Simple File Transfer
```js
const UDX = require('udx-native')
const fs = require('fs')
async function sendFile(filePath, port, host) {
const u = new UDX()
const socket = u.createSocket()
await socket.bind()
const stream = u.createStream(1)
stream.connect(socket, 2, port, host)
const file = fs.createReadStream(filePath)
file.on('data', (chunk) => {
stream.write(chunk)
})
file.on('end', () => {
stream.end()
})
await new Promise((resolve) => {
stream.on('close', resolve)
})
await socket.close()
}
async function receiveFile(filePath, port) {
const u = new UDX()
const socket = u.createSocket()
await socket.bind(port)
const file = fs.createWriteStream(filePath)
const stream = u.createStream(2)
// Wait for connection
socket.on('message', () => {
// Connection established
})
stream.on('data', (data) => {
file.write(data)
})
stream.on('end', () => {
file.end()
stream.end()
})
stream.on('close', () => {
socket.close()
})
}
// Usage
// Receiver: receiveFile('received.bin', 8080)
// Sender: sendFile('file.bin', 8080, '127.0.0.1')
```
### Example 2: P2P Chat
```js
const UDX = require('udx-native')
const readline = require('readline')
class P2PChat {
constructor() {
this.udx = new UDX()
this.socket = null
this.stream = null
this.peers = new Map()
}
async start(port = 0) {
this.socket = this.udx.createSocket()
await this.socket.bind(port)
console.log('Listening on:', this.socket.address())
this.socket.on('message', (msg, from) => {
console.log(`\n[${from.host}:${from.port}] ${msg.toString()}`)
process.stdout.write('> ')
})
}
async connect(host, port) {
const stream = this.udx.createStream(Date.now())
stream.connect(this.socket, Date.now() + 1, port, host)
stream.on('data', (data) => {
console.log(`\n[Peer] ${data.toString()}`)
process.stdout.write('> ')
})
this.stream = stream
}
send(message) {
if (this.stream && this.stream.connected) {
this.stream.write(Buffer.from(message))
} else {
// Fallback to socket
// (would need to track peer addresses)
}
}
async close() {
if (this.stream) await this.stream.end()
if (this.socket) await this.socket.close()
}
}
// Usage
const chat = new P2PChat()
async function main() {
const port = parseInt(process.argv[2]) || 0
await chat.start(port)
if (process.argv[3] && process.argv[4]) {
await chat.connect(process.argv[3], parseInt(process.argv[4]))
}
const rl = readline.createInterface({
input: process.stdin,
output: process.stdout
})
rl.setPrompt('> ')
rl.prompt()
rl.on('line', (line) => {
chat.send(line)
rl.prompt()
})
rl.on('close', () => {
chat.close()
process.exit(0)
})
}
main()
```
### Example 3: Game State Sync
```js
const UDX = require('udx-native')
class GameNetwork {
constructor(tickRate = 60) {
this.udx = new UDX()
this.socket = null
this.streams = new Map()
this.tickRate = tickRate
this.players = new Map()
}
async host(port) {
this.socket = this.udx.createSocket()
await this.socket.bind(port)
console.log('Game server on port', port)
// Accept connections
this.socket.on('message', (msg, from) => {
this.handleMessage(msg, from)
})
}
async connect(host, port) {
this.socket = this.udx.createSocket()
await this.socket.bind()
const stream = this.udx.createStream(1)
stream.connect(this.socket, 2, port, host)
stream.setInteractive(true) // Low latency mode
stream.on('data', (data) => {
const state = JSON.parse(data)
this.onGameState(state)
})
this.streams.set('server', stream)
// Send input at tick rate
setInterval(() => {
this.sendInput()
}, 1000 / this.tickRate)
}
sendGameState(state) {
const data = Buffer.from(JSON.stringify(state))
for (const stream of this.streams.values()) {
if (stream.connected) {
stream.trySend(data)
}
}
}
sendInput() {
const input = this.collectInput()
const stream = this.streams.get('server')
if (stream && stream.connected) {
stream.trySend(Buffer.from(JSON.stringify({ type: 'input', input })))
}
}
collectInput() {
// Get current input state
return { timestamp: Date.now() }
}
onGameState(state) {
// Apply received game state
console.log('Received state:', state)
}
async close() {
for (const stream of this.streams.values()) {
await stream.end()
}
if (this.socket) await this.socket.close()
}
}
```
### Example 4: Network Diagnostics
```js
const UDX = require('udx-native')
class NetworkDiagnostics {
constructor() {
this.udx = new UDX()
}
async ping(host, port, count = 4) {
const socket = this.udx.createSocket()
await socket.bind()
const results = []
for (let i = 0; i < count; i++) {
const start = Date.now()
await socket.send(Buffer.from(`ping:${i}`), port, host)
// Wait for response with timeout
const response = await Promise.race([
new Promise((resolve) => {
socket.once('message', (msg) => resolve(msg))
}),
new Promise((_, reject) =>
setTimeout(() => reject(new Error('Timeout')), 1000)
)
]).catch(() => null)
if (response) {
const rtt = Date.now() - start
results.push({ seq: i, rtt, success: true })
} else {
results.push({ seq: i, rtt: null, success: false })
}
await new Promise(r => setTimeout(r, 1000))
}
await socket.close()
// Print results
console.log(`Ping statistics for ${host}:${port}:`)
const successful = results.filter(r => r.success)
console.log(` Transmitted: ${count}, Received: ${successful.length}`)
if (successful.length > 0) {
const times = successful.map(r => r.rtt)
console.log(` Min: ${Math.min(...times)}ms, Max: ${Math.max(...times)}ms`)
console.log(` Avg: ${times.reduce((a, b) => a + b) / times.length}ms`)
}
return results
}
async bandwidthTest(host, port, duration = 10000) {
const socket = this.udx.createSocket()
await socket.bind()
const stream = this.udx.createStream(1)
stream.connect(socket, 2, port, host)
let bytesSent = 0
const startTime = Date.now()
// Send data as fast as possible
const interval = setInterval(() => {
if (Date.now() - startTime >= duration) {
clearInterval(interval)
stream.end()
return
}
const data = Buffer.alloc(1400) // MTU-sized packets
stream.write(data)
bytesSent += data.length
}, 1)
await new Promise((resolve) => {
stream.on('close', resolve)
})
await socket.close()
const elapsed = (Date.now() - startTime) / 1000
const mbps = (bytesSent * 8 / 1000000) / elapsed
console.log(`Bandwidth test results:`)
console.log(` Duration: ${elapsed.toFixed(2)}s`)
console.log(` Sent: ${(bytesSent / 1024 / 1024).toFixed(2)} MB`)
console.log(` Throughput: ${mbps.toFixed(2)} Mbps`)
return { bytesSent, duration: elapsed, mbps }
}
listInterfaces() {
const interfaces = this.udx.networkInterfaces()
console.log('Network interfaces:')
interfaces.forEach(iface => {
console.log(` ${iface.name}: ${iface.host} (${iface.family === 4 ? 'IPv4' : 'IPv6'})`)
})
return interfaces
}
}
// Usage
const diag = new NetworkDiagnostics()
diag.listInterfaces()
// diag.ping('127.0.0.1', 8080)
// diag.bandwidthTest('127.0.0.1', 8080)
```
## Best Practices
### Use Interactive Mode for Low Latency
```js
stream.setInteractive(true)
```
### Handle MTU Limits
```js
stream.on('mtu-exceeded', () => {
console.warn('Packet too large, fragment or reduce size')
})
```
### Monitor Stream Health
```js
console.log('RTT:', stream.rtt)
console.log('Inflight:', stream.inflight)
console.log('CWND:', stream.cwnd)
```
### Proper Cleanup
```js
process.on('SIGINT', async () => {
for (const stream of streams) {
await stream.end()
}
await socket.close()
process.exit(0)
})
```
## Integration with Other Modules
### With Protomux
```js
const Protomux = require('protomux')
const UDX = require('udx-native')
const u = new UDX()
const socket = u.createSocket()
await socket.bind()
const stream = u.createStream(1)
const mux = new Protomux(stream)
```
### With HyperDHT
```js
const DHT = require('hyperdht')
const UDX = require('udx-native')
const dht = new DHT()
const u = new UDX()
// UDX can work with DHT for hole punching
```
## Development Setup
```bash
# Install dependencies
npm install -g bare-runtime bare-make
# Clone and build
git clone https://github.com/holepunchto/udx-native
cd udx-native
npm install
# Generate build files
bare-make generate
# Build
bare-make build
# Install
bare-make install
```
## License
Apache-2.0
---
**Module Type**: Networking Library | **Ecosystem Role**: Transport Layer | **Dependencies**: libudx