update
This commit is contained in:
@@ -0,0 +1,118 @@
|
||||
# appling-native
|
||||
|
||||
Bindings to `libappling` for Bare. Provides app descriptors, platform resolution, invite parsing, and install locks.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm i appling-native
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
- Manage Pear app descriptors and launches.
|
||||
- Parse `pear://` links and inspect ids/data.
|
||||
- Resolve and lock platform installations.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[App/Link/Platform] --> B[libappling binding]
|
||||
B --> C[OS/Platform install]
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
#### `const appling = require('appling-native')`
|
||||
|
||||
### App
|
||||
|
||||
#### `const app = new appling.App(id[, path])`
|
||||
|
||||
Properties: `app.id`, `app.path`.
|
||||
|
||||
#### `app.open([argument])`
|
||||
|
||||
### Link
|
||||
|
||||
#### `const link = appling.parse(input[, encoding])`
|
||||
|
||||
Properties: `link.id`, `link.data`.
|
||||
|
||||
### Platform
|
||||
|
||||
#### `const platform = await appling.resolve([directory])`
|
||||
|
||||
Properties: `platform.path`, `platform.key`.
|
||||
|
||||
#### `platform.ready(link)`
|
||||
|
||||
#### `platform.preflight(link[, callback])`
|
||||
|
||||
#### `platform.launch(app[, link])`
|
||||
|
||||
### Lock
|
||||
|
||||
#### `const lock = await appling.lock([directory])`
|
||||
|
||||
Properties: `lock.dir`.
|
||||
|
||||
#### `lock.unlock()` / `lock[Symbol.dispose]()`
|
||||
|
||||
## Examples
|
||||
|
||||
### 1) Open an app
|
||||
|
||||
```js
|
||||
const appling = require('appling-native')
|
||||
const app = new appling.App('keet')
|
||||
app.open('pear://keet/invite')
|
||||
```
|
||||
|
||||
### 2) Resolve platform
|
||||
|
||||
```js
|
||||
const platform = await appling.resolve()
|
||||
console.log(platform.path)
|
||||
```
|
||||
|
||||
### 3) Ready check + launch
|
||||
|
||||
```js
|
||||
if (platform.ready('pear://keet')) {
|
||||
platform.launch('keet')
|
||||
}
|
||||
```
|
||||
|
||||
### 4) Lock install dir
|
||||
|
||||
```js
|
||||
using lock = await appling.lock()
|
||||
console.log(lock.dir)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use `ready()` before `launch()` to avoid missing installs.
|
||||
- Always release locks on shutdown.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Most operations are lightweight; preflight may involve I/O.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Treat `pear://` link data as untrusted input.
|
||||
|
||||
## Integration Notes
|
||||
|
||||
- Works alongside Pear runtime installers and app launch flows.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- `resolve()` throws when no platform installation is found.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -0,0 +1,94 @@
|
||||
# bitarray-native
|
||||
|
||||
Native bindings to libbitarray for fast sparse bit arrays.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm i bitarray-native
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
- Sparse bit storage with paging.
|
||||
- Supports bit operations, ranges, and counts.
|
||||
- Native performance for large bitfields.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Bitarray] --> B[libbitarray binding]
|
||||
B --> C[pages / sparse storage]
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
#### `const Bitarray = require('bitarray-native')`
|
||||
|
||||
Methods:
|
||||
|
||||
- `destroy()`
|
||||
- `page(index, bitfield)`
|
||||
- `insert(bitfield[, start])`, `clear(bitfield[, start])`
|
||||
- `get(bit)`, `set(bit[, value])`, `unset(bit)`
|
||||
- `setBatch(bits[, value])`, `unsetBatch(bits)`
|
||||
- `fill(value[, start[, end]])`
|
||||
- `findFirst`, `firstSet`, `firstUnset`, `findLast`, `lastSet`, `lastUnset`
|
||||
- `count`, `countSet`, `countUnset`
|
||||
|
||||
#### `Bitarray.constants.BYTES_PER_PAGE`
|
||||
|
||||
## Examples
|
||||
|
||||
### 1) Basic usage
|
||||
|
||||
```js
|
||||
const Bitarray = require('bitarray-native')
|
||||
const b = new Bitarray()
|
||||
b.set(1234, true)
|
||||
console.log(b.get(1234))
|
||||
```
|
||||
|
||||
### 2) Range fill
|
||||
|
||||
```js
|
||||
b.fill(true, 0, 1024)
|
||||
```
|
||||
|
||||
### 3) Count bits
|
||||
|
||||
```js
|
||||
const n = b.countSet(0, 2048)
|
||||
```
|
||||
|
||||
### 4) Cleanup
|
||||
|
||||
```js
|
||||
b.destroy()
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Call `destroy()` to release native resources.
|
||||
- Use `page()` for bulk page updates.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Bulk operations are much faster than per-bit updates.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Validate indexes and bitfield sizes to avoid RangeErrors.
|
||||
|
||||
## Integration Notes
|
||||
|
||||
- Used by `bitarray-universal` as the native backend.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Throws on invalid indices or bitfield sizes.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -0,0 +1,68 @@
|
||||
# crc-native
|
||||
|
||||
Native CRC32 bindings for JavaScript (Node/Bare).
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm i crc-native
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
- Computes CRC32 of a buffer using native bindings.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[Buffer] --> B[crc32]
|
||||
B --> C[uint32]
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
#### `const { crc32 } = require('crc-native')`
|
||||
|
||||
#### `crc32(buffer)`
|
||||
|
||||
Returns CRC32 as a number.
|
||||
|
||||
## Examples
|
||||
|
||||
### 1) Compute CRC
|
||||
|
||||
```js
|
||||
const { crc32 } = require('crc-native')
|
||||
const sum = crc32(Buffer.from('hello'))
|
||||
```
|
||||
|
||||
### 2) Compare checksums
|
||||
|
||||
```js
|
||||
if (crc32(bufA) !== crc32(bufB)) throw new Error('mismatch')
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use for integrity checks, not cryptographic security.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Native binding is fast; good for large buffers.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- CRC32 is not secure against adversarial collisions.
|
||||
|
||||
## Integration Notes
|
||||
|
||||
- Suitable for file integrity and protocol checksums.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Throws if buffer type is invalid.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -0,0 +1,86 @@
|
||||
# fx-native
|
||||
|
||||
JavaScript bindings for `libfx`, providing a lightweight native UI runtime.
|
||||
|
||||
## Install
|
||||
|
||||
```bash
|
||||
npm install fx-native
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
- App lifecycle and window management.
|
||||
- Views, text, images, and web views.
|
||||
- Runs on main thread with worker messaging.
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
flowchart LR
|
||||
A[App] --> B[Window]
|
||||
B --> C[View/Text/Image/WebView]
|
||||
```
|
||||
|
||||
## API
|
||||
|
||||
#### `const app = App.shared()` / `app.run()` / `app.broadcast(buffer)` / `app.destroy()`
|
||||
|
||||
#### `const window = new Window(x, y, width, height, options)`
|
||||
|
||||
#### `const view = new View(x, y, width, height)`
|
||||
|
||||
#### `const text = new Text(x, y, width, height)`
|
||||
|
||||
#### `const image = new Image(x, y, width, height)`
|
||||
|
||||
#### `const webView = new WebView(x, y, width, height)`
|
||||
|
||||
## Examples
|
||||
|
||||
### 1) Create a window
|
||||
|
||||
```js
|
||||
const { App, Window } = require('fx-native')
|
||||
const app = App.shared()
|
||||
const win = new Window(0, 0, 800, 600)
|
||||
app.run()
|
||||
```
|
||||
|
||||
### 2) Add a view
|
||||
|
||||
```js
|
||||
const view = new View(0, 0, 800, 600)
|
||||
win.appendChild(view)
|
||||
```
|
||||
|
||||
### 3) Load a web page
|
||||
|
||||
```js
|
||||
const web = new WebView(0, 0, 800, 600)
|
||||
web.loadURL('https://example.com')
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Create windows/views on the main thread only.
|
||||
|
||||
## Performance Notes
|
||||
|
||||
- Keep UI updates batched to reduce native calls.
|
||||
|
||||
## Security Considerations
|
||||
|
||||
- Validate URLs loaded in web views.
|
||||
|
||||
## Integration Notes
|
||||
|
||||
- Backed by `libfx`.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Errors surface via exceptions or event handlers.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -0,0 +1,87 @@
|
||||
# quickbit-native
|
||||
|
||||
Native bindings for fast bitfield operations.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
npm i quickbit-native
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
`quickbit-native` provides high-performance bit operations and indexes for efficient scanning of bitfields.
|
||||
|
||||
## API
|
||||
|
||||
### Bit operations
|
||||
|
||||
- `get(field, bit)`
|
||||
- `set(field, bit[, value])`
|
||||
- `fill(field, value[, start[, end]])`
|
||||
- `clear(field, chunk)`
|
||||
- `findFirst(field, value[, position])`
|
||||
- `findLast(field, value[, position])`
|
||||
|
||||
### Index
|
||||
|
||||
- `Index.from(fieldOrChunks)`
|
||||
- `index.update(bit)`
|
||||
- `index.skipFirst(value[, position])`
|
||||
- `index.skipLast(value[, position])`
|
||||
|
||||
## Examples
|
||||
|
||||
### Set and get bits
|
||||
|
||||
```js
|
||||
const quickbit = require('quickbit-native')
|
||||
|
||||
const field = Buffer.alloc(8)
|
||||
quickbit.set(field, 3, 1)
|
||||
console.log(quickbit.get(field, 3))
|
||||
```
|
||||
|
||||
### Find first set bit
|
||||
|
||||
```js
|
||||
const quickbit = require('quickbit-native')
|
||||
|
||||
const field = Buffer.alloc(8)
|
||||
quickbit.set(field, 10, 1)
|
||||
const pos = quickbit.findFirst(field, 1)
|
||||
```
|
||||
|
||||
### Use an index
|
||||
|
||||
```js
|
||||
const quickbit = require('quickbit-native')
|
||||
|
||||
const index = quickbit.Index.from(field)
|
||||
index.update(10)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Pre-allocate buffers for predictable memory use.
|
||||
- Use `Index` for repeated scans over large bitfields.
|
||||
|
||||
## Performance
|
||||
|
||||
- Native SIMD operations provide fast scans and updates.
|
||||
|
||||
## Security
|
||||
|
||||
- Validate bit positions to avoid out-of-range writes.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Ensure buffers are the expected size.
|
||||
|
||||
## Integration
|
||||
|
||||
- Used by Hypercore and block indexes.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -0,0 +1,593 @@
|
||||
# rabin-native - Content-Defined Chunking
|
||||
|
||||
## Overview
|
||||
|
||||
rabin-native provides JavaScript bindings for the Rabin fingerprinting algorithm, enabling content-defined chunking of data streams. This algorithm is essential for deduplication systems, as it identifies chunk boundaries based on data content rather than fixed positions, allowing identical chunks to be detected even when shifted within a file.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Content-defined chunking**: Chunk boundaries determined by data patterns
|
||||
- **Variable chunk sizes**: Configurable min/max chunk sizes
|
||||
- **Streaming API**: Process data incrementally
|
||||
- **High performance**: Native C++ implementation via librabin
|
||||
- **Deduplication support**: Identical content produces identical chunks
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **Distributed storage**: Efficient synchronization with rsync-like algorithms
|
||||
- **Version control**: Detect moved or shifted content
|
||||
- **Backup systems**: Deduplicate data across versions
|
||||
- **P2P file sharing**: Resumable downloads and efficient seeding
|
||||
- **Hypercore**: Used internally for block deduplication
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph LR
|
||||
subgraph "Input Stream"
|
||||
DATA[Raw Data<br/>Files/Streams]
|
||||
end
|
||||
|
||||
subgraph "Rabin Chunking"
|
||||
CHUNKER[Chunker<br/>Sliding Window]
|
||||
RABIN[Rabin Fingerprint<br/>Polynomial Hash]
|
||||
BOUNDARY[Boundary Detection<br/>Pattern Match]
|
||||
end
|
||||
|
||||
subgraph "Output"
|
||||
CHUNKS[Variable Chunks<br/>Content-Defined]
|
||||
META[Chunk Metadata<br/>Offset + Length]
|
||||
end
|
||||
|
||||
DATA --> CHUNKER
|
||||
CHUNKER --> RABIN
|
||||
RABIN --> BOUNDARY
|
||||
BOUNDARY --> CHUNKS
|
||||
BOUNDARY --> META
|
||||
```
|
||||
|
||||
### How Rabin Chunking Works
|
||||
|
||||
1. **Sliding Window**: A fixed-size window slides over the data
|
||||
2. **Fingerprint Calculation**: Rabin polynomial hash computed for each window
|
||||
3. **Boundary Detection**: When fingerprint matches a pattern (e.g., low bits = 0), a boundary is declared
|
||||
4. **Chunk Extraction**: Data between boundaries forms a chunk
|
||||
5. **Variable Size**: Chunks average to target size but vary based on content
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install rabin-native
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Chunking
|
||||
|
||||
```js
|
||||
const rabin = require('rabin-native')
|
||||
|
||||
const chunker = new rabin.Chunker()
|
||||
const chunks = []
|
||||
|
||||
// Push data incrementally
|
||||
for (const chunk of chunker.push(data)) {
|
||||
chunks.push(chunk)
|
||||
}
|
||||
|
||||
// Get final chunk
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) chunks.push(lastChunk)
|
||||
|
||||
console.log(`Split into ${chunks.length} chunks`)
|
||||
```
|
||||
|
||||
### Chunking a File
|
||||
|
||||
```js
|
||||
const fs = require('fs')
|
||||
const rabin = require('rabin-native')
|
||||
|
||||
const chunker = new rabin.Chunker()
|
||||
const stream = fs.createReadStream('large-file.bin')
|
||||
|
||||
const chunks = []
|
||||
|
||||
stream.on('data', (data) => {
|
||||
for (const chunk of chunker.push(data)) {
|
||||
chunks.push(chunk)
|
||||
console.log(`Chunk: offset=${chunk.offset}, length=${chunk.length}`)
|
||||
}
|
||||
})
|
||||
|
||||
stream.on('end', () => {
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) chunks.push(lastChunk)
|
||||
|
||||
console.log(`Total chunks: ${chunks.length}`)
|
||||
})
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### Chunker
|
||||
|
||||
Main class for Rabin chunking operations.
|
||||
|
||||
#### `new rabin.Chunker([options])`
|
||||
|
||||
Create a new Rabin chunker instance.
|
||||
|
||||
**Parameters:**
|
||||
- `options` (object, optional): Configuration options
|
||||
- `minSize` (number): Minimum chunk size in bytes (default: 512 KiB)
|
||||
- `maxSize` (number): Maximum chunk size in bytes (default: 8 MiB)
|
||||
|
||||
**Returns:** `Chunker` instance
|
||||
|
||||
#### `chunker.push(data)`
|
||||
|
||||
Push data into the chunker.
|
||||
|
||||
**Parameters:**
|
||||
- `data` (Buffer): Data to process
|
||||
|
||||
**Returns:** Iterator yielding chunk objects
|
||||
|
||||
Each chunk object:
|
||||
```js
|
||||
{
|
||||
length: number, // Size of chunk in bytes
|
||||
offset: number // Offset within the stream
|
||||
}
|
||||
```
|
||||
|
||||
#### `chunker.end()`
|
||||
|
||||
Finalize the chunker and return any trailing data.
|
||||
|
||||
**Returns:** Chunk object or `null` if no trailing data
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Example 1: Basic File Deduplication
|
||||
|
||||
```js
|
||||
const fs = require('fs')
|
||||
const crypto = require('crypto')
|
||||
const rabin = require('rabin-native')
|
||||
|
||||
async function deduplicateFile(filepath) {
|
||||
const chunker = new rabin.Chunker({
|
||||
minSize: 64 * 1024, // 64 KiB minimum
|
||||
maxSize: 1024 * 1024 // 1 MiB maximum
|
||||
})
|
||||
|
||||
const chunks = []
|
||||
const chunkHashes = new Map()
|
||||
|
||||
const stream = fs.createReadStream(filepath)
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
stream.on('data', (data) => {
|
||||
for (const chunk of chunker.push(data)) {
|
||||
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
|
||||
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
|
||||
|
||||
if (!chunkHashes.has(hash)) {
|
||||
chunkHashes.set(hash, chunkData)
|
||||
}
|
||||
|
||||
chunks.push({ hash, ...chunk })
|
||||
}
|
||||
})
|
||||
|
||||
stream.on('end', () => {
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) {
|
||||
// Process final chunk
|
||||
chunks.push(lastChunk)
|
||||
}
|
||||
|
||||
resolve({
|
||||
totalChunks: chunks.length,
|
||||
uniqueChunks: chunkHashes.size,
|
||||
deduplicationRatio: chunks.length / chunkHashes.size,
|
||||
chunks
|
||||
})
|
||||
})
|
||||
|
||||
stream.on('error', reject)
|
||||
})
|
||||
}
|
||||
|
||||
// Usage
|
||||
deduplicateFile('./myfile.dat').then(result => {
|
||||
console.log(`Deduplication ratio: ${result.deduplicationRatio.toFixed(2)}x`)
|
||||
})
|
||||
```
|
||||
|
||||
### Example 2: Resumable File Upload
|
||||
|
||||
```js
|
||||
const rabin = require('rabin-native')
|
||||
const fs = require('fs')
|
||||
|
||||
class ResumableUploader {
|
||||
constructor(chunkSize = { min: 256 * 1024, max: 2 * 1024 * 1024 }) {
|
||||
this.chunker = new rabin.Chunker(chunkSize)
|
||||
this.uploadedChunks = new Set()
|
||||
}
|
||||
|
||||
async uploadFile(filepath, uploadChunk) {
|
||||
const stream = fs.createReadStream(filepath)
|
||||
const pendingChunks = []
|
||||
|
||||
return new Promise((resolve, reject) => {
|
||||
stream.on('data', (data) => {
|
||||
for (const chunk of this.chunker.push(data)) {
|
||||
const chunkId = `${chunk.offset}-${chunk.length}`
|
||||
|
||||
if (!this.uploadedChunks.has(chunkId)) {
|
||||
pendingChunks.push(uploadChunk(chunkId, chunk, data))
|
||||
}
|
||||
}
|
||||
})
|
||||
|
||||
stream.on('end', async () => {
|
||||
const lastChunk = this.chunker.end()
|
||||
if (lastChunk) {
|
||||
const chunkId = `final-${lastChunk.length}`
|
||||
pendingChunks.push(uploadChunk(chunkId, lastChunk))
|
||||
}
|
||||
|
||||
await Promise.all(pendingChunks)
|
||||
resolve()
|
||||
})
|
||||
|
||||
stream.on('error', reject)
|
||||
})
|
||||
}
|
||||
|
||||
markUploaded(chunkId) {
|
||||
this.uploadedChunks.add(chunkId)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const uploader = new ResumableUploader()
|
||||
|
||||
// Simulate server that tracks uploaded chunks
|
||||
const serverChunks = new Set()
|
||||
|
||||
uploader.uploadFile('./large-file.zip', async (id, meta, buffer) => {
|
||||
if (!serverChunks.has(id)) {
|
||||
console.log(`Uploading chunk ${id} (${meta.length} bytes)`)
|
||||
// await uploadToServer(id, buffer)
|
||||
serverChunks.add(id)
|
||||
uploader.markUploaded(id)
|
||||
} else {
|
||||
console.log(`Skipping already uploaded chunk ${id}`)
|
||||
}
|
||||
})
|
||||
```
|
||||
|
||||
### Example 3: Content-Addressed Storage
|
||||
|
||||
```js
|
||||
const rabin = require('rabin-native')
|
||||
const crypto = require('crypto')
|
||||
|
||||
class ContentAddressedStore {
|
||||
constructor() {
|
||||
this.chunks = new Map()
|
||||
}
|
||||
|
||||
async store(data) {
|
||||
const chunker = new rabin.Chunker({
|
||||
minSize: 32 * 1024,
|
||||
maxSize: 256 * 1024
|
||||
})
|
||||
|
||||
const chunkIds = []
|
||||
|
||||
for (const chunk of chunker.push(data)) {
|
||||
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
|
||||
const id = crypto.createHash('sha256').update(chunkData).digest('hex')
|
||||
|
||||
if (!this.chunks.has(id)) {
|
||||
this.chunks.set(id, chunkData)
|
||||
}
|
||||
|
||||
chunkIds.push(id)
|
||||
}
|
||||
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) {
|
||||
const lastData = data.slice(-lastChunk.length)
|
||||
const id = crypto.createHash('sha256').update(lastData).digest('hex')
|
||||
|
||||
if (!this.chunks.has(id)) {
|
||||
this.chunks.set(id, lastData)
|
||||
}
|
||||
|
||||
chunkIds.push(id)
|
||||
}
|
||||
|
||||
return {
|
||||
rootHash: crypto.createHash('sha256').update(data).digest('hex'),
|
||||
chunks: chunkIds
|
||||
}
|
||||
}
|
||||
|
||||
retrieve(chunkIds) {
|
||||
const chunks = chunkIds.map(id => this.chunks.get(id))
|
||||
return Buffer.concat(chunks)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const store = new ContentAddressedStore()
|
||||
|
||||
const data = Buffer.alloc(1024 * 1024)
|
||||
data.fill('A') // 1MB of data
|
||||
|
||||
const { rootHash, chunks } = await store.store(data)
|
||||
console.log(`Stored as ${chunks.length} chunks`)
|
||||
|
||||
// Store similar data - only new chunks are added
|
||||
const data2 = Buffer.concat([data, Buffer.from('B')])
|
||||
const result2 = await store.store(data2)
|
||||
console.log(`Similar data: ${result2.chunks.length} chunks (many reused)`)
|
||||
```
|
||||
|
||||
### Example 4: Delta Sync Algorithm
|
||||
|
||||
```js
|
||||
const rabin = require('rabin-native')
|
||||
const crypto = require('crypto')
|
||||
|
||||
class DeltaSync {
|
||||
constructor() {
|
||||
this.chunkIndex = new Map() // hash -> [files]
|
||||
}
|
||||
|
||||
indexFile(filepath, data) {
|
||||
const chunker = new rabin.Chunker()
|
||||
const chunks = []
|
||||
|
||||
for (const chunk of chunker.push(data)) {
|
||||
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
|
||||
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
|
||||
|
||||
if (!this.chunkIndex.has(hash)) {
|
||||
this.chunkIndex.set(hash, [])
|
||||
}
|
||||
this.chunkIndex.get(hash).push({ filepath, offset: chunk.offset })
|
||||
|
||||
chunks.push(hash)
|
||||
}
|
||||
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) {
|
||||
const lastData = data.slice(-lastChunk.length)
|
||||
const hash = crypto.createHash('sha256').update(lastData).digest('hex')
|
||||
|
||||
if (!this.chunkIndex.has(hash)) {
|
||||
this.chunkIndex.set(hash, [])
|
||||
}
|
||||
this.chunkIndex.get(hash).push({
|
||||
filepath,
|
||||
offset: data.length - lastChunk.length
|
||||
})
|
||||
|
||||
chunks.push(hash)
|
||||
}
|
||||
|
||||
return chunks
|
||||
}
|
||||
|
||||
computeDelta(oldData, newData) {
|
||||
const oldChunks = this.getChunkHashes(oldData)
|
||||
const newChunks = this.getChunkHashes(newData)
|
||||
|
||||
const unchanged = []
|
||||
const changed = []
|
||||
|
||||
for (let i = 0; i < newChunks.length; i++) {
|
||||
if (oldChunks.includes(newChunks[i])) {
|
||||
unchanged.push({ index: i, hash: newChunks[i] })
|
||||
} else {
|
||||
changed.push({ index: i, hash: newChunks[i] })
|
||||
}
|
||||
}
|
||||
|
||||
return { unchanged, changed }
|
||||
}
|
||||
|
||||
getChunkHashes(data) {
|
||||
const chunker = new rabin.Chunker()
|
||||
const hashes = []
|
||||
|
||||
for (const chunk of chunker.push(data)) {
|
||||
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
|
||||
const hash = crypto.createHash('sha256').update(chunkData).digest('hex')
|
||||
hashes.push(hash)
|
||||
}
|
||||
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) {
|
||||
const lastData = data.slice(-lastChunk.length)
|
||||
const hash = crypto.createHash('sha256').update(lastData).digest('hex')
|
||||
hashes.push(hash)
|
||||
}
|
||||
|
||||
return hashes
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const sync = new DeltaSync()
|
||||
|
||||
const v1 = Buffer.from('Hello World! This is version 1.')
|
||||
const v2 = Buffer.from('Hello World! This is version 2 with changes.')
|
||||
|
||||
sync.indexFile('doc.txt', v1)
|
||||
const delta = sync.computeDelta(v1, v2)
|
||||
|
||||
console.log(`Unchanged chunks: ${delta.unchanged.length}`)
|
||||
console.log(`Changed chunks: ${delta.changed.length}`)
|
||||
```
|
||||
|
||||
## Configuration Options
|
||||
|
||||
### Choosing Chunk Sizes
|
||||
|
||||
```js
|
||||
// Small chunks - better deduplication, more overhead
|
||||
const small = new rabin.Chunker({
|
||||
minSize: 16 * 1024, // 16 KiB
|
||||
maxSize: 128 * 1024 // 128 KiB
|
||||
})
|
||||
|
||||
// Medium chunks - balanced (default)
|
||||
const medium = new rabin.Chunker()
|
||||
// min: 512 KiB, max: 8 MiB
|
||||
|
||||
// Large chunks - less overhead, less granular
|
||||
const large = new rabin.Chunker({
|
||||
minSize: 2 * 1024 * 1024, // 2 MiB
|
||||
maxSize: 16 * 1024 * 1024 // 16 MiB
|
||||
})
|
||||
```
|
||||
|
||||
### Trade-offs
|
||||
|
||||
| Size | Deduplication | Overhead | Use Case |
|
||||
|------|--------------|----------|----------|
|
||||
| Small | Excellent | High | Source code, small files |
|
||||
| Medium | Good | Medium | General purpose |
|
||||
| Large | Moderate | Low | Large media files |
|
||||
|
||||
## Performance Characteristics
|
||||
|
||||
- **Throughput**: Processes data at ~100-500 MB/s depending on hardware
|
||||
- **Memory**: O(1) - processes streaming data without buffering entire file
|
||||
- **CPU**: Single-threaded, can be parallelized across multiple files
|
||||
- **Chunk variance**: Typically ±25% around average of (min + max) / 2
|
||||
|
||||
## Integration with Other Modules
|
||||
|
||||
### With Hypercore
|
||||
|
||||
```js
|
||||
const Hypercore = require('hypercore')
|
||||
const rabin = require('rabin-native')
|
||||
|
||||
const core = new Hypercore('./my-core')
|
||||
|
||||
function appendWithChunking(data) {
|
||||
const chunker = new rabin.Chunker()
|
||||
|
||||
for (const chunk of chunker.push(data)) {
|
||||
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
|
||||
core.append(chunkData)
|
||||
}
|
||||
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) {
|
||||
const lastData = data.slice(-lastChunk.length)
|
||||
core.append(lastData)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### With Hyperdrive
|
||||
|
||||
```js
|
||||
const Hyperdrive = require('hyperdrive')
|
||||
const rabin = require('rabin-native')
|
||||
|
||||
const drive = new Hyperdrive('./my-drive')
|
||||
|
||||
async function putFileChunked(path, data) {
|
||||
const chunker = new rabin.Chunker()
|
||||
const chunks = []
|
||||
|
||||
for (const chunk of chunker.push(data)) {
|
||||
const chunkData = data.slice(chunk.offset, chunk.offset + chunk.length)
|
||||
chunks.push(chunkData)
|
||||
}
|
||||
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) {
|
||||
const lastData = data.slice(-lastChunk.length)
|
||||
chunks.push(lastData)
|
||||
}
|
||||
|
||||
// Store chunk index and chunks
|
||||
await drive.put(path, Buffer.concat(chunks))
|
||||
await drive.put(`${path}.chunks`, JSON.stringify(chunks.map(c => c.length)))
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Consistent Configuration
|
||||
|
||||
```js
|
||||
// Use same settings across your application
|
||||
const CHUNK_CONFIG = {
|
||||
minSize: 256 * 1024,
|
||||
maxSize: 2 * 1024 * 1024
|
||||
}
|
||||
|
||||
// Reuse configuration everywhere
|
||||
const chunker = new rabin.Chunker(CHUNK_CONFIG)
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```js
|
||||
try {
|
||||
const chunker = new rabin.Chunker()
|
||||
|
||||
for (const chunk of chunker.push(largeBuffer)) {
|
||||
if (chunk.length > MAX_SAFE_SIZE) {
|
||||
throw new Error('Chunk too large')
|
||||
}
|
||||
processChunk(chunk)
|
||||
}
|
||||
} catch (err) {
|
||||
console.error('Chunking failed:', err)
|
||||
}
|
||||
```
|
||||
|
||||
### Resource Management
|
||||
|
||||
```js
|
||||
// For processing many files
|
||||
async function* chunkFiles(filePaths) {
|
||||
for (const path of filePaths) {
|
||||
const chunker = new rabin.Chunker()
|
||||
const data = await fs.promises.readFile(path)
|
||||
|
||||
const chunks = []
|
||||
for (const chunk of chunker.push(data)) {
|
||||
chunks.push(chunk)
|
||||
}
|
||||
|
||||
const lastChunk = chunker.end()
|
||||
if (lastChunk) chunks.push(lastChunk)
|
||||
|
||||
yield { path, chunks }
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
**Module Type**: Algorithm/Utility | **Ecosystem Role**: Data Processing | **Dependencies**: librabin (native)
|
||||
@@ -0,0 +1,546 @@
|
||||
# rocksdb-native - RocksDB Bindings for JavaScript
|
||||
|
||||
## Overview
|
||||
|
||||
rocksdb-native provides JavaScript bindings for RocksDB, Facebook's high-performance embedded database. It offers a fast key-value store with advanced features like column families, snapshots, and atomic transactions.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Native bindings**: Direct access to RocksDB C++ library
|
||||
- **Batch operations**: Atomic read/write batches
|
||||
- **Column families**: Organized data storage
|
||||
- **Snapshots**: Point-in-time database views
|
||||
- **Async API**: Non-blocking operations
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **High-performance storage**: Low-latency key-value operations
|
||||
- **Large datasets**: Efficient handling of terabyte-scale data
|
||||
- **Indexed storage**: Fast lookups and range scans
|
||||
- **Blockchain indexing**: Store and query blockchain data
|
||||
- **Hypercore storage**: Backend for Hypercore and Hyperbee
|
||||
|
||||
## Architecture
|
||||
|
||||
```mermaid
|
||||
graph TB
|
||||
subgraph "JavaScript"
|
||||
APP[Application]
|
||||
API[rocksdb-native API]
|
||||
end
|
||||
|
||||
subgraph "Native Layer"
|
||||
BIND[RocksDB Bindings]
|
||||
NAPI[Node-API]
|
||||
end
|
||||
|
||||
subgraph "Storage Engine"
|
||||
ROCKS[RocksDB C++]
|
||||
MEM[MemTable]
|
||||
SST[SSTables]
|
||||
WAL[Write-Ahead Log]
|
||||
end
|
||||
|
||||
APP --> API
|
||||
API --> BIND
|
||||
BIND --> NAPI
|
||||
NAPI --> ROCKS
|
||||
ROCKS --> MEM
|
||||
ROCKS --> SST
|
||||
ROCKS --> WAL
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install rocksdb-native
|
||||
```
|
||||
|
||||
**Note:** On Linux, `libatomic` must be installed:
|
||||
|
||||
```bash
|
||||
# Ubuntu/Debian
|
||||
sudo apt-get install libatomic1
|
||||
|
||||
# CentOS/RHEL/Fedora
|
||||
sudo yum install libatomic
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
```js
|
||||
const RocksDB = require('rocksdb-native')
|
||||
|
||||
// Open database
|
||||
const db = new RocksDB('./example.db')
|
||||
|
||||
// Write data
|
||||
const w = db.write()
|
||||
w.put('hello', 'world')
|
||||
await w.flush()
|
||||
|
||||
// Read data
|
||||
const r = db.read()
|
||||
const p = r.get('hello')
|
||||
r.flush()
|
||||
|
||||
console.log(await p) // 'world'
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### RocksDB
|
||||
|
||||
Main database class.
|
||||
|
||||
#### `new RocksDB(path[, options])`
|
||||
|
||||
Open or create a database.
|
||||
|
||||
**Parameters:**
|
||||
- `path` (string): Database directory path
|
||||
- `options` (object, optional):
|
||||
- `createIfMissing` (boolean): Create if doesn't exist (default: true)
|
||||
- `errorIfExists` (boolean): Error if exists (default: false)
|
||||
|
||||
**Example:**
|
||||
```js
|
||||
const db = new RocksDB('./mydb', {
|
||||
createIfMissing: true
|
||||
})
|
||||
```
|
||||
|
||||
### Write Batch
|
||||
|
||||
#### `db.write()`
|
||||
|
||||
Create a write batch.
|
||||
|
||||
**Returns:** WriteBatch
|
||||
|
||||
#### `write.put(key, value)`
|
||||
|
||||
Add a put operation to the batch.
|
||||
|
||||
**Parameters:**
|
||||
- `key` (string | Buffer): Key
|
||||
- `value` (string | Buffer): Value
|
||||
|
||||
**Returns:** WriteBatch (for chaining)
|
||||
|
||||
#### `write.delete(key)`
|
||||
|
||||
Add a delete operation to the batch.
|
||||
|
||||
**Parameters:**
|
||||
- `key` (string | Buffer): Key to delete
|
||||
|
||||
**Returns:** WriteBatch (for chaining)
|
||||
|
||||
#### `await write.flush()`
|
||||
|
||||
Execute all operations in the batch.
|
||||
|
||||
**Returns:** Promise<void>
|
||||
|
||||
### Read Snapshot
|
||||
|
||||
#### `db.read()`
|
||||
|
||||
Create a read snapshot.
|
||||
|
||||
**Returns:** ReadSnapshot
|
||||
|
||||
#### `snapshot.get(key)`
|
||||
|
||||
Get a value from the snapshot.
|
||||
|
||||
**Parameters:**
|
||||
- `key` (string | Buffer): Key to look up
|
||||
|
||||
**Returns:** Promise<Buffer | null>
|
||||
|
||||
#### `snapshot.flush()`
|
||||
|
||||
Finalize the snapshot (required after gets).
|
||||
|
||||
**Returns:** void
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Example 1: Key-Value Store
|
||||
|
||||
```js
|
||||
const RocksDB = require('rocksdb-native')
|
||||
|
||||
class KeyValueStore {
|
||||
constructor(dbPath) {
|
||||
this.db = new RocksDB(dbPath)
|
||||
}
|
||||
|
||||
async set(key, value) {
|
||||
const w = this.db.write()
|
||||
w.put(key, value)
|
||||
await w.flush()
|
||||
}
|
||||
|
||||
async get(key) {
|
||||
const r = this.db.read()
|
||||
const p = r.get(key)
|
||||
r.flush()
|
||||
return await p
|
||||
}
|
||||
|
||||
async delete(key) {
|
||||
const w = this.db.write()
|
||||
w.delete(key)
|
||||
await w.flush()
|
||||
}
|
||||
|
||||
async batch(operations) {
|
||||
const w = this.db.write()
|
||||
|
||||
for (const op of operations) {
|
||||
if (op.type === 'put') {
|
||||
w.put(op.key, op.value)
|
||||
} else if (op.type === 'del') {
|
||||
w.delete(op.key)
|
||||
}
|
||||
}
|
||||
|
||||
await w.flush()
|
||||
}
|
||||
|
||||
async close() {
|
||||
// Cleanup if needed
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const store = new KeyValueStore('./kv-store')
|
||||
|
||||
await store.set('user:1', JSON.stringify({ name: 'Alice', age: 30 }))
|
||||
await store.set('user:2', JSON.stringify({ name: 'Bob', age: 25 }))
|
||||
|
||||
const user1 = JSON.parse(await store.get('user:1'))
|
||||
console.log(user1) // { name: 'Alice', age: 30 }
|
||||
|
||||
await store.close()
|
||||
```
|
||||
|
||||
### Example 2: Counter Store
|
||||
|
||||
```js
|
||||
const RocksDB = require('rocksdb-native')
|
||||
|
||||
class CounterStore {
|
||||
constructor(dbPath) {
|
||||
this.db = new RocksDB(dbPath)
|
||||
}
|
||||
|
||||
async increment(key, amount = 1) {
|
||||
const r = this.db.read()
|
||||
const p = r.get(key)
|
||||
r.flush()
|
||||
|
||||
const current = parseInt(await p || '0', 10)
|
||||
const next = current + amount
|
||||
|
||||
const w = this.db.write()
|
||||
w.put(key, String(next))
|
||||
await w.flush()
|
||||
|
||||
return next
|
||||
}
|
||||
|
||||
async decrement(key, amount = 1) {
|
||||
return this.increment(key, -amount)
|
||||
}
|
||||
|
||||
async get(key) {
|
||||
const r = this.db.read()
|
||||
const p = r.get(key)
|
||||
r.flush()
|
||||
|
||||
const val = await p
|
||||
return val ? parseInt(val, 10) : 0
|
||||
}
|
||||
|
||||
async reset(key) {
|
||||
const w = this.db.write()
|
||||
w.put(key, '0')
|
||||
await w.flush()
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const counters = new CounterStore('./counters')
|
||||
|
||||
await counters.increment('visits') // 1
|
||||
await counters.increment('visits') // 2
|
||||
await counters.increment('visits', 5) // 7
|
||||
|
||||
const visits = await counters.get('visits')
|
||||
console.log('Total visits:', visits)
|
||||
```
|
||||
|
||||
### Example 3: Time-Series Store
|
||||
|
||||
```js
|
||||
const RocksDB = require('rocksdb-native')
|
||||
|
||||
class TimeSeriesStore {
|
||||
constructor(dbPath) {
|
||||
this.db = new RocksDB(dbPath)
|
||||
}
|
||||
|
||||
key(metric, timestamp) {
|
||||
return `${metric}:${timestamp.toString().padStart(15, '0')}`
|
||||
}
|
||||
|
||||
async write(metric, value, timestamp = Date.now()) {
|
||||
const w = this.db.write()
|
||||
w.put(
|
||||
this.key(metric, timestamp),
|
||||
JSON.stringify({ value, timestamp })
|
||||
)
|
||||
await w.flush()
|
||||
}
|
||||
|
||||
async readRange(metric, start, end) {
|
||||
const results = []
|
||||
const startKey = this.key(metric, start)
|
||||
const endKey = this.key(metric, end)
|
||||
|
||||
// Note: rocksdb-native doesn't expose iterators directly
|
||||
// In production, you'd use column families or prefix iteration
|
||||
// This is a simplified example
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
async getLatest(metric) {
|
||||
// Simplified - would use reverse iteration in production
|
||||
const r = this.db.read()
|
||||
const p = r.get(this.key(metric, Date.now()))
|
||||
r.flush()
|
||||
|
||||
const data = await p
|
||||
return data ? JSON.parse(data) : null
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const ts = new TimeSeriesStore('./timeseries')
|
||||
|
||||
// Write metrics
|
||||
await ts.write('cpu.usage', 45.2)
|
||||
await ts.write('cpu.usage', 52.1)
|
||||
await ts.write('memory.usage', 78.5)
|
||||
|
||||
const latest = await ts.getLatest('cpu.usage')
|
||||
console.log('Latest CPU:', latest)
|
||||
```
|
||||
|
||||
### Example 4: Document Store
|
||||
|
||||
```js
|
||||
const RocksDB = require('rocksdb-native')
|
||||
|
||||
class DocumentStore {
|
||||
constructor(dbPath) {
|
||||
this.db = new RocksDB(dbPath)
|
||||
}
|
||||
|
||||
async insert(collection, id, doc) {
|
||||
const key = `${collection}:${id}`
|
||||
const w = this.db.write()
|
||||
w.put(key, JSON.stringify(doc))
|
||||
await w.flush()
|
||||
return id
|
||||
}
|
||||
|
||||
async findById(collection, id) {
|
||||
const key = `${collection}:${id}`
|
||||
const r = this.db.read()
|
||||
const p = r.get(key)
|
||||
r.flush()
|
||||
|
||||
const data = await p
|
||||
return data ? JSON.parse(data) : null
|
||||
}
|
||||
|
||||
async update(collection, id, updates) {
|
||||
const existing = await this.findById(collection, id)
|
||||
if (!existing) return null
|
||||
|
||||
const updated = { ...existing, ...updates }
|
||||
await this.insert(collection, id, updated)
|
||||
return updated
|
||||
}
|
||||
|
||||
async delete(collection, id) {
|
||||
const key = `${collection}:${id}`
|
||||
const w = this.db.write()
|
||||
w.delete(key)
|
||||
await w.flush()
|
||||
}
|
||||
|
||||
async batchInsert(collection, docs) {
|
||||
const w = this.db.write()
|
||||
const ids = []
|
||||
|
||||
for (const doc of docs) {
|
||||
const id = this.generateId()
|
||||
const key = `${collection}:${id}`
|
||||
w.put(key, JSON.stringify(doc))
|
||||
ids.push(id)
|
||||
}
|
||||
|
||||
await w.flush()
|
||||
return ids
|
||||
}
|
||||
|
||||
generateId() {
|
||||
return Date.now().toString(36) + Math.random().toString(36).substr(2)
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const docs = new DocumentStore('./documents')
|
||||
|
||||
// Insert
|
||||
const userId = await docs.insert('users', null, {
|
||||
name: 'Alice',
|
||||
email: '[email protected]',
|
||||
createdAt: Date.now()
|
||||
})
|
||||
|
||||
// Find
|
||||
const user = await docs.findById('users', userId)
|
||||
console.log('User:', user)
|
||||
|
||||
// Update
|
||||
await docs.update('users', userId, { lastLogin: Date.now() })
|
||||
|
||||
// Batch insert
|
||||
const posts = [
|
||||
{ title: 'Post 1', content: 'Content 1' },
|
||||
{ title: 'Post 2', content: 'Content 2' }
|
||||
]
|
||||
const postIds = await docs.batchInsert('posts', posts)
|
||||
console.log('Created posts:', postIds)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Batch Operations
|
||||
|
||||
```js
|
||||
// Good: Batch multiple operations
|
||||
const w = db.write()
|
||||
for (const item of items) {
|
||||
w.put(item.key, item.value)
|
||||
}
|
||||
await w.flush()
|
||||
|
||||
// Less efficient: Individual writes
|
||||
for (const item of items) {
|
||||
const w = db.write()
|
||||
w.put(item.key, item.value)
|
||||
await w.flush()
|
||||
}
|
||||
```
|
||||
|
||||
### Read Snapshots
|
||||
|
||||
```js
|
||||
// Always call flush() after reads
|
||||
const r = db.read()
|
||||
const p1 = r.get('key1')
|
||||
const p2 = r.get('key2')
|
||||
r.flush() // Required!
|
||||
|
||||
const [val1, val2] = await Promise.all([p1, p2])
|
||||
```
|
||||
|
||||
### Error Handling
|
||||
|
||||
```js
|
||||
try {
|
||||
const w = db.write()
|
||||
w.put('key', 'value')
|
||||
await w.flush()
|
||||
} catch (err) {
|
||||
if (err.message.includes('IO error')) {
|
||||
console.error('Disk error:', err)
|
||||
} else {
|
||||
console.error('Database error:', err)
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Resource Cleanup
|
||||
|
||||
```js
|
||||
// Ensure proper cleanup
|
||||
process.on('SIGINT', async () => {
|
||||
// RocksDB handles cleanup on process exit
|
||||
// But explicit close is better for tests
|
||||
process.exit(0)
|
||||
})
|
||||
```
|
||||
|
||||
## Integration with Other Modules
|
||||
|
||||
### With Hypercore
|
||||
|
||||
```js
|
||||
const RocksDB = require('rocksdb-native')
|
||||
const Hypercore = require('hypercore')
|
||||
|
||||
const db = new RocksDB('./core-storage')
|
||||
const core = new Hypercore(storage)
|
||||
|
||||
// Custom storage using RocksDB
|
||||
function createRocksStorage(db) {
|
||||
return {
|
||||
read: async (offset, size) => {
|
||||
const r = db.read()
|
||||
const p = r.get(`block:${offset}`)
|
||||
r.flush()
|
||||
return await p
|
||||
},
|
||||
write: async (offset, data) => {
|
||||
const w = db.write()
|
||||
w.put(`block:${offset}`, data)
|
||||
await w.flush()
|
||||
}
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### With Hyperbee
|
||||
|
||||
```js
|
||||
const RocksDB = require('rocksdb-native')
|
||||
const Hyperbee = require('hyperbee')
|
||||
|
||||
const db = new RocksDB('./bee-storage')
|
||||
// Hyperbee uses RocksDB as underlying storage
|
||||
```
|
||||
|
||||
## Performance Tips
|
||||
|
||||
1. **Use batches** for multiple writes
|
||||
2. **Compact keys** - shorter keys use less memory
|
||||
3. **Async operations** - don't block the event loop
|
||||
4. **Proper flush** - always flush read snapshots
|
||||
5. **Monitor space** - RocksDB compacts automatically but uses disk space
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
**Module Type**: Storage | **Ecosystem Role**: Database Engine | **Dependencies**: librocksdb
|
||||
@@ -0,0 +1,65 @@
|
||||
# simdle-native
|
||||
|
||||
Native SIMD bit operations bindings.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
npm i simdle-native
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
Provides fast SIMD helpers for bitwise operations on buffers.
|
||||
|
||||
## API
|
||||
|
||||
### Functions
|
||||
|
||||
- `allo`, `allz`, `and`, `clear`, `clo`, `clz`, `cnt`, `cto`, `ctz`, `not`, `or`, `sum`, `xor`
|
||||
|
||||
## Examples
|
||||
|
||||
### Count set bits
|
||||
|
||||
```js
|
||||
const simdle = require('simdle-native')
|
||||
|
||||
const count = simdle.cnt(Buffer.from([0xff, 0x00]))
|
||||
```
|
||||
|
||||
### AND two buffers
|
||||
|
||||
```js
|
||||
const out = simdle.and(a, b)
|
||||
```
|
||||
|
||||
### Clear bits
|
||||
|
||||
```js
|
||||
simdle.clear(buf, 0, 8)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Ensure buffer sizes match for binary ops.
|
||||
|
||||
## Performance
|
||||
|
||||
- SIMD acceleration makes bit ops fast on supported CPUs.
|
||||
|
||||
## Security
|
||||
|
||||
- Validate inputs to avoid out-of-range access.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Functions may throw on invalid buffer sizes.
|
||||
|
||||
## Integration
|
||||
|
||||
- Used in low-level indexing and bitmap logic.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -0,0 +1,21 @@
|
||||
# sodium-native - Crypto Bindings
|
||||
|
||||
## Overview
|
||||
|
||||
High-perf libsodium addon. Ed25519/XChaCha20Poly/blake2b.
|
||||
|
||||
**Ex**:
|
||||
|
||||
```js
|
||||
const sodium = require('sodium-native')
|
||||
sodium.crypto_sign_keypair(pub, priv)
|
||||
sodium.crypto_sign_detached(sig, msg, priv)
|
||||
```
|
||||
|
||||
**Perf**: SIMD opt, faster than universal.
|
||||
|
||||
**Hyper**: Core signing, noise.
|
||||
|
||||
**Prebuilds**: Multi-arch.
|
||||
|
||||
**Source**: github/holepunchto/sodium-native
|
||||
@@ -0,0 +1,73 @@
|
||||
# sqlite3-native
|
||||
|
||||
Async SQLite3 bindings with VFS support.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
npm i sqlite3-native
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
Provides a native SQLite3 driver with async APIs and optional VFS integrations.
|
||||
|
||||
## API
|
||||
|
||||
### `new SQLite3()`
|
||||
|
||||
### `db.exec(sql)`
|
||||
|
||||
- **returns** query results
|
||||
|
||||
## Examples
|
||||
|
||||
### Create a table and insert data
|
||||
|
||||
```js
|
||||
const SQLite3 = require('sqlite3-native')
|
||||
|
||||
const db = new SQLite3()
|
||||
await db.exec('create table test (id integer, name text)')
|
||||
await db.exec("insert into test values (1, 'hello')")
|
||||
```
|
||||
|
||||
### Query rows
|
||||
|
||||
```js
|
||||
const rows = await db.exec('select * from test')
|
||||
```
|
||||
|
||||
### Use in a service
|
||||
|
||||
```js
|
||||
async function init() {
|
||||
const db = new SQLite3()
|
||||
await db.exec('pragma journal_mode=wal')
|
||||
return db
|
||||
}
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Use WAL mode for concurrent reads/writes.
|
||||
|
||||
## Performance
|
||||
|
||||
- Performance depends on disk I/O and query complexity.
|
||||
|
||||
## Security
|
||||
|
||||
- Use parameterized queries to avoid injection.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Catch errors from `exec` and handle transactions.
|
||||
|
||||
## Integration
|
||||
|
||||
- Suitable for local persistence in Pear apps.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -0,0 +1,70 @@
|
||||
# tt-native
|
||||
|
||||
PTY bindings for Node.js via libtt.
|
||||
|
||||
## Install
|
||||
|
||||
```sh
|
||||
npm i tt-native
|
||||
```
|
||||
|
||||
## Overview
|
||||
|
||||
`tt-native` provides a pseudo-terminal interface. `spawn` returns a duplex stream representing the PTY.
|
||||
|
||||
## API
|
||||
|
||||
### `spawn(file[, args][, options])`
|
||||
|
||||
- **options**: `width`, `height`, `env`, `cwd`
|
||||
- Returns `pty` stream with `pid`, `width`, `height`
|
||||
- `pty.resize(width, height)`
|
||||
- `pty.kill([signal])`
|
||||
- `pty.on('exit', fn)`
|
||||
|
||||
## Examples
|
||||
|
||||
### Spawn a shell
|
||||
|
||||
```js
|
||||
const { spawn } = require('tt-native')
|
||||
|
||||
const pty = spawn('bash', [], { width: 80, height: 24 })
|
||||
pty.pipe(process.stdout)
|
||||
```
|
||||
|
||||
### Resize PTY
|
||||
|
||||
```js
|
||||
pty.resize(100, 30)
|
||||
```
|
||||
|
||||
### Kill process
|
||||
|
||||
```js
|
||||
pty.kill('SIGTERM')
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
- Handle `exit` to cleanup resources.
|
||||
|
||||
## Performance
|
||||
|
||||
- PTY throughput depends on OS and process.
|
||||
|
||||
## Security
|
||||
|
||||
- Avoid spawning untrusted commands.
|
||||
|
||||
## Error Handling
|
||||
|
||||
- Handle spawn failures and permission issues.
|
||||
|
||||
## Integration
|
||||
|
||||
- Useful for terminal UIs and process control.
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
@@ -0,0 +1,627 @@
|
||||
# udx-native - Reliable UDP Streams
|
||||
|
||||
## Overview
|
||||
|
||||
udx-native provides reliable, multiplexed, and congestion-controlled streams over UDP. It's designed specifically for peer-to-peer networking with no handshakes, no encryption, and minimal overhead - just fast, composable streams and messages.
|
||||
|
||||
### Key Features
|
||||
|
||||
- **Reliable streams**: TCP-like reliability over UDP
|
||||
- **Multiplexing**: Multiple streams per socket
|
||||
- **Congestion control**: Automatic bandwidth adaptation
|
||||
- **Zero handshakes**: Direct stream creation
|
||||
- **Message support**: Datagram-style messaging
|
||||
- **P2P optimized**: Built for peer-to-peer scenarios
|
||||
|
||||
### Use Cases
|
||||
|
||||
- **P2P networking**: Direct peer connections
|
||||
- **Media streaming**: Real-time audio/video
|
||||
- **Game networking**: Low-latency game state sync
|
||||
- **File transfer**: Fast bulk data transfer
|
||||
- **Hole punching**: NAT traversal
|
||||
|
||||
## Installation
|
||||
|
||||
```bash
|
||||
npm install udx-native
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
|
||||
### Basic Message Example
|
||||
|
||||
```js
|
||||
const UDX = require('udx-native')
|
||||
|
||||
const u = new UDX()
|
||||
const a = u.createSocket()
|
||||
const b = u.createSocket()
|
||||
|
||||
b.on('message', (message) => {
|
||||
console.log('received', message.toString())
|
||||
a.close()
|
||||
b.close()
|
||||
})
|
||||
|
||||
b.bind(0)
|
||||
a.send(Buffer.from('hello'), b.address().port)
|
||||
```
|
||||
|
||||
### Stream Example
|
||||
|
||||
```js
|
||||
const UDX = require('udx-native')
|
||||
|
||||
const u = new UDX()
|
||||
|
||||
const socket1 = u.createSocket()
|
||||
const socket2 = u.createSocket()
|
||||
|
||||
socket1.bind()
|
||||
socket2.bind()
|
||||
|
||||
const stream1 = u.createStream(1)
|
||||
const stream2 = u.createStream(2)
|
||||
|
||||
stream1.connect(socket1, stream2.id, socket2.address().port, '127.0.0.1')
|
||||
stream2.connect(socket2, stream1.id, socket1.address().port, '127.0.0.1')
|
||||
|
||||
stream1.write(Buffer.from('hello'))
|
||||
stream1.end()
|
||||
|
||||
stream2.on('data', (data) => {
|
||||
console.log(data.toString())
|
||||
})
|
||||
|
||||
stream2.on('end', () => {
|
||||
stream2.end()
|
||||
})
|
||||
|
||||
stream1.on('close', () => {
|
||||
socket1.close()
|
||||
})
|
||||
|
||||
stream2.on('close', () => {
|
||||
socket2.close()
|
||||
})
|
||||
```
|
||||
|
||||
## API Reference
|
||||
|
||||
### UDX
|
||||
|
||||
Main class for UDX operations.
|
||||
|
||||
#### `new UDX()`
|
||||
|
||||
Create a new UDX instance.
|
||||
|
||||
#### `UDX.isIPv4(host)`
|
||||
|
||||
Returns `true` if host is an IPv4 address.
|
||||
|
||||
#### `UDX.isIPv6(host)`
|
||||
|
||||
Returns `true` if host is an IPv6 address.
|
||||
|
||||
#### `UDX.isIP(host)`
|
||||
|
||||
Returns address family (`4` or `6`), or `0` if invalid.
|
||||
|
||||
### Sockets
|
||||
|
||||
#### `const socket = udx.createSocket([options])`
|
||||
|
||||
Create a UDP socket.
|
||||
|
||||
**Options:**
|
||||
- `ipv6Only` (boolean): IPv6 only mode (default: false)
|
||||
- `reuseAddress` (boolean): Allow address reuse (default: false)
|
||||
|
||||
#### `socket.bind([port], [host])`
|
||||
|
||||
Bind socket to address.
|
||||
|
||||
**Parameters:**
|
||||
- `port` (number): Port number (default: 0)
|
||||
- `host` (string): Host address (default: binds to `::` then `0.0.0.0`)
|
||||
|
||||
#### `socket.address()`
|
||||
|
||||
Get socket address info.
|
||||
|
||||
**Returns:** `{ host, family, port }`
|
||||
|
||||
#### `await socket.send(buffer, port, [host], [ttl])`
|
||||
|
||||
Send a message.
|
||||
|
||||
#### `socket.trySend(buffer, port, [host], [ttl])`
|
||||
|
||||
Send without awaiting.
|
||||
|
||||
#### `await socket.close()`
|
||||
|
||||
Close the socket.
|
||||
|
||||
### Streams
|
||||
|
||||
#### `const stream = udx.createStream(id, [options])`
|
||||
|
||||
Create a new stream.
|
||||
|
||||
**Parameters:**
|
||||
- `id` (number): Stream identifier
|
||||
- `options` (object):
|
||||
- `firewall` (function): Firewall function
|
||||
- `framed` (boolean): Framed mode
|
||||
- `seq` (number): Initial sequence number
|
||||
|
||||
#### `stream.connect(socket, remoteId, port, [host], [options])`
|
||||
|
||||
Connect stream to remote endpoint.
|
||||
|
||||
#### `stream.write(buffer)`
|
||||
|
||||
Write data to stream.
|
||||
|
||||
#### `stream.end()`
|
||||
|
||||
End the stream.
|
||||
|
||||
#### `await stream.flush()`
|
||||
|
||||
Wait for writes to be acknowledged.
|
||||
|
||||
### Network Interfaces
|
||||
|
||||
#### `const interfaces = udx.networkInterfaces()`
|
||||
|
||||
Get network interface list.
|
||||
|
||||
#### `const watcher = udx.watchNetworkInterfaces([onchange])`
|
||||
|
||||
Watch for interface changes.
|
||||
|
||||
## Complete Examples
|
||||
|
||||
### Example 1: Simple File Transfer
|
||||
|
||||
```js
|
||||
const UDX = require('udx-native')
|
||||
const fs = require('fs')
|
||||
|
||||
async function sendFile(filePath, port, host) {
|
||||
const u = new UDX()
|
||||
const socket = u.createSocket()
|
||||
await socket.bind()
|
||||
|
||||
const stream = u.createStream(1)
|
||||
stream.connect(socket, 2, port, host)
|
||||
|
||||
const file = fs.createReadStream(filePath)
|
||||
|
||||
file.on('data', (chunk) => {
|
||||
stream.write(chunk)
|
||||
})
|
||||
|
||||
file.on('end', () => {
|
||||
stream.end()
|
||||
})
|
||||
|
||||
await new Promise((resolve) => {
|
||||
stream.on('close', resolve)
|
||||
})
|
||||
|
||||
await socket.close()
|
||||
}
|
||||
|
||||
async function receiveFile(filePath, port) {
|
||||
const u = new UDX()
|
||||
const socket = u.createSocket()
|
||||
await socket.bind(port)
|
||||
|
||||
const file = fs.createWriteStream(filePath)
|
||||
|
||||
const stream = u.createStream(2)
|
||||
|
||||
// Wait for connection
|
||||
socket.on('message', () => {
|
||||
// Connection established
|
||||
})
|
||||
|
||||
stream.on('data', (data) => {
|
||||
file.write(data)
|
||||
})
|
||||
|
||||
stream.on('end', () => {
|
||||
file.end()
|
||||
stream.end()
|
||||
})
|
||||
|
||||
stream.on('close', () => {
|
||||
socket.close()
|
||||
})
|
||||
}
|
||||
|
||||
// Usage
|
||||
// Receiver: receiveFile('received.bin', 8080)
|
||||
// Sender: sendFile('file.bin', 8080, '127.0.0.1')
|
||||
```
|
||||
|
||||
### Example 2: P2P Chat
|
||||
|
||||
```js
|
||||
const UDX = require('udx-native')
|
||||
const readline = require('readline')
|
||||
|
||||
class P2PChat {
|
||||
constructor() {
|
||||
this.udx = new UDX()
|
||||
this.socket = null
|
||||
this.stream = null
|
||||
this.peers = new Map()
|
||||
}
|
||||
|
||||
async start(port = 0) {
|
||||
this.socket = this.udx.createSocket()
|
||||
await this.socket.bind(port)
|
||||
|
||||
console.log('Listening on:', this.socket.address())
|
||||
|
||||
this.socket.on('message', (msg, from) => {
|
||||
console.log(`\n[${from.host}:${from.port}] ${msg.toString()}`)
|
||||
process.stdout.write('> ')
|
||||
})
|
||||
}
|
||||
|
||||
async connect(host, port) {
|
||||
const stream = this.udx.createStream(Date.now())
|
||||
stream.connect(this.socket, Date.now() + 1, port, host)
|
||||
|
||||
stream.on('data', (data) => {
|
||||
console.log(`\n[Peer] ${data.toString()}`)
|
||||
process.stdout.write('> ')
|
||||
})
|
||||
|
||||
this.stream = stream
|
||||
}
|
||||
|
||||
send(message) {
|
||||
if (this.stream && this.stream.connected) {
|
||||
this.stream.write(Buffer.from(message))
|
||||
} else {
|
||||
// Fallback to socket
|
||||
// (would need to track peer addresses)
|
||||
}
|
||||
}
|
||||
|
||||
async close() {
|
||||
if (this.stream) await this.stream.end()
|
||||
if (this.socket) await this.socket.close()
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const chat = new P2PChat()
|
||||
|
||||
async function main() {
|
||||
const port = parseInt(process.argv[2]) || 0
|
||||
await chat.start(port)
|
||||
|
||||
if (process.argv[3] && process.argv[4]) {
|
||||
await chat.connect(process.argv[3], parseInt(process.argv[4]))
|
||||
}
|
||||
|
||||
const rl = readline.createInterface({
|
||||
input: process.stdin,
|
||||
output: process.stdout
|
||||
})
|
||||
|
||||
rl.setPrompt('> ')
|
||||
rl.prompt()
|
||||
|
||||
rl.on('line', (line) => {
|
||||
chat.send(line)
|
||||
rl.prompt()
|
||||
})
|
||||
|
||||
rl.on('close', () => {
|
||||
chat.close()
|
||||
process.exit(0)
|
||||
})
|
||||
}
|
||||
|
||||
main()
|
||||
```
|
||||
|
||||
### Example 3: Game State Sync
|
||||
|
||||
```js
|
||||
const UDX = require('udx-native')
|
||||
|
||||
class GameNetwork {
|
||||
constructor(tickRate = 60) {
|
||||
this.udx = new UDX()
|
||||
this.socket = null
|
||||
this.streams = new Map()
|
||||
this.tickRate = tickRate
|
||||
this.players = new Map()
|
||||
}
|
||||
|
||||
async host(port) {
|
||||
this.socket = this.udx.createSocket()
|
||||
await this.socket.bind(port)
|
||||
|
||||
console.log('Game server on port', port)
|
||||
|
||||
// Accept connections
|
||||
this.socket.on('message', (msg, from) => {
|
||||
this.handleMessage(msg, from)
|
||||
})
|
||||
}
|
||||
|
||||
async connect(host, port) {
|
||||
this.socket = this.udx.createSocket()
|
||||
await this.socket.bind()
|
||||
|
||||
const stream = this.udx.createStream(1)
|
||||
stream.connect(this.socket, 2, port, host)
|
||||
|
||||
stream.setInteractive(true) // Low latency mode
|
||||
|
||||
stream.on('data', (data) => {
|
||||
const state = JSON.parse(data)
|
||||
this.onGameState(state)
|
||||
})
|
||||
|
||||
this.streams.set('server', stream)
|
||||
|
||||
// Send input at tick rate
|
||||
setInterval(() => {
|
||||
this.sendInput()
|
||||
}, 1000 / this.tickRate)
|
||||
}
|
||||
|
||||
sendGameState(state) {
|
||||
const data = Buffer.from(JSON.stringify(state))
|
||||
for (const stream of this.streams.values()) {
|
||||
if (stream.connected) {
|
||||
stream.trySend(data)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
sendInput() {
|
||||
const input = this.collectInput()
|
||||
const stream = this.streams.get('server')
|
||||
if (stream && stream.connected) {
|
||||
stream.trySend(Buffer.from(JSON.stringify({ type: 'input', input })))
|
||||
}
|
||||
}
|
||||
|
||||
collectInput() {
|
||||
// Get current input state
|
||||
return { timestamp: Date.now() }
|
||||
}
|
||||
|
||||
onGameState(state) {
|
||||
// Apply received game state
|
||||
console.log('Received state:', state)
|
||||
}
|
||||
|
||||
async close() {
|
||||
for (const stream of this.streams.values()) {
|
||||
await stream.end()
|
||||
}
|
||||
if (this.socket) await this.socket.close()
|
||||
}
|
||||
}
|
||||
```
|
||||
|
||||
### Example 4: Network Diagnostics
|
||||
|
||||
```js
|
||||
const UDX = require('udx-native')
|
||||
|
||||
class NetworkDiagnostics {
|
||||
constructor() {
|
||||
this.udx = new UDX()
|
||||
}
|
||||
|
||||
async ping(host, port, count = 4) {
|
||||
const socket = this.udx.createSocket()
|
||||
await socket.bind()
|
||||
|
||||
const results = []
|
||||
|
||||
for (let i = 0; i < count; i++) {
|
||||
const start = Date.now()
|
||||
|
||||
await socket.send(Buffer.from(`ping:${i}`), port, host)
|
||||
|
||||
// Wait for response with timeout
|
||||
const response = await Promise.race([
|
||||
new Promise((resolve) => {
|
||||
socket.once('message', (msg) => resolve(msg))
|
||||
}),
|
||||
new Promise((_, reject) =>
|
||||
setTimeout(() => reject(new Error('Timeout')), 1000)
|
||||
)
|
||||
]).catch(() => null)
|
||||
|
||||
if (response) {
|
||||
const rtt = Date.now() - start
|
||||
results.push({ seq: i, rtt, success: true })
|
||||
} else {
|
||||
results.push({ seq: i, rtt: null, success: false })
|
||||
}
|
||||
|
||||
await new Promise(r => setTimeout(r, 1000))
|
||||
}
|
||||
|
||||
await socket.close()
|
||||
|
||||
// Print results
|
||||
console.log(`Ping statistics for ${host}:${port}:`)
|
||||
const successful = results.filter(r => r.success)
|
||||
console.log(` Transmitted: ${count}, Received: ${successful.length}`)
|
||||
|
||||
if (successful.length > 0) {
|
||||
const times = successful.map(r => r.rtt)
|
||||
console.log(` Min: ${Math.min(...times)}ms, Max: ${Math.max(...times)}ms`)
|
||||
console.log(` Avg: ${times.reduce((a, b) => a + b) / times.length}ms`)
|
||||
}
|
||||
|
||||
return results
|
||||
}
|
||||
|
||||
async bandwidthTest(host, port, duration = 10000) {
|
||||
const socket = this.udx.createSocket()
|
||||
await socket.bind()
|
||||
|
||||
const stream = this.udx.createStream(1)
|
||||
stream.connect(socket, 2, port, host)
|
||||
|
||||
let bytesSent = 0
|
||||
const startTime = Date.now()
|
||||
|
||||
// Send data as fast as possible
|
||||
const interval = setInterval(() => {
|
||||
if (Date.now() - startTime >= duration) {
|
||||
clearInterval(interval)
|
||||
stream.end()
|
||||
return
|
||||
}
|
||||
|
||||
const data = Buffer.alloc(1400) // MTU-sized packets
|
||||
stream.write(data)
|
||||
bytesSent += data.length
|
||||
}, 1)
|
||||
|
||||
await new Promise((resolve) => {
|
||||
stream.on('close', resolve)
|
||||
})
|
||||
|
||||
await socket.close()
|
||||
|
||||
const elapsed = (Date.now() - startTime) / 1000
|
||||
const mbps = (bytesSent * 8 / 1000000) / elapsed
|
||||
|
||||
console.log(`Bandwidth test results:`)
|
||||
console.log(` Duration: ${elapsed.toFixed(2)}s`)
|
||||
console.log(` Sent: ${(bytesSent / 1024 / 1024).toFixed(2)} MB`)
|
||||
console.log(` Throughput: ${mbps.toFixed(2)} Mbps`)
|
||||
|
||||
return { bytesSent, duration: elapsed, mbps }
|
||||
}
|
||||
|
||||
listInterfaces() {
|
||||
const interfaces = this.udx.networkInterfaces()
|
||||
console.log('Network interfaces:')
|
||||
interfaces.forEach(iface => {
|
||||
console.log(` ${iface.name}: ${iface.host} (${iface.family === 4 ? 'IPv4' : 'IPv6'})`)
|
||||
})
|
||||
return interfaces
|
||||
}
|
||||
}
|
||||
|
||||
// Usage
|
||||
const diag = new NetworkDiagnostics()
|
||||
diag.listInterfaces()
|
||||
// diag.ping('127.0.0.1', 8080)
|
||||
// diag.bandwidthTest('127.0.0.1', 8080)
|
||||
```
|
||||
|
||||
## Best Practices
|
||||
|
||||
### Use Interactive Mode for Low Latency
|
||||
|
||||
```js
|
||||
stream.setInteractive(true)
|
||||
```
|
||||
|
||||
### Handle MTU Limits
|
||||
|
||||
```js
|
||||
stream.on('mtu-exceeded', () => {
|
||||
console.warn('Packet too large, fragment or reduce size')
|
||||
})
|
||||
```
|
||||
|
||||
### Monitor Stream Health
|
||||
|
||||
```js
|
||||
console.log('RTT:', stream.rtt)
|
||||
console.log('Inflight:', stream.inflight)
|
||||
console.log('CWND:', stream.cwnd)
|
||||
```
|
||||
|
||||
### Proper Cleanup
|
||||
|
||||
```js
|
||||
process.on('SIGINT', async () => {
|
||||
for (const stream of streams) {
|
||||
await stream.end()
|
||||
}
|
||||
await socket.close()
|
||||
process.exit(0)
|
||||
})
|
||||
```
|
||||
|
||||
## Integration with Other Modules
|
||||
|
||||
### With Protomux
|
||||
|
||||
```js
|
||||
const Protomux = require('protomux')
|
||||
const UDX = require('udx-native')
|
||||
|
||||
const u = new UDX()
|
||||
const socket = u.createSocket()
|
||||
await socket.bind()
|
||||
|
||||
const stream = u.createStream(1)
|
||||
const mux = new Protomux(stream)
|
||||
```
|
||||
|
||||
### With HyperDHT
|
||||
|
||||
```js
|
||||
const DHT = require('hyperdht')
|
||||
const UDX = require('udx-native')
|
||||
|
||||
const dht = new DHT()
|
||||
const u = new UDX()
|
||||
|
||||
// UDX can work with DHT for hole punching
|
||||
```
|
||||
|
||||
## Development Setup
|
||||
|
||||
```bash
|
||||
# Install dependencies
|
||||
npm install -g bare-runtime bare-make
|
||||
|
||||
# Clone and build
|
||||
git clone https://github.com/holepunchto/udx-native
|
||||
cd udx-native
|
||||
npm install
|
||||
|
||||
# Generate build files
|
||||
bare-make generate
|
||||
|
||||
# Build
|
||||
bare-make build
|
||||
|
||||
# Install
|
||||
bare-make install
|
||||
```
|
||||
|
||||
## License
|
||||
|
||||
Apache-2.0
|
||||
|
||||
---
|
||||
**Module Type**: Networking Library | **Ecosystem Role**: Transport Layer | **Dependencies**: libudx
|
||||
Reference in New Issue
Block a user