Add Supercomputer category: 12 P2P resource-sharing modules (164 total)
Introduce a new production category for distributed system resource pooling over Hyperswarm — CPU, RAM, GPU, bandwidth, disk, jobs, cache, and egress. New modules (supercomputer/): - hyper-p2p-capacity-registry — gossip CPU/RAM/GPU/disk/bandwidth ads - hyper-p2p-cpu-share — CPU millisecond credit pool - hyper-p2p-ram-pool — RAM byte lend/claim/release - hyper-p2p-gpu-slot — GPU slot register/reserve - hyper-p2p-bandwidth-share — shared up/down Mbps - hyper-p2p-disk-stripe — striped block shards - hyper-p2p-job-dispatcher — submit/claim/complete jobs - hyper-p2p-thermal-guard — load/temperature throttle signals - hyper-p2p-cache-farm — distributed LRU cache - hyper-p2p-net-gateway — peer internet egress routing - hyper-p2p-cluster-affinity — hardware tag placement scoring - hyper-p2p-work-stealer — shard queues + work stealing Each package: index.js, tests, README, api/architecture docs, examples/basic.js. Shared layering guide: _shared/SUPERCOMPUTER_LAYERS.md. Registry, module-paths, MODULE_CATEGORIES, and workspace README updated to 164. Parent workspace also adds docs/supercomputer hub and demo-supercomputer-mesh (outside this repo). Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Raven Scott
+16
-1
@@ -1,6 +1,6 @@
|
|||||||
# Module categories (v2)
|
# Module categories (v2)
|
||||||
|
|
||||||
152 modules in **29** category directories. All **production**.
|
164 modules in **30** category directories. All **production**.
|
||||||
|
|
||||||
Source of truth: [`MODULE_REGISTRY.yaml`](MODULE_REGISTRY.yaml).
|
Source of truth: [`MODULE_REGISTRY.yaml`](MODULE_REGISTRY.yaml).
|
||||||
|
|
||||||
@@ -182,6 +182,21 @@ Source of truth: [`MODULE_REGISTRY.yaml`](MODULE_REGISTRY.yaml).
|
|||||||
- 🟢 **hyper-p2p-peer-scheduler** — `peer-scheduler/v1` (production)
|
- 🟢 **hyper-p2p-peer-scheduler** — `peer-scheduler/v1` (production)
|
||||||
- 🟢 **hyper-p2p-topic-lease** — `topic-lease/v1` (production)
|
- 🟢 **hyper-p2p-topic-lease** — `topic-lease/v1` (production)
|
||||||
|
|
||||||
|
## Supercomputer (`supercomputer`) — 12 modules
|
||||||
|
|
||||||
|
- 🟢 **hyper-p2p-bandwidth-share** — `bandwidth-share/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-cache-farm** — `cache-farm/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-capacity-registry** — `capacity-registry/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-cluster-affinity** — `cluster-affinity/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-cpu-share** — `cpu-share/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-disk-stripe** — `disk-stripe/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-gpu-slot** — `gpu-slot/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-job-dispatcher** — `job-dispatcher/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-net-gateway** — `net-gateway/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-ram-pool** — `ram-pool/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-thermal-guard** — `thermal-guard/v1` (production)
|
||||||
|
- 🟢 **hyper-p2p-work-stealer** — `work-stealer/v1` (production)
|
||||||
|
|
||||||
## State & CRDTs (`state-crdts`) — 10 modules
|
## State & CRDTs (`state-crdts`) — 10 modules
|
||||||
|
|
||||||
- 🟢 **hyper-p2p-conflict-set** — `conflict-set/v1` (production)
|
- 🟢 **hyper-p2p-conflict-set** — `conflict-set/v1` (production)
|
||||||
|
|||||||
+130
-2
@@ -1,4 +1,4 @@
|
|||||||
# Module registry — source of truth (152 modules)
|
# Module registry — source of truth (164 modules)
|
||||||
# tiers: production | scaffold
|
# tiers: production | scaffold
|
||||||
version: 1
|
version: 1
|
||||||
categories: {
|
categories: {
|
||||||
@@ -29,7 +29,8 @@ categories: {
|
|||||||
"observability": "Observability",
|
"observability": "Observability",
|
||||||
"pear-platform": "Pear / Bare platform",
|
"pear-platform": "Pear / Bare platform",
|
||||||
"applications-collab": "Applications (collaboration)",
|
"applications-collab": "Applications (collaboration)",
|
||||||
"applications-economy": "Applications (economy)"
|
"applications-economy": "Applications (economy)",
|
||||||
|
"supercomputer": "Supercomputer"
|
||||||
}
|
}
|
||||||
modules:
|
modules:
|
||||||
- name: hyper-p2p-protocol-handshake
|
- name: hyper-p2p-protocol-handshake
|
||||||
@@ -1706,3 +1707,130 @@ modules:
|
|||||||
- hyper-p2p-distributed-lock
|
- hyper-p2p-distributed-lock
|
||||||
composes_with:
|
composes_with:
|
||||||
- hyper-p2p-topic-lease
|
- hyper-p2p-topic-lease
|
||||||
|
- name: hyper-p2p-capacity-registry
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: capacity-registry/v1
|
||||||
|
class: HyperP2PCapacityRegistry
|
||||||
|
summary: Gossip registry of peer CPU, RAM, GPU, disk, and bandwidth capacity.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-cluster-affinity
|
||||||
|
- hyper-p2p-job-dispatcher
|
||||||
|
- name: hyper-p2p-cpu-share
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: cpu-share/v1
|
||||||
|
class: HyperP2PCpuShare
|
||||||
|
summary: P2P CPU millisecond credit donate and consume pool.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-job-dispatcher
|
||||||
|
- hyper-p2p-thermal-guard
|
||||||
|
- name: hyper-p2p-ram-pool
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: ram-pool/v1
|
||||||
|
class: HyperP2PRamPool
|
||||||
|
summary: RAM byte lending with claim leases across peers.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-capacity-registry
|
||||||
|
- name: hyper-p2p-gpu-slot
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: gpu-slot/v1
|
||||||
|
class: HyperP2PGpuSlot
|
||||||
|
summary: GPU slot registration and reservation gossip.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-job-dispatcher
|
||||||
|
- name: hyper-p2p-bandwidth-share
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: bandwidth-share/v1
|
||||||
|
class: HyperP2PBandwidthShare
|
||||||
|
summary: Shared uplink and downlink Mbps offers per peer.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-bandwidth-broker
|
||||||
|
- hyper-p2p-net-gateway
|
||||||
|
- name: hyper-p2p-disk-stripe
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: disk-stripe/v1
|
||||||
|
class: HyperP2PDiskStripe
|
||||||
|
summary: Striped block storage shards across the mesh.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-multipath-fanout
|
||||||
|
- name: hyper-p2p-job-dispatcher
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: job-dispatcher/v1
|
||||||
|
class: HyperP2PJobDispatcher
|
||||||
|
summary: Distributed submit, claim, and complete job queue.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-work-stealer
|
||||||
|
- hyper-p2p-cluster-affinity
|
||||||
|
- name: hyper-p2p-thermal-guard
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: thermal-guard/v1
|
||||||
|
class: HyperP2PThermalGuard
|
||||||
|
summary: CPU, RAM, and temperature load throttle signals.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-cpu-share
|
||||||
|
- hyper-p2p-congestion-signal
|
||||||
|
- name: hyper-p2p-cache-farm
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: cache-farm/v1
|
||||||
|
class: HyperP2PCacheFarm
|
||||||
|
summary: Distributed LRU cache with gossip invalidation.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-gossip-mesh
|
||||||
|
- name: hyper-p2p-net-gateway
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: net-gateway/v1
|
||||||
|
class: HyperP2PNetGateway
|
||||||
|
summary: Peer internet egress gateway registration and routing.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-bandwidth-share
|
||||||
|
- name: hyper-p2p-cluster-affinity
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: cluster-affinity/v1
|
||||||
|
class: HyperP2PClusterAffinity
|
||||||
|
summary: Hardware tag and latency workload placement scoring.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-capacity-registry
|
||||||
|
- hyper-p2p-anycast-selector
|
||||||
|
- name: hyper-p2p-work-stealer
|
||||||
|
category: supercomputer
|
||||||
|
tier: production
|
||||||
|
status: implemented
|
||||||
|
protocol: work-stealer/v1
|
||||||
|
class: HyperP2PWorkStealer
|
||||||
|
summary: Shard task queues with idle-peer work stealing.
|
||||||
|
holepunch_refs: []
|
||||||
|
composes_with:
|
||||||
|
- hyper-p2p-job-dispatcher
|
||||||
|
- hyper-p2p-peer-scheduler
|
||||||
|
|||||||
@@ -1,6 +1,6 @@
|
|||||||
# Hyper-P2P module workspace
|
# Hyper-P2P module workspace
|
||||||
|
|
||||||
This directory is the **module library** for the repository: **152** independent npm packages for [Bare](https://github.com/holepunchto/bare) and Pear. Each package is a small, composable peer-to-peer primitive—discovery, messaging, storage, coordination, security, or application logic—built on shared Hyperswarm and Protomux conventions.
|
This directory is the **module library** for the repository: **164** independent npm packages for [Bare](https://github.com/holepunchto/bare) and Pear. Each package is a small, composable peer-to-peer primitive—discovery, messaging, storage, coordination, security, or application logic—built on shared Hyperswarm and Protomux conventions.
|
||||||
|
|
||||||
The parent repository adds cross-cutting guides under `docs/`, composition examples under `examples/`, and the unified test runner under `real_tests/`. Start at the [repository README](../README.md) for clone, CI, and top-level commands.
|
The parent repository adds cross-cutting guides under `docs/`, composition examples under `examples/`, and the unified test runner under `real_tests/`. Start at the [repository README](../README.md) for clone, CI, and top-level commands.
|
||||||
|
|
||||||
@@ -25,7 +25,7 @@ Naming convention: **`hyper-p2p-<feature>`** (or `hyper-<pear-feature>` for Pear
|
|||||||
|
|
||||||
## How modules are grouped
|
## How modules are grouped
|
||||||
|
|
||||||
Packages live under **30 functional categories** (one folder per category). Category folder names are stable ids (e.g. `messaging-pubsub`, `storage-hypercore`); they describe **what the code does**, not release history.
|
Packages live under **31 functional categories** (one folder per category). Category folder names are stable ids (e.g. `messaging-pubsub`, `storage-hypercore`); they describe **what the code does**, not release history.
|
||||||
|
|
||||||
| Category | Count | Role |
|
| Category | Count | Role |
|
||||||
|----------|------:|------|
|
|----------|------:|------|
|
||||||
@@ -44,6 +44,7 @@ Packages live under **30 functional categories** (one folder per category). Cate
|
|||||||
| Consensus & coordination | 5 | Locks, leases, quorum, Raft-lite, causal consensus |
|
| Consensus & coordination | 5 | Locks, leases, quorum, Raft-lite, causal consensus |
|
||||||
| Scheduling & queues | 5 | Activity queue, peer scheduler, topic lease, cron, deadlines |
|
| Scheduling & queues | 5 | Activity queue, peer scheduler, topic lease, cron, deadlines |
|
||||||
| Measurement & rate control | 4 | Token buckets, histograms, sketches, SLA budgets |
|
| Measurement & rate control | 4 | Token buckets, histograms, sketches, SLA budgets |
|
||||||
|
| Supercomputer | 12 | CPU/RAM/GPU/bandwidth/disk pools, jobs, cache, gateway, affinity |
|
||||||
| Indexes & search | 8 | Bloom, fulltext, graph, inverted, vector, LSH, trie, spatial |
|
| Indexes & search | 8 | Bloom, fulltext, graph, inverted, vector, LSH, trie, spatial |
|
||||||
| Storage (Hypercore) | 7 | Replication, fetch, forks, merkle, audit, bitfield, seed policy |
|
| Storage (Hypercore) | 7 | Replication, fetch, forks, merkle, audit, bitfield, seed policy |
|
||||||
| Storage (Hyperbee) | 5 | Batch write, diff follow, range watch, secondary index, tombstones |
|
| Storage (Hyperbee) | 5 | Batch write, diff follow, range watch, secondary index, tombstones |
|
||||||
@@ -58,7 +59,7 @@ Packages live under **30 functional categories** (one folder per category). Cate
|
|||||||
| Oracles | 1 | Decentralized oracle gossip |
|
| Oracles | 1 | Decentralized oracle gossip |
|
||||||
| Time & ordering | 1 | Temporal index |
|
| Time & ordering | 1 | Temporal index |
|
||||||
|
|
||||||
**152** modules total. The authoritative list with protocol ids and paths is in **`MODULE_REGISTRY.yaml`**; **`module-paths.json`** maps package name → category path for tooling and tests.
|
**164** modules total. The authoritative list with protocol ids and paths is in **`MODULE_REGISTRY.yaml`**; **`module-paths.json`** maps package name → category path for tooling and tests.
|
||||||
|
|
||||||
For a per-module index with protocol ids, use **`MODULE_CATEGORIES.md`** in this directory.
|
For a per-module index with protocol ids, use **`MODULE_CATEGORIES.md`** in this directory.
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,28 @@
|
|||||||
|
# Supercomputer resource layers
|
||||||
|
|
||||||
|
P2P modules for pooling compute, memory, network, and storage across a Hyperswarm topic.
|
||||||
|
|
||||||
|
```text
|
||||||
|
Job / workload (job-dispatcher, work-stealer)
|
||||||
|
↓
|
||||||
|
Placement (cluster-affinity, capacity-registry, thermal-guard)
|
||||||
|
↓
|
||||||
|
Resource pools (cpu-share, ram-pool, gpu-slot, disk-stripe, cache-farm)
|
||||||
|
↓
|
||||||
|
Egress (bandwidth-share, net-gateway)
|
||||||
|
↓
|
||||||
|
Hyperswarm + Protomux gossip
|
||||||
|
```
|
||||||
|
|
||||||
|
| Layer | Modules |
|
||||||
|
|-------|---------|
|
||||||
|
| Registry | `hyper-p2p-capacity-registry` |
|
||||||
|
| Compute | `hyper-p2p-cpu-share`, `hyper-p2p-gpu-slot` |
|
||||||
|
| Memory | `hyper-p2p-ram-pool`, `hyper-p2p-cache-farm` |
|
||||||
|
| Network | `hyper-p2p-bandwidth-share`, `hyper-p2p-net-gateway` |
|
||||||
|
| Storage | `hyper-p2p-disk-stripe` |
|
||||||
|
| Scheduling | `hyper-p2p-job-dispatcher`, `hyper-p2p-work-stealer` |
|
||||||
|
| Safety | `hyper-p2p-thermal-guard` |
|
||||||
|
| Placement | `hyper-p2p-cluster-affinity` |
|
||||||
|
|
||||||
|
Pairs with `hyper-p2p-bandwidth-broker`, `hyper-p2p-topic-lease`, and `hyper-p2p-quorum-pool`.
|
||||||
@@ -12,6 +12,7 @@
|
|||||||
"hyper-p2p-autobase-view-sync": "storage-autobase/hyper-p2p-autobase-view-sync",
|
"hyper-p2p-autobase-view-sync": "storage-autobase/hyper-p2p-autobase-view-sync",
|
||||||
"hyper-p2p-autobase-writer-lease": "storage-autobase/hyper-p2p-autobase-writer-lease",
|
"hyper-p2p-autobase-writer-lease": "storage-autobase/hyper-p2p-autobase-writer-lease",
|
||||||
"hyper-p2p-bandwidth-broker": "network-stack/hyper-p2p-bandwidth-broker",
|
"hyper-p2p-bandwidth-broker": "network-stack/hyper-p2p-bandwidth-broker",
|
||||||
|
"hyper-p2p-bandwidth-share": "supercomputer/hyper-p2p-bandwidth-share",
|
||||||
"hyper-p2p-bee-batch-write": "storage-hyperbee/hyper-p2p-bee-batch-write",
|
"hyper-p2p-bee-batch-write": "storage-hyperbee/hyper-p2p-bee-batch-write",
|
||||||
"hyper-p2p-bee-diff-follow": "storage-hyperbee/hyper-p2p-bee-diff-follow",
|
"hyper-p2p-bee-diff-follow": "storage-hyperbee/hyper-p2p-bee-diff-follow",
|
||||||
"hyper-p2p-bee-range-watch": "storage-hyperbee/hyper-p2p-bee-range-watch",
|
"hyper-p2p-bee-range-watch": "storage-hyperbee/hyper-p2p-bee-range-watch",
|
||||||
@@ -21,11 +22,14 @@
|
|||||||
"hyper-p2p-blind-relay-bridge": "network-transport/hyper-p2p-blind-relay-bridge",
|
"hyper-p2p-blind-relay-bridge": "network-transport/hyper-p2p-blind-relay-bridge",
|
||||||
"hyper-p2p-bloom-gossip": "indexes-search/hyper-p2p-bloom-gossip",
|
"hyper-p2p-bloom-gossip": "indexes-search/hyper-p2p-bloom-gossip",
|
||||||
"hyper-p2p-bucket-rate-limit": "measurement-rate-control/hyper-p2p-bucket-rate-limit",
|
"hyper-p2p-bucket-rate-limit": "measurement-rate-control/hyper-p2p-bucket-rate-limit",
|
||||||
|
"hyper-p2p-cache-farm": "supercomputer/hyper-p2p-cache-farm",
|
||||||
"hyper-p2p-capabilities": "core-infrastructure/hyper-p2p-capabilities",
|
"hyper-p2p-capabilities": "core-infrastructure/hyper-p2p-capabilities",
|
||||||
"hyper-p2p-capability-discovery": "network-discovery/hyper-p2p-capability-discovery",
|
"hyper-p2p-capability-discovery": "network-discovery/hyper-p2p-capability-discovery",
|
||||||
|
"hyper-p2p-capacity-registry": "supercomputer/hyper-p2p-capacity-registry",
|
||||||
"hyper-p2p-causal-consensus": "consensus-coordination/hyper-p2p-causal-consensus",
|
"hyper-p2p-causal-consensus": "consensus-coordination/hyper-p2p-causal-consensus",
|
||||||
"hyper-p2p-circuit-breaker": "routing-advanced/hyper-p2p-circuit-breaker",
|
"hyper-p2p-circuit-breaker": "routing-advanced/hyper-p2p-circuit-breaker",
|
||||||
"hyper-p2p-circuit-loom": "network-stack/hyper-p2p-circuit-loom",
|
"hyper-p2p-circuit-loom": "network-stack/hyper-p2p-circuit-loom",
|
||||||
|
"hyper-p2p-cluster-affinity": "supercomputer/hyper-p2p-cluster-affinity",
|
||||||
"hyper-p2p-collab-room": "applications-collab/hyper-p2p-collab-room",
|
"hyper-p2p-collab-room": "applications-collab/hyper-p2p-collab-room",
|
||||||
"hyper-p2p-compact-codec-bridge": "encoding-wire/hyper-p2p-compact-codec-bridge",
|
"hyper-p2p-compact-codec-bridge": "encoding-wire/hyper-p2p-compact-codec-bridge",
|
||||||
"hyper-p2p-conflict-set": "state-crdts/hyper-p2p-conflict-set",
|
"hyper-p2p-conflict-set": "state-crdts/hyper-p2p-conflict-set",
|
||||||
@@ -39,6 +43,7 @@
|
|||||||
"hyper-p2p-core-priority-fetch": "storage-hypercore/hyper-p2p-core-priority-fetch",
|
"hyper-p2p-core-priority-fetch": "storage-hypercore/hyper-p2p-core-priority-fetch",
|
||||||
"hyper-p2p-core-replicator": "storage-hypercore/hyper-p2p-core-replicator",
|
"hyper-p2p-core-replicator": "storage-hypercore/hyper-p2p-core-replicator",
|
||||||
"hyper-p2p-core-seed-policy": "storage-hypercore/hyper-p2p-core-seed-policy",
|
"hyper-p2p-core-seed-policy": "storage-hypercore/hyper-p2p-core-seed-policy",
|
||||||
|
"hyper-p2p-cpu-share": "supercomputer/hyper-p2p-cpu-share",
|
||||||
"hyper-p2p-crdt-grow-only-set": "state-crdts/hyper-p2p-crdt-grow-only-set",
|
"hyper-p2p-crdt-grow-only-set": "state-crdts/hyper-p2p-crdt-grow-only-set",
|
||||||
"hyper-p2p-crdt-lww-register": "state-crdts/hyper-p2p-crdt-lww-register",
|
"hyper-p2p-crdt-lww-register": "state-crdts/hyper-p2p-crdt-lww-register",
|
||||||
"hyper-p2p-crdt-map": "state-crdts/hyper-p2p-crdt-map",
|
"hyper-p2p-crdt-map": "state-crdts/hyper-p2p-crdt-map",
|
||||||
@@ -55,6 +60,7 @@
|
|||||||
"hyper-p2p-dedup-filter": "messaging-gossip/hyper-p2p-dedup-filter",
|
"hyper-p2p-dedup-filter": "messaging-gossip/hyper-p2p-dedup-filter",
|
||||||
"hyper-p2p-dht-bootstrap-hint": "network-transport/hyper-p2p-dht-bootstrap-hint",
|
"hyper-p2p-dht-bootstrap-hint": "network-transport/hyper-p2p-dht-bootstrap-hint",
|
||||||
"hyper-p2p-discovery-health": "network-discovery/hyper-p2p-discovery-health",
|
"hyper-p2p-discovery-health": "network-discovery/hyper-p2p-discovery-health",
|
||||||
|
"hyper-p2p-disk-stripe": "supercomputer/hyper-p2p-disk-stripe",
|
||||||
"hyper-p2p-distributed-event-bus": "messaging-gossip/hyper-p2p-distributed-event-bus",
|
"hyper-p2p-distributed-event-bus": "messaging-gossip/hyper-p2p-distributed-event-bus",
|
||||||
"hyper-p2p-distributed-lock": "consensus-coordination/hyper-p2p-distributed-lock",
|
"hyper-p2p-distributed-lock": "consensus-coordination/hyper-p2p-distributed-lock",
|
||||||
"hyper-p2p-document-line-lock": "applications-collab/hyper-p2p-document-line-lock",
|
"hyper-p2p-document-line-lock": "applications-collab/hyper-p2p-document-line-lock",
|
||||||
@@ -71,12 +77,14 @@
|
|||||||
"hyper-p2p-fulltext-lite": "indexes-search/hyper-p2p-fulltext-lite",
|
"hyper-p2p-fulltext-lite": "indexes-search/hyper-p2p-fulltext-lite",
|
||||||
"hyper-p2p-geo-hint-router": "routing-advanced/hyper-p2p-geo-hint-router",
|
"hyper-p2p-geo-hint-router": "routing-advanced/hyper-p2p-geo-hint-router",
|
||||||
"hyper-p2p-gossip-mesh": "messaging-gossip/hyper-p2p-gossip-mesh",
|
"hyper-p2p-gossip-mesh": "messaging-gossip/hyper-p2p-gossip-mesh",
|
||||||
|
"hyper-p2p-gpu-slot": "supercomputer/hyper-p2p-gpu-slot",
|
||||||
"hyper-p2p-graph-index": "indexes-search/hyper-p2p-graph-index",
|
"hyper-p2p-graph-index": "indexes-search/hyper-p2p-graph-index",
|
||||||
"hyper-p2p-gravity-well": "experimental/hyper-p2p-gravity-well",
|
"hyper-p2p-gravity-well": "experimental/hyper-p2p-gravity-well",
|
||||||
"hyper-p2p-health-probe": "observability/hyper-p2p-health-probe",
|
"hyper-p2p-health-probe": "observability/hyper-p2p-health-probe",
|
||||||
"hyper-p2p-histogram-gossip": "measurement-rate-control/hyper-p2p-histogram-gossip",
|
"hyper-p2p-histogram-gossip": "measurement-rate-control/hyper-p2p-histogram-gossip",
|
||||||
"hyper-p2p-intent-router": "routing-paths/hyper-p2p-intent-router",
|
"hyper-p2p-intent-router": "routing-paths/hyper-p2p-intent-router",
|
||||||
"hyper-p2p-inverted-index": "indexes-search/hyper-p2p-inverted-index",
|
"hyper-p2p-inverted-index": "indexes-search/hyper-p2p-inverted-index",
|
||||||
|
"hyper-p2p-job-dispatcher": "supercomputer/hyper-p2p-job-dispatcher",
|
||||||
"hyper-p2p-key-rotation": "trust-security/hyper-p2p-key-rotation",
|
"hyper-p2p-key-rotation": "trust-security/hyper-p2p-key-rotation",
|
||||||
"hyper-p2p-leader-lease": "consensus-coordination/hyper-p2p-leader-lease",
|
"hyper-p2p-leader-lease": "consensus-coordination/hyper-p2p-leader-lease",
|
||||||
"hyper-p2p-link-probe": "network-stack/hyper-p2p-link-probe",
|
"hyper-p2p-link-probe": "network-stack/hyper-p2p-link-probe",
|
||||||
@@ -91,6 +99,7 @@
|
|||||||
"hyper-p2p-multipath-fanout": "network-stack/hyper-p2p-multipath-fanout",
|
"hyper-p2p-multipath-fanout": "network-stack/hyper-p2p-multipath-fanout",
|
||||||
"hyper-p2p-multisig-threshold": "trust-security/hyper-p2p-multisig-threshold",
|
"hyper-p2p-multisig-threshold": "trust-security/hyper-p2p-multisig-threshold",
|
||||||
"hyper-p2p-mycelium-pool": "experimental/hyper-p2p-mycelium-pool",
|
"hyper-p2p-mycelium-pool": "experimental/hyper-p2p-mycelium-pool",
|
||||||
|
"hyper-p2p-net-gateway": "supercomputer/hyper-p2p-net-gateway",
|
||||||
"hyper-p2p-noise-session-wrap": "network-transport/hyper-p2p-noise-session-wrap",
|
"hyper-p2p-noise-session-wrap": "network-transport/hyper-p2p-noise-session-wrap",
|
||||||
"hyper-p2p-overlay-topology": "network-stack/hyper-p2p-overlay-topology",
|
"hyper-p2p-overlay-topology": "network-stack/hyper-p2p-overlay-topology",
|
||||||
"hyper-p2p-paradox-clock": "experimental/hyper-p2p-paradox-clock",
|
"hyper-p2p-paradox-clock": "experimental/hyper-p2p-paradox-clock",
|
||||||
@@ -106,6 +115,7 @@
|
|||||||
"hyper-p2p-qos-topic": "messaging-pubsub/hyper-p2p-qos-topic",
|
"hyper-p2p-qos-topic": "messaging-pubsub/hyper-p2p-qos-topic",
|
||||||
"hyper-p2p-quorum-pool": "consensus-coordination/hyper-p2p-quorum-pool",
|
"hyper-p2p-quorum-pool": "consensus-coordination/hyper-p2p-quorum-pool",
|
||||||
"hyper-p2p-raft-lite": "consensus-coordination/hyper-p2p-raft-lite",
|
"hyper-p2p-raft-lite": "consensus-coordination/hyper-p2p-raft-lite",
|
||||||
|
"hyper-p2p-ram-pool": "supercomputer/hyper-p2p-ram-pool",
|
||||||
"hyper-p2p-reactive-state": "state-crdts/hyper-p2p-reactive-state",
|
"hyper-p2p-reactive-state": "state-crdts/hyper-p2p-reactive-state",
|
||||||
"hyper-p2p-relay-tunnel": "routing-paths/hyper-p2p-relay-tunnel",
|
"hyper-p2p-relay-tunnel": "routing-paths/hyper-p2p-relay-tunnel",
|
||||||
"hyper-p2p-reputation-system": "trust-security/hyper-p2p-reputation-system",
|
"hyper-p2p-reputation-system": "trust-security/hyper-p2p-reputation-system",
|
||||||
@@ -133,6 +143,7 @@
|
|||||||
"hyper-p2p-task-orchestrator": "agents-workflows/hyper-p2p-task-orchestrator",
|
"hyper-p2p-task-orchestrator": "agents-workflows/hyper-p2p-task-orchestrator",
|
||||||
"hyper-p2p-temporal-index": "time-ordering/hyper-p2p-temporal-index",
|
"hyper-p2p-temporal-index": "time-ordering/hyper-p2p-temporal-index",
|
||||||
"hyper-p2p-tension-field": "experimental/hyper-p2p-tension-field",
|
"hyper-p2p-tension-field": "experimental/hyper-p2p-tension-field",
|
||||||
|
"hyper-p2p-thermal-guard": "supercomputer/hyper-p2p-thermal-guard",
|
||||||
"hyper-p2p-time-capsule": "experimental/hyper-p2p-time-capsule",
|
"hyper-p2p-time-capsule": "experimental/hyper-p2p-time-capsule",
|
||||||
"hyper-p2p-topic-announcer": "network-discovery/hyper-p2p-topic-announcer",
|
"hyper-p2p-topic-announcer": "network-discovery/hyper-p2p-topic-announcer",
|
||||||
"hyper-p2p-topic-channel": "messaging-pubsub/hyper-p2p-topic-channel",
|
"hyper-p2p-topic-channel": "messaging-pubsub/hyper-p2p-topic-channel",
|
||||||
@@ -147,6 +158,7 @@
|
|||||||
"hyper-p2p-whisper-mesh": "experimental/hyper-p2p-whisper-mesh",
|
"hyper-p2p-whisper-mesh": "experimental/hyper-p2p-whisper-mesh",
|
||||||
"hyper-p2p-whiteboard-op": "applications-collab/hyper-p2p-whiteboard-op",
|
"hyper-p2p-whiteboard-op": "applications-collab/hyper-p2p-whiteboard-op",
|
||||||
"hyper-p2p-wire-registry": "encoding-wire/hyper-p2p-wire-registry",
|
"hyper-p2p-wire-registry": "encoding-wire/hyper-p2p-wire-registry",
|
||||||
|
"hyper-p2p-work-stealer": "supercomputer/hyper-p2p-work-stealer",
|
||||||
"hyper-p2p-workflow-graph": "agents-workflows/hyper-p2p-workflow-graph",
|
"hyper-p2p-workflow-graph": "agents-workflows/hyper-p2p-workflow-graph",
|
||||||
"hyper-pear-runtime-session": "pear-platform/hyper-pear-runtime-session",
|
"hyper-pear-runtime-session": "pear-platform/hyper-pear-runtime-session",
|
||||||
"hyper-pear-update-gossip": "pear-platform/hyper-pear-update-gossip",
|
"hyper-pear-update-gossip": "pear-platform/hyper-pear-update-gossip",
|
||||||
|
|||||||
@@ -0,0 +1,73 @@
|
|||||||
|
# Supercomputer
|
||||||
|
|
||||||
|
**Path:** `modules/supercomputer/` · **Modules:** 12 (all production)
|
||||||
|
|
||||||
|
Distributed **resource pooling** over Hyperswarm: advertise capacity, share CPU/RAM/GPU/bandwidth/disk, dispatch jobs, route internet egress, and place workloads by hardware affinity. Hub: [`../../docs/supercomputer/README.md`](../../docs/supercomputer/README.md). Layering: [`../_shared/SUPERCOMPUTER_LAYERS.md`](../_shared/SUPERCOMPUTER_LAYERS.md).
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Building a **volunteer or federated compute mesh** on one topic: peers donate idle CPU milliseconds, RAM, GPU slots, and uplink; schedulers claim work and respect thermal/load guards.
|
||||||
|
|
||||||
|
## When not to use
|
||||||
|
|
||||||
|
Single-machine orchestration (use local process APIs). Untrusted open networks without `trust-security` and rate limits.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```bash
|
||||||
|
bare ../../examples/demo-supercomputer-mesh/index.js
|
||||||
|
```
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PCapacityRegistry } = require('hyper-p2p-capacity-registry')
|
||||||
|
const { HyperP2PCpuShare } = require('hyper-p2p-cpu-share')
|
||||||
|
const { HyperP2PJobDispatcher } = require('hyper-p2p-job-dispatcher')
|
||||||
|
|
||||||
|
const reg = new HyperP2PCapacityRegistry({ topic: 'super-mesh' })
|
||||||
|
const cpu = new HyperP2PCpuShare({ topic: 'super-mesh' })
|
||||||
|
const jobs = new HyperP2PJobDispatcher({ topic: 'super-mesh' })
|
||||||
|
await reg.ready()
|
||||||
|
reg.advertise({ cpuCores: 8, ramMb: 16384, gpuSlots: 1, upMbps: 100 })
|
||||||
|
cpu.donate(reg.peerId, 60_000)
|
||||||
|
const job = jobs.submitJob({ kind: 'render', cpuMs: 5000, ramMb: 512 })
|
||||||
|
```
|
||||||
|
|
||||||
|
## Modules by layer
|
||||||
|
|
||||||
|
### Registry & placement
|
||||||
|
|
||||||
|
| Module | Protocol | Role |
|
||||||
|
|--------|----------|------|
|
||||||
|
| [hyper-p2p-capacity-registry](./hyper-p2p-capacity-registry/) | `capacity-registry/v1` | Gossip CPU/RAM/GPU/disk/bandwidth ads |
|
||||||
|
| [hyper-p2p-cluster-affinity](./hyper-p2p-cluster-affinity/) | `cluster-affinity/v1` | Tag + latency scoring for placement |
|
||||||
|
| [hyper-p2p-thermal-guard](./hyper-p2p-thermal-guard/) | `thermal-guard/v1` | CPU/RAM/temp throttle signals |
|
||||||
|
|
||||||
|
### Resource pools
|
||||||
|
|
||||||
|
| Module | Protocol | Role |
|
||||||
|
|--------|----------|------|
|
||||||
|
| [hyper-p2p-cpu-share](./hyper-p2p-cpu-share/) | `cpu-share/v1` | CPU millisecond credits |
|
||||||
|
| [hyper-p2p-ram-pool](./hyper-p2p-ram-pool/) | `ram-pool/v1` | RAM byte lend / claim / release |
|
||||||
|
| [hyper-p2p-gpu-slot](./hyper-p2p-gpu-slot/) | `gpu-slot/v1` | GPU slot register / reserve |
|
||||||
|
| [hyper-p2p-bandwidth-share](./hyper-p2p-bandwidth-share/) | `bandwidth-share/v1` | Shared up/down Mbps |
|
||||||
|
| [hyper-p2p-disk-stripe](./hyper-p2p-disk-stripe/) | `disk-stripe/v1` | Striped block shards |
|
||||||
|
| [hyper-p2p-cache-farm](./hyper-p2p-cache-farm/) | `cache-farm/v1` | Distributed LRU cache |
|
||||||
|
|
||||||
|
### Workloads & egress
|
||||||
|
|
||||||
|
| Module | Protocol | Role |
|
||||||
|
|--------|----------|------|
|
||||||
|
| [hyper-p2p-job-dispatcher](./hyper-p2p-job-dispatcher/) | `job-dispatcher/v1` | Submit / claim / complete jobs |
|
||||||
|
| [hyper-p2p-work-stealer](./hyper-p2p-work-stealer/) | `work-stealer/v1` | Shard queues + steal |
|
||||||
|
| [hyper-p2p-net-gateway](./hyper-p2p-net-gateway/) | `net-gateway/v1` | Peer internet egress credits |
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
Typical stack: `capacity-registry` → `cluster-affinity` + `thermal-guard` → `cpu-share` / `ram-pool` / `gpu-slot` → `job-dispatcher` or `work-stealer`. Pair with [`hyper-p2p-bandwidth-broker`](../network-stack/hyper-p2p-bandwidth-broker/) and [`hyper-p2p-topic-lease`](../scheduling-queues/hyper-p2p-topic-lease/).
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
cd hyper-p2p-capacity-registry && npm install && npm test
|
||||||
|
../../real_tests/run-all.sh --tier=production
|
||||||
|
```
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (bandwidth-share/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-bandwidth-share
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `bandwidth-share/v1` · **Export:** `HyperP2PBandwidthShare`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Shared uplink and downlink Mbps offers per peer.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PBandwidthShare } = require('hyper-p2p-bandwidth-share')
|
||||||
|
const mod = new HyperP2PBandwidthShare({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-bandwidth-share
|
||||||
|
|
||||||
|
**Protocol:** `bandwidth-share/v1` · **Export:** `HyperP2PBandwidthShare`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Shared uplink and downlink Mbps offers per peer.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PBandwidthShare(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'bandwidth-share/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `bandwidth-share/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-bandwidth-share
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `bandwidth-share/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PBandwidthShare]
|
||||||
|
Mod --> Mux[Protomux bandwidth-share/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PBandwidthShare } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PBandwidthShare()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'bandwidth-share/v1'
|
||||||
|
|
||||||
|
class HyperP2PBandwidthShare extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { offered: 0, consumed: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._offers = new Map()
|
||||||
|
this._used = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_pid (peerId) {
|
||||||
|
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
offer (peerId, upMbps, downMbps) {
|
||||||
|
if (upMbps < 0 || downMbps < 0) throw new Error('Mbps must be non-negative')
|
||||||
|
const id = this._pid(peerId)
|
||||||
|
const offer = { peerId: id, upMbps, downMbps, at: Date.now() }
|
||||||
|
this._offers.set(id, offer)
|
||||||
|
this._stats.offered++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'offer', offer })
|
||||||
|
this.emit('offer', offer)
|
||||||
|
return offer
|
||||||
|
}
|
||||||
|
|
||||||
|
consume (peerId, bytes, direction = 'up') {
|
||||||
|
if (!peerId) throw new Error('peerId required')
|
||||||
|
if (bytes < 0) throw new Error('bytes must be non-negative')
|
||||||
|
const id = this._pid(peerId)
|
||||||
|
const offer = this._offers.get(id)
|
||||||
|
if (!offer) return false
|
||||||
|
const key = id + ':' + direction
|
||||||
|
const used = (this._used.get(key) || 0) + bytes
|
||||||
|
const capMbps = direction === 'down' ? offer.downMbps : offer.upMbps
|
||||||
|
const capBytes = (capMbps * 125000)
|
||||||
|
if (used > capBytes) return false
|
||||||
|
this._used.set(key, used)
|
||||||
|
this._stats.consumed += bytes
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'consume', peerId: id, bytes, direction, used })
|
||||||
|
this.emit('consume', { peerId: id, bytes, direction, used })
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
getOffer (peerId) { return this._offers.get(this._pid(peerId)) || null }
|
||||||
|
|
||||||
|
listOffers () { return [...this._offers.values()] }
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'offer' && data.offer) this._offers.set(data.offer.peerId, data.offer)
|
||||||
|
else if (data?.type === 'consume') {
|
||||||
|
const key = data.peerId + ':' + data.direction
|
||||||
|
const cur = this._used.get(key) || 0
|
||||||
|
if (data.used > cur) this._used.set(key, data.used)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, offers: this._offers.size, protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._offers.clear()
|
||||||
|
this._used.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PBandwidthShare, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-bandwidth-share",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "Shared uplink and downlink capacity for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,21 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PBandwidthShare } = require('../index.js')
|
||||||
|
|
||||||
|
test('bandwidth-share: offer consume', async (t) => {
|
||||||
|
const b = new HyperP2PBandwidthShare()
|
||||||
|
b.offer('peer-a', 100, 100)
|
||||||
|
t.ok(b.consume('peer-a', 1000, 'up'))
|
||||||
|
await b.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('bandwidth-share: validation', async (t) => {
|
||||||
|
const b = new HyperP2PBandwidthShare()
|
||||||
|
try {
|
||||||
|
b.offer('p', -1, 10)
|
||||||
|
t.fail('expected throw')
|
||||||
|
} catch (e) {
|
||||||
|
t.ok(e instanceof Error)
|
||||||
|
}
|
||||||
|
await b.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (cache-farm/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-cache-farm
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `cache-farm/v1` · **Export:** `HyperP2PCacheFarm`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Distributed LRU cache with gossip invalidation.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PCacheFarm } = require('hyper-p2p-cache-farm')
|
||||||
|
const mod = new HyperP2PCacheFarm({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-cache-farm
|
||||||
|
|
||||||
|
**Protocol:** `cache-farm/v1` · **Export:** `HyperP2PCacheFarm`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Distributed LRU cache with gossip invalidation.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PCacheFarm(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'cache-farm/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `cache-farm/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-cache-farm
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `cache-farm/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PCacheFarm]
|
||||||
|
Mod --> Mux[Protomux cache-farm/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PCacheFarm } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PCacheFarm()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'cache-farm/v1'
|
||||||
|
|
||||||
|
class HyperP2PCacheFarm extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { puts: 0, hits: 0, misses: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.maxEntries = opts.maxEntries ?? 4096
|
||||||
|
this._cache = new Map()
|
||||||
|
this._order = []
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_evict () {
|
||||||
|
while (this._cache.size > this.maxEntries && this._order.length) {
|
||||||
|
const k = this._order.shift()
|
||||||
|
this._cache.delete(k)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
put (key, value, ttlMs = null) {
|
||||||
|
if (key == null) throw new Error('key required')
|
||||||
|
const entry = { key: String(key), value, ttlMs, at: Date.now(), expiresAt: ttlMs ? Date.now() + ttlMs : null }
|
||||||
|
if (!this._cache.has(entry.key)) this._order.push(entry.key)
|
||||||
|
else {
|
||||||
|
const i = this._order.indexOf(entry.key)
|
||||||
|
if (i >= 0) this._order.splice(i, 1)
|
||||||
|
this._order.push(entry.key)
|
||||||
|
}
|
||||||
|
this._cache.set(entry.key, entry)
|
||||||
|
this._stats.puts++
|
||||||
|
this._evict()
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'put', entry })
|
||||||
|
this.emit('put', entry)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
get (key) {
|
||||||
|
const entry = this._cache.get(String(key))
|
||||||
|
if (!entry) {
|
||||||
|
this._stats.misses++
|
||||||
|
return undefined
|
||||||
|
}
|
||||||
|
if (entry.expiresAt && Date.now() > entry.expiresAt) {
|
||||||
|
this._cache.delete(entry.key)
|
||||||
|
this._stats.misses++
|
||||||
|
return undefined
|
||||||
|
}
|
||||||
|
const i = this._order.indexOf(entry.key)
|
||||||
|
if (i >= 0) { this._order.splice(i, 1); this._order.push(entry.key) }
|
||||||
|
this._stats.hits++
|
||||||
|
return entry.value
|
||||||
|
}
|
||||||
|
|
||||||
|
invalidate (key) {
|
||||||
|
const k = String(key)
|
||||||
|
const ok = this._cache.delete(k)
|
||||||
|
if (ok) {
|
||||||
|
const i = this._order.indexOf(k)
|
||||||
|
if (i >= 0) this._order.splice(i, 1)
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'invalidate', key: k })
|
||||||
|
this.emit('invalidate', { key: k })
|
||||||
|
}
|
||||||
|
return ok
|
||||||
|
}
|
||||||
|
|
||||||
|
size () { return this._cache.size }
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'put' && data.entry) this.put(data.entry.key, data.entry.value, data.entry.ttlMs)
|
||||||
|
else if (data?.type === 'invalidate') this._cache.delete(data.key)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, entries: this._cache.size, protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._cache.clear()
|
||||||
|
this._order = []
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PCacheFarm, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-cache-farm",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "Distributed LRU cache farm for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PCacheFarm } = require('../index.js')
|
||||||
|
|
||||||
|
test('cache-farm: put get invalidate', async (t) => {
|
||||||
|
const c = new HyperP2PCacheFarm()
|
||||||
|
c.put('k1', { v: 1 })
|
||||||
|
t.is(c.get('k1').v, 1)
|
||||||
|
t.ok(c.invalidate('k1'))
|
||||||
|
t.is(c.get('k1'), undefined)
|
||||||
|
await c.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('cache-farm: validation', async (t) => {
|
||||||
|
const c = new HyperP2PCacheFarm()
|
||||||
|
try {
|
||||||
|
c.put(null, 1)
|
||||||
|
t.fail('expected throw')
|
||||||
|
} catch (e) {
|
||||||
|
t.ok(e instanceof Error)
|
||||||
|
}
|
||||||
|
await c.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (capacity-registry/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-capacity-registry
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `capacity-registry/v1` · **Export:** `HyperP2PCapacityRegistry`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Gossip registry of peer CPU, RAM, GPU, disk, and bandwidth capacity.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PCapacityRegistry } = require('hyper-p2p-capacity-registry')
|
||||||
|
const mod = new HyperP2PCapacityRegistry({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-capacity-registry
|
||||||
|
|
||||||
|
**Protocol:** `capacity-registry/v1` · **Export:** `HyperP2PCapacityRegistry`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Gossip registry of peer CPU, RAM, GPU, disk, and bandwidth capacity.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PCapacityRegistry(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'capacity-registry/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `capacity-registry/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-capacity-registry
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `capacity-registry/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PCapacityRegistry]
|
||||||
|
Mod --> Mux[Protomux capacity-registry/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PCapacityRegistry } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PCapacityRegistry()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,104 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'capacity-registry/v1'
|
||||||
|
|
||||||
|
class HyperP2PCapacityRegistry extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { ops: 0, errors: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._peers = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_pid (peerId) {
|
||||||
|
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
advertise (resources = {}, peerId = null) {
|
||||||
|
const id = this._pid(peerId || this.peerId)
|
||||||
|
const rec = {
|
||||||
|
peerId: id,
|
||||||
|
cpuCores: Math.max(0, resources.cpuCores ?? 0),
|
||||||
|
ramMb: Math.max(0, resources.ramMb ?? 0),
|
||||||
|
diskGb: Math.max(0, resources.diskGb ?? 0),
|
||||||
|
gpuSlots: Math.max(0, resources.gpuSlots ?? 0),
|
||||||
|
upMbps: Math.max(0, resources.upMbps ?? 0),
|
||||||
|
downMbps: Math.max(0, resources.downMbps ?? 0),
|
||||||
|
tags: resources.tags || [],
|
||||||
|
at: Date.now()
|
||||||
|
}
|
||||||
|
this._peers.set(id, rec)
|
||||||
|
this._stats.ops++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'advertise', record: rec })
|
||||||
|
this.emit('advertise', rec)
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
lookup (peerId) {
|
||||||
|
return this._peers.get(this._pid(peerId)) || null
|
||||||
|
}
|
||||||
|
|
||||||
|
listPeers () { return [...this._peers.values()] }
|
||||||
|
|
||||||
|
bestMatch (requirements = {}) {
|
||||||
|
let best = null
|
||||||
|
let bestScore = -1
|
||||||
|
for (const rec of this._peers.values()) {
|
||||||
|
if (requirements.cpuCores != null && rec.cpuCores < requirements.cpuCores) continue
|
||||||
|
if (requirements.ramMb != null && rec.ramMb < requirements.ramMb) continue
|
||||||
|
if (requirements.gpuSlots != null && rec.gpuSlots < requirements.gpuSlots) continue
|
||||||
|
const score = rec.cpuCores + rec.ramMb / 1024 + rec.gpuSlots * 4 + rec.upMbps / 10
|
||||||
|
if (score > bestScore) { bestScore = score; best = rec }
|
||||||
|
}
|
||||||
|
return best
|
||||||
|
}
|
||||||
|
|
||||||
|
merge (remote) {
|
||||||
|
if (!remote || !remote.records) return 0
|
||||||
|
let n = 0
|
||||||
|
for (const rec of remote.records) {
|
||||||
|
const cur = this._peers.get(rec.peerId)
|
||||||
|
if (!cur || rec.at > cur.at) {
|
||||||
|
this._peers.set(rec.peerId, rec)
|
||||||
|
n++
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (n) this.emit('merge', { updated: n })
|
||||||
|
return n
|
||||||
|
}
|
||||||
|
|
||||||
|
toJSON () { return { records: this.listPeers() } }
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair,
|
||||||
|
topic: this.topic,
|
||||||
|
protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'advertise' && data.record) this.merge({ records: [data.record] })
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, peers: this._peers.size, protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._peers.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PCapacityRegistry, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-capacity-registry",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "Peer resource advertisement registry for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,25 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PCapacityRegistry } = require('../index.js')
|
||||||
|
|
||||||
|
test('capacity-registry: advertise and match', async (t) => {
|
||||||
|
const r = new HyperP2PCapacityRegistry()
|
||||||
|
r.advertise({ cpuCores: 8, ramMb: 16384 })
|
||||||
|
const best = r.bestMatch({ cpuCores: 4, ramMb: 8192 })
|
||||||
|
t.ok(best)
|
||||||
|
t.is(best.cpuCores, 8)
|
||||||
|
await r.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('capacity-registry: getStats', async (t) => {
|
||||||
|
const r = new HyperP2PCapacityRegistry()
|
||||||
|
t.is(r.getStats().protocol, 'capacity-registry/v1')
|
||||||
|
await r.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('capacity-registry: close idempotent', async (t) => {
|
||||||
|
const m = new HyperP2PCapacityRegistry()
|
||||||
|
await m.close()
|
||||||
|
await m.close()
|
||||||
|
t.pass()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (cluster-affinity/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-cluster-affinity
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `cluster-affinity/v1` · **Export:** `HyperP2PClusterAffinity`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Hardware tag and latency workload placement scoring.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PClusterAffinity } = require('hyper-p2p-cluster-affinity')
|
||||||
|
const mod = new HyperP2PClusterAffinity({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-cluster-affinity
|
||||||
|
|
||||||
|
**Protocol:** `cluster-affinity/v1` · **Export:** `HyperP2PClusterAffinity`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Hardware tag and latency workload placement scoring.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PClusterAffinity(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'cluster-affinity/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `cluster-affinity/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-cluster-affinity
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `cluster-affinity/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PClusterAffinity]
|
||||||
|
Mod --> Mux[Protomux cluster-affinity/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PClusterAffinity } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PClusterAffinity()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,86 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'cluster-affinity/v1'
|
||||||
|
|
||||||
|
class HyperP2PClusterAffinity extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { tags: 0, ranks: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._tags = new Map()
|
||||||
|
this._latency = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_pid (peerId) {
|
||||||
|
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
tagPeer (peerId, tags = [], latencyMs = null) {
|
||||||
|
const id = this._pid(peerId)
|
||||||
|
const rec = { peerId: id, tags: [...tags], at: Date.now() }
|
||||||
|
this._tags.set(id, rec)
|
||||||
|
if (latencyMs != null) this._latency.set(id, latencyMs)
|
||||||
|
this._stats.tags++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'tag', record: rec, latencyMs })
|
||||||
|
this.emit('tag', rec)
|
||||||
|
return rec
|
||||||
|
}
|
||||||
|
|
||||||
|
score (peerId, requiredTags = []) {
|
||||||
|
const id = this._pid(peerId)
|
||||||
|
const rec = this._tags.get(id)
|
||||||
|
if (!rec) return 0
|
||||||
|
let match = 0
|
||||||
|
for (const t of requiredTags) if (rec.tags.includes(t)) match++
|
||||||
|
const lat = this._latency.get(id) ?? 9999
|
||||||
|
return match * 1000 - lat
|
||||||
|
}
|
||||||
|
|
||||||
|
rank (requiredTags = []) {
|
||||||
|
const out = []
|
||||||
|
for (const rec of this._tags.values()) {
|
||||||
|
out.push({ peerId: rec.peerId, score: this.score(rec.peerId, requiredTags), tags: rec.tags })
|
||||||
|
}
|
||||||
|
out.sort((a, b) => b.score - a.score)
|
||||||
|
this._stats.ranks++
|
||||||
|
this.emit('rank', { requiredTags, results: out })
|
||||||
|
return out
|
||||||
|
}
|
||||||
|
|
||||||
|
listTagged () { return [...this._tags.values()] }
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'tag' && data.record) {
|
||||||
|
this._tags.set(data.record.peerId, data.record)
|
||||||
|
if (data.latencyMs != null) this._latency.set(data.record.peerId, data.latencyMs)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, peers: this._tags.size, protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._tags.clear()
|
||||||
|
this._latency.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PClusterAffinity, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-cluster-affinity",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "Hardware tag workload placement for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,18 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PClusterAffinity } = require('../index.js')
|
||||||
|
|
||||||
|
test('cluster-affinity: rank by tags', async (t) => {
|
||||||
|
const a = new HyperP2PClusterAffinity()
|
||||||
|
a.tagPeer('p1', ['gpu', 'linux'], 10)
|
||||||
|
a.tagPeer('p2', ['cpu'], 50)
|
||||||
|
const ranked = a.rank(['gpu'])
|
||||||
|
t.is(ranked[0].peerId, 'p1')
|
||||||
|
await a.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('cluster-affinity: getStats', async (t) => {
|
||||||
|
const a = new HyperP2PClusterAffinity()
|
||||||
|
t.is(a.getStats().protocol, 'cluster-affinity/v1')
|
||||||
|
await a.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (cpu-share/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-cpu-share
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `cpu-share/v1` · **Export:** `HyperP2PCpuShare`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
P2P CPU millisecond credit donate and consume pool.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PCpuShare } = require('hyper-p2p-cpu-share')
|
||||||
|
const mod = new HyperP2PCpuShare({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-cpu-share
|
||||||
|
|
||||||
|
**Protocol:** `cpu-share/v1` · **Export:** `HyperP2PCpuShare`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
P2P CPU millisecond credit donate and consume pool.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PCpuShare(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'cpu-share/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `cpu-share/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-cpu-share
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `cpu-share/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PCpuShare]
|
||||||
|
Mod --> Mux[Protomux cpu-share/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PCpuShare } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PCpuShare()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,78 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'cpu-share/v1'
|
||||||
|
|
||||||
|
class HyperP2PCpuShare extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { donated: 0, consumed: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._balances = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_pid (peerId) {
|
||||||
|
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
donate (peerId, cpuMs) {
|
||||||
|
if (cpuMs == null || cpuMs < 0) throw new Error('cpuMs must be non-negative')
|
||||||
|
const id = this._pid(peerId)
|
||||||
|
const next = (this._balances.get(id) || 0) + cpuMs
|
||||||
|
this._balances.set(id, next)
|
||||||
|
this._stats.donated += cpuMs
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'donate', peerId: id, balance: next })
|
||||||
|
this.emit('donate', { peerId: id, cpuMs, balance: next })
|
||||||
|
return next
|
||||||
|
}
|
||||||
|
|
||||||
|
consume (peerId, cpuMs) {
|
||||||
|
if (cpuMs == null || cpuMs < 0) throw new Error('cpuMs must be non-negative')
|
||||||
|
const id = this._pid(peerId)
|
||||||
|
const cur = this._balances.get(id) || 0
|
||||||
|
if (cur < cpuMs) return false
|
||||||
|
const next = cur - cpuMs
|
||||||
|
this._balances.set(id, next)
|
||||||
|
this._stats.consumed += cpuMs
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'consume', peerId: id, balance: next })
|
||||||
|
this.emit('consume', { peerId: id, cpuMs, balance: next })
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
balance (peerId) { return this._balances.get(this._pid(peerId)) || 0 }
|
||||||
|
|
||||||
|
listPeerIds () { return [...this._balances.keys()] }
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'donate' || data?.type === 'consume') {
|
||||||
|
const cur = this._balances.get(data.peerId) || 0
|
||||||
|
if (data.balance > cur) this._balances.set(data.peerId, data.balance)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, peers: this._balances.size, protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._balances.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PCpuShare, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-cpu-share",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "Distributed CPU millisecond credit pool for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PCpuShare } = require('../index.js')
|
||||||
|
|
||||||
|
test('cpu-share: donate and consume', async (t) => {
|
||||||
|
const c = new HyperP2PCpuShare()
|
||||||
|
c.donate('peer-a', 1000)
|
||||||
|
t.ok(c.consume('peer-a', 400))
|
||||||
|
t.not(c.consume('peer-a', 700))
|
||||||
|
await c.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('cpu-share: validation', async (t) => {
|
||||||
|
const c = new HyperP2PCpuShare()
|
||||||
|
try {
|
||||||
|
c.donate('p', -1)
|
||||||
|
t.fail('expected throw')
|
||||||
|
} catch (e) {
|
||||||
|
t.ok(e instanceof Error)
|
||||||
|
}
|
||||||
|
await c.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('cpu-share: close idempotent', async (t) => {
|
||||||
|
const m = new HyperP2PCpuShare()
|
||||||
|
await m.close()
|
||||||
|
await m.close()
|
||||||
|
t.pass()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (disk-stripe/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-disk-stripe
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `disk-stripe/v1` · **Export:** `HyperP2PDiskStripe`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Striped block storage shards across the mesh.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PDiskStripe } = require('hyper-p2p-disk-stripe')
|
||||||
|
const mod = new HyperP2PDiskStripe({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-disk-stripe
|
||||||
|
|
||||||
|
**Protocol:** `disk-stripe/v1` · **Export:** `HyperP2PDiskStripe`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Striped block storage shards across the mesh.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PDiskStripe(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'disk-stripe/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `disk-stripe/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-disk-stripe
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `disk-stripe/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PDiskStripe]
|
||||||
|
Mod --> Mux[Protomux disk-stripe/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PDiskStripe } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PDiskStripe()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,101 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'disk-stripe/v1'
|
||||||
|
|
||||||
|
class HyperP2PDiskStripe extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { stripes: 0, shards: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._stripes = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
allocateStripe (sizeBytes, pathCount = 2, stripeId = null) {
|
||||||
|
if (sizeBytes == null || sizeBytes < 1) throw new Error('sizeBytes required')
|
||||||
|
if (pathCount < 1 || pathCount > 32) throw new Error('pathCount must be 1-32')
|
||||||
|
const id = stripeId || b4a.toString(require('hypercore-crypto').hash(b4a.from(String(sizeBytes) + Date.now())), 'hex').slice(0, 16)
|
||||||
|
const shardSize = Math.ceil(sizeBytes / pathCount)
|
||||||
|
const stripe = {
|
||||||
|
stripeId: id,
|
||||||
|
sizeBytes,
|
||||||
|
pathCount,
|
||||||
|
shardSize,
|
||||||
|
shards: new Map(),
|
||||||
|
createdAt: Date.now()
|
||||||
|
}
|
||||||
|
this._stripes.set(id, stripe)
|
||||||
|
this._stats.stripes++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'allocate', stripe: { ...stripe, shards: null } })
|
||||||
|
this.emit('allocate', stripe)
|
||||||
|
return id
|
||||||
|
}
|
||||||
|
|
||||||
|
writeShard (stripeId, idx, data, peerId = null) {
|
||||||
|
const stripe = this._stripes.get(stripeId)
|
||||||
|
if (!stripe) throw new Error('unknown stripe')
|
||||||
|
if (idx < 0 || idx >= stripe.pathCount) throw new Error('invalid shard index')
|
||||||
|
const buf = b4a.isBuffer(data) ? data : b4a.from(data)
|
||||||
|
const shard = {
|
||||||
|
idx,
|
||||||
|
data: b4a.toString(buf, 'hex'),
|
||||||
|
peerId: peerId || this.peerId,
|
||||||
|
at: Date.now()
|
||||||
|
}
|
||||||
|
stripe.shards.set(idx, shard)
|
||||||
|
this._stats.shards++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'shard', stripeId, shard })
|
||||||
|
this.emit('shard', { stripeId, shard })
|
||||||
|
return shard
|
||||||
|
}
|
||||||
|
|
||||||
|
readShard (stripeId, idx) {
|
||||||
|
const stripe = this._stripes.get(stripeId)
|
||||||
|
if (!stripe) return null
|
||||||
|
const s = stripe.shards.get(idx)
|
||||||
|
return s ? { ...s, data: b4a.from(s.data, 'hex') } : null
|
||||||
|
}
|
||||||
|
|
||||||
|
isComplete (stripeId) {
|
||||||
|
const stripe = this._stripes.get(stripeId)
|
||||||
|
return stripe ? stripe.shards.size === stripe.pathCount : false
|
||||||
|
}
|
||||||
|
|
||||||
|
listStripeIds () { return [...this._stripes.keys()] }
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'allocate' && data.stripe) {
|
||||||
|
const s = { ...data.stripe, shards: new Map() }
|
||||||
|
this._stripes.set(s.stripeId, s)
|
||||||
|
} else if (data?.type === 'shard') {
|
||||||
|
const stripe = this._stripes.get(data.stripeId)
|
||||||
|
if (stripe) stripe.shards.set(data.shard.idx, data.shard)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, open: this._stripes.size, protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._stripes.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PDiskStripe, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-disk-stripe",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "Striped block storage across peers for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PDiskStripe } = require('../index.js')
|
||||||
|
|
||||||
|
test('disk-stripe: allocate and shard', async (t) => {
|
||||||
|
const d = new HyperP2PDiskStripe()
|
||||||
|
const id = d.allocateStripe(1024, 2)
|
||||||
|
d.writeShard(id, 0, 'aa')
|
||||||
|
d.writeShard(id, 1, 'bb')
|
||||||
|
t.ok(d.isComplete(id))
|
||||||
|
await d.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('disk-stripe: validation', async (t) => {
|
||||||
|
const d = new HyperP2PDiskStripe()
|
||||||
|
try {
|
||||||
|
d.allocateStripe(0)
|
||||||
|
t.fail('expected throw')
|
||||||
|
} catch (e) {
|
||||||
|
t.ok(e instanceof Error)
|
||||||
|
}
|
||||||
|
await d.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (gpu-slot/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-gpu-slot
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `gpu-slot/v1` · **Export:** `HyperP2PGpuSlot`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
GPU slot registration and reservation gossip.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PGpuSlot } = require('hyper-p2p-gpu-slot')
|
||||||
|
const mod = new HyperP2PGpuSlot({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-gpu-slot
|
||||||
|
|
||||||
|
**Protocol:** `gpu-slot/v1` · **Export:** `HyperP2PGpuSlot`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
GPU slot registration and reservation gossip.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PGpuSlot(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'gpu-slot/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `gpu-slot/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-gpu-slot
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `gpu-slot/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PGpuSlot]
|
||||||
|
Mod --> Mux[Protomux gpu-slot/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PGpuSlot } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PGpuSlot()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'gpu-slot/v1'
|
||||||
|
|
||||||
|
class HyperP2PGpuSlot extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { registered: 0, reserved: 0, released: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._slots = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_pid (peerId) {
|
||||||
|
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
registerSlot (slotId, caps = {}) {
|
||||||
|
if (!slotId) throw new Error('slotId required')
|
||||||
|
const slot = {
|
||||||
|
slotId: String(slotId),
|
||||||
|
owner: this.peerId,
|
||||||
|
vramMb: caps.vramMb ?? 0,
|
||||||
|
computeUnits: caps.computeUnits ?? 1,
|
||||||
|
tags: caps.tags || [],
|
||||||
|
state: 'free',
|
||||||
|
holder: null,
|
||||||
|
at: Date.now()
|
||||||
|
}
|
||||||
|
this._slots.set(slot.slotId, slot)
|
||||||
|
this._stats.registered++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'register', slot })
|
||||||
|
this.emit('register', slot)
|
||||||
|
return slot
|
||||||
|
}
|
||||||
|
|
||||||
|
reserve (slotId, peerId = null) {
|
||||||
|
const slot = this._slots.get(String(slotId))
|
||||||
|
if (!slot || slot.state !== 'free') return null
|
||||||
|
slot.state = 'reserved'
|
||||||
|
slot.holder = this._pid(peerId || this.peerId)
|
||||||
|
slot.at = Date.now()
|
||||||
|
this._stats.reserved++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'reserve', slotId: slot.slotId, holder: slot.holder })
|
||||||
|
this.emit('reserve', slot)
|
||||||
|
return slot
|
||||||
|
}
|
||||||
|
|
||||||
|
release (slotId) {
|
||||||
|
const slot = this._slots.get(String(slotId))
|
||||||
|
if (!slot) return false
|
||||||
|
slot.state = 'free'
|
||||||
|
slot.holder = null
|
||||||
|
this._stats.released++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'release', slotId: slot.slotId })
|
||||||
|
this.emit('release', slot)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
listSlots () { return [...this._slots.values()] }
|
||||||
|
|
||||||
|
freeSlots () { return this.listSlots().filter((s) => s.state === 'free') }
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'register' && data.slot) this._slots.set(data.slot.slotId, data.slot)
|
||||||
|
else if (data?.type === 'reserve') {
|
||||||
|
const s = this._slots.get(data.slotId)
|
||||||
|
if (s && s.state === 'free') { s.state = 'reserved'; s.holder = data.holder }
|
||||||
|
} else if (data?.type === 'release') {
|
||||||
|
const s = this._slots.get(data.slotId)
|
||||||
|
if (s) { s.state = 'free'; s.holder = null }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
slots: this._slots.size,
|
||||||
|
free: this.freeSlots().length,
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._slots.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PGpuSlot, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-gpu-slot",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "GPU slot registration and reservation for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PGpuSlot } = require('../index.js')
|
||||||
|
|
||||||
|
test('gpu-slot: register reserve release', async (t) => {
|
||||||
|
const g = new HyperP2PGpuSlot()
|
||||||
|
g.registerSlot('gpu0', { vramMb: 8192 })
|
||||||
|
const s = g.reserve('gpu0')
|
||||||
|
t.ok(s)
|
||||||
|
t.ok(g.release('gpu0'))
|
||||||
|
await g.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('gpu-slot: validation', async (t) => {
|
||||||
|
const g = new HyperP2PGpuSlot()
|
||||||
|
try {
|
||||||
|
g.registerSlot(null)
|
||||||
|
t.fail('expected throw')
|
||||||
|
} catch (e) {
|
||||||
|
t.ok(e instanceof Error)
|
||||||
|
}
|
||||||
|
await g.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (job-dispatcher/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-job-dispatcher
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `job-dispatcher/v1` · **Export:** `HyperP2PJobDispatcher`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Distributed submit, claim, and complete job queue.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PJobDispatcher } = require('hyper-p2p-job-dispatcher')
|
||||||
|
const mod = new HyperP2PJobDispatcher({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-job-dispatcher
|
||||||
|
|
||||||
|
**Protocol:** `job-dispatcher/v1` · **Export:** `HyperP2PJobDispatcher`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Distributed submit, claim, and complete job queue.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PJobDispatcher(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'job-dispatcher/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `job-dispatcher/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-job-dispatcher
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `job-dispatcher/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PJobDispatcher]
|
||||||
|
Mod --> Mux[Protomux job-dispatcher/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PJobDispatcher } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PJobDispatcher()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,107 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'job-dispatcher/v1'
|
||||||
|
|
||||||
|
class HyperP2PJobDispatcher extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { submitted: 0, claimed: 0, completed: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._jobs = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_pid (peerId) {
|
||||||
|
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
submitJob (spec = {}) {
|
||||||
|
const jobId = b4a.toString(require('hypercore-crypto').hash(b4a.from(JSON.stringify(spec) + Date.now())), 'hex').slice(0, 16)
|
||||||
|
const job = {
|
||||||
|
jobId,
|
||||||
|
spec,
|
||||||
|
state: 'pending',
|
||||||
|
submitter: this.peerId,
|
||||||
|
worker: null,
|
||||||
|
result: null,
|
||||||
|
cpuMs: spec.cpuMs ?? 0,
|
||||||
|
ramMb: spec.ramMb ?? 0,
|
||||||
|
at: Date.now()
|
||||||
|
}
|
||||||
|
this._jobs.set(jobId, job)
|
||||||
|
this._stats.submitted++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'submit', job })
|
||||||
|
this.emit('submit', job)
|
||||||
|
return job
|
||||||
|
}
|
||||||
|
|
||||||
|
claimJob (workerId = null) {
|
||||||
|
const wid = this._pid(workerId || this.peerId)
|
||||||
|
for (const job of this._jobs.values()) {
|
||||||
|
if (job.state !== 'pending') continue
|
||||||
|
job.state = 'running'
|
||||||
|
job.worker = wid
|
||||||
|
job.claimedAt = Date.now()
|
||||||
|
this._stats.claimed++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'claim', jobId: job.jobId, worker: wid })
|
||||||
|
this.emit('claim', job)
|
||||||
|
return job
|
||||||
|
}
|
||||||
|
return null
|
||||||
|
}
|
||||||
|
|
||||||
|
completeJob (jobId, result = null) {
|
||||||
|
const job = this._jobs.get(jobId)
|
||||||
|
if (!job || job.state !== 'running') return false
|
||||||
|
job.state = 'done'
|
||||||
|
job.result = result
|
||||||
|
job.completedAt = Date.now()
|
||||||
|
this._stats.completed++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'complete', jobId, result })
|
||||||
|
this.emit('complete', job)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
getJob (jobId) { return this._jobs.get(jobId) || null }
|
||||||
|
|
||||||
|
pendingCount () {
|
||||||
|
return [...this._jobs.values()].filter((j) => j.state === 'pending').length
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'submit' && data.job) this._jobs.set(data.job.jobId, data.job)
|
||||||
|
else if (data?.type === 'claim') {
|
||||||
|
const j = this._jobs.get(data.jobId)
|
||||||
|
if (j && j.state === 'pending') { j.state = 'running'; j.worker = data.worker }
|
||||||
|
} else if (data?.type === 'complete') {
|
||||||
|
const j = this._jobs.get(data.jobId)
|
||||||
|
if (j) { j.state = 'done'; j.result = data.result }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, jobs: this._jobs.size, pending: this.pendingCount(), protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._jobs.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PJobDispatcher, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-job-dispatcher",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "Distributed compute job queue for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,19 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PJobDispatcher } = require('../index.js')
|
||||||
|
|
||||||
|
test('job-dispatcher: submit claim complete', async (t) => {
|
||||||
|
const j = new HyperP2PJobDispatcher()
|
||||||
|
const job = j.submitJob({ kind: 'hash', cpuMs: 100 })
|
||||||
|
const claimed = j.claimJob('worker-1')
|
||||||
|
t.ok(claimed)
|
||||||
|
t.ok(j.completeJob(job.jobId, { ok: true }))
|
||||||
|
await j.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('job-dispatcher: pending count', async (t) => {
|
||||||
|
const j = new HyperP2PJobDispatcher()
|
||||||
|
j.submitJob({ a: 1 })
|
||||||
|
t.is(j.pendingCount(), 1)
|
||||||
|
await j.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (net-gateway/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-net-gateway
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `net-gateway/v1` · **Export:** `HyperP2PNetGateway`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
Peer internet egress gateway registration and routing.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PNetGateway } = require('hyper-p2p-net-gateway')
|
||||||
|
const mod = new HyperP2PNetGateway({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-net-gateway
|
||||||
|
|
||||||
|
**Protocol:** `net-gateway/v1` · **Export:** `HyperP2PNetGateway`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
Peer internet egress gateway registration and routing.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PNetGateway(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'net-gateway/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `net-gateway/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-net-gateway
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `net-gateway/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PNetGateway]
|
||||||
|
Mod --> Mux[Protomux net-gateway/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PNetGateway } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PNetGateway()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,93 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'net-gateway/v1'
|
||||||
|
|
||||||
|
class HyperP2PNetGateway extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { registered: 0, routed: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._gateways = new Map()
|
||||||
|
this._usage = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_pid (peerId) {
|
||||||
|
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
registerGateway (peerId, caps = {}) {
|
||||||
|
const id = this._pid(peerId)
|
||||||
|
const gw = {
|
||||||
|
peerId: id,
|
||||||
|
maxBytes: caps.maxBytes ?? 1_000_000_000,
|
||||||
|
regions: caps.regions || [],
|
||||||
|
nat: caps.nat ?? 'unknown',
|
||||||
|
at: Date.now()
|
||||||
|
}
|
||||||
|
this._gateways.set(id, gw)
|
||||||
|
this._usage.set(id, 0)
|
||||||
|
this._stats.registered++
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'register', gateway: gw })
|
||||||
|
this.emit('register', gw)
|
||||||
|
return gw
|
||||||
|
}
|
||||||
|
|
||||||
|
requestRoute (bytes, preferPeerId = null) {
|
||||||
|
if (bytes < 0) throw new Error('bytes must be non-negative')
|
||||||
|
let gw = preferPeerId ? this._gateways.get(this._pid(preferPeerId)) : null
|
||||||
|
if (!gw) {
|
||||||
|
for (const g of this._gateways.values()) {
|
||||||
|
const used = this._usage.get(g.peerId) || 0
|
||||||
|
if (used + bytes <= g.maxBytes) { gw = g; break }
|
||||||
|
}
|
||||||
|
}
|
||||||
|
if (!gw) return null
|
||||||
|
const used = (this._usage.get(gw.peerId) || 0) + bytes
|
||||||
|
if (used > gw.maxBytes) return null
|
||||||
|
this._usage.set(gw.peerId, used)
|
||||||
|
this._stats.routed += bytes
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'route', peerId: gw.peerId, bytes, used })
|
||||||
|
this.emit('route', { gateway: gw.peerId, bytes, used })
|
||||||
|
return gw
|
||||||
|
}
|
||||||
|
|
||||||
|
listGateways () { return [...this._gateways.values()] }
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'register' && data.gateway) {
|
||||||
|
this._gateways.set(data.gateway.peerId, data.gateway)
|
||||||
|
this._usage.set(data.gateway.peerId, this._usage.get(data.gateway.peerId) || 0)
|
||||||
|
} else if (data?.type === 'route') {
|
||||||
|
const cur = this._usage.get(data.peerId) || 0
|
||||||
|
if (data.used > cur) this._usage.set(data.peerId, data.used)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return { ...this._stats, gateways: this._gateways.size, protocol: PROTOCOL }
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._gateways.clear()
|
||||||
|
this._usage.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PNetGateway, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-net-gateway",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "Peer internet egress gateway credits for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PNetGateway } = require('../index.js')
|
||||||
|
|
||||||
|
test('net-gateway: register route', async (t) => {
|
||||||
|
const g = new HyperP2PNetGateway()
|
||||||
|
g.registerGateway('gw-1', { maxBytes: 1_000_000 })
|
||||||
|
const gw = g.requestRoute(5000)
|
||||||
|
t.ok(gw)
|
||||||
|
await g.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('net-gateway: validation', async (t) => {
|
||||||
|
const g = new HyperP2PNetGateway()
|
||||||
|
try {
|
||||||
|
g.requestRoute(-1)
|
||||||
|
t.fail('expected throw')
|
||||||
|
} catch (e) {
|
||||||
|
t.ok(e instanceof Error)
|
||||||
|
}
|
||||||
|
await g.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (ram-pool/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-ram-pool
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `ram-pool/v1` · **Export:** `HyperP2PRamPool`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
RAM byte lending with claim leases across peers.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PRamPool } = require('hyper-p2p-ram-pool')
|
||||||
|
const mod = new HyperP2PRamPool({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-ram-pool
|
||||||
|
|
||||||
|
**Protocol:** `ram-pool/v1` · **Export:** `HyperP2PRamPool`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
RAM byte lending with claim leases across peers.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PRamPool(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'ram-pool/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `ram-pool/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-ram-pool
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `ram-pool/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PRamPool]
|
||||||
|
Mod --> Mux[Protomux ram-pool/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
@@ -0,0 +1,10 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const { HyperP2PRamPool } = require('../index.js')
|
||||||
|
|
||||||
|
async function main () {
|
||||||
|
const mod = new HyperP2PRamPool()
|
||||||
|
console.log('stats', mod.getStats())
|
||||||
|
await mod.close()
|
||||||
|
console.log('done')
|
||||||
|
}
|
||||||
|
main().catch(console.error)
|
||||||
@@ -0,0 +1,105 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const EventEmitter = require('bare-events')
|
||||||
|
const b4a = require('b4a')
|
||||||
|
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
|
||||||
|
|
||||||
|
const PROTOCOL = 'ram-pool/v1'
|
||||||
|
|
||||||
|
class HyperP2PRamPool extends EventEmitter {
|
||||||
|
constructor (opts = {}) {
|
||||||
|
super()
|
||||||
|
this._stats = { lent: 0, claimed: 0, released: 0 }
|
||||||
|
this.topic = opts.topic || null
|
||||||
|
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
|
||||||
|
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
|
||||||
|
this._pool = new Map()
|
||||||
|
this._leases = new Map()
|
||||||
|
this.swarm = null
|
||||||
|
this._peerMsgs = null
|
||||||
|
}
|
||||||
|
|
||||||
|
_pid (peerId) {
|
||||||
|
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
|
||||||
|
}
|
||||||
|
|
||||||
|
lend (peerId, bytes) {
|
||||||
|
if (bytes == null || bytes < 0) throw new Error('bytes must be non-negative')
|
||||||
|
const id = this._pid(peerId)
|
||||||
|
const next = (this._pool.get(id) || 0) + bytes
|
||||||
|
this._pool.set(id, next)
|
||||||
|
this._stats.lent += bytes
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'lend', peerId: id, pool: next })
|
||||||
|
this.emit('lend', { peerId: id, bytes, pool: next })
|
||||||
|
return next
|
||||||
|
}
|
||||||
|
|
||||||
|
claim (bytes, borrowerId = null) {
|
||||||
|
if (bytes == null || bytes < 0) throw new Error('bytes must be non-negative')
|
||||||
|
const borrower = this._pid(borrowerId || this.peerId)
|
||||||
|
let best = null
|
||||||
|
for (const [id, avail] of this._pool) {
|
||||||
|
if (id === borrower) continue
|
||||||
|
if (avail >= bytes && (!best || avail > best.avail)) best = { id, avail }
|
||||||
|
}
|
||||||
|
if (!best) return null
|
||||||
|
const next = best.avail - bytes
|
||||||
|
this._pool.set(best.id, next)
|
||||||
|
const leaseId = b4a.toString(require('hypercore-crypto').hash(b4a.from(best.id + borrower + Date.now())), 'hex').slice(0, 12)
|
||||||
|
const lease = { leaseId, lender: best.id, borrower, bytes, at: Date.now() }
|
||||||
|
this._leases.set(leaseId, lease)
|
||||||
|
this._stats.claimed += bytes
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'claim', lease, pool: next })
|
||||||
|
this.emit('claim', lease)
|
||||||
|
return lease
|
||||||
|
}
|
||||||
|
|
||||||
|
release (leaseId) {
|
||||||
|
const lease = this._leases.get(leaseId)
|
||||||
|
if (!lease) return false
|
||||||
|
this._leases.delete(leaseId)
|
||||||
|
this._pool.set(lease.lender, (this._pool.get(lease.lender) || 0) + lease.bytes)
|
||||||
|
this._stats.released += lease.bytes
|
||||||
|
if (this._peerMsgs) gossipSend(this, { type: 'release', leaseId })
|
||||||
|
this.emit('release', lease)
|
||||||
|
return true
|
||||||
|
}
|
||||||
|
|
||||||
|
poolTotal () {
|
||||||
|
let t = 0
|
||||||
|
for (const v of this._pool.values()) t += v
|
||||||
|
return t
|
||||||
|
}
|
||||||
|
|
||||||
|
async ready () {
|
||||||
|
if (this.swarm || !this.topic) return this
|
||||||
|
await initModuleSwarm(this, {
|
||||||
|
keyPair: this.keyPair, topic: this.topic, protocol: PROTOCOL,
|
||||||
|
onmessage: (data) => {
|
||||||
|
if (data?.type === 'lend') this._pool.set(data.peerId, data.pool)
|
||||||
|
else if (data?.type === 'claim' && data.lease) this._leases.set(data.lease.leaseId, data.lease)
|
||||||
|
else if (data?.type === 'release') this._leases.delete(data.leaseId)
|
||||||
|
}
|
||||||
|
})
|
||||||
|
return this
|
||||||
|
}
|
||||||
|
|
||||||
|
getStats () {
|
||||||
|
return {
|
||||||
|
...this._stats,
|
||||||
|
lenders: this._pool.size,
|
||||||
|
leases: this._leases.size,
|
||||||
|
poolTotal: this.poolTotal(),
|
||||||
|
protocol: PROTOCOL
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async close () {
|
||||||
|
if (this.swarm) await this.swarm.destroy().catch(() => {})
|
||||||
|
this.swarm = null
|
||||||
|
this._pool.clear()
|
||||||
|
this._leases.clear()
|
||||||
|
this.emit('closed')
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
module.exports = { HyperP2PRamPool, PROTOCOL }
|
||||||
+1774
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,23 @@
|
|||||||
|
{
|
||||||
|
"name": "hyper-p2p-ram-pool",
|
||||||
|
"version": "0.3.0",
|
||||||
|
"description": "P2P RAM byte lending and leases for Bare/Pear P2P supercomputer mesh.",
|
||||||
|
"main": "index.js",
|
||||||
|
"type": "commonjs",
|
||||||
|
"license": "Apache-2.0",
|
||||||
|
"scripts": { "test": "brittle-bare test/test.js" },
|
||||||
|
"dependencies": {
|
||||||
|
"bare-events": "^2.8.0",
|
||||||
|
"bare-process": "^4.4.0",
|
||||||
|
"b4a": "^1.6.7",
|
||||||
|
"hypercore-crypto": "^3.0.0",
|
||||||
|
"protomux": "^3.0.0",
|
||||||
|
"compact-encoding": "^2.0.0"
|
||||||
|
},
|
||||||
|
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
|
||||||
|
"devDependencies": { "brittle": "^3.0.0" },
|
||||||
|
"imports": {
|
||||||
|
"process": { "bare": "bare-process", "default": "process" },
|
||||||
|
"events": { "bare": "bare-events", "default": "events" }
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,23 @@
|
|||||||
|
require('bare-process/global')
|
||||||
|
const test = require('brittle')
|
||||||
|
const { HyperP2PRamPool } = require('../index.js')
|
||||||
|
|
||||||
|
test('ram-pool: lend and claim', async (t) => {
|
||||||
|
const p = new HyperP2PRamPool()
|
||||||
|
p.lend('lender-1', 1_000_000)
|
||||||
|
const lease = p.claim(500_000, 'borrower-1')
|
||||||
|
t.ok(lease)
|
||||||
|
t.ok(p.release(lease.leaseId))
|
||||||
|
await p.close()
|
||||||
|
})
|
||||||
|
|
||||||
|
test('ram-pool: validation', async (t) => {
|
||||||
|
const p = new HyperP2PRamPool()
|
||||||
|
try {
|
||||||
|
p.lend('x', -1)
|
||||||
|
t.fail('expected throw')
|
||||||
|
} catch (e) {
|
||||||
|
t.ok(e instanceof Error)
|
||||||
|
}
|
||||||
|
await p.close()
|
||||||
|
})
|
||||||
@@ -0,0 +1,5 @@
|
|||||||
|
# Changelog
|
||||||
|
|
||||||
|
## 0.3.0
|
||||||
|
|
||||||
|
- Initial Supercomputer category release (thermal-guard/v1).
|
||||||
@@ -0,0 +1,30 @@
|
|||||||
|
# hyper-p2p-thermal-guard
|
||||||
|
|
||||||
|
Production **Supercomputer** module: pool P2P system resources over Hyperswarm.
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `thermal-guard/v1` · **Export:** `HyperP2PThermalGuard`
|
||||||
|
|
||||||
|
## When to use
|
||||||
|
|
||||||
|
CPU, RAM, and temperature load throttle signals.
|
||||||
|
|
||||||
|
## Quick start
|
||||||
|
|
||||||
|
```js
|
||||||
|
const { HyperP2PThermalGuard } = require('hyper-p2p-thermal-guard')
|
||||||
|
const mod = new HyperP2PThermalGuard({ topic: 'my-super-mesh' })
|
||||||
|
await mod.ready()
|
||||||
|
// ...
|
||||||
|
await mod.close()
|
||||||
|
```
|
||||||
|
|
||||||
|
## Docs
|
||||||
|
|
||||||
|
- [docs/api.md](docs/api.md)
|
||||||
|
- [docs/architecture.md](docs/architecture.md)
|
||||||
|
|
||||||
|
## Test
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,36 @@
|
|||||||
|
# API: hyper-p2p-thermal-guard
|
||||||
|
|
||||||
|
**Protocol:** `thermal-guard/v1` · **Export:** `HyperP2PThermalGuard`
|
||||||
|
|
||||||
|
## Overview
|
||||||
|
|
||||||
|
CPU, RAM, and temperature load throttle signals.
|
||||||
|
|
||||||
|
## Constructor
|
||||||
|
|
||||||
|
```js
|
||||||
|
const mod = new HyperP2PThermalGuard(opts)
|
||||||
|
```
|
||||||
|
|
||||||
|
| Option | Type | Default | Description |
|
||||||
|
|--------|------|---------|-------------|
|
||||||
|
| `topic` | Buffer \| string \| null | `null` | Hyperswarm topic |
|
||||||
|
| `keyPair` | KeyPair | random | Discovery identity |
|
||||||
|
|
||||||
|
## Methods
|
||||||
|
|
||||||
|
See [`index.js`](../index.js) for the full method list. All modules implement `getStats()`, `async ready()`, and `async close()`.
|
||||||
|
|
||||||
|
## getStats()
|
||||||
|
|
||||||
|
Returns `{ ...stats, protocol: 'thermal-guard/v1' }` plus module-specific counters.
|
||||||
|
|
||||||
|
## P2P
|
||||||
|
|
||||||
|
When `topic` is set, `ready()` joins Hyperswarm and opens Protomux `thermal-guard/v1`.
|
||||||
|
|
||||||
|
## Testing
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npm install && npm test
|
||||||
|
```
|
||||||
@@ -0,0 +1,14 @@
|
|||||||
|
# Architecture: hyper-p2p-thermal-guard
|
||||||
|
|
||||||
|
**Category:** Supercomputer · **Protocol:** `thermal-guard/v1`
|
||||||
|
|
||||||
|
```mermaid
|
||||||
|
flowchart LR
|
||||||
|
App[Application] --> Mod[HyperP2PThermalGuard]
|
||||||
|
Mod --> Mux[Protomux thermal-guard/v1]
|
||||||
|
Mux --> Swarm[Hyperswarm]
|
||||||
|
```
|
||||||
|
|
||||||
|
## Composition
|
||||||
|
|
||||||
|
See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md) and [`../README.md`](../README.md).
|
||||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user