Unify supercomputer mesh into one virtual machine via cluster-fabric

Add hyper-p2p-cluster-fabric: aggregates every peer's CPU, RAM, GPU, disk,
and bandwidth into a single logical supercomputer (clusterId, asOneMachine).

- getVirtualMachine() — total vs available capacity across all nodes
- publishNode() — join the giant computer; syncs attached pool modules
- reserveCluster() — greedy multi-peer allocation for one workload
- runClusterJob() — reserve + fan-out jobs across slices
- Gossip cluster-node / cluster-reserve / cluster-run on Protomux

Enhance hyper-p2p-capacity-registry with clusterTotals() for the same
aggregate view at the registry layer.

Update SUPERCOMPUTER_LAYERS, category README, demo (3 racks → 44 cores,
155 GB RAM, 5 GPUs as one machine). Registry now 165 modules.

Co-authored-by: Cursor <[email protected]>
This commit is contained in:
Raven Scott
2026-05-21 00:41:57 -04:00
co-authored by Cursor
parent 693385bd7c
commit 03151a4cfc
17 changed files with 2442 additions and 22 deletions
@@ -0,0 +1,5 @@
# Changelog
## 0.3.0
- Initial release: unified virtual supercomputer (`cluster-fabric/v1`).
@@ -0,0 +1,54 @@
# hyper-p2p-cluster-fabric
**The unified supercomputer** — combines every peer on a topic into **one logical machine** with summed CPU, RAM, GPU, disk, and bandwidth.
**Category:** Supercomputer · **Protocol:** `cluster-fabric/v1` · **Export:** `HyperP2PClusterFabric`
## When to use
You want the mesh to behave as a **single giant computer**, not isolated pools. Call `getVirtualMachine()` for total capacity, `reserveCluster()` to allocate across nodes, and `runClusterJob()` to execute on the combined fabric.
## Quick start
```js
const { HyperP2PClusterFabric } = require('hyper-p2p-cluster-fabric')
const { HyperP2PCapacityRegistry } = require('hyper-p2p-capacity-registry')
const { HyperP2PCpuShare } = require('hyper-p2p-cpu-share')
const { HyperP2PJobDispatcher } = require('hyper-p2p-job-dispatcher')
const fabric = new HyperP2PClusterFabric({
topic: 'my-supercomputer',
registry: new HyperP2PCapacityRegistry({ topic: 'my-supercomputer' }),
cpu: new HyperP2PCpuShare({ topic: 'my-supercomputer' }),
jobs: new HyperP2PJobDispatcher({ topic: 'my-supercomputer' })
})
await fabric.ready()
fabric.publishNode({ cpuCores: 8, ramMb: 16384, gpuSlots: 1, cpuMsDonated: 60_000 })
const vm = fabric.getVirtualMachine()
console.log(vm.total.cpuCores, 'cores across', vm.nodes, 'nodes')
fabric.runClusterJob({ kind: 'train', cpuMs: 100_000, ramMb: 8192 })
```
## Key APIs
| Method | Purpose |
|--------|---------|
| `publishNode(resources)` | Join the giant computer; syncs attached pools |
| `getVirtualMachine()` | `{ total, available, nodes, asOneMachine: true }` |
| `clusterTotals()` | Summed capacity of all nodes |
| `availableTotals()` | Free capacity after reservations |
| `reserveCluster(spec)` | Multi-peer greedy allocation |
| `runClusterJob(spec)` | Reserve + fan-out jobs to slices |
## Docs
- [docs/api.md](docs/api.md)
- [docs/architecture.md](docs/architecture.md)
## Test
```bash
npm install && npm test
```
@@ -0,0 +1,52 @@
# API: hyper-p2p-cluster-fabric
**Protocol:** `cluster-fabric/v1` · **Export:** `HyperP2PClusterFabric`
## Overview
Top-level **virtual supercomputer** facade. Every peer that `publishNode()` becomes part of one machine identified by `clusterId`. Workloads draw from **combined** CPU/RAM/GPU/disk across the mesh.
## Constructor
```js
const fabric = new HyperP2PClusterFabric({
topic,
clusterId, // optional; derived from topic hash
registry, // optional HyperP2PCapacityRegistry
cpu, ram, gpu, jobs, thermal, affinity, bandwidth, disk // optional attached modules
})
```
## Methods
### `publishNode(resources, peerId?) → nodeRecord`
Registers a node on the fabric; forwards to attached modules (`advertise`, `donate`, `lend`, `offer`, etc.).
### `getVirtualMachine() → { clusterId, name, nodes, total, available, asOneMachine }`
Single view of the **entire** mesh as one computer.
### `clusterTotals()` / `availableTotals()`
Aggregated capacity and remaining free resources.
### `reserveCluster({ cpuMs, ramMb, gpuSlots, diskGb }) → reservation`
Greedy placement across peers; returns `{ reservationId, slices: [{ peerId, cpuMs, ramMb, ... }] }`.
### `releaseReservation(reservationId) → boolean`
### `runClusterJob(spec) → { reservationId, slices, jobIds }`
Calls `reserveCluster` then `jobs.submitJob` per slice when `jobs` is attached.
### `listNodes()` / `mergeCluster(remote)` / `getStats()` / `ready()` / `close()`
## Events
`node`, `reserve`, `release`, `run`, `merge`, `closed`
## Composition
Sits above all other `supercomputer/*` modules. See [`../../_shared/SUPERCOMPUTER_LAYERS.md`](../../_shared/SUPERCOMPUTER_LAYERS.md).
@@ -0,0 +1,29 @@
# Architecture: hyper-p2p-cluster-fabric
```text
┌─────────────────────────────┐
│ HyperP2PClusterFabric │
│ (one virtual machine) │
└──────────────┬──────────────┘
┌──────────┼──────────┬──────────┬──────────┐
▼ ▼ ▼ ▼ ▼
capacity cpu-share ram-pool gpu-slot job-dispatcher
registry ...
└──────────┴──────────┴──────────┴──────────┘
Hyperswarm topic
```
## Wire messages
| type | Purpose |
|------|---------|
| `cluster-node` | Gossip node capacity into fabric view |
| `cluster-reserve` | Shared reservation across mesh |
| `cluster-release` | Release reservation |
| `cluster-run` | Announced cluster job fan-out |
## State
- `_nodes` — per-peer contributed capacity + usage
- `_reservations` — active multi-peer allocations
- `_used` — cluster-wide reserved totals
@@ -0,0 +1,11 @@
require('bare-process/global')
const { HyperP2PClusterFabric } = require('../index.js')
async function main () {
const fabric = new HyperP2PClusterFabric()
fabric.publishNode({ cpuCores: 8, ramMb: 16384, gpuSlots: 1, cpuMsDonated: 50_000 })
console.log('virtual machine', fabric.getVirtualMachine())
await fabric.close()
console.log('done')
}
main().catch(console.error)
@@ -0,0 +1,347 @@
require('bare-process/global')
const EventEmitter = require('bare-events')
const b4a = require('b4a')
const { initModuleSwarm, gossipSend } = require('../../_shared/p2p-bare.js')
const PROTOCOL = 'cluster-fabric/v1'
/**
* Unifies every peer on a topic into one logical supercomputer.
* Aggregates CPU, RAM, GPU, disk, and bandwidth; reserves and runs workloads
* across multiple nodes as a single cluster allocation.
*/
class HyperP2PClusterFabric extends EventEmitter {
constructor (opts = {}) {
super()
this._stats = { publishes: 0, reservations: 0, jobs: 0 }
this.topic = opts.topic || null
this.keyPair = opts.keyPair || require('hypercore-crypto').keyPair()
this.peerId = b4a.toString(this.keyPair.publicKey, 'hex')
this.clusterId = opts.clusterId || b4a.toString(
require('hypercore-crypto').hash(b4a.from(String(this.topic || 'local-cluster'))),
'hex'
).slice(0, 16)
this.registry = opts.registry || null
this.cpu = opts.cpu || null
this.ram = opts.ram || null
this.gpu = opts.gpu || null
this.jobs = opts.jobs || null
this.thermal = opts.thermal || null
this.affinity = opts.affinity || null
this.bandwidth = opts.bandwidth || null
this.disk = opts.disk || null
this._nodes = new Map()
this._reservations = new Map()
this._used = { cpuMs: 0, ramMb: 0, gpuSlots: 0, diskGb: 0 }
this.swarm = null
this._peerMsgs = null
}
_pid (peerId) {
return typeof peerId === 'string' ? peerId : b4a.toString(peerId, 'hex')
}
_nodeRecord (peerId, resources = {}) {
return {
peerId: this._pid(peerId),
cpuCores: Math.max(0, resources.cpuCores ?? 0),
ramMb: Math.max(0, resources.ramMb ?? 0),
diskGb: Math.max(0, resources.diskGb ?? 0),
gpuSlots: Math.max(0, resources.gpuSlots ?? 0),
upMbps: Math.max(0, resources.upMbps ?? 0),
downMbps: Math.max(0, resources.downMbps ?? 0),
cpuMsDonated: Math.max(0, resources.cpuMsDonated ?? 0),
cpuMsUsed: Math.max(0, resources.cpuMsUsed ?? 0),
ramMbLent: Math.max(0, resources.ramMbLent ?? 0),
ramMbUsed: Math.max(0, resources.ramMbUsed ?? 0),
tags: resources.tags || [],
at: Date.now()
}
}
/**
* Join the mesh as one node of the giant computer — advertises capacity
* and optionally syncs into attached pool modules.
*/
publishNode (resources = {}, peerId = null) {
const rec = this._nodeRecord(peerId || this.peerId, resources)
this._nodes.set(rec.peerId, rec)
this._stats.publishes++
if (this.registry) this.registry.advertise(resources, rec.peerId)
if (this.cpu && resources.cpuMsDonated) this.cpu.donate(rec.peerId, resources.cpuMsDonated)
if (this.ram && resources.ramMbLent) this.ram.lend(rec.peerId, resources.ramMbLent * 1024 * 1024)
if (this.gpu && resources.gpuSlots) {
for (let i = 0; i < resources.gpuSlots; i++) {
this.gpu.registerSlot(`${rec.peerId}-gpu${i}`, { vramMb: resources.vramMb ?? 8192, tags: resources.tags })
}
}
if (this.bandwidth && (resources.upMbps || resources.downMbps)) {
this.bandwidth.offer(rec.peerId, resources.upMbps ?? 0, resources.downMbps ?? 0)
}
if (this.affinity) this.affinity.tagPeer(rec.peerId, resources.tags || [], resources.latencyMs ?? null)
if (this._peerMsgs) gossipSend(this, { type: 'cluster-node', clusterId: this.clusterId, node: rec })
this.emit('node', rec)
return rec
}
_syncFromRegistry () {
if (!this.registry) return
for (const rec of this.registry.listPeers()) {
const cur = this._nodes.get(rec.peerId)
if (!cur || rec.at > cur.at) {
this._nodes.set(rec.peerId, this._nodeRecord(rec.peerId, {
cpuCores: rec.cpuCores,
ramMb: rec.ramMb,
diskGb: rec.diskGb,
gpuSlots: rec.gpuSlots,
upMbps: rec.upMbps,
downMbps: rec.downMbps,
tags: rec.tags,
at: rec.at
}))
}
}
}
/**
* One virtual machine: sum of every node's contributed resources.
*/
clusterTotals () {
this._syncFromRegistry()
const t = {
clusterId: this.clusterId,
nodes: 0,
cpuCores: 0,
ramMb: 0,
diskGb: 0,
gpuSlots: 0,
upMbps: 0,
downMbps: 0,
cpuMsPool: 0,
cpuMsUsed: 0,
ramMbPool: 0,
ramMbUsed: 0
}
for (const n of this._nodes.values()) {
if (this.thermal && this.thermal.shouldThrottle(n.peerId)) continue
t.nodes++
t.cpuCores += n.cpuCores
t.ramMb += n.ramMb
t.diskGb += n.diskGb
t.gpuSlots += n.gpuSlots
t.upMbps += n.upMbps
t.downMbps += n.downMbps
t.cpuMsPool += n.cpuMsDonated
t.cpuMsUsed += n.cpuMsUsed
t.ramMbPool += n.ramMbLent
t.ramMbUsed += n.ramMbUsed
}
if (this.cpu) {
let pool = 0
for (const id of this.cpu.listPeerIds()) pool += this.cpu.balance(id)
t.cpuMsPool = Math.max(t.cpuMsPool, pool)
}
return t
}
/**
* What the unified supercomputer still has free (cluster minus reservations).
*/
availableTotals () {
const t = this.clusterTotals()
return {
...t,
cpuCores: Math.max(0, t.cpuCores - Math.ceil(this._used.cpuMs / 1000)),
ramMb: Math.max(0, t.ramMb - this._used.ramMb),
gpuSlots: Math.max(0, t.gpuSlots - this._used.gpuSlots),
diskGb: Math.max(0, t.diskGb - this._used.diskGb),
cpuMs: Math.max(0, t.cpuMsPool - t.cpuMsUsed - this._used.cpuMs),
ramMbFree: Math.max(0, t.ramMbPool - t.ramMbUsed - this._used.ramMb)
}
}
/**
* Greedy multi-peer reservation — one workload drawn from the combined cluster.
*/
reserveCluster (spec = {}) {
const need = {
cpuMs: Math.max(0, spec.cpuMs ?? 0),
ramMb: Math.max(0, spec.ramMb ?? 0),
gpuSlots: Math.max(0, spec.gpuSlots ?? 0),
diskGb: Math.max(0, spec.diskGb ?? 0)
}
const avail = this.availableTotals()
if (need.cpuMs > avail.cpuMs) throw new Error('cluster insufficient cpuMs')
if (need.ramMb > avail.ramMb && need.ramMb > avail.ramMbFree) throw new Error('cluster insufficient ramMb')
if (need.gpuSlots > avail.gpuSlots) throw new Error('cluster insufficient gpuSlots')
if (need.diskGb > avail.diskGb) throw new Error('cluster insufficient diskGb')
const ranked = [...this._nodes.values()]
.filter((n) => !this.thermal || !this.thermal.shouldThrottle(n.peerId))
.sort((a, b) => (b.cpuCores + b.ramMb / 512) - (a.cpuCores + a.ramMb / 512))
const slices = []
let left = { ...need }
for (const n of ranked) {
if (left.cpuMs <= 0 && left.ramMb <= 0 && left.gpuSlots <= 0 && left.diskGb <= 0) break
const slice = {
peerId: n.peerId,
cpuMs: Math.min(left.cpuMs, n.cpuMsDonated || n.cpuCores * 60_000),
ramMb: Math.min(left.ramMb, n.ramMb - n.ramMbUsed),
gpuSlots: Math.min(left.gpuSlots, n.gpuSlots),
diskGb: Math.min(left.diskGb, n.diskGb)
}
if (slice.cpuMs <= 0 && slice.ramMb <= 0 && slice.gpuSlots <= 0 && slice.diskGb <= 0) continue
if (this.cpu && slice.cpuMs) this.cpu.consume(slice.peerId, slice.cpuMs)
left.cpuMs -= slice.cpuMs
left.ramMb -= slice.ramMb
left.gpuSlots -= slice.gpuSlots
left.diskGb -= slice.diskGb
n.cpuMsUsed += slice.cpuMs
n.ramMbUsed += slice.ramMb
slices.push(slice)
}
if (left.cpuMs > 0 || left.ramMb > 0 || left.gpuSlots > 0 || left.diskGb > 0) {
throw new Error('could not place reservation across cluster')
}
const reservationId = b4a.toString(
require('hypercore-crypto').hash(b4a.from(JSON.stringify(need) + Date.now())),
'hex'
).slice(0, 16)
const reservation = { reservationId, clusterId: this.clusterId, spec: need, slices, at: Date.now() }
this._reservations.set(reservationId, reservation)
this._used.cpuMs += need.cpuMs
this._used.ramMb += need.ramMb
this._used.gpuSlots += need.gpuSlots
this._used.diskGb += need.diskGb
this._stats.reservations++
if (this._peerMsgs) gossipSend(this, { type: 'cluster-reserve', reservation })
this.emit('reserve', reservation)
return reservation
}
releaseReservation (reservationId) {
const r = this._reservations.get(reservationId)
if (!r) return false
for (const s of r.slices) {
const n = this._nodes.get(s.peerId)
if (n) {
n.cpuMsUsed = Math.max(0, n.cpuMsUsed - s.cpuMs)
n.ramMbUsed = Math.max(0, n.ramMbUsed - s.ramMb)
}
}
this._used.cpuMs = Math.max(0, this._used.cpuMs - r.spec.cpuMs)
this._used.ramMb = Math.max(0, this._used.ramMb - r.spec.ramMb)
this._used.gpuSlots = Math.max(0, this._used.gpuSlots - r.spec.gpuSlots)
this._used.diskGb = Math.max(0, this._used.diskGb - r.spec.diskGb)
this._reservations.delete(reservationId)
if (this._peerMsgs) gossipSend(this, { type: 'cluster-release', reservationId })
this.emit('release', { reservationId })
return true
}
/**
* Run one logical job on the combined supercomputer (may fan out to N peers).
*/
runClusterJob (spec = {}) {
const reservation = this.reserveCluster(spec)
const jobIds = []
if (this.jobs) {
for (const slice of reservation.slices) {
const job = this.jobs.submitJob({
...spec,
kind: spec.kind || 'cluster',
targetPeer: slice.peerId,
cpuMs: slice.cpuMs,
ramMb: slice.ramMb,
gpuSlots: slice.gpuSlots,
clusterId: this.clusterId,
reservationId: reservation.reservationId
})
jobIds.push(job.jobId)
}
}
this._stats.jobs++
const run = { reservationId: reservation.reservationId, clusterId: this.clusterId, slices: reservation.slices, jobIds }
if (this._peerMsgs) gossipSend(this, { type: 'cluster-run', run })
this.emit('run', run)
return run
}
getVirtualMachine () {
const t = this.clusterTotals()
const a = this.availableTotals()
return {
clusterId: this.clusterId,
name: `HyperP2P-Cluster-${this.clusterId}`,
nodes: t.nodes,
total: t,
available: a,
asOneMachine: true
}
}
listNodes () { return [...this._nodes.values()] }
mergeCluster (remote) {
if (!remote || !remote.nodes) return 0
let n = 0
for (const node of remote.nodes) {
const cur = this._nodes.get(node.peerId)
if (!cur || node.at > cur.at) {
this._nodes.set(node.peerId, node)
n++
}
}
if (n) this.emit('merge', { updated: n })
return n
}
async ready () {
if (this.swarm || !this.topic) return this
await initModuleSwarm(this, {
keyPair: this.keyPair,
topic: this.topic,
protocol: PROTOCOL,
onmessage: (data) => {
if (data?.type === 'cluster-node' && data.node) {
if (data.clusterId === this.clusterId) this._nodes.set(data.node.peerId, data.node)
} else if (data?.type === 'cluster-reserve' && data.reservation) {
this._reservations.set(data.reservation.reservationId, data.reservation)
} else if (data?.type === 'cluster-release') {
this._reservations.delete(data.reservationId)
}
}
})
return this
}
getStats () {
const t = this.clusterTotals()
return {
...this._stats,
clusterId: this.clusterId,
nodes: t.nodes,
reservations: this._reservations.size,
protocol: PROTOCOL
}
}
async close () {
if (this.swarm) await this.swarm.destroy().catch(() => {})
this.swarm = null
this._nodes.clear()
this._reservations.clear()
this.emit('closed')
}
}
module.exports = { HyperP2PClusterFabric, PROTOCOL }
File diff suppressed because it is too large Load Diff
@@ -0,0 +1,23 @@
{
"name": "hyper-p2p-cluster-fabric",
"version": "0.3.0",
"description": "Unified virtual supercomputer registry for Bare/Pear P2P supercomputer mesh.",
"main": "index.js",
"type": "commonjs",
"license": "Apache-2.0",
"scripts": { "test": "brittle-bare test/test.js" },
"dependencies": {
"bare-events": "^2.8.0",
"bare-process": "^4.4.0",
"b4a": "^1.6.7",
"hypercore-crypto": "^3.0.0",
"protomux": "^3.0.0",
"compact-encoding": "^2.0.0"
},
"peerDependencies": { "hyperswarm": "^4.0.0", "bare": ">=1.0.0" },
"devDependencies": { "brittle": "^3.0.0" },
"imports": {
"process": { "bare": "bare-process", "default": "process" },
"events": { "bare": "bare-events", "default": "events" }
}
}
@@ -0,0 +1,52 @@
require('bare-process/global')
const test = require('brittle')
const { HyperP2PClusterFabric } = require('../index.js')
const { HyperP2PCapacityRegistry } = require('../../hyper-p2p-capacity-registry/index.js')
const { HyperP2PCpuShare } = require('../../hyper-p2p-cpu-share/index.js')
const { HyperP2PJobDispatcher } = require('../../hyper-p2p-job-dispatcher/index.js')
test('cluster-fabric: combines nodes into one VM', async (t) => {
const reg = new HyperP2PCapacityRegistry()
const cpu = new HyperP2PCpuShare()
const jobs = new HyperP2PJobDispatcher()
const fabric = new HyperP2PClusterFabric({ registry: reg, cpu, jobs })
fabric.publishNode({ cpuCores: 8, ramMb: 16384, gpuSlots: 1, cpuMsDonated: 60_000 }, 'node-a')
fabric.publishNode({ cpuCores: 16, ramMb: 32768, gpuSlots: 2, cpuMsDonated: 120_000 }, 'node-b')
const vm = fabric.getVirtualMachine()
t.is(vm.asOneMachine, true)
t.is(vm.nodes, 2)
t.is(vm.total.cpuCores, 24)
t.is(vm.total.ramMb, 49152)
t.is(vm.total.gpuSlots, 3)
const run = fabric.runClusterJob({ kind: 'render', cpuMs: 5000, ramMb: 4096 })
t.ok(run.jobIds.length >= 1)
t.ok(run.slices.length >= 1)
await fabric.close()
await reg.close()
await cpu.close()
await jobs.close()
})
test('cluster-fabric: clusterTotals standalone', async (t) => {
const f = new HyperP2PClusterFabric()
f.publishNode({ cpuCores: 4, ramMb: 8192 })
f.publishNode({ cpuCores: 4, ramMb: 8192 }, 'peer-2')
t.is(f.clusterTotals().cpuCores, 8)
await f.close()
})
test('cluster-fabric: validation', async (t) => {
const f = new HyperP2PClusterFabric()
f.publishNode({ cpuCores: 1, ramMb: 512, cpuMsDonated: 100 })
try {
f.reserveCluster({ cpuMs: 999_999_999 })
t.fail('expected throw')
} catch (e) {
t.ok(e instanceof Error)
}
await f.close()
})