ravenclaw/research
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
Files
5d8f3ff8e718191039d90f0dd8ccf19aa7f45394
research/architecture
T
History
Raven Scott 62e2227499 update
2026-02-20 01:28:45 -05:00
..
bare-network-stack.md
Update
2026-02-19 19:57:17 -05:00
hyper-stack-overview.md
Update
2026-02-19 19:57:17 -05:00
README.md
update
2026-02-20 01:28:45 -05:00

README.md

Complete Holepunch Architecture Stack

Overview

The Holepunch ecosystem is a comprehensive peer-to-peer (P2P) application stack spanning 554+ repositories. It provides everything needed to build, distribute, and run distributed applications across desktop and mobile platforms.

flowchart TB
    subgraph "Application Layer"
        APP["📱 Applications"]
        KEET["Keet"]
        SHELL["Hypershell"]
        PASS["Autopass"]
        PEARAPP["Pear Apps"]
    end
    
    subgraph "Platform Layer (Pear)"
        PEAR["🍐 Pear Runtime"]
        SIDE["Sidecar"]
        IPC["IPC/Bridge"]
        PACK["Pack/Bundle"]
        UPD["Updater"]
    end
    
    subgraph "Runtime Layer (Bare)"
        BARE["⚡ Bare JS Runtime"]
        BMOD["Module System"]
        BADD["Native Addons"]
        BBUN["Bundling"]
        BT["Threading"]
    end
    
    subgraph "Data Layer (Hypercore)"
        CORE["📦 Hypercore"]
        BEE["🌳 Hyperbee"]
        DRIVE["📁 Hyperdrive"]
        AUTO["🔗 Autobase"]
        BLOB["Hyperblobs"]
    end
    
    subgraph "Network Layer (Hyperswarm)"
        SWARM["🌐 Hyperswarm"]
        DHT["🔍 HyperDHT"]
        SEC["Secret Streams"]
        RELAY["DHT Relay"]
    end
    
    subgraph "Protocol Layer"
        PMUX["Protomux"]
        RPC["HRPC"]
        ENC["Compact Encoding"]
    end
    
    subgraph "Native Layer"
        LIBJS["libjs (JS Engine)"]
        LIBUDX["libudx (UDP)"]
        SODIUM["sodium-native"]
        ROCKS["rocksdb-native"]
    end
    
    APP --> PEAR
    KEET --> PEAR
    SHELL --> PEAR
    PASS --> PEAR
    PEARAPP --> PEAR
    
    PEAR --> BARE
    PEAR --> SIDE
    PEAR --> IPC
    PEAR --> PACK
    PEAR --> UPD
    
    BARE --> CORE
    BARE --> SWARM
    BARE --> BMOD
    BARE --> BADD
    
    CORE --> BEE
    CORE --> DRIVE
    CORE --> AUTO
    DRIVE --> BLOB
    
    SWARM --> DHT
    SWARM --> SEC
    SWARM --> RELAY
    
    DHT --> PMUX
    SEC --> PMUX
    PMUX --> RPC
    PMUX --> ENC
    
    BARE --> LIBJS
    DHT --> LIBUDX
    SEC --> SODIUM
    CORE --> ROCKS

1. Application Layer (Pear Platform)

1.1 Pear Runtime

The Pear platform is the flagship application runtime for building and distributing P2P applications.

Core Components:

Module Purpose Location
pear Main platform (CLI, sidecar, subsystems) Main repo
pear-api Runtime API interface API layer
pear-cli Command-line interface CLI layer
pear-cmd Command implementations Commands
pear-sidecar Background service Sidecar
pear-bridge IPC bridge IPC
pear-ipc Inter-process communication IPC
pear-desktop Desktop UI runtime UI
pear-terminal Terminal UI runtime UI
pear-electron Electron integration UI

Build & Distribution:

Module Purpose
pear-bundle Application bundling
pear-pack Package creation
pear-build Build system
pear-stage Staging for release
pear-release Release management
pear-updater Update system
pear-appling Application packaging

Storage & Data:

Module Purpose
pear-hyperdb Database integration
pear-appdrive Application storage
pear-seed Seeding utilities

Utilities:

Module Purpose
pear-inspect Debugging/inspection
pear-doctor Diagnostics
pear-info System information
pear-logger Logging
pear-init Project scaffolding
pear-templates Starter templates

2. Runtime Layer (Bare)

2.1 Bare JavaScript Runtime

Bare is a minimal, cross-platform JavaScript runtime designed for embedding and P2P applications.

flowchart LR
    subgraph "Bare Architecture"
        JS["JavaScript Code"]
        MOD["Module System<br/>bare-module*"]
        API["Core APIs"]
        ADD["Native Addons<br/>bare-addon*"]
        BUN["Bundler<br/>bare-bundle*"]
    end
    
    subgraph "Platform Support"
        IOS["iOS<br/>bare-ios"]
        AND["Android<br/>bare-android"]
        DESK["Desktop<br/>bare-kit"]
        WEB["WebKit<br/>bare-web-kit"]
    end
    
    JS --> MOD
    MOD --> API
    MOD --> ADD
    API --> BUN
    ADD --> IOS
    ADD --> AND
    ADD --> DESK
    ADD --> WEB

Core I/O Modules:

Module Purpose Node.js Equivalent
bare-fs File system fs
bare-path Path utilities path
bare-os OS interfaces os
bare-stream Streams stream
bare-buffer Buffers buffer
bare-events Event emitters events
bare-timers Timers timers

Network Modules:

Module Purpose Protocol
bare-tcp TCP sockets TCP
bare-udp UDP sockets UDP
bare-tls TLS/SSL TLS 1.3
bare-http1 HTTP/1.1 HTTP
bare-https HTTPS HTTPS
bare-fetch Fetch API HTTP/HTTPS
bare-ws WebSocket WS/WSS

Module System:

Module Purpose
bare-module Module loading
bare-module-resolve Module resolution
bare-module-lexer Module parsing
bare-module-traverse Module traversal

Bundling & Distribution:

Module Purpose
bare-bundle Code bundling
bare-pack Package creation
bare-unpack Package extraction
bare-bundle-compile Compile bundles
bare-bundle-evaluate Execute bundles

Concurrency:

Module Purpose
bare-thread Worker threads
bare-worker Web Workers
bare-channel Inter-thread communication
bare-atomics Atomic operations

Platform Bindings:

Module Platform Purpose
bare-ios iOS iOS runtime
bare-android Android Android runtime
bare-kit Mobile/Desktop Cross-platform toolkit
bare-kit-swift iOS Swift bindings
bare-ndk Android NDK support
bare-apk Android APK packaging
bare-win-ui Windows Windows UI
bare-gtk Linux GTK bindings
bare-sdl Cross-platform SDL bindings

3. Data Layer (Hypercore Ecosystem)

3.1 Core Data Structures

flowchart TB
    subgraph "Data Layer"
        CORE["Hypercore<br/>Append-only Log"]
        CS["Corestore<br/>Multi-core Management"]
        
        subgraph "Built on Core"
            BEE["Hyperbee<br/>B-tree Index"]
            DRIVE["Hyperdrive<br/>Filesystem"]
            AUTO["Autobase<br/>Multi-writer"]
        end
        
        subgraph "Supporting"
            BLOB["Hyperblobs"]
            ENC["Encryption"]
            CRYPTO["Crypto"]
        end
    end
    
    CS --> CORE
    CORE --> BEE
    CORE --> DRIVE
    CORE --> AUTO
    DRIVE --> BLOB
    CORE --> ENC
    CORE --> CRYPTO

Hypercore - Distributed append-only log:

Feature Implementation
Structure Merkle tree with signed roots
Replication Bitfield-based peer sync
Storage RocksDB backend
Encryption Per-block encryption
Signatures Ed25519

Extensions:

Module Purpose
hypercore-crypto Cryptographic operations
hypercore-encryption Block encryption
hypercore-storage Storage backends
hypercore-byte-stream Streaming access
hypercore-messages Message protocols
hypercore-stats Metrics
hypercore-id-encoding Key encoding
hypercore-sign Signing utilities
hypercore-blob-server Blob serving
hypercore-audit Audit trails

Hyperbee - B-tree on Hypercore:

Feature Description
Structure Sorted key-value store
Index B-tree over Hypercore
Diffing hyperbee-diff-stream
Use case Database indices, file trees

Hyperdrive - Distributed filesystem:

Feature Implementation
Structure Hyperbee (metadata) + Hyperblobs (content)
Blobs Separate blob storage
Versioning Complete history
Diffing Drive-to-drive comparisons

Extensions:

Module Purpose
hyperdrive-next Next-gen features
hyperdrive-profiler Performance profiling
hyperdrive-swarm-test Network testing
hyperblobs Blob storage

Autobase - Multi-writer data structure:

Feature Description
Ordering Causal (Lamport timestamps)
Merge Automatic view merging
Use case Collaborative editing, CRDTs

Extensions:

Module Purpose
autobase-discovery Peer discovery
autobase-light-writer Lightweight writers
autobase-example Example implementations

Corestore - Multi-core management:

Feature Description
Storage Deduplicated core storage
Namespacing Key namespaces
Efficiency Batch operations

4. Network Layer (Hyperswarm)

4.1 P2P Networking Stack

flowchart TB
    subgraph "Networking Stack"
        SWARM["Hyperswarm<br/>High-level API"]
        DHT["HyperDHT<br/>Distributed Hash Table"]
        RPC["DHT-RPC<br/>Remote Procedure Call"]
        
        subgraph "Connection"
            SEC["Secret Stream<br/>Encryption"]
            CAP["Capability<br/>Auth"]
            RELAY["DHT Relay<br/>NAT Traversal"]
        end
        
        subgraph "Transport"
            UDX["libudx<br/>UDP Transport"]
            TCP["TCP Fallback"]
        end
    end
    
    SWARM --> DHT
    DHT --> RPC
    SWARM --> SEC
    SWARM --> CAP
    SWARM --> RELAY
    DHT --> UDX
    DHT --> TCP

Hyperswarm - High-level P2P API:

Feature Description
Discovery Topic-based peer discovery
Hole punching NAT traversal
Encryption Noise protocol
Pooling Connection management

Extensions:

Module Purpose
hyperswarm-secret-stream Encrypted streams
hyperswarm-dht-relay Relay servers
hyperswarm-capability Capability auth
hyperswarm-seeders Seeder infrastructure
hyperswarm-testnet Test networks
hyperswarm-stats Metrics
hyperswarm-doctor Diagnostics

HyperDHT - Distributed Hash Table:

Feature Description
Structure Kademlia DHT
Keys Ed25519 public keys
Persistence Persistent node IDs
Queries Find peers by key

Extensions:

Module Purpose
hyperdht-stats DHT metrics
http-dht-proxy HTTP proxy
http-forward-host Host forwarding

5. Protocol Layer

5.1 Protocol Multiplexing

flowchart LR
    STREAM["Encrypted Stream"]
    MUX["Protomux<br/>Multiplexer"]
    
    subgraph "Channels"
        RPC["HRPC<br/>RPC"]
        WAKE["Wakeup"]
        CUSTOM["Custom Protocols"]
    end
    
    STREAM --> MUX
    MUX --> RPC
    MUX --> WAKE
    MUX --> CUSTOM

Protomux - Protocol multiplexing:

Feature Description
Multiplexing Multiple protocols over single stream
Framing Length-prefixed messages
Backpressure Flow control

Extensions:

Module Purpose
protomux-rpc RPC framework
protomux-rpc-client RPC client
protomux-rpc-client-pool Connection pooling
protomux-rpc-middleware Middleware support
protomux-rpc-router Request routing
protomux-wakeup Wake protocol

Compact Encoding - Serialization:

Feature Description
Format Compact binary
Schema Type definitions
Cross-language Multiple bindings

Extensions:

Module Purpose
compact-encoding-bitfield Bitfield encoding
compact-encoding-net Network types
compact-encoding-struct Struct encoding
compact-encoding-variant Variant types
compact-encoding-golang Go bindings
compact-encoding-swift Swift bindings

6. Native Layer

6.1 C/C++ Libraries

flowchart TB
    subgraph "Native Libraries"
        LIBJS["libjs<br/>JS Engine Abstraction"]
        LIBUDX["libudx<br/>UDP Transport"]
        LIBRPC["librpc<br/>RPC"]
        
        subgraph "Crypto"
            SODIUM["sodium-native<br/>NaCl"]
            NOISE["noise-handshake"]
        end
        
        subgraph "Storage"
            ROCKS["rocksdb-native<br/>RocksDB"]
            MEM["libmem"]
        end
        
        subgraph "System"
            FS["libfs"]
            PATH["libpath"]
            LOG["liblog"]
        end
    end
    
    LIBJS --> BARE
    LIBUDX --> DHT
    LIBRPC --> RPC
    SODIUM --> SEC
    ROCKS --> CORE

Core Libraries:

Library Purpose Language
libjs JS engine abstraction C
libnapi Native addon interface C
libjstl JS template library C++
libudx UDP data transfer C
librpc RPC communication C
libmem Memory utilities C
liblog Logging C
libpath Path utilities C
libfs File system C
libhex Hex encoding C
liburl URL parsing C
libutf UTF utilities C
libtls TLS/SSL C
libz32 Base32 encoding C
libbitarray Bit arrays C
libquickbit Bit operations C
libsimdle SIMD utilities C
libcompact Compact encoding C
libjson JSON parsing C
libcrc CRC algorithms C
librabin Rabin fingerprinting C
librocksdb RocksDB bindings C
libdaemon Daemon utilities C
libdynload Dynamic loading C
libprng PRNG C

7. Data Flow Examples

7.1 Application Startup Flow

sequenceDiagram
    participant User
    participant App as Application
    participant Pear as Pear Runtime
    participant Bare as Bare JS
    participant Swarm as Hyperswarm
    participant DHT as HyperDHT
    participant Core as Hypercore
    
    User->>App: Launch app
    App->>Pear: pear.run()
    Pear->>Bare: Initialize runtime
    Bare->>Core: Load hypercores
    Core->>Core: Verify integrity
    App->>Swarm: Join topics
    Swarm->>DHT: Announce presence
    DHT-->>Swarm: Peer candidates
    Swarm->>Swarm: Hole punch
    Swarm-->>App: Connected peers
    App-->>User: Ready

7.2 Data Replication Flow

sequenceDiagram
    participant A as Peer A
    participant Swarm as Hyperswarm
    participant B as Peer B
    participant CoreA as Hypercore A
    participant CoreB as Hypercore B
    
    A->>Swarm: Join discoveryKey
    B->>Swarm: Join discoveryKey
    Swarm->>A: Found peer B
    Swarm->>B: Found peer A
    A->>B: Noise handshake
    B->>A: Noise handshake
    A->>CoreA: Get bitfield
    CoreA-->>A: Bitfield
    A->>B: Send bitfield
    B->>CoreB: Compare bitfields
    CoreB-->>B: Need blocks X-Y
    B->>A: Request blocks X-Y
    A->>CoreA: Read blocks X-Y
    CoreA-->>A: Block data
    A->>B: Send blocks
    B->>CoreB: Write blocks

7.3 Multi-writer Autobase Flow

sequenceDiagram
    participant W1 as Writer 1
    participant W2 as Writer 2
    participant W3 as Writer 3
    participant Auto as Autobase
    participant View as Derived View
    
    W1->>Auto: Write entry A
    Auto->>Auto: Assign LSN=1
    W2->>Auto: Write entry B
    Auto->>Auto: Assign LSN=2
    W3->>Auto: Write entry C
    Auto->>Auto: Assign LSN=3
    
    Auto->>Auto: Causal ordering
    Auto->>View: Apply in order
    View->>View: A → B → C
    View-->>W1: Updated view
    View-->>W2: Updated view
    View-->>W3: Updated view

8. Build System Architecture

8.1 CMake Toolchain

flowchart TB
    subgraph "CMake Build System"
        CMAKE["CMake Configuration"]
        
        subgraph "Platform Modules"
            IOS["cmake-ios"]
            MAC["cmake-macos"]
            WIN["cmake-windows"]
            AND["cmake-android"]
        end
        
        subgraph "Build Tools"
            BARE["cmake-bare"]
            PEAR["cmake-pear"]
            NAPI["cmake-napi"]
        end
        
        subgraph "Distribution"
            IMG["cmake-app-image"]
            MSIX["cmake-msix"]
        end
    end
    
    CMAKE --> IOS
    CMAKE --> MAC
    CMAKE --> WIN
    CMAKE --> AND
    CMAKE --> BARE
    CMAKE --> PEAR
    CMAKE --> NAPI
    CMAKE --> IMG
    CMAKE --> MSIX

9. Application Examples

9.1 Real-world Applications

Application Stack Used Description
Keet Full stack P2P chat, video & audio calls - no sign-up, seed-phrase identity
PearPass Hyperdrive P2P password & secrets manager
Hypershell Hyperswarm P2P shell/SSH alternative
Autopass Hyperdrive P2P password manager
Simple Seeder Hyperdrive File seeding utility
Hyperbeam Hyperswarm P2P data transfer
Hyperssh Hyperswarm SSH over Hyperswarm

9.2 CLI Tools

Tool Description
Hypershell CLI for P2P E2E encrypted shells
Hypertele Swiss-army knife proxy powered by HyperDHT
Hyperbeam One-to-one E2E encrypted internet pipe
Hyperssh Run SSH over the DHT
Drives CLI for Hyperdrive & Localdrive operations

9.3 Example Projects

Project Technologies Purpose
filesharing-app-example Hyperdrive + Hyperswarm File sharing
filesharing-react-app-example React + Hyperdrive React file sharing
autobase-example Autobase Multi-writer demo
autopass-mobile-example React Native Mobile password manager
pear-expo-hello-world Expo + Pear Mobile Pear app
hrpc-modular-example HRPC Modular RPC demo

10. Key Design Principles

10.1 Architecture Principles

  1. Modularity: Each component is independently usable
  2. Composability: Components stack and compose cleanly
  3. P2P-First: Designed for peer-to-peer from the ground up
  4. Offline-First: Works without central servers
  5. End-to-End Encryption: Security built-in, not bolted-on
  6. Cross-Platform: Desktop, mobile, and embedded support
  7. Native Performance: Critical paths in native code
  8. JavaScript Ergonomics: High-level APIs in JavaScript

10.2 Data Principles

  1. Immutable Logs: Hypercore append-only for integrity
  2. Merkle Trees: Cryptographic verification
  3. Bitfield Sync: Efficient replication
  4. Deduplication: Corestore shares storage
  5. Encryption: Per-block encryption support

10.3 Network Principles

  1. DHT Discovery: Decentralized peer finding
  2. Hole Punching: NAT traversal
  3. Noise Protocol: Modern encryption
  4. UDX Transport: Custom UDP for performance
  5. Connection Pooling: Efficient resource use

Repository Statistics

  • Total Repositories: 554+
  • Core Ecosystem: 100+ hyper* modules
  • Bare Runtime: 100+ bare* modules
  • Pear Platform: 80+ pear* modules
  • Native Libraries: 40+ lib* modules
  • CMake System: 30+ cmake* modules

See Also

Reference in New Issue View Git Blame Copy Permalink