research/architecture

Complete Holepunch Architecture Stack

Overview

The Holepunch ecosystem is a comprehensive peer-to-peer (P2P) application stack spanning 600+ repositories (see research/scripts/module-inventory.json for the mirrored set). It provides everything needed to build, distribute, and run distributed applications across desktop and mobile platforms.

flowchart TB
    subgraph "Application Layer"
        APP["📱 Applications"]
        KEET["Keet"]
        SHELL["Hypershell"]
        PASS["Autopass"]
        PEARAPP["Pear Apps"]
    end
    
    subgraph "Platform Layer (Pear)"
        PEAR["🍐 Pear Runtime"]
        SIDE["Sidecar"]
        IPC["IPC/Bridge"]
        PACK["Pack/Bundle"]
        UPD["Updater"]
    end
    
    subgraph "Runtime Layer (Bare)"
        BARE["⚡ Bare JS Runtime"]
        BMOD["Module System"]
        BADD["Native Addons"]
        BBUN["Bundling"]
        BT["Threading"]
    end
    
    subgraph "Data Layer (Hypercore)"
        CORE["📦 Hypercore"]
        BEE["🌳 Hyperbee"]
        DRIVE["📁 Hyperdrive"]
        AUTO["🔗 Autobase"]
        BLOB["Hyperblobs"]
    end
    
    subgraph "Network Layer (Hyperswarm)"
        SWARM["🌐 Hyperswarm"]
        DHT["🔍 HyperDHT"]
        SEC["Secret Streams"]
        RELAY["DHT Relay"]
    end
    
    subgraph "Protocol Layer"
        PMUX["Protomux"]
        RPC["HRPC"]
        ENC["Compact Encoding"]
    end
    
    subgraph "Native Layer"
        LIBJS["libjs (JS Engine)"]
        LIBUDX["libudx (UDP)"]
        SODIUM["sodium-native"]
        ROCKS["rocksdb-native"]
    end
    
    APP --> PEAR
    KEET --> PEAR
    SHELL --> PEAR
    PASS --> PEAR
    PEARAPP --> PEAR
    
    PEAR --> BARE
    PEAR --> SIDE
    PEAR --> IPC
    PEAR --> PACK
    PEAR --> UPD
    
    BARE --> CORE
    BARE --> SWARM
    BARE --> BMOD
    BARE --> BADD
    
    CORE --> BEE
    CORE --> DRIVE
    CORE --> AUTO
    DRIVE --> BLOB
    
    SWARM --> DHT
    SWARM --> SEC
    SWARM --> RELAY
    
    DHT --> PMUX
    SEC --> PMUX
    PMUX --> RPC
    PMUX --> ENC
    
    BARE --> LIBJS
    DHT --> LIBUDX
    SEC --> SODIUM
    CORE --> ROCKS

---

1. Application Layer (Pear Platform)

1.1 Pear Runtime

The Pear platform is the flagship application runtime for building and distributing P2P applications.

Core Components:

Module	Purpose	Location
pear	Main platform (CLI, sidecar, subsystems)	Main repo
pear-api	Runtime API interface	API layer
pear-cli	Command-line interface	CLI layer
pear-cmd	Command implementations	Commands
pear-sidecar	Background service	Sidecar
pear-bridge	IPC bridge	IPC
pear-ipc	Inter-process communication	IPC
pear-desktop	Desktop UI runtime	UI
pear-terminal	Terminal UI runtime	UI
pear-electron	Electron integration	UI

Build & Distribution:

Module	Purpose
pear-bundle	Application bundling
pear-pack	Package creation
pear-build	Build system
pear-stage	Staging for release
pear-release	Release management
pear-updater	Update system
pear-appling	Application packaging

Storage & Data:

Module	Purpose
pear-hyperdb	Database integration
pear-appdrive	Application storage
pear-seed	Seeding utilities

Utilities:

Module	Purpose
pear-inspect	Debugging/inspection
pear-doctor	Diagnostics
pear-info	System information
pear-logger	Logging
pear-init	Project scaffolding
pear-templates	Starter templates

---

2. Runtime Layer (Bare)

2.1 Bare JavaScript Runtime

Bare is a minimal, cross-platform JavaScript runtime designed for embedding and P2P applications.

flowchart LR
    subgraph "Bare Architecture"
        JS["JavaScript Code"]
        MOD["Module System<br/>bare-module*"]
        API["Core APIs"]
        ADD["Native Addons<br/>bare-addon*"]
        BUN["Bundler<br/>bare-bundle*"]
    end
    
    subgraph "Platform Support"
        IOS["iOS<br/>bare-ios"]
        AND["Android<br/>bare-android"]
        DESK["Desktop<br/>bare-kit"]
        WEB["WebKit<br/>bare-web-kit"]
    end
    
    JS --> MOD
    MOD --> API
    MOD --> ADD
    API --> BUN
    ADD --> IOS
    ADD --> AND
    ADD --> DESK
    ADD --> WEB

Core I/O Modules:

Module	Purpose	Node.js Equivalent
bare-fs	File system	fs
bare-path	Path utilities	path
bare-os	OS interfaces	os
bare-stream	Streams	stream
bare-buffer	Buffers	buffer
bare-events	Event emitters	events
bare-timers	Timers	timers

Network Modules:

Module	Purpose	Protocol
bare-tcp	TCP sockets	TCP
bare-udp	UDP sockets	UDP
bare-tls	TLS/SSL	TLS 1.3
bare-http1	HTTP/1.1	HTTP
bare-https	HTTPS	HTTPS
bare-fetch	Fetch API	HTTP/HTTPS
bare-ws	WebSocket	WS/WSS

Module System:

Module	Purpose
bare-module	Module loading
bare-module-resolve	Module resolution
bare-module-lexer	Module parsing
bare-module-traverse	Module traversal

Bundling & Distribution:

Module	Purpose
bare-bundle	Code bundling
bare-pack	Package creation
bare-unpack	Package extraction
bare-bundle-compile	Compile bundles
bare-bundle-evaluate	Execute bundles

Concurrency:

Module	Purpose
bare-thread	Worker threads
bare-worker	Web Workers
bare-channel	Inter-thread communication
bare-atomics	Atomic operations

Platform Bindings:

Module	Platform	Purpose
bare-ios	iOS	iOS runtime
bare-android	Android	Android runtime
bare-kit	Mobile/Desktop	Cross-platform toolkit
bare-kit-swift	iOS	Swift bindings
bare-ndk	Android	NDK support
bare-apk	Android	APK packaging
bare-win-ui	Windows	Windows UI
bare-gtk	Linux	GTK bindings
bare-sdl	Cross-platform	SDL bindings

---

3. Data Layer (Hypercore Ecosystem)

3.1 Core Data Structures

flowchart TB
    subgraph "Data Layer"
        CORE["Hypercore<br/>Append-only Log"]
        CS["Corestore<br/>Multi-core Management"]
        
        subgraph "Built on Core"
            BEE["Hyperbee<br/>B-tree Index"]
            DRIVE["Hyperdrive<br/>Filesystem"]
            AUTO["Autobase<br/>Multi-writer"]
        end
        
        subgraph "Supporting"
            BLOB["Hyperblobs"]
            ENC["Encryption"]
            CRYPTO["Crypto"]
        end
    end
    
    CS --> CORE
    CORE --> BEE
    CORE --> DRIVE
    CORE --> AUTO
    DRIVE --> BLOB
    CORE --> ENC
    CORE --> CRYPTO

Hypercore - Distributed append-only log:

Feature	Implementation
Structure	Merkle tree with signed roots
Replication	Bitfield-based peer sync
Storage	RocksDB backend
Encryption	Per-block encryption
Signatures	Ed25519

Extensions:

Module	Purpose
hypercore-crypto	Cryptographic operations
hypercore-encryption	Block encryption
hypercore-storage	Storage backends
hypercore-byte-stream	Streaming access
hypercore-messages	Message protocols
hypercore-stats	Metrics
hypercore-id-encoding	Key encoding
hypercore-sign	Signing utilities
hypercore-blob-server	Blob serving
hypercore-audit	Audit trails

Hyperbee - B-tree on Hypercore:

Feature	Description
Structure	Sorted key-value store
Index	B-tree over Hypercore
Diffing	hyperbee-diff-stream
Use case	Database indices, file trees

Hyperdrive - Distributed filesystem:

Feature	Implementation
Structure	Hyperbee (metadata) + Hyperblobs (content)
Blobs	Separate blob storage
Versioning	Complete history
Diffing	Drive-to-drive comparisons

Extensions:

Module	Purpose
hyperdrive-next	Next-gen features
hyperdrive-profiler	Performance profiling
hyperdrive-swarm-test	Network testing
hyperblobs	Blob storage

Autobase - Multi-writer data structure:

Feature	Description
Ordering	Causal (Lamport timestamps)
Merge	Automatic view merging
Use case	Collaborative editing, CRDTs

Extensions:

Module	Purpose
autobase-discovery	Peer discovery
autobase-light-writer	Lightweight writers
autobase-example	Example implementations

Corestore - Multi-core management:

Feature	Description
Storage	Deduplicated core storage
Namespacing	Key namespaces
Efficiency	Batch operations

---

4. Network Layer (Hyperswarm)

4.1 P2P Networking Stack

flowchart TB
    subgraph "Networking Stack"
        SWARM["Hyperswarm<br/>High-level API"]
        DHT["HyperDHT<br/>Distributed Hash Table"]
        RPC["DHT-RPC<br/>Remote Procedure Call"]
        
        subgraph "Connection"
            SEC["Secret Stream<br/>Encryption"]
            CAP["Capability<br/>Auth"]
            RELAY["DHT Relay<br/>NAT Traversal"]
        end
        
        subgraph "Transport"
            UDX["libudx<br/>UDP Transport"]
            TCP["TCP Fallback"]
        end
    end
    
    SWARM --> DHT
    DHT --> RPC
    SWARM --> SEC
    SWARM --> CAP
    SWARM --> RELAY
    DHT --> UDX
    DHT --> TCP

Hyperswarm - High-level P2P API:

Feature	Description
Discovery	Topic-based peer discovery
Hole punching	NAT traversal
Encryption	Noise protocol
Pooling	Connection management

Extensions:

Module	Purpose
hyperswarm-secret-stream	Encrypted streams
hyperswarm-dht-relay	Relay servers
hyperswarm-capability	Capability auth
hyperswarm-seeders	Seeder infrastructure
hyperswarm-testnet	Test networks
hyperswarm-stats	Metrics
hyperswarm-doctor	Diagnostics

HyperDHT - Distributed Hash Table:

Feature	Description
Structure	Kademlia DHT
Keys	Ed25519 public keys
Persistence	Persistent node IDs
Queries	Find peers by key

Extensions:

Module	Purpose
hyperdht-stats	DHT metrics
http-dht-proxy	HTTP proxy
http-forward-host	Host forwarding

---

5. Protocol Layer

5.1 Protocol Multiplexing

flowchart LR
    STREAM["Encrypted Stream"]
    MUX["Protomux<br/>Multiplexer"]
    
    subgraph "Channels"
        RPC["HRPC<br/>RPC"]
        WAKE["Wakeup"]
        CUSTOM["Custom Protocols"]
    end
    
    STREAM --> MUX
    MUX --> RPC
    MUX --> WAKE
    MUX --> CUSTOM

Protomux - Protocol multiplexing:

Feature	Description
Multiplexing	Multiple protocols over single stream
Framing	Length-prefixed messages
Backpressure	Flow control

Extensions:

Module	Purpose
protomux-rpc	RPC framework
protomux-rpc-client	RPC client
protomux-rpc-client-pool	Connection pooling
protomux-rpc-middleware	Middleware support
protomux-rpc-router	Request routing
protomux-wakeup	Wake protocol

Compact Encoding - Serialization:

Feature	Description
Format	Compact binary
Schema	Type definitions
Cross-language	Multiple bindings

Extensions:

Module	Purpose
compact-encoding-bitfield	Bitfield encoding
compact-encoding-net	Network types
compact-encoding-struct	Struct encoding
compact-encoding-variant	Variant types
compact-encoding-golang	Go bindings
compact-encoding-swift	Swift bindings

---

6. Native Layer

6.1 C/C++ Libraries

flowchart TB
    subgraph "Native Libraries"
        LIBJS["libjs<br/>JS Engine Abstraction"]
        LIBUDX["libudx<br/>UDP Transport"]
        LIBRPC["librpc<br/>RPC"]
        
        subgraph "Crypto"
            SODIUM["sodium-native<br/>NaCl"]
            NOISE["noise-handshake"]
        end
        
        subgraph "Storage"
            ROCKS["rocksdb-native<br/>RocksDB"]
            MEM["libmem"]
        end
        
        subgraph "System"
            FS["libfs"]
            PATH["libpath"]
            LOG["liblog"]
        end
    end
    
    LIBJS --> BARE
    LIBUDX --> DHT
    LIBRPC --> RPC
    SODIUM --> SEC
    ROCKS --> CORE

Core Libraries:

Library	Purpose	Language
libjs	JS engine abstraction	C
libnapi	Native addon interface	C
libjstl	JS template library	C++
libudx	UDP data transfer	C
librpc	RPC communication	C
libmem	Memory utilities	C
liblog	Logging	C
libpath	Path utilities	C
libfs	File system	C
libhex	Hex encoding	C
liburl	URL parsing	C
libutf	UTF utilities	C
libtls	TLS/SSL	C
libz32	Base32 encoding	C
libbitarray	Bit arrays	C
libquickbit	Bit operations	C
libsimdle	SIMD utilities	C
libcompact	Compact encoding	C
libjson	JSON parsing	C
libcrc	CRC algorithms	C
librabin	Rabin fingerprinting	C
librocksdb	RocksDB bindings	C
libdaemon	Daemon utilities	C
libdynload	Dynamic loading	C
libprng	PRNG	C

---

7. Data Flow Examples

7.1 Application Startup Flow

sequenceDiagram
    participant User
    participant App as Application
    participant Pear as Pear Runtime
    participant Bare as Bare JS
    participant Swarm as Hyperswarm
    participant DHT as HyperDHT
    participant Core as Hypercore
    
    User->>App: Launch app
    App->>Pear: pear.run()
    Pear->>Bare: Initialize runtime
    Bare->>Core: Load hypercores
    Core->>Core: Verify integrity
    App->>Swarm: Join topics
    Swarm->>DHT: Announce presence
    DHT-->>Swarm: Peer candidates
    Swarm->>Swarm: Hole punch
    Swarm-->>App: Connected peers
    App-->>User: Ready

7.2 Data Replication Flow

sequenceDiagram
    participant A as Peer A
    participant Swarm as Hyperswarm
    participant B as Peer B
    participant CoreA as Hypercore A
    participant CoreB as Hypercore B
    
    A->>Swarm: Join discoveryKey
    B->>Swarm: Join discoveryKey
    Swarm->>A: Found peer B
    Swarm->>B: Found peer A
    A->>B: Noise handshake
    B->>A: Noise handshake
    A->>CoreA: Get bitfield
    CoreA-->>A: Bitfield
    A->>B: Send bitfield
    B->>CoreB: Compare bitfields
    CoreB-->>B: Need blocks X-Y
    B->>A: Request blocks X-Y
    A->>CoreA: Read blocks X-Y
    CoreA-->>A: Block data
    A->>B: Send blocks
    B->>CoreB: Write blocks

7.3 Multi-writer Autobase Flow

sequenceDiagram
    participant W1 as Writer 1
    participant W2 as Writer 2
    participant W3 as Writer 3
    participant Auto as Autobase
    participant View as Derived View
    
    W1->>Auto: Write entry A
    Auto->>Auto: Assign LSN=1
    W2->>Auto: Write entry B
    Auto->>Auto: Assign LSN=2
    W3->>Auto: Write entry C
    Auto->>Auto: Assign LSN=3
    
    Auto->>Auto: Causal ordering
    Auto->>View: Apply in order
    View->>View: A → B → C
    View-->>W1: Updated view
    View-->>W2: Updated view
    View-->>W3: Updated view

---

8. Build System Architecture

8.1 CMake Toolchain

flowchart TB
    subgraph "CMake Build System"
        CMAKE["CMake Configuration"]
        
        subgraph "Platform Modules"
            IOS["cmake-ios"]
            MAC["cmake-macos"]
            WIN["cmake-windows"]
            AND["cmake-android"]
        end
        
        subgraph "Build Tools"
            BARE["cmake-bare"]
            PEAR["cmake-pear"]
            NAPI["cmake-napi"]
        end
        
        subgraph "Distribution"
            IMG["cmake-app-image"]
            MSIX["cmake-msix"]
        end
    end
    
    CMAKE --> IOS
    CMAKE --> MAC
    CMAKE --> WIN
    CMAKE --> AND
    CMAKE --> BARE
    CMAKE --> PEAR
    CMAKE --> NAPI
    CMAKE --> IMG
    CMAKE --> MSIX

---

9. Application Examples

9.1 Real-world Applications

Application	Stack Used	Description
Keet	Full stack	P2P chat, video & audio calls - no sign-up, seed-phrase identity
PearPass	Hyperdrive	P2P password & secrets manager
Hypershell	Hyperswarm	P2P shell/SSH alternative
Autopass	Hyperdrive	P2P password manager
Simple Seeder	Hyperdrive	File seeding utility
Hyperbeam	Hyperswarm	P2P data transfer
Hyperssh	Hyperswarm	SSH over Hyperswarm

9.2 CLI Tools

Tool	Description
Hypershell	CLI for P2P E2E encrypted shells
Hypertele	Swiss-army knife proxy powered by HyperDHT
Hyperbeam	One-to-one E2E encrypted internet pipe
Hyperssh	Run SSH over the DHT
Drives	CLI for Hyperdrive & Localdrive operations

9.3 Example Projects

Project	Technologies	Purpose
filesharing-app-example	Hyperdrive + Hyperswarm	File sharing
filesharing-react-app-example	React + Hyperdrive	React file sharing
autobase-example	Autobase	Multi-writer demo
autopass-mobile-example	React Native	Mobile password manager
pear-expo-hello-world	Expo + Pear	Mobile Pear app
hrpc-modular-example	HRPC	Modular RPC demo

---

10. Key Design Principles

10.1 Architecture Principles

1. Modularity: Each component is independently usable
2. Composability: Components stack and compose cleanly
3. P2P-First: Designed for peer-to-peer from the ground up
4. Offline-First: Works without central servers
5. End-to-End Encryption: Security built-in, not bolted-on
6. Cross-Platform: Desktop, mobile, and embedded support
7. Native Performance: Critical paths in native code
8. JavaScript Ergonomics: High-level APIs in JavaScript

10.2 Data Principles

1. Immutable Logs: Hypercore append-only for integrity
2. Merkle Trees: Cryptographic verification
3. Bitfield Sync: Efficient replication
4. Deduplication: Corestore shares storage
5. Encryption: Per-block encryption support

10.3 Network Principles

1. DHT Discovery: Decentralized peer finding
2. Hole Punching: NAT traversal
3. Noise Protocol: Modern encryption
4. UDX Transport: Custom UDP for performance
5. Connection Pooling: Efficient resource use

---

Repository Statistics

• Total Repositories: 600+
• Core Ecosystem: 100+ hyper* modules
• Bare Runtime: 100+ bare* modules
• Pear Platform: 80+ pear* modules
• Native Libraries: 40+ lib* modules
• CMake System: 30+ cmake* modules

---

See Also

• Module inventory - clone-to-doc map for the holepunchto_repos mirror
• Core Concepts - Detailed concept explanations
• Building Tools - Build and development tools
• Interconnections - How components compose
• Existing Projects - Real-world applications
• Project Ideas - Opportunities for extension