research/core-concepts

Core Concepts

Fundamental concepts underlying the Holepunch peer-to-peer ecosystem.

Table of Contents

1. Feeds and Append-Only Logs
2. Merkle Trees and Verification
3. Replication and Sync
4. Cryptographic Primitives
5. Hole Punching and NAT Traversal
6. Distributed Hash Table (DHT)
7. Multi-Writer and Autobase
8. CRDTs and Causal Consistency
9. Encryption and Security
10. Runtime Architecture

---

1. Feeds and Append-Only Logs

1.1 What is a Feed?

A feed is an append-only log - a data structure where data can only be added to the end, never modified or deleted from the middle.

Feed Structure:
┌─────────────────────────────────────────────────────┐
│ Block 0 │ Block 1 │ Block 2 │ Block 3 │ Block 4 │ ...│
│  Data   │  Data   │  Data   │  Data   │  Data   │    │
│  [0-99] │[100-199]│[200-299]│[300-399]│[400-499]│    │
└─────────────────────────────────────────────────────┘
   ↑       ↑       ↑       ↑       ↑
   │       │       │       │       │
   └───────┴───────┴───────┴───────┘
         Merkle Tree Root

1.2 Hypercore Feed

Hypercore is the primary feed implementation in the Holepunch ecosystem.

Key Properties:

Property	Description
Append-only	Data can only be added, never modified
Signed	Each root is signed with Ed25519
Verifiable	Merkle tree enables integrity verification
Replicated	Bitfield-based sync between peers
Encrypted	Optional per-block encryption

Block Structure:

const Hypercore = require('hypercore')
const core = new Hypercore('./my-feed')

// Append data
await core.append(Buffer.from('Hello World'))
await core.append(Buffer.from('Block 2'))

// Read by index
const block = await core.get(0) // 'Hello World'

// Get metadata
console.log(core.length)     // 2 (number of blocks)
console.log(core.byteLength) // Total bytes
console.log(core.key)        // Public key (discovery)
console.log(core.discoveryKey) // Hashed key for DHT

1.3 Why Append-Only?

Benefit	Explanation
Integrity	History cannot be altered
Replication	Simple sync protocol
Verification	Cryptographic proof of contents
Caching	Blocks can be cached indefinitely
Conflict-free	No merge conflicts

---

2. Merkle Trees and Verification

2.1 Merkle Tree Structure

A Merkle tree is a binary tree where each leaf is a data block, and each parent is the hash of its children.

         Root Hash (signed)
              │
        ┌─────┴─────┐
        │           │
    ┌───┴───┐   ┌───┴───┐
    │       │   │       │
  ┌─┴─┐   ┌─┴─┐ ┌─┴─┐   x
  │   │   │   │ │   │
 B0   B1 B2   B3 B4

B0-4 = Data blocks (leaves)
x    = Empty placeholder (balanced tree)

2.2 Verification Process

flowchart LR
    PEER["Peer A"] --> TREE["Request Merkle Tree"]
    TREE --> VERIFY["Verify Signatures"]
    VERIFY --> CHECK["Check Block Hashes"]
    CHECK --> TRUST["Trust Verified"]
    
    PEER --> REQUEST["Request Block 2"]
    REQUEST --> PROOF["Get Proof: Hash(B2)+Hash(B3)+Parent"]
    PROOF --> VERIFY2["Verify against Root"]
    VERIFY2 --> CONFIRM["Block Valid"]

Proof verification:

// Get a block with proof
const { block, proof } = await core.get(2, { value: true, proof: true })

// Verify the proof
const verified = Hypercore.verifyProof(proof, block, core.key)
console.log(verified) // true

2.3 Partial Verification

Scenario	Approach
Verify single block	Get Merkle proof path
Verify range	Get subtree proof
Verify integrity	Check root signature
Detect tampering	Hash mismatch

---

3. Replication and Sync

3.1 Bitfield Synchronization

Peers use bitfields to indicate which blocks they have:

Peer A bitfield: [1, 1, 1, 1, 0, 0, 0]  (has blocks 0-3)
Peer B bitfield: [1, 1, 0, 0, 1, 1, 0]  (has blocks 0-1, 4-5)

Missing:
- Peer A needs: 4, 5
- Peer B needs: 2, 3

3.2 Replication Protocol

sequenceDiagram
    participant A as Peer A
    participant B as Peer B
    
    A->>B: Send bitfield [1,1,1,1,0,0,0]
    B->>A: Send bitfield [1,1,0,0,1,1,0]
    
    A->>B: Request blocks 4, 5
    B->>A: Send block 4 (with proof)
    B->>A: Send block 5 (with proof)
    
    B->>A: Request blocks 2, 3
    A->>B: Send block 2 (with proof)
    A->>B: Send block 3 (with proof)
    
    Note over A,B: Both peers have all blocks

3.3 Live Replication

Continuous sync:

// Create replication stream
const stream = core.replicate(true) // true = initiator

// Pipe to network connection
stream.pipe(socket).pipe(stream)

// Live sync - new blocks automatically replicated
core.on('append', () => {
  console.log('New blocks, auto-syncing...')
})

3.4 Replication Modes

Mode	Description	Use Case
Sparse	Request specific blocks	Random access
Linear	Sequential from start	Full sync
Live	Continuous updates	Real-time
Eager	Pre-fetch blocks	Low latency

---

4. Cryptographic Primitives

4.1 Key Types

Key	Algorithm	Purpose
Feed Key	Ed25519	Sign feed updates
Discovery Key	Blake2b	DHT lookup (hashed)
Encryption Key	XSalsa20	Block encryption
Noise Keys	Curve25519	Connection encryption

4.2 Feed Key Derivation

const Hypercore = require('hypercore')
const crypto = require('hypercore-crypto')

// Generate new keypair
const keyPair = crypto.keyPair()

// Create feed with key
const core = new Hypercore('./feed', keyPair.publicKey, {
  keyPair: keyPair,
  secretKey: keyPair.secretKey
})

// Derive discovery key (for DHT)
const discoveryKey = crypto.discoveryKey(keyPair.publicKey)

4.3 Signature Flow

flowchart LR
    DATA["New Block Data"] --> HASH["Hash Block"]
    HASH --> TREE["Update Merkle Tree"]
    TREE --> ROOT["New Root Hash"]
    ROOT --> SIGN["Sign with Ed25519"]
    SIGN --> STORE["Store Signature"]
    STORE --> REPLICATE["Replicate to Peers"]

4.4 Security Properties

Property	Mechanism	Guarantee
Authenticity	Ed25519 signatures	Only key owner can append
Integrity	Merkle tree	Tampering detectable
Privacy	XSalsa20 encryption	Content confidential
Forward secrecy	Noise protocol	Past messages safe

---

5. Hole Punching and NAT Traversal

5.1 The NAT Problem

Network Address Translation (NAT) hides private IPs behind public ones, making direct P2P connection difficult.

Internet
    │
    ▼
┌──────────────────┐
│  Public IP       │
│  203.0.113.1     │
│  (NAT Router)    │
└──────────────────┘
    │
    ▼
┌──────────────────┐
│  Private Network │
│  192.168.1.x     │
│  (Home Network)  │
└──────────────────┘

5.2 Hole Punching Technique

Simultaneous open: Both peers try to connect to each other at the same time.

sequenceDiagram
    participant A as Peer A (NAT A)
    participant DHT as DHT
    participant B as Peer B (NAT B)
    participant Relay as Relay (fallback)
    
    A->>DHT: Announce public address
    B->>DHT: Announce public address
    
    A->>DHT: Lookup B's address
    DHT-->>A: B's public addr
    B->>DHT: Lookup A's address
    DHT-->>B: A's public addr
    
    A->>B: SYN (blocked by NAT B)
    B->>A: SYN (blocked by NAT A)
    
    Note over A,B: Both NATs see outgoing SYN,<br/>allow return traffic
    
    B-->>A: SYN-ACK (NAT A allows)
    A-->>B: ACK (NAT B allows)
    
    Note over A,B: Direct connection established!
    
    alt Hole punching fails
        A->>Relay: Connect via relay
        B->>Relay: Connect via relay
        Relay->>Relay: Relay traffic
    end

5.3 NAT Types

Type	Behavior	Punchable?
Full Cone	Any external can connect	✅ Yes
Restricted Cone	Must send first	✅ Yes
Port Restricted	Must send to specific port	✅ Yes
Symmetric	Random ports per peer	❌ Hard

5.4 Implementation in Hyperswarm

const Hyperswarm = require('hyperswarm')
const swarm = new Hyperswarm()

// Automatic hole punching
swarm.join(topic, { server: true, client: true })

swarm.on('connection', (conn, info) => {
  console.log('Connected via hole punching!')
  console.log('Direct:', info.direct) // true if direct
  console.log('Nat type:', info.nat)
})

---

6. Distributed Hash Table (DHT)

6.1 Kademlia DHT

HyperDHT implements the Kademlia protocol for decentralized peer discovery.

Kademlia Structure:

              ┌──────────────┐
              │    Root      │
              │   Bucket     │
              │   (0-159)    │
              └──────┬───────┘
                     │
        ┌────────────┼────────────┐
        │            │            │
   ┌────┴────┐  ┌────┴────┐  ┌────┴────┐
   │Bucket 0 │  │Bucket 1 │  │Bucket 2 │
   │ (0-79)  │  │(80-119) │  │(120-159)│
   └────┬────┘  └────┬────┘  └────┬────┘
        │            │            │
     Peers        Peers        Peers

6.2 Peer Discovery

How it works:

1. Announce: Peer stores its address at key K
2. Lookup: Other peers query key K
3. Find: DHT routes to closest nodes
4. Connect: Peers exchange connection info

const DHT = require('hyperdht')
const dht = new DHT()

// Announce on a key
const keyPair = DHT.keyPair()
await dht.announce(keyPair.publicKey, { host: '1.2.3.4', port: 1234 })

// Lookup peers
const peers = await dht.lookup(keyPair.publicKey)
for await (const peer of peers) {
  console.log('Found peer:', peer.host, peer.port)
}

6.3 DHT Operations

Operation	Description
announce(key, address)	Register as provider
lookup(key)	Find providers
unannounce(key)	Remove registration
query(target)	Find nodes near target

---

7. Multi-Writer and Autobase

7.1 The Multi-Writer Problem

Traditional Hypercore: Single writer per feed. How do multiple users collaborate?

Problem:

User A's Feed:  [A1][A2][A3]
                     ↓
User B's Feed:  [B1][B2]
                     ↓
User C's Feed:  [C1][C2][C3][C4]

Goal: Merge into single consistent view

7.2 Autobase Solution

Autobase merges multiple Hypercores into a single causal stream.

flowchart TB
    subgraph "Input Feeds"
        A["Feed A<br/>[A1,A2,A3]"]
        B["Feed B<br/>[B1,B2]"]
        C["Feed C<br/>[C1,C2,C3,C4]"]
    end
    
    subgraph "Autobase"
        MERGE["Causal Merge<br/>(Lamport timestamps)"]
        ORDER["Total Order<br/>(Linearized)"]
    end
    
    subgraph "Output"
        VIEW["Derived View<br/>[A1,B1,C1,A2,B2,C2,A3,C3,C4]"]
    end
    
    A --> MERGE
    B --> MERGE
    C --> MERGE
    MERGE --> ORDER
    ORDER --> VIEW

7.3 Causal Ordering

Lamport Timestamps track causality:

const Autobase = require('autobase')

// Create autobase with multiple inputs
const base = new Autobase({
  inputs: [feedA, feedB, feedC],
  localInput: feedA // Our feed
})

// Append causally
await base.append('Hello from A')

// Autobase assigns LSN (Linearized Sequence Number)
// Order: Causal + Deterministic tie-breaking

7.4 Autobase Properties

Property	Guarantee
Causality	If A happened before B, A < B in order
Consistency	All peers see same linearized order
Availability	Writers can append independently
Conflict-free	No merge conflicts

---

8. CRDTs and Causal Consistency

8.1 What are CRDTs?

Conflict-free Replicated Data Types - Data structures that merge consistently without coordination.

Types of CRDTs:

Type	Examples	Merge Strategy
State-based	G-Set, LWW-Register	Union, last-writer-wins
Op-based	Counter, Set	Operation replay
Delta-state	All types	Delta synchronization

8.2 Autobase as CRDT Foundation

Building CRDTs on Autobase:

const Autobase = require('autobase')
const c = require('compact-encoding')

// Define CRDT operations
const OpType = {
  INSERT: 0,
  DELETE: 1,
  UPDATE: 2
}

// Operation encoding
const Op = c.struct({
  type: c.uint,
  key: c.string,
  value: c.raw
})

// Apply to view
const view = base.view
base.on('append', async (node) => {
  const op = c.decode(Op, node.value)
  
  switch (op.type) {
    case OpType.INSERT: view.set(op.key, op.value); break
    case OpType.DELETE: view.delete(op.key); break
    case OpType.UPDATE: view.merge(op.key, op.value); break
  }
})

8.3 Causal Consistency

Consistency model:

Timeline:

Process A: ───[Write x=1]────[Read x=1]────────────
                 ↓
Process B: ───────────[Read x=1]───[Write x=2]───
                        ↓
Process C: ───────────────────[Read x=2]─────────

Guarantees:
- Causal reads: B reads x=1 (caused by A)
- Causal writes: C reads x=2 (caused by B)

---

9. Encryption and Security

9.1 Feed Encryption

Hypercore supports per-block encryption:

const Hypercore = require('hypercore')
const crypto = require('hypercore-crypto')

// Generate encryption key
const encryptionKey = crypto.randomBytes(32)

// Create encrypted feed
const core = new Hypercore('./encrypted', {
  encryptionKey: encryptionKey
})

// Append encrypted data
await core.append(Buffer.from('Secret data'))

// Only peers with encryptionKey can read

9.2 Noise Protocol

Connection encryption using Noise:

Noise Handshake:

Initiator                 Responder
─────────                 ─────────
  │                           │
  ├── e ---------------------> │  (ephemeral key)
  │                           │
  │<── e, ee, s, es ---------│  (ephemeral + static)
  │                           │
  ├── s, se ---------------->│  (static key)
  │                           │
  │<── [encrypted data]──────│  (secure channel)

9.3 Security Layers

Layer	Mechanism	Protection
Feed	XSalsa20	Content encryption
Connection	Noise_XX	Transport encryption
Authentication	Ed25519	Identity verification
Integrity	Merkle tree	Tamper detection

9.4 Capability-Based Security

Fine-grained access control:

const capabilities = require('hyperswarm-capability')

// Create capability
const cap = capabilities.create({
  key: feed.key,
  allow: ['read', 'append'],
  expires: Date.now() + 3600000 // 1 hour
})

// Verify capability
const valid = capabilities.verify(cap, feed.key)
if (valid) {
  // Grant access
}

---

10. Runtime Architecture

10.1 Bare Runtime

Bare is a minimal JavaScript runtime optimized for P2P applications.

flowchart TB
    subgraph "Bare Runtime"
        JS["JS Engine<br/>V8/JSC/QuickJS"]
        MOD["Module System"]
        API["Core APIs"]
        ADDON["Native Addons"]
        
        subgraph "Built-in Modules"
            FS["bare-fs"]
            NET["bare-tcp/udp"]
            STREAM["bare-stream"]
            CRYPTO["bare-crypto"]
        end
    end
    
    subgraph "Native Layer"
        LIBJS["libjs"]
        LIBUDX["libudx"]
        SODIUM["sodium-native"]
    end
    
    JS --> MOD
    MOD --> API
    API --> FS
    API --> NET
    API --> STREAM
    API --> CRYPTO
    MOD --> ADDON
    ADDON --> LIBJS
    NET --> LIBUDX
    CRYPTO --> SODIUM

10.2 Pear Runtime

Pear extends Bare with application lifecycle management.

CLI vs embeddable packages: the pear binary is the command-line interface for init, dev, staging, seeding, releases, and more. The pear-runtime, pear-runtime-updater, and related npm packages are what you embed when you need a runtime inside your own shipped binary with P2P OTA-style updates. The legacy pear run CLI subcommand is deprecated from Pear v2.4+; prefer pear dev locally and the pear-runtime family when embedding (upstream CHANGELOG). Concept guide: Pear platform and runtime. Module articles: pear-runtime, Pear (KB). Composition overview: Interconnections §0.

flowchart TB
    subgraph pearPlatform [PearPlatform]
        APP[Application]
        subgraph pearServices [PearServices]
            IPC[IPC_Bridge]
            SIDE[Sidecar]
            PACK[Bundle_Pack]
            UPD[Updater]
        end
        subgraph uiLayer [UI]
            DESK[Desktop]
            TERM[Terminal]
            ELEC[Electron]
        end
    end
    subgraph bareRuntime [BareRuntime]
        BARE[BareJS]
    end
    APP --> IPC
    IPC --> SIDE
    SIDE --> PACK
    PACK --> UPD
    APP --> DESK
    APP --> TERM
    APP --> ELEC
    APP --> BARE

10.3 Runtime Comparison

Feature	Bare	Pear	Node.js
Size	~30MB	~50MB	~100MB
Startup	Fast	Fast	Slower
P2P	Native	Native	Via modules
Updates	Manual	Built-in	Manual
Mobile	✅	✅	❌
Distribution	Bundles	Apps	Packages

---

Summary

Concept	Key Takeaway
Feeds	Append-only logs with cryptographic integrity
Merkle Trees	Efficient verification of data integrity
Replication	Bitfield-based P2P synchronization
Crypto	Ed25519 signatures, Noise encryption
Hole Punching	NAT traversal for direct P2P
DHT	Decentralized peer discovery
Autobase	Multi-writer causal ordering
CRDTs	Conflict-free collaborative data
Security	Layered encryption and authentication

---

See Also

• Pear platform and runtime (concept) - CLI vs pear-runtime npm, Pear v2.4+ notes
• Architecture - System architecture
• Interconnections - Component composition
• Building Tools - Development tools
• Existing Projects - Real-world examples