holesail-browser/docs/ARCHITECTURE.md

# Architecture

Holesail Browser is composed of three parts: a browser extension, a native host process, and the Holesail P2P network. The extension and native host communicate via Chrome's native messaging protocol; the native host manages all tunnel connections and runs a local HTTPS proxy that the browser routes virtual host traffic through.

## Component overview

```
┌─────────────────────────────────────────────────────────────────┐
│  Browser (Chrome / Firefox)                                     │
│                                                                 │
│  ┌──────────────┐    ┌──────────────────────────────────────┐   │
│  │ background.js│    │ dashboard/dashboard.html             │   │
│  │ (service     │◄──►│ (management UI — virtual hosts,      │   │
│  │  worker)     │    │  SSH, RDP, backups, settings, logs)  │   │
│  └──────┬───────┘    └──────────────────────────────────────┘   │
│         │ PAC script: *.hole.sail → PROXY 127.0.0.1:8442        │
│         │             *.custom.tld → PROXY 127.0.0.1:8442       │
│         │ Native messaging: chrome.runtime.connectNative(...)   │
└─────────┼───────────────────────────────────────────────────────┘
          │ stdin/stdout (4-byte length-prefixed JSON)
          ▼
┌─────────────────────────────────────────────────────────────────┐
│  Native Host  (~/.holesail-browser/holesail-browser-host)       │
│                                                                 │
│  ┌──────────┐  ┌──────────────────┐  ┌──────────────────────┐   │
│  │ host.js  │  │ holesail-        │  │ certificate-         │   │
│  │ (command │  │ manager.js       │  │ authority.js         │   │
│  │ dispatch)│  │ (tunnel lifecycle│  │ (root CA + per-TLD   │   │
│  └────┬─────┘  │  + state.json)   │  │  wildcard certs,     │   │
│       │        └────────┬─────────┘  │  keychain install)   │   │
│       │                 │            └──────────────────────┘   │
│  ┌────▼─────────────────▼──────────────────────────────────┐    │
│  │  https-proxy.js          connect-proxy.js               │    │
│  │  127.0.0.1:8443          127.0.0.1:8442                 │    │
│  │  (SNI-aware TLS,         (HTTP CONNECT handler,         │    │
│  │   per-TLD wildcard        pipes to 127.0.0.1:8443)      │    │
│  │   certs, HTTP forwarding)                               │    │
│  └─────────────────────────────────────────────────────────┘    │
│                                                                 │
│  ┌──────────────┐  ┌──────────────┐  ┌──────────────────────┐   │
│  │ ssh-manager  │  │ rdp-manager  │  │ backup-manager       │   │
│  │ (PTY + WS    │  │ (VNC/RDP +   │  │ (tar.gz snapshots)   │   │
│  │  bridge)     │  │  WS bridge)  │  │                      │   │
│  └──────────────┘  └──────────────┘  └──────────────────────┘   │
└─────────────────────────────────────────────────────────────────┘
          │
          │  Holesail P2P (Noise protocol, DHT)
          ▼
┌──────────────────────────────────────────────────────────────┐
│  Remote peers (HTTP servers, SSH daemons, VNC/RDP servers)   │
└──────────────────────────────────────────────────────────────┘
```

## Source files

### Native host (`native-host/`)

#### Top-level files

| File | Purpose |
|------|---------|
| `index.mjs` | Entry point. Bootstraps Bare globals, creates the native messaging `messenger`, wires `handleMessage` from `host.js`. Handles `SIGTERM`/`SIGINT` for graceful shutdown. |
| `messenger.js` | Chrome/Firefox native messaging framing: 4-byte little-endian length prefix + UTF-8 JSON body. Max 1 MB per message. |
| `host.js` | Thin shim — re-exports `handleMessage` and `cleanup` from `host/message-router.js`. Kept at the top level so `index.mjs` requires it unchanged. |
| `holesail-manager.js` | Thin shim — re-exports the full API from `holesail-manager/index.js`. Kept at the top level so `host/message-router.js` requires it unchanged. |
| `https-proxy.js` | SNI-aware HTTPS reverse proxy on `127.0.0.1:8443`. Uses a pure-JS TLS ClientHello parser to extract the SNI hostname from each incoming connection, derives the wildcard parent domain, and presents a per-TLD wildcard cert. Supports any hostname depth (e.g. `i.love.hole.sail`). Returns a 502 HTML page for unknown hostnames. |
| `connect-proxy.js` | HTTP CONNECT proxy on `127.0.0.1:8442`. Accepts `CONNECT hostname:443`, replies `200 Connection established`, then pipes the raw TCP stream to `127.0.0.1:8443`. |
| `certificate-authority.js` | Generates a 2048-bit RSA root CA (10-year validity) using `node-forge`. Signs per-TLD wildcard domain certs on demand (1-year). Installs the CA into the OS trust store. Fingerprint-verifies to detect stale entries. |
| `ssh-manager.js` | Per SSH session: starts a Holesail client tunnel, spawns `ssh` with a real PTY via `tt-native` (`forkpty`), starts a `bare-ws` WebSocket server, and bridges PTY ↔ WebSocket. SSH is spawned only after the browser WebSocket client connects and sends a ready-signal. Password delivery uses `SSH_ASKPASS` + a named FIFO — public key auth is tried first; a password prompt appears in xterm.js only if key auth fails and no password is saved. |
| `rdp-manager.js` | Per RDP/VNC session: starts a Holesail client tunnel, starts a WebSocket server. VNC: transparent byte pipe. RDP: `node-rdpjs-2` client, converts bitmap updates to JSON. |
| `backup-manager.js` | Creates/restores `tar.gz` backups of `state.json` + all certificates. Supports create, list, restore, delete, and auto-prune by retention count. |

#### `host/` — host orchestration modules

| File | Purpose |
|------|---------|
| `paths.js` | `resolveBase()` — detects standalone binary vs. dev mode and returns the correct base directory. Exports `BASE_DIR` and `STORAGE_PATH` constants used by all other host modules. |
| `logger.js` | `log()` and `debugLog()` — timestamped writes to `holesail-browser.log` (or `BRIDGE_SWARM_LOG` override) and `process.stderr`. Respects `HOLESAIL_DEBUG` env var. |
| `startup.js` | `initStartup()` — deferred proxy startup sequence: loads persisted settings, waits for CA readiness, starts the HTTPS and CONNECT proxies, then calls `restorePersistedTunnels()`. Exports promise accessors used by the message router. |
| `message-router.js` | `handleMessageAsync()` — the 30-case `switch` that dispatches every browser message to the appropriate manager. Wires all managers together on first require. Exports `handleMessage` and `cleanup`. |

#### `holesail-manager/` — tunnel and state sub-modules

| File | Purpose |
|------|---------|
| `state.js` | `loadState()`, `saveStateSync()`, `buildDefaultState()`. Handles `state.json` read/write and migration from the legacy `holesail-persist.json` format. |
| `settings.js` | `getSettings()`, `updateSettings()`, `getProxyPort()`, `setProxyPort()`. Owns `currentSettings` and `runtimeProxyPort`. |
| `connections.js` | `getSshConnections/set`, `getRdpConnections/set`. Owns the in-memory SSH and RDP connection lists (persisted in `state.json`, not live sessions). |
| `port-allocator.js` | `allocateTunnelPort()`, `releaseTunnelPort()`. Manages the virtual-host tunnel port pool starting at 19000 with a free list for recycling. |
| `servers.js` | `startServer()`, `stopServer()`, `getServers()`. Manages Holesail server tunnels (server mode — exposes a local port to the P2P network). |
| `virtual-hosts.js` | `setVirtualHost()`, `removeVirtualHost()`, `getVirtualHosts()`, `getLocalBackend()`, `getLocalPortForHostname()`, `getVirtualHostMap()`. Manages virtual host client tunnels routed through the HTTPS proxy. |
| `service-tunnels.js` | `startServiceTunnel()`, `stopServiceTunnel()`, `getServiceTunnels()`. Manages direct TCP client tunnels (not HTTP-proxied). |
| `index.js` | Assembles all sub-modules, injects shared `saveState` and `emit` callbacks, exposes `restorePersistedState()` and `cleanup()`, and re-exports the complete original `holesail-manager.js` API. |

### Extension (`extension/`)

#### Top-level files

| File | Purpose |
|------|---------|
| `manifest.json` | Manifest V3. Permissions: `nativeMessaging`, `proxy`, `declarativeNetRequest`, `tabs`, `notifications`. Optional host permissions requested at runtime for custom TLDs. |
| `background.js` | Service worker entry point. Declares constants (`DEBUG_VERBOSE`, `browser` shim), then loads all background modules via `importScripts()` in dependency order, applies the initial PAC script, and connects to the native host. |
| `content.js` | Minimal content script. Relays `holesail-host-disconnect` to the page as a `CustomEvent`. |
| `wrong-domain.html` | Error page for `*.host.test` (common typo), redirected via `declarativeNetRequest`. |

#### `background/` — service worker modules

| File | Purpose |
|------|---------|
| `logs.js` | In-memory log ring buffer (500 entries), `log()`/`debugLog()` helpers, `broadcastLogs()` to open dashboard tabs, `dashboardTabs` set. |
| `state.js` | All shared mutable state: `extensionState`, `activeConnections`, `tabSwarms`, `swarmRefCount`, `pacConfirmedActive`, `notifyOnDisconnect`, port defaults. |
| `proxy.js` | `applyPAC()` — builds and installs the PAC script; `clearProxy()`; `getActiveTlds()` helper; `proxy.settings.onChange` listener to re-apply if overridden. |
| `native-messaging.js` | `connect()`, `send()`, `scheduleReconnect()`, `retryGetStateForConnectProxy()`. Owns the `port` reference, `pending` map, `subscribedTabs` set, and all `port.onMessage`/`port.onDisconnect` logic. |
| `tab-lifecycle.js` | `tabs.onRemoved` listener — decrements swarm ref counts, destroys swarms when their last tab closes, cleans up `subscribedTabs` and `dashboardTabs`. |
| `message-router.js` | `runtime.onMessage` dispatcher — handles `registerSwarm`, `send`, `subscribe`/`unsubscribe`, `registerDashboard`/`unregisterDashboard`, and `getState` actions. |

#### `dashboard/` — management UI

| File | Purpose |
|------|---------|
| `dashboard.html` | HTML shell + ordered `<script>` tags. Links `dashboard.css` and vendor scripts. No inline styles or logic. |
| `dashboard.css` | All dashboard styles (~900 lines), extracted from the original inline `<style>` block. |
| `refresh.js` | Top-level `refresh()` orchestrator — fetches state from background, syncs SSH/RDP connections and settings, calls all page `update*` functions. |
| `events.js` | `setupEvents()` — wires toggle switches, settings save/reset, and calls each page's `setup*Events()` function. |

**`dashboard/core/`**

| File | Purpose |
|------|---------|
| `utils.js` | `$()`, `log()`, `timeAgo()`, `formatUptime()`, `truncate()`, `escapeHtml()` |
| `state.js` | All dashboard-level state variables: `currentState`, `settings`, `SETTINGS_DEFAULTS` |
| `messaging.js` | `sendToNative()` and `fetchState()` — wraps `chrome.runtime.sendMessage` |
| `navigation.js` | `PAGE_TITLES`, `navigateTo()`, `setupNavigation()` |
| `init.js` | Entry point. Dashboard context guard, manifest version display, calls all `setup*` functions, starts polling interval. |

**`dashboard/data/`**

| File | Purpose |
|------|---------|
| `tlds.js` | `REAL_TLDS` and `REAL_SLD_TLDS` sets (~300 lines of data) |
| `hostname-validator.js` | `isValidVhostHostname()`, `extractBaseDomain()`, `extractActiveTlds()` |

**`dashboard/ui/`**

| File | Purpose |
|------|---------|
| `toast.js` | `showToast()`, `copyToClipboard()` |
| `modal.js` | `openModal()`, `closeModal()`, `showModalError()`, global close/escape handlers |
| `state-tag.js` | `stateTag()` — renders coloured state badge HTML |

**`dashboard/pages/`**

| File | Purpose |
|------|---------|
| `overview.js` | `OvList` class, `initOvLists()`, `updateDashboard()`, quick action button handlers |
| `virtual-hosts.js` | `updateConnectionsTable()`, `setupVirtualHostEvents()` |
| `servers.js` | `updateSwarmsTable()`, `setupServerEvents()` |
| `service-tunnels.js` | `updateServiceTunnelsTable()`, `setupServiceTunnelEvents()` |
| `proxy-ca.js` | `updateTabsTable()`, cert validator, `setupCertValidator()`, CA install logic |
| `backups.js` | `updateBackupsTable()`, `refreshBackups()`, `setupBackupEvents()` |
| `settings.js` | `updateSettingsUI()`, `saveSettings()` |
| `ssh.js` | SSH state, `renderSshGrid()`, `connectSsh()`, xterm.js lifecycle, `setupSshEvents()` |
| `rdp.js` | RDP/VNC state, `renderRdpGrid()`, `initVncViewer()`, `initRdpViewer()`, `setupRdpEvents()` |
| `logs.js` | Log buffer, `updateLogsDisplay()`, filter/auto-scroll, `setupLogsEvents()` |

## Proxy architecture

The browser cannot connect directly to a custom HTTPS server via a proxy — it sends a `CONNECT` request instead. This requires two proxy layers:

```
Browser navigates to https://myapp.hole.sail/
        │
        │  PAC script (applied by background/proxy.js):
        │    *.hole.sail      →  PROXY 127.0.0.1:8442
        │    *.custom.tld     →  PROXY 127.0.0.1:8442  (custom TLDs)
        │    everything else  →  DIRECT
        ▼
CONNECT proxy  (connect-proxy.js)  127.0.0.1:8442
        │  Browser sends:  CONNECT myapp.hole.sail:443 HTTP/1.1
        │  Proxy replies:  HTTP/1.1 200 Connection established
        │  Pipes raw TCP stream to 127.0.0.1:8443
        ▼
HTTPS proxy  (https-proxy.js)  127.0.0.1:8443
        │  Peeks TLS ClientHello → extracts SNI: "myapp.hole.sail"
        │  Derives wildcard parent: "hole.sail"
        │  Selects cert: wildcard.hole.sail/ (*.hole.sail)
        │  TLS handshake with SNI-matched cert
        │  Reads Host header: myapp.hole.sail
        │  Calls holesailManager.getLocalBackend('myapp.hole.sail')
        │    → { host: '127.0.0.1', port: 19042 }
        │  Proxies HTTP request to 127.0.0.1:19042
        ▼
Holesail client tunnel  (holesail-manager.js)  127.0.0.1:19042
        │  P2P connection via Noise protocol over Holesail DHT
        ▼
Remote peer (HTTP server)
```

### Deep hostname example

```
Browser navigates to https://i.love.hole.sail/
        │
        │  PAC: dnsDomainIs(host, ".hole.sail") → PROXY 127.0.0.1:8442
        ▼
CONNECT proxy  127.0.0.1:8442
        │  CONNECT i.love.hole.sail:443
        ▼
HTTPS proxy  127.0.0.1:8443
        │  SNI: "i.love.hole.sail"
        │  Wildcard parent: "love.hole.sail"
        │  Cert: wildcard.love.hole.sail/ (*.love.hole.sail)
        │  Host header → backend lookup → proxy
        ▼
Holesail tunnel → remote peer
```

### SNI implementation

Because `bare-tls` does not expose `SSL_CTX_set_tlsext_servername_callback`, SNI is implemented entirely in JavaScript:

1. Raw TCP connections are accepted via `bare-tcp`
2. The first data chunk (TLS ClientHello) is read and parsed with a pure-JS TLS record parser (RFC 5246 extension type `0x0000`)
3. The SNI hostname is extracted; its wildcard parent is derived by stripping the leftmost label
4. `certificate-authority.getOrCreateWildcardCert(parent)` returns (or generates) a wildcard cert for that parent
5. A `bare-tls.Socket` is created with that cert; the already-read ClientHello bytes are replayed into it so the handshake proceeds normally
6. A `bare-http1.ServerConnection` wraps the TLS socket for HTTP parsing

Each unique wildcard parent gets its own cert directory under `holesail-browser-certs/`, generated on first connection and cached for subsequent ones.

The PAC script is applied immediately on browser startup using the default port (8442), then updated once the native host reports its actual configured port. It is re-applied if overridden by another extension.

## Custom TLDs

Virtual hosts can use any private TLD — not just `.hole.sail`. The dashboard validates that the hostname:

- Has at least 3 labels (e.g. `app.hole.sail`, not `hole.sail`)
- Uses only letters, digits, and hyphens per label
- Does not use a real public TLD (`.com`, `.net`, `.co.uk`, etc.)

When a virtual host with a new TLD is added:

1. The PAC script is updated to include `dnsDomainIs(host, ".new.tld")` — traffic is routed through the CONNECT proxy
2. The browser requests `optional_host_permissions` for `*://*.new.tld/*` at runtime
3. On the first HTTPS connection to any hostname under that TLD, the HTTPS proxy generates a wildcard cert for the exact wildcard parent on demand

No proxy restart is needed when adding new TLDs.

## Native messaging protocol

All messages are JSON objects framed with a 4-byte little-endian length prefix (Chrome native messaging format). The extension sends requests; the native host sends responses and events.

**Request:**
```json
{ "id": 42, "type": "setVirtualHost", "payload": { "hostname": "myapp.hole.sail", "hsUrl": "hs://abc..." } }
```

**Response:**
```json
{ "id": 42, "type": "response", "payload": { "ok": true, "hostname": "myapp.hole.sail", "localPort": 19000 } }
```

**Event (unsolicited):**
```json
{ "type": "event", "event": "tunnelReady", "payload": { "hostname": "myapp.hole.sail", "localPort": 19000 } }
```

See [NATIVE-HOST.md](NATIVE-HOST.md) for the full message type reference.

## Port allocation

| Port | Component | Description |
|------|-----------|-------------|
| 8443 | `https-proxy.js` | HTTPS proxy (SNI-aware TLS termination) |
| 8442 | `connect-proxy.js` | CONNECT proxy (PAC target) |
| 19000+ | `holesail-manager.js` | Virtual host and service tunnel client ports |
| 20000+ | `ssh-manager.js` | SSH Holesail client tunnel ports |
| 21000+ | `ssh-manager.js` | SSH WebSocket server ports |
| 22000+ | `rdp-manager.js` | RDP/VNC Holesail client tunnel ports |
| 23000+ | `rdp-manager.js` | RDP/VNC WebSocket server ports |

All ports are bound to `127.0.0.1` only.

## State persistence

All state is owned by the native host and persisted to `state.json` next to the binary:

```
~/.holesail-browser/
├── holesail-browser-host           # binary
├── holesail-browser.log            # log file
├── holesail-browser-certs/
│   ├── ca.key.pem                  # root CA private key
│   ├── ca.cert.pem                 # root CA certificate
│   ├── wildcard.hole.sail/         # cert for *.hole.sail (default TLD)
│   │   ├── key.pem
│   │   └── cert.pem                # leaf cert + CA chain
│   ├── wildcard.haha.wooo/         # cert for *.haha.wooo (custom TLD)
│   │   ├── key.pem
│   │   └── cert.pem
│   └── wildcard.love.hole.sail/    # cert for *.love.hole.sail (deep hostname)
│       ├── key.pem
│       └── cert.pem
└── holesail-browser-storage/
    ├── state.json                  # all persistent state
    └── backups/                    # tar.gz backup archives
```

`state.json` schema (version 2):

```json
{
  "version": 2,
  "settings": {
    "proxyPort": 8443,
    "connectProxyPort": 8442,
    "readyTimeoutMs": 0,
    "notifyOnDisconnect": true,
    "debug": false,
    "disableOnFileUrls": false,
    "backupRetention": 5
  },
  "nextServerId": 0,
  "nextServiceTunnelId": 0,
  "servers": [
    { "id": "server_1", "port": 3000, "host": "127.0.0.1", "secure": true, "udp": false, "label": "My Web App" }
  ],
  "virtualHosts": [
    { "hostname": "myapp.hole.sail", "hsUrl": "hs://abc123..." },
    { "hostname": "i.love.hole.sail", "hsUrl": "hs://def456..." },
    { "hostname": "api.haha.wooo", "hsUrl": "hs://ghi789..." }
  ],
  "serviceTunnels": [
    { "id": "svc-1", "label": "Postgres", "hsUrl": "hs://def456...", "localPort": 5432 }
  ],
  "sshConnections": [
    { "id": "ssh-abc123", "label": "My Server", "hsUrl": "hs://xyz...", "username": "root", "passwordB64": "cGFzc3dvcmQ=" }
  ],
  "rdpConnections": [
    { "id": "rdp-abc123", "label": "Work PC", "hsUrl": "hs://xyz...", "type": "vnc", "port": 5901, "width": 1280, "height": 720, "username": "", "passwordB64": "cGFzc3dvcmQ=" }
  ]
}
```

## Native host lifecycle

Chrome spawns the native host process when the background service worker first calls `connectNative`. The process exits when Chrome disconnects (e.g. browser closed, service worker killed). On next connection, Chrome spawns a fresh process which restores all tunnels from `state.json`.

If port 8443 or 8442 is already in use when the native host starts, it exits immediately — this prevents ghost instances with no tunnel state from accumulating.

## Build system

The native host is built with the [Bare](https://github.com/holepunchto/bare) runtime:

1. `bare-pack` bundles the JS module graph (resolving `node:*` imports via `bare-node-*` shims)
2. `bare-build` embeds the bundle into a pre-built Bare runtime binary, producing a self-contained executable
3. Native addons (`.bare` files) are embedded and extracted to a content-addressed temp directory at runtime

Targets: `darwin-arm64`, `darwin-x64`, `linux-arm64`, `linux-x64`, `win32-x64`.