BridgeSwarm/docs/AGENT.md

Agent (grok-class harness)

BridgeSwarm.agent is a Bare JS agent loop modeled on Grok Build: sample → tool calls → results → repeat. Inference is QVAC (tools: true). Multimodal models can see attached stills and a live desktop ring; image/video generation and click/keyboard computer-use tools are not implemented. The agent requires Settings → Enable QVAC; create / prompt fail until that toggle is on.

Embed

const session = await BridgeSwarm.agent.create({
  model: 'qwen3.5-4b',
  cwd: 'default',
  permissionMode: 'ask' // ask | allowlist | always-approve
})
session.on((ev) => { /* agent_message_chunk | tool_call | permission | ask_user | plan_approval | end */ })
await session.prompt('List the workspace and summarize', {
  planMode: false,
  permissionMode: 'ask',
})
await session.prompt('What is in this screenshot?', {
  images: [{ dataUrl: canvas.toDataURL('image/png') }],
})
await session.cancel()
await session.dispose()

Always-approve is only honored for origins listed under Settings → Agent always-approve.

Multimodal catalog models (Qwen 3.5/3.6, Gemma 4) load a matching mmproj (Q8_0 or F16). If a vision GGUF is already in memory without a projector, the next prompt reloads it with mmproj. prompt(..., { images }) accepts data URLs or { dataBase64, mime } (jpeg/png/webp/gif, max 4). A separate capture app can POST stills to http://127.0.0.1:11435/v1/vision/frames (up to 5 FPS). HTTPS pages (Pip) cannot hit that loopback — use BridgeSwarm.qvac.pushDesktopFrame({ dataUrl }) over native messaging instead. Each agent turn sees the latest frame unless prompt(..., { desktopVision: false }). Stream frames are not written to session JSONL. The host writes images under $BRIDGE_SWARM_STORAGE/qvac/vision/ and passes QVAC attachments: [{ path }]. Image/video generation is still not implemented. Text-only models (gpt-oss-20b, qwen3-* INST) refuse images.

Tools

Must-have: read_file, write_file, search_replace, grep, list_dir, run_terminal_cmd, todo_write

• search_replace requires a unique old_string unless replace_all is true. An empty old_string creates a file only if it is missing or empty. The result includes ±3 lines of context.
• grep prefers rg when present (glob + output_mode: content | files_with_matches | count) and falls back to a JS walker.
• Independent tools in one model turn run in parallel. write_file / search_replace that share a path are serialized. ask_user_question, exit_plan_mode, update_goal, enter_plan_mode, and task stay sequential.
• Long tool results and web_fetch are truncated head+tail with a recovery marker.

Also: task / send_subagent_message / get_task_output / wait_tasks / kill_task (subagents share the loaded model; wait_tasks actually waits), web_search, web_fetch (opt-in), memory_search / memory_get / memory_write, enter_plan_mode / exit_plan_mode, ask_user_question, update_goal, search_tool / use_tool (MCP HTTP: initialize + tools/list on register, then tools/call; stdio requires a trusted origin and is not spawned by default)

Shell is cwd-jailed via bare-subprocess (host-internal, not a page pack). MCP HTTP tools must be public URLs (net policy). Handshake failures are surfaced once as a system reminder.

Permissions

permissionMode: 'ask' (default) prompts the page. Agent Studio offers Allow, Always allow (persist this command/path prefix next to sessions), and Deny. Remembered git status / git diff prefixes skip the prompt without turning on full-shell YOLO. session.permit(jobId, toolCallId, true | false | 'always').

Subagents

task({ prompt, subagent_type }) with explore (read-only, default) or general (same write gate as the parent). At most 2 concurrent child tasks (one GPU). The child history is stored on the task record; send_subagent_message continues that inner loop (up to 8 turns). task still blocks the parent until the child finishes.

Custom tools from the page

Handlers stay in the tab (they are never sent to the host). The host only stores the JSON schema and, when the model calls the tool, waits for toolResult.

const session = await BridgeSwarm.agent.create({
  model: 'qwen3.5-4b',
  tools: [
    {
      name: 'get_cart',
      description: 'Return the in-page shopping cart',
      parameters: { type: 'object', properties: {} },
      execute: () => window.cart,
    },
  ],
})
await session.addTool({
  name: 'highlight',
  description: 'Highlight a line in the page editor',
  parameters: { type: 'object', properties: { line: { type: 'number' } }, required: ['line'] },
  execute: (args) => { editor.highlight(args.line); return 'ok'; },
})
await session.prompt('Add a README.md then highlight line 1')

Built-in names (read_file, write_file, …) cannot be overwritten while host workspace tools are on. Max 32 custom tools per session. Events include tool_request when the page must run a handler.

Disable host workspace (page / container agents)

By default the agent ships host-jail tools against $BRIDGE_SWARM_STORAGE/agent/<origin-hash>/. Embedders that own their own filesystem (a container, a panel, an in-page editor) should turn that off so the model never sees or shells the native host:

const session = await BridgeSwarm.agent.create({
  model: 'qwen3.5-4b',
  hostWorkspace: false,          // or hostTools: false
  workspace: 'dlinux-container', // prompt label only — not a host path
  builtinTools: false,           // optional: drop todo/task/web/MCP builtins too
  tools: [
    {
      name: 'read_file',
      description: 'Read a file inside the container',
      parameters: { type: 'object', properties: { path: { type: 'string' } }, required: ['path'] },
      execute: (args) => container.read(args.path),
    },
    {
      name: 'run_terminal_cmd',
      description: 'Run a command in the container',
      parameters: { type: 'object', properties: { command: { type: 'string' } }, required: ['command'] },
      execute: (args) => container.exec(args.command),
    },
  ],
})

With hostWorkspace: false:

• Host read_file / write_file / search_replace / grep / list_dir / run_terminal_cmd / memory_* are not offered and will not execute on the host.
• Those names may be registered as page tools (handlers stay in the tab).
• cwd / workspace is a label for the system prompt, not a path under $BRIDGE_SWARM_STORAGE.
• Subagents do not get host filesystem tools.
• Session JSONL still lives under $BRIDGE_SWARM_STORAGE/agent/sessions/ (harness metadata only).

builtinTools: false removes remaining host-executed builtins (todo_write, task, web_search, MCP, …). Pass an array of names to keep a subset.

Pages cannot grant extra host roots via create — extra absolute roots still come only from Settings → Agent workspace roots.

Sandbox

Default cwd: $BRIDGE_SWARM_STORAGE/agent/<origin-hash>/. Extra absolute roots: Settings → Agent workspace roots (pushed to the host as grants).

Sessions

JSONL under $BRIDGE_SWARM_STORAGE/agent/sessions/. History is compacted near 85% of the model context with one tool-free summary complete (five sections: goal, done, files, open work, next). Degenerate summaries fall back to drop-oldest. Compacted history is rewritten so a reload does not restore the uncompacted log. After compact, the loop injects reminders (todos, plan mode, AGENTS.md) and a short [memory] keyword block. Each prompt also refreshes the system message (AGENTS.md) plus a cached git status -sb sidecar (~30s) when the cwd is a git work tree. plan.md, plan-mode snapshot, and goal status live on the session summary. Allow/deny patterns persist in $BRIDGE_SWARM_STORAGE/agent/permission-rules.json.

Call session.dispose() (or BridgeSwarm.agent.destroy(sessionId)) when the page is done — that deletes the session dir, drops page-tool schemas, and schedules a GPU unload if nothing else is running. The loaded model is also released after 5 minutes idle (not mid-turn: permission / ask-user / generation hold VRAM until the turn ends). The next prompt or qvac.load brings the same model back.

Page-tool schemas are stored on the session summary so a later load / prompt restores them without stacking a second model.

ACP-shaped events

Host emits cap-chunk with pack: 'agent' and type in agent_message_chunk, agent_thought_chunk, tool_call, tool_result, permission, ask_user, plan_approval, plan_update, goal_update, end.

end.reason is one of stop | max_turns | stuck | cancelled | goal_complete | goal_blocked.

Plan mode

enter_plan_mode / session.prompt(text, { planMode: true }) switches the session to plan mode. While active:

• run_terminal_cmd is not offered; write_file / search_replace may only edit the session plan.md (host session dir, streamed as plan_update).
• Other writes are rejected at execute time, including page-registered write_file / search_replace / run_terminal_cmd.
• The model is reminded to end the turn with ask_user_question or exit_plan_mode.

exit_plan_mode emits plan_approval and blocks until the page calls session.planDecision('approve' | 'reject'). Approve injects “plan approved, implement it”; reject stays in plan mode.

session.on((ev) => {
  if (ev.type === 'plan_approval') {
    session.planDecision('approve')
  }
})
await session.prompt('Plan a README then implement it', { planMode: true })

Ask the user

ask_user_question emits ask_user and waits. Reply with session.answer(toolCallId, choice).

session.on((ev) => {
  if (ev.type === 'ask_user') session.answer(ev.toolCallId, ev.options[0])
})

Goals

Done is not the model stopping tool calls. Start a goal with create({ goal: '…' }) or prompt(text, { goal: '…' }). The agent must update_goal({ completed: true }) after todos are finished. Open todos reject that call (premature stop). A single verifier complete then either ends with reason: 'goal_complete' or injects gaps and continues. update_goal({ blocked_reason }) ends with goal_blocked. Pass { verify: false } to skip the verifier.

If a goal is active and the model stops with open work, the loop injects a continuation instead of end stop.

Prompt options

session.prompt(text, { planMode, permissionMode, system, webFetch, goal, verify }) forwards those fields to the host.

Models, devices, enable toggle, and BridgeSwarm.qvac chat (page-owned tools, no workspace shell) are documented in QVAC.md.