174 lines
10 KiB
Markdown
174 lines
10 KiB
Markdown
# Agent (grok-class harness)
|
|
|
|
`BridgeSwarm.agent` is a Bare JS agent loop modeled on Grok Build: sample → tool calls → results → repeat. Inference is QVAC (`tools: true`). Multimodal models can **see** attached stills and a live desktop ring; image/video **generation** and click/keyboard computer-use tools are not implemented. The agent requires **Settings → Enable QVAC**; `create` / `prompt` fail until that toggle is on.
|
|
|
|
## Embed
|
|
|
|
```js
|
|
const session = await BridgeSwarm.agent.create({
|
|
model: 'qwen3.5-4b',
|
|
cwd: 'default',
|
|
permissionMode: 'ask' // ask | allowlist | always-approve
|
|
})
|
|
session.on((ev) => { /* agent_message_chunk | tool_call | permission | ask_user | plan_approval | end */ })
|
|
await session.prompt('List the workspace and summarize', {
|
|
planMode: false,
|
|
permissionMode: 'ask',
|
|
})
|
|
await session.prompt('What is in this screenshot?', {
|
|
images: [{ dataUrl: canvas.toDataURL('image/png') }],
|
|
})
|
|
await session.cancel()
|
|
await session.dispose()
|
|
```
|
|
|
|
Always-approve is only honored for origins listed under **Settings → Agent always-approve**.
|
|
|
|
Multimodal catalog models (Qwen 3.5/3.6, Gemma 4) load a matching mmproj (Q8_0 or F16). If a vision GGUF is already in memory without a projector, the next `prompt` reloads it with mmproj. `prompt(..., { images })` accepts data URLs or `{ dataBase64, mime }` (jpeg/png/webp/gif, max 4). A separate capture app can POST stills to `http://127.0.0.1:11435/v1/vision/frames` (up to 5 FPS). **HTTPS pages (Pip) cannot hit that loopback** — use `BridgeSwarm.qvac.pushDesktopFrame({ dataUrl })` over native messaging instead. Each agent turn sees the latest frame unless `prompt(..., { desktopVision: false })`. Stream frames are not written to session JSONL. The host writes images under `$BRIDGE_SWARM_STORAGE/qvac/vision/` and passes QVAC `attachments: [{ path }]`. Image/video **generation** is still not implemented. Text-only models (`gpt-oss-20b`, `qwen3-*` INST) refuse images.
|
|
|
|
## Tools
|
|
|
|
Must-have: `read_file`, `write_file`, `search_replace`, `grep`, `list_dir`, `run_terminal_cmd`, `todo_write`
|
|
|
|
- `search_replace` requires a unique `old_string` unless `replace_all` is true. An empty `old_string` creates a file only if it is missing or empty. The result includes ±3 lines of context.
|
|
- `grep` prefers `rg` when present (glob + `output_mode`: `content` | `files_with_matches` | `count`) and falls back to a JS walker.
|
|
- Independent tools in one model turn run in parallel. `write_file` / `search_replace` that share a path are serialized. `ask_user_question`, `exit_plan_mode`, `update_goal`, `enter_plan_mode`, and `task` stay sequential.
|
|
- Long tool results and `web_fetch` are truncated head+tail with a recovery marker.
|
|
|
|
Also: `task` / `send_subagent_message` / `get_task_output` / `wait_tasks` / `kill_task` (subagents share the loaded model; `wait_tasks` actually waits), `web_search`, `web_fetch` (opt-in), `memory_search` / `memory_get` / `memory_write`, `enter_plan_mode` / `exit_plan_mode`, `ask_user_question`, `update_goal`, `search_tool` / `use_tool` (MCP HTTP: `initialize` + `tools/list` on register, then `tools/call`; stdio requires a trusted origin and is not spawned by default)
|
|
|
|
Shell is cwd-jailed via `bare-subprocess` (host-internal, not a page pack). MCP HTTP tools must be public URLs (`net` policy). Handshake failures are surfaced once as a system reminder.
|
|
|
|
## Permissions
|
|
|
|
`permissionMode: 'ask'` (default) prompts the page. Agent Studio offers **Allow**, **Always allow** (persist this command/path prefix next to sessions), and **Deny**. Remembered `git status` / `git diff` prefixes skip the prompt without turning on full-shell YOLO. `session.permit(jobId, toolCallId, true | false | 'always')`.
|
|
|
|
## Subagents
|
|
|
|
`task({ prompt, subagent_type })` with `explore` (read-only, default) or `general` (same write gate as the parent). At most **2** concurrent child tasks (one GPU). The child history is stored on the task record; `send_subagent_message` continues that inner loop (up to 8 turns). `task` still **blocks the parent** until the child finishes.
|
|
|
|
## Custom tools from the page
|
|
|
|
Handlers stay in the tab (they are never sent to the host). The host only stores the JSON schema and, when the model calls the tool, waits for `toolResult`.
|
|
|
|
```js
|
|
const session = await BridgeSwarm.agent.create({
|
|
model: 'qwen3.5-4b',
|
|
tools: [
|
|
{
|
|
name: 'get_cart',
|
|
description: 'Return the in-page shopping cart',
|
|
parameters: { type: 'object', properties: {} },
|
|
execute: () => window.cart,
|
|
},
|
|
],
|
|
})
|
|
await session.addTool({
|
|
name: 'highlight',
|
|
description: 'Highlight a line in the page editor',
|
|
parameters: { type: 'object', properties: { line: { type: 'number' } }, required: ['line'] },
|
|
execute: (args) => { editor.highlight(args.line); return 'ok'; },
|
|
})
|
|
await session.prompt('Add a README.md then highlight line 1')
|
|
```
|
|
|
|
Built-in names (`read_file`, `write_file`, …) cannot be overwritten **while host workspace tools are on**. Max 32 custom tools per session. Events include `tool_request` when the page must run a handler.
|
|
|
|
## Disable host workspace (page / container agents)
|
|
|
|
By default the agent ships host-jail tools against `$BRIDGE_SWARM_STORAGE/agent/<origin-hash>/`. Embedders that own their own filesystem (a container, a panel, an in-page editor) should turn that off so the model never sees or shells the native host:
|
|
|
|
```js
|
|
const session = await BridgeSwarm.agent.create({
|
|
model: 'qwen3.5-4b',
|
|
hostWorkspace: false, // or hostTools: false
|
|
workspace: 'dlinux-container', // prompt label only — not a host path
|
|
builtinTools: false, // optional: drop todo/task/web/MCP builtins too
|
|
tools: [
|
|
{
|
|
name: 'read_file',
|
|
description: 'Read a file inside the container',
|
|
parameters: { type: 'object', properties: { path: { type: 'string' } }, required: ['path'] },
|
|
execute: (args) => container.read(args.path),
|
|
},
|
|
{
|
|
name: 'run_terminal_cmd',
|
|
description: 'Run a command in the container',
|
|
parameters: { type: 'object', properties: { command: { type: 'string' } }, required: ['command'] },
|
|
execute: (args) => container.exec(args.command),
|
|
},
|
|
],
|
|
})
|
|
```
|
|
|
|
With `hostWorkspace: false`:
|
|
|
|
- Host `read_file` / `write_file` / `search_replace` / `grep` / `list_dir` / `run_terminal_cmd` / `memory_*` are **not** offered and will not execute on the host.
|
|
- Those names may be registered as page tools (handlers stay in the tab).
|
|
- `cwd` / `workspace` is a label for the system prompt, not a path under `$BRIDGE_SWARM_STORAGE`.
|
|
- Subagents do not get host filesystem tools.
|
|
- Session JSONL still lives under `$BRIDGE_SWARM_STORAGE/agent/sessions/` (harness metadata only).
|
|
|
|
`builtinTools: false` removes remaining host-executed builtins (`todo_write`, `task`, `web_search`, MCP, …). Pass an array of names to keep a subset.
|
|
|
|
Pages cannot grant extra host roots via `create` — extra absolute roots still come only from **Settings → Agent workspace roots**.
|
|
|
|
## Sandbox
|
|
|
|
Default cwd: `$BRIDGE_SWARM_STORAGE/agent/<origin-hash>/`. Extra absolute roots: Settings → Agent workspace roots (pushed to the host as grants).
|
|
|
|
## Sessions
|
|
|
|
JSONL under `$BRIDGE_SWARM_STORAGE/agent/sessions/`. History is compacted near **85%** of the model context with one tool-free summary complete (five sections: goal, done, files, open work, next). Degenerate summaries fall back to drop-oldest. Compacted history is rewritten so a reload does not restore the uncompacted log. After compact, the loop injects reminders (todos, plan mode, AGENTS.md) and a short `[memory]` keyword block. Each prompt also refreshes the system message (AGENTS.md) plus a cached `git status -sb` sidecar (~30s) when the cwd is a git work tree. `plan.md`, plan-mode snapshot, and goal status live on the session summary. Allow/deny patterns persist in `$BRIDGE_SWARM_STORAGE/agent/permission-rules.json`.
|
|
|
|
Call `session.dispose()` (or `BridgeSwarm.agent.destroy(sessionId)`) when the page is done — that deletes the session dir, drops page-tool schemas, and schedules a GPU unload if nothing else is running. The loaded model is also released after **5 minutes idle** (not mid-turn: permission / ask-user / generation hold VRAM until the turn ends). The next `prompt` or `qvac.load` brings the same model back.
|
|
|
|
Page-tool schemas are stored on the session summary so a later `load` / `prompt` restores them without stacking a second model.
|
|
|
|
## ACP-shaped events
|
|
|
|
Host emits `cap-chunk` with `pack: 'agent'` and `type` in `agent_message_chunk`, `agent_thought_chunk`, `tool_call`, `tool_result`, `permission`, `ask_user`, `plan_approval`, `plan_update`, `goal_update`, `end`.
|
|
|
|
`end.reason` is one of `stop` | `max_turns` | `stuck` | `cancelled` | `goal_complete` | `goal_blocked`.
|
|
|
|
## Plan mode
|
|
|
|
`enter_plan_mode` / `session.prompt(text, { planMode: true })` switches the session to plan mode. While active:
|
|
|
|
- `run_terminal_cmd` is not offered; `write_file` / `search_replace` may only edit the session `plan.md` (host session dir, streamed as `plan_update`).
|
|
- Other writes are rejected at execute time, including page-registered `write_file` / `search_replace` / `run_terminal_cmd`.
|
|
- The model is reminded to end the turn with `ask_user_question` or `exit_plan_mode`.
|
|
|
|
`exit_plan_mode` emits `plan_approval` and **blocks** until the page calls `session.planDecision('approve' | 'reject')`. Approve injects “plan approved, implement it”; reject stays in plan mode.
|
|
|
|
```js
|
|
session.on((ev) => {
|
|
if (ev.type === 'plan_approval') {
|
|
session.planDecision('approve')
|
|
}
|
|
})
|
|
await session.prompt('Plan a README then implement it', { planMode: true })
|
|
```
|
|
|
|
## Ask the user
|
|
|
|
`ask_user_question` emits `ask_user` and **waits**. Reply with `session.answer(toolCallId, choice)`.
|
|
|
|
```js
|
|
session.on((ev) => {
|
|
if (ev.type === 'ask_user') session.answer(ev.toolCallId, ev.options[0])
|
|
})
|
|
```
|
|
|
|
## Goals
|
|
|
|
Done is **not** the model stopping tool calls. Start a goal with `create({ goal: '…' })` or `prompt(text, { goal: '…' })`. The agent must `update_goal({ completed: true })` after todos are finished. Open todos reject that call (premature stop). A single verifier `complete` then either ends with `reason: 'goal_complete'` or injects gaps and continues. `update_goal({ blocked_reason })` ends with `goal_blocked`. Pass `{ verify: false }` to skip the verifier.
|
|
|
|
If a goal is active and the model stops with open work, the loop injects a continuation instead of `end stop`.
|
|
|
|
## Prompt options
|
|
|
|
`session.prompt(text, { planMode, permissionMode, system, webFetch, goal, verify })` forwards those fields to the host.
|
|
|
|
Models, devices, enable toggle, and `BridgeSwarm.qvac` chat (page-owned tools, no workspace shell) are documented in [QVAC.md](QVAC.md).
|