obsidian

vendor/agent-harness/agent-workspace/AGENTS.md

@@ -41,7 +41,7 @@ Workspace skills live in `skills/<name>/SKILL.md`. When a request matches a skil
 - `read_file` / `list_dir` / `grep` / `write_file` / `search_replace` — workspace files.
 - `run_terminal_cmd` — local shell. Public HTTP via curl or wget is blocked; use `browser`.
 - The only web tool is `browser`, the headed Jarvis Chromium window. `web_search`, `web_fetch`, and the other search tools are removed. Before a multi-step browse, `read_file` `skills/browser/SKILL.md`.
-- `browser` actions: `navigate` (needs `url`; returns page text), `snapshot`, `click` (`ref` from the last snapshot), `type` (`ref` + `text`, optional `submit`), `press` (`key`), `scroll` (`dy`), `wait` (`ms`). To search, navigate to a public search url, then click a result ref. Snapshot or navigate first. Refs change after every click. Do not use `cu_observe` or the shell for websites.
+- `browser` actions: `navigate` (needs `url`; returns page text), `snapshot`, `click` (`ref` from the last snapshot), `type` (`ref` + `text`, optional `submit`), `press` (`key`), `scroll` (`dy`), `wait` (`ms`). To search, navigate to DuckDuckGo (`https://duckduckgo.com/?q=QUERY`), then click a result ref. Snapshot or navigate first. Refs change after every click. Do not use `cu_observe` or the shell for websites.
 - Desktop and computer-use tools are registered by Jarvis. After Settings → Computer use → Allow now, call `cu_observe`, then `cu_click` / `cu_type`. Do not paste tool JSON into chat.
 - Webcam: Settings → Camera → Allow now. Call `webcam`. The still is attached; do not speak the file path. This is not `cu_observe`.
 - `ask_user_question` — wait for a user choice.

vendor/agent-harness/agent-workspace/TOOLS.md

@@ -21,7 +21,7 @@ Local notes for this Jarvis session. This file is guidance, not an allowlist.
 
 - `browser` is the only web tool. It drives one headed Playwright Chromium window.
 - `web_search`, `google_search`, `fetch_page`, `web_fetch`, `wiki_search`, `hn_search`, and `code_search` are removed. Do not call them.
-- To search, `navigate` to `https://duckduckgo.com/?q=QUERY` or a Google search url, then `click` a result ref.
+- To search, `navigate` to `https://duckduckgo.com/?q=QUERY`, then `click` a result ref.
 - `navigate` and `snapshot` return page `text`. Read it, then decide the next action.
 - Cookie walls, forms, leftover challenges: `read_file` `skills/browser/SKILL.md` for the playbook.
 - `browser` actions: `navigate` + `url`, `snapshot`, `click`/`type` with `ref` from the last snapshot, `press` + `key`, `scroll` + `dy`, `wait` + `ms`.

vendor/agent-harness/agent-workspace/skills/browser/SKILL.md

@@ -7,7 +7,7 @@ description: Drive the headed Jarvis Chromium window. This is the only web tool.
 
 `browser` is the only way to the public web. `web_search`, `google_search`, `fetch_page`, `web_fetch`, `wiki_search`, `hn_search`, and `code_search` are removed. Do not call them. Do not use `cu_observe`, `cu_click`, curl, or wget for websites.
 
-Chrome runs on this computer. After a navigate or snapshot, read the page `text` and decide the next click. If Groq is enabled, that thinking step uses Groq. The crawl itself stays local.
+Chrome runs on this computer. After a navigate or snapshot, read the page `text` and decide the next click. If Groq is enabled, that thinking step uses Groq. The crawl itself stays local. The window loads uBlock Origin Lite and Dark Reader, so pages are darkened and many ads are already gone. Cookie banners can still appear.
 
 ## `browser` actions
 
@@ -29,12 +29,21 @@ Private, loopback, and metadata hosts are blocked before Chromium starts.
 
 Do not call a search tool. Navigate to a public search url, then click a result:
 
-- Web: `https://duckduckgo.com/?q=QUERY` or `https://www.google.com/search?q=QUERY`
+- Web: `https://duckduckgo.com/?q=QUERY`. The window also opens on `https://duckduckgo.com/`.
 - Wikipedia: `https://en.wikipedia.org/w/index.php?search=QUERY`
 - Hacker News: `https://hn.algolia.com/?q=QUERY`
 - GitHub: `https://github.com/search?q=QUERY&type=repositories`
 - This computer's public IP: `https://ifconfig.me/ip`
 
+## Answer from the page
+
+A `navigate` or `snapshot` result is the page, not a hint. Before you speak:
+
+1. Read `page_text` from start to end. It is the full readable page after Chrome scrolled it, not only the first screen.
+2. Use `headings`, `tables`, and `visible` labels. The images are slices of that same page from top to bottom. They do not replace the text.
+3. If the page is a search or link list, speak a report from the titles and snippets on that page. The browser window stays open. You may open the best result afterward for more detail. Do not stop after thinking.
+4. Draft a complete answer from those facts. Include names, numbers, dates, and the points the page states. Do not stop after one sentence when the page has more.
+
 ## Loop
 
 1. `navigate` or `snapshot` so you have fresh `refs` and page `text`.

vendor/agent-harness/agent/compaction.js

@@ -181,10 +181,22 @@ function rebuildHistory(messages, summary) {
   );
 }
 
-function truncateMsg(m, maxChars) {
+function isBrowserReading(msg) {
+  if (!msg || (msg.role !== 'tool' && msg.role !== 'function')) return false;
+  if (msg.name === 'browser') return true;
+  return String(msg.content || '').includes('\npage_text:\n');
+}
+
+function truncateMsg(m, maxChars, floor) {
   if (!m) return m;
   const copy = Object.assign({}, m);
   const c = copy.content;
+  if (isBrowserReading(copy) && typeof c === 'string') {
+    const cap = Math.max(maxChars, floor || 12000);
+    if (c.length <= cap) return copy;
+    copy.content = c.slice(0, Math.max(0, cap - 80)) + '\n\n[rest of page omitted; answer from the page text above]\n';
+    return copy;
+  }
   if (typeof c === 'string' && c.length > maxChars) {
     copy.content = truncate.truncateWithMarker(c, maxChars);
   } else if (Array.isArray(c)) {
@@ -236,12 +248,14 @@ function heuristicCompact(messages, opts) {
   const assemble = () => systems.concat(summary ? [summary] : [], request ? [request] : [], groups.flat());
   let keep = assemble();
   // Shrink bulky results first; preserve call ids, arguments and ordering.
+  const browserFloor = opts.browserFloor > 0 ? opts.browserFloor : 12000;
   for (const cap of [1600, 600, 180]) {
     if (!overHistoryBudget(keep, budget)) break;
-    groups = groups.map((group) => group.map((m) => truncateMsg(m, cap)));
+    groups = groups.map((group) => group.map((m) => truncateMsg(m, cap, browserFloor)));
     keep = assemble();
   }
   while (overHistoryBudget(keep, budget) && groups.length) {
+    if (groups.length === 1 && groups[0].some(isBrowserReading)) break;
     groups.shift();
     keep = assemble();
   }

vendor/agent-harness/agent/loop.js

@@ -85,6 +85,7 @@ function historyCompactOpts(session, toolDefs, ctxSize, sidecarTokens, budget, a
       voice: !!budget.voice,
       systemTokens: systemTokens(session.history),
     }),
+    browserFloor: browserResultCap(),
     tools: toolDefs,
     voice: !!budget.voice,
   };
@@ -99,6 +100,19 @@ function toolResultCap(budget) {
   return Math.min(max, Math.max(min, Math.floor(ctx * ratio)));
 }
 
+function browserResultCap() {
+  const ctx = loadedCtxSize();
+  if (typeof engine.remoteActive === 'function' && engine.remoteActive()) {
+    return Math.min(64000, Math.max(16000, Math.floor(ctx * 0.35)));
+  }
+  return Math.min(12000, Math.max(6000, Math.floor(ctx * 0.45)));
+}
+
+function resultCap(budget, name) {
+  if (name === 'browser') return browserResultCap();
+  return toolResultCap(budget);
+}
+
 function emitCompactDone(emit, session, jobId, toolDefs, ctxSize, beforeUsage, method) {
   const afterUsage = compaction.usage(session.history, toolDefs, ctxSize);
   emitUpdate(emit, session.id, jobId, {
@@ -290,10 +304,13 @@ function pushHistory(session, msg) {
 
 function pushVisionFollowUp(session, out) {
   if (!out || typeof out !== 'object' || !Array.isArray(out.images) || !out.images.length) return;
+  // Page screenshots stay off the local model. A follow-up user message would
+  // also skip Groq and leave the turn hanging with no spoken reply.
+  if (typeof engine.remoteVision !== 'function' || !engine.remoteVision()) return;
   const [followUp] = engine.prepareVisionHistory([
     {
       role: 'user',
-      content: engine.VISION_FOLLOWUP_QUESTION,
+      content: out.visionQuestion || engine.VISION_FOLLOWUP_QUESTION,
       images: out.images.slice(0, 4),
     },
   ]);
@@ -628,7 +645,7 @@ async function runTurn(ctx) {
     } catch (err) {
       out = { error: err.message };
     }
-    const rendered = truncate.renderToolResult(out, toolResultCap(budget));
+    const rendered = truncate.renderToolResult(out, resultCap(budget, name));
     const toolMsg = { role: 'tool', name, content: rendered, tool_call_id: toolCallId };
     pushHistory(session, toolMsg);
     pushVisionFollowUp(session, out);
@@ -729,8 +746,8 @@ async function runTurn(ctx) {
               webProcess: webTurn,
               toolDialect: webTurn ? 'openai' : catalog.toolDialectFor(session.model),
               desktopVision: payload && payload.desktopVision === false ? false : undefined,
-              timeoutMs: budget.completeTimeoutMs,
-              idleMs: budget.completeIdleMs,
+              timeoutMs: (webTurn || webProcess.needsGroqWebProcess(history)) ? (budget.completeTimeoutMs || 90000) : budget.completeTimeoutMs,
+              idleMs: (webTurn || webProcess.needsGroqWebProcess(history)) ? (budget.completeIdleMs || 45000) : budget.completeIdleMs,
             },
             (ev) => {
               if (ev.type === 'contentDelta') {
@@ -806,6 +823,10 @@ async function runTurn(ctx) {
         if (calls.length) assistant.tool_calls = calls;
         pushHistory(session, assistant);
       }
+      if (!calls.length && result && !String(result.text || '').trim()) {
+        const report = webProcess.pageReport(webProcess.latestBrowserReading(session.history));
+        if (report) result.text = report;
+      }
       if (!calls.length) {
         const goalActive = goalMod.isActive(session.goal);
         if (goalActive && goalNudges < MAX_GOAL_NUDGES && !budget.answerOnly) {
@@ -942,6 +963,14 @@ async function runTurn(ctx) {
     if (err && err.message === 'cancelled') {
       return endTurn(emit, session, jobId, tracker, { reason: 'cancelled', text: lastText });
     }
+    const report = webProcess.pageReport(webProcess.latestBrowserReading(session.history));
+    if (report) {
+      return endTurn(emit, session, jobId, tracker, {
+        reason: 'stop',
+        text: report,
+        turns: lastText ? undefined : 1,
+      });
+    }
     throw err;
   }
 }

vendor/agent-harness/agent/truncate.js

@@ -29,6 +29,9 @@ function renderToolResult(out, maxChars) {
     payload = Object.assign({}, out);
     delete payload.images;
   }
+  if (payload && typeof payload.reading === 'string' && payload.reading) {
+    return truncateWithMarker(payload.reading, maxChars != null ? maxChars : 40000);
+  }
   const raw = typeof payload === 'string' ? payload : JSON.stringify(payload);
   return truncateWithMarker(raw, maxChars != null ? maxChars : 12000);
 }

vendor/agent-harness/lib/qvac.js

@@ -34,6 +34,10 @@ function remoteActive() {
   return remote.provider === 'groq';
 }
 
+function remoteVision() {
+  return remoteActive() && groq.visionModel(remote.model);
+}
+
 function groqLoaded() {
   return {
     modelId: 'groq:' + remote.model,
@@ -664,6 +668,7 @@ module.exports = {
   resources,
   configureRemote,
   remoteActive,
+  remoteVision,
   VISION_FOLLOWUP_QUESTION,
   prepareVisionHistory,
   hold,

vendor/agent-harness/lib/web-process.js

@@ -38,9 +38,14 @@ function isRetiredWeb(name) {
   return RETIRED_WEB.has(canonicalWeb(name));
 }
 
+const { duckduckgoSearchUrl } = require('../../../browser-use/startpage.cjs');
+
 function searchUrl(query, engine) {
   const q = encodeURIComponent(String(query || '').trim());
   const id = fold(engine) || 'duckduckgo';
+  if (id === 'duckduckgo' || id === 'ddg' || id === 'startpage' || id === 'start' || id === 'auto') {
+    return duckduckgoSearchUrl(query);
+  }
   if (id === 'google') return 'https://www.google.com/search?q=' + q;
   if (id === 'bing') return 'https://www.bing.com/search?q=' + q;
   if (id === 'wikipedia' || id === 'wiki') return 'https://en.wikipedia.org/w/index.php?search=' + q;
@@ -50,7 +55,7 @@ function searchUrl(query, engine) {
   if (id === 'mdn') return 'https://developer.mozilla.org/en-US/search?q=' + q;
   if (id === 'stackoverflow') return 'https://stackoverflow.com/search?q=' + q;
   if (id === 'arxiv') return 'https://arxiv.org/search/?query=' + q + '&searchtype=all';
-  return 'https://duckduckgo.com/?q=' + q;
+  return duckduckgoSearchUrl(query);
 }
 
 function asBrowserCall(name, args) {
@@ -60,11 +65,10 @@ function asBrowserCall(name, args) {
   if (canonical === 'web_fetch' || canonical === 'fetch_page') {
     return { name: 'browser', arguments: { action: 'navigate', url: String(input.url || input.href || '') } };
   }
-  const engine = canonical === 'google_search' ? 'google'
-    : canonical === 'wiki_search' ? 'wikipedia'
+  const engine = canonical === 'wiki_search' ? 'wikipedia'
     : canonical === 'hn_search' ? 'hn'
     : canonical === 'code_search' ? (input.engine || 'github')
-    : (input.engine || 'duckduckgo');
+    : 'duckduckgo';
   return { name: 'browser', arguments: { action: 'navigate', url: searchUrl(input.query || input.q, engine) } };
 }
 
@@ -73,11 +77,19 @@ function callName(call) {
   return call.name || (call.function && call.function.name) || '';
 }
 
+function isPageVisionFollowUp(msg) {
+  if (!msg || msg.role !== 'user') return false;
+  const text = String(msg.content || '');
+  if (/slices of one open page|open browser page|page_text is the full readable page/i.test(text)) return true;
+  return Array.isArray(msg.attachments) && msg.attachments.length > 0 && /do not only describe/i.test(text);
+}
+
 function needsGroqWebProcess(history) {
   const list = Array.isArray(history) ? history : [];
   for (let i = list.length - 1; i >= 0; i--) {
     const msg = list[i];
     if (!msg || !msg.role) continue;
+    if (msg.role === 'user' && isPageVisionFollowUp(msg)) continue;
     if (msg.role === 'user') return false;
     if (msg.role === 'tool' || msg.role === 'function') return callName(msg) === 'browser' || msg.name === 'browser';
     if (msg.role === 'assistant' && Array.isArray(msg.tool_calls) && msg.tool_calls.length) {
@@ -96,7 +108,42 @@ function browserToolsOnly(tools) {
   });
 }
 
-const GROQ_WEB_HINT = 'This turn is web-page processing only. Chrome already crawled on this computer. Think about the latest browser result, then speak the answer or call browser again to click, type, or open the next page. Do not call file, shell, desktop, or camera tools. Those stay on the local model.';
+const GROQ_WEB_HINT = 'This turn is web-page processing only. Chrome already scrolled the open page and left the window open. page_text is the full readable page, not just the first screen. Read it from start to end. Draft the spoken report now from those facts: names, numbers, dates, titles, and the points the page states. A search-results page is enough for that report. You may open one result afterward for more detail, but do not stop after thinking and do not wait for another page before speaking. Do not call file, shell, desktop, or camera tools. Those stay on the local model.';
+
+function latestBrowserReading(history) {
+  const list = Array.isArray(history) ? history : [];
+  for (let i = list.length - 1; i >= 0; i--) {
+    const msg = list[i];
+    if (!msg || (msg.role !== 'tool' && msg.role !== 'function')) continue;
+    if (msg.name !== 'browser' && !String(msg.content || '').includes('\npage_text:\n')) continue;
+    return String(msg.content || '');
+  }
+  return '';
+}
+
+function pageReport(reading) {
+  const raw = String(reading || '');
+  const split = raw.split('\npage_text:\n');
+  const body = (split.length > 1 ? split.slice(1).join('\npage_text:\n') : raw)
+    .split('\nrefs:')[0]
+    .split('\nRead page_text')[0];
+  const lines = body.split('\n').map((line) => line.replace(/\s+/g, ' ').trim()).filter((line) => {
+    if (!line || line.length < 12) return false;
+    if (/^(url|title|coverage|headings|tables|visible|page_text):/.test(line)) return false;
+    if (line === '(no readable text yet)') return false;
+    if (/cookie|privacy policy|sign in|log in|accept all/i.test(line)) return false;
+    return true;
+  });
+  const picked = [];
+  for (const line of lines) {
+    const clean = line.replace(/^[-*]\s*/, '');
+    if (picked.some((item) => item === clean)) continue;
+    picked.push(clean);
+    if (picked.length >= 8) break;
+  }
+  if (!picked.length) return '';
+  return 'Here is what the open page says. ' + picked.join('. ').replace(/\.\./g, '.');
+}
 
 module.exports = {
   RETIRED_WEB,
@@ -104,6 +151,9 @@ module.exports = {
   isRetiredWeb,
   asBrowserCall,
   needsGroqWebProcess,
+  isPageVisionFollowUp,
+  pageReport,
+  latestBrowserReading,
   browserToolsOnly,
   GROQ_WEB_HINT,
   searchUrl,