@@ -23,7 +23,8 @@ Important modules:
|
||||
- `agent/web-search.js`: public search and fetch through the Playwright sidecar in `browser-use/`. The Bare daemon never loads Playwright.
|
||||
- `agent/sessions.js`: persisted session summaries, history, updates, and plan files.
|
||||
- `agent/memory.js`: local short/long-term memory notes.
|
||||
- `lib/qvac.js`: lazy `@qvac/sdk` import, model loading, completion streaming, vision attachments, cancellation, and lifecycle close.
|
||||
- `lib/qvac.js`: lazy `@qvac/sdk` import, model loading, completion streaming, vision attachments, optional Groq dispatch, cancellation, and lifecycle close.
|
||||
- `lib/groq.js`: host-pinned Groq Chat Completions for the agent loop (`api.groq.com`). Tools still run locally.
|
||||
- `lib/catalog.js`: friendly model ids mapped to QVAC SDK constants.
|
||||
|
||||
## Start and embed
|
||||
|
||||
+9
-15
@@ -63,27 +63,21 @@ highlight and agent cursor), not a waveform overlay.
|
||||
|
||||
## Search tools
|
||||
|
||||
Search and fetch open a headed Playwright Chromium window owned by
|
||||
`browser-use/helper.js`. The Bare daemon never loads Playwright; it talks to
|
||||
that Node sidecar over JSON lines. Builtin tool names stay the same. There is
|
||||
no HTML/RSS scraper fallback and no SearXNG.
|
||||
The only public web tool is `browser`. It opens a headed Playwright Chromium
|
||||
window owned by `browser-use/helper.js`. The Bare daemon never loads Playwright;
|
||||
it talks to that Node sidecar over JSON lines. `web_search`, `google_search`,
|
||||
`fetch_page`, `web_fetch`, `wiki_search`, `hn_search`, and `code_search` are
|
||||
not registered. There is no HTML/RSS scraper fallback and no SearXNG.
|
||||
|
||||
| Tool | Behavior |
|
||||
| --- | --- |
|
||||
| `web_search` | Public search in the Jarvis Chromium window. Default `engine=auto` tries DuckDuckGo, then Google. Pin `engine` to retry one backend. |
|
||||
| `google_search` | Google first, then DuckDuckGo |
|
||||
| `fetch_page` | Open a public http(s) URL in Chromium and return readable text, links, headings, and metadata; `offset`, `max_chars`, and `find` continue reading |
|
||||
| `web_fetch` | Same as `fetch_page`, including IP lookup pages such as ifconfig.me |
|
||||
| `wiki_search` | Wikipedia in Chromium |
|
||||
| `hn_search` | Hacker News in Chromium |
|
||||
| `code_search` | GitHub, npm, and MDN in Chromium |
|
||||
| `browser` | Drive the same Chromium window. Actions: `navigate` (`url`), `snapshot` (numbered `refs`), `click`/`type` (`ref` from the last snapshot), `press` (`key`), `scroll` (`dy`), `wait` (`ms`). Snapshot before every click. Cookie walls and leftover challenges use this tool, not computer-use. |
|
||||
| `browser` | The only web tool. Actions: `navigate` (`url`, returns page text), `snapshot` (numbered `refs` plus page text), `click`/`type` (`ref` from the last snapshot), `press` (`key`), `scroll` (`dy`), `wait` (`ms`). Search by navigating to a public search URL, then click a result ref. Cookie walls and leftover challenges use this tool, not computer-use. |
|
||||
|
||||
Those tools run JavaScript and wait about 20s on bot-check pages. If a result
|
||||
`browser` runs JavaScript and waits about 20s on bot-check pages. If a result
|
||||
has `challenge: true`, finish the prompt in the Jarvis browser window and call
|
||||
the tool again. Private, loopback, and metadata URLs fail before Chromium
|
||||
starts. Shell HTTP (`curl`/`wget` over public hosts) remains blocked; use these
|
||||
tools instead. Install the browser once with `npm run browser:install`. The
|
||||
starts. Shell HTTP (`curl`/`wget` over public hosts) remains blocked; use
|
||||
`browser` instead. Install the browser once with `npm run browser:install`. The
|
||||
daemon looks up Node via `JARVIS_BROWSER_NODE` because `jarvisd` itself runs
|
||||
under Bare.
|
||||
|
||||
|
||||
@@ -46,7 +46,8 @@ flowchart TB
|
||||
| Control Center | settings, model fit, permissions | inference loop or hidden data deletion |
|
||||
| Computer-use helpers | portal sessions, frames, AT-SPI, EIS/legacy backend | model planning or lock-screen bypass |
|
||||
| Agent harness | session history, planning, memory, tool loop, permissions | direct QVAC ownership or direct desktop hacks |
|
||||
| QVAC master | SDK worker, GPU admission, lifecycle, serialization, cancellation | cloud APIs or independent model instances |
|
||||
| QVAC master | SDK worker, GPU admission, lifecycle, serialization, cancellation | a second QVAC runtime |
|
||||
| Groq (opt-in) | Agent-loop chat completions only, host-pinned to api.groq.com | TTS, ASR, tools, or a second planner |
|
||||
|
||||
Production `jarvisd` is the packaged Bare process (`daemon/bare-entry.js` via
|
||||
`packaging/jarvisd.service`). Node 22 is for tests, packaging, and the harness
|
||||
@@ -56,7 +57,9 @@ CLI, not the live daemon.
|
||||
|
||||
Control messages and small state changes use session D-Bus. Audio streams and
|
||||
large screenshots use PipeWire, temporary files, or the local IPC channel. The
|
||||
daemon never sends screen frames or microphone data to a remote service.
|
||||
daemon never sends microphone audio to a remote service. Screen frames stay
|
||||
local unless the user opts into Groq and selects a Groq vision model, which
|
||||
may attach webcam stills to that chat request.
|
||||
|
||||
```mermaid
|
||||
sequenceDiagram
|
||||
|
||||
@@ -7,7 +7,7 @@ that is not present.
|
||||
|
||||
| Domain | Tools | Execution owner | Typical permission |
|
||||
| --- | --- | --- | --- |
|
||||
| Conversation | `chat`, `plan`, `summarize`, `rewrite`, `code` | Harness + QVAC master | read |
|
||||
| Conversation | `chat`, `plan`, `summarize`, `rewrite`, `code` | Harness + local QVAC, or opt-in Groq for the agent loop | read |
|
||||
| Retrieval | `embed`, `remember`, `ask-my-files`, workspaces | QVAC master + local stores | read/write |
|
||||
| Vision | screenshot, `look-at-this`, `ocr`, `what-is-this`, `webcam` | Portal + QVAC master | read |
|
||||
| Speech | `dictate`, `transcribe-file`, `meeting`, `speak`, `clone-voice` | PipeWire + QVAC master | read/write |
|
||||
@@ -15,7 +15,7 @@ that is not present.
|
||||
| Media | `imagine`, `edit-image`, `make-video`, `compose` | Job runner + QVAC master | write |
|
||||
| Training | `teach-me` | Isolated job + QVAC master | dangerous |
|
||||
| Lab | `bci`, `robot`, `world` | Explicit lab adapters | read/write/dangerous |
|
||||
| Search | `web_search`, `google_search`, `fetch_page`, `web_fetch`, `wiki_search`, `hn_search`, `code_search`, `browser` | Playwright Chromium sidecar | read (no extra keys) |
|
||||
| Search | `browser` | Local Playwright Chromium sidecar. Groq may think about the page text; the crawl stays local | read (no extra keys) |
|
||||
| Desktop | launch, focus, window, clipboard, media, settings | GNOME/GIO/D-Bus | read/write |
|
||||
| Computer use | `cu.observe`, `cu.find`, `cu.act`, `cu.click`, `cu.type`, `cu.key` | CU session | computer-use |
|
||||
|
||||
@@ -26,7 +26,7 @@ flowchart TB
|
||||
J --> M[Memory\nembeddings · RAG · batch]
|
||||
J --> P[Perception\nscreenshot · OCR · webcam]
|
||||
J --> V[Voice\nwake · ASR · TTS · translation]
|
||||
J --> W[Search\nweb_search · fetch_page · browser]
|
||||
J --> W[Search\nbrowser]
|
||||
J --> G[Media\nimage · video · music]
|
||||
J --> D[Desktop\nGNOME · AT-SPI · computer use]
|
||||
J --> L[Lab\nLoRA · BCI · VLA · ABot-World]
|
||||
|
||||
@@ -43,16 +43,16 @@ it with `# completed`. Computer-use tools are custom harness tools backed by
|
||||
|
||||
## Adding a tool
|
||||
|
||||
1. Put the implementation in the appropriate skill module. Public web search
|
||||
and page fetch already live in
|
||||
`vendor/agent-harness/agent/web-search.js` (`web_search`, `google_search`,
|
||||
`fetch_page`, `web_fetch`, `wiki_search`, `hn_search`, `code_search`), which
|
||||
calls the Playwright sidecar in `browser-use/`. Extra clicks use the custom
|
||||
`browser` gateway from `skills/browser-tools.js`. Webcam stills use
|
||||
`skills/webcam-tools.js` after Settings → Camera → Allow now.
|
||||
1. Put the implementation in the appropriate skill module. Public web access is
|
||||
only the custom `browser` gateway from `skills/browser-tools.js`, which
|
||||
calls the Playwright sidecar in `browser-use/`. Do not register
|
||||
`web_search` or `web_fetch`. Webcam stills use `skills/webcam-tools.js`
|
||||
after Settings → Camera → Allow now.
|
||||
2. Export a JSON-schema tool with a stable name, description, timeout, and
|
||||
permission level.
|
||||
3. Route QVAC work through the master facade.
|
||||
3. Route QVAC work through the master facade. Opt-in Groq is only for
|
||||
thinking about a `browser` result. Other tool turns stay on local QVAC.
|
||||
Skills still do not call Groq.
|
||||
4. Add a focused fixture test with a fake master or backend.
|
||||
5. Update [API](api.md), the capability matrix, and the roadmap if the tool is
|
||||
user-visible.
|
||||
|
||||
@@ -26,7 +26,10 @@ It is not started alongside `jarvisd` in production and must remain bound to
|
||||
## Lifecycle
|
||||
|
||||
1. Read `QVAC_CONFIG_PATH` and the selected profile.
|
||||
2. Query GPU visibility and model fit before loading.
|
||||
2. Query GPU visibility and model fit before loading the local chat model.
|
||||
Opt-in Groq agent inference skips this local chat load; ASR and TTS still
|
||||
use auxiliary QVAC models on the same worker.
|
||||
3. Admit work through interactive, computer-vision, background, or
|
||||
3. Admit work through interactive, computer-vision, background, or
|
||||
maintenance lanes.
|
||||
4. Load one model configuration at a time and serialize worker access.
|
||||
|
||||
@@ -1,14 +1,15 @@
|
||||
# Security and privacy
|
||||
|
||||
The default posture is local inference, explicit writes, and fail-closed
|
||||
computer use. No telemetry or cloud inference is required. Model endpoints
|
||||
stay on localhost. Public `web_search`, `google_search`, `fetch_page`,
|
||||
`web_fetch`, `wiki_search`, `hn_search`, `code_search`, and `browser` are allowed
|
||||
by default without a confirmation prompt and without extra API keys. They run in
|
||||
a Jarvis-owned Chromium window. Private, loopback, and metadata URLs are
|
||||
blocked before the browser starts. There is no SearXNG dependency. Shell
|
||||
commands that open public HTTP (curl, wget) remain blocked by the runtime; use
|
||||
the web tools instead.
|
||||
computer use. No telemetry or cloud inference is required. Local QVAC chat
|
||||
stays on this computer. An opt-in Settings switch can send **only web-page
|
||||
thinking** to Groq (`agentInference: "groq"`). Chrome still crawls locally.
|
||||
Speech, wake, files, shell, desktop, and camera stay on the local model. Groq
|
||||
is off by default. The only public web tool is `browser`. It does not wait for
|
||||
confirmation and does not need an extra API key. It runs in a Jarvis-owned
|
||||
Chromium window. Private, loopback, and metadata URLs are blocked before the
|
||||
browser starts. There is no SearXNG dependency. Shell commands that open public
|
||||
HTTP (curl, wget) remain blocked by the runtime; use `browser` instead.
|
||||
|
||||
```mermaid
|
||||
flowchart TD
|
||||
@@ -36,6 +37,11 @@ flowchart TD
|
||||
- Voice references, memory, and model caches are user-owned files.
|
||||
- QVAC binds to localhost; bearer tokens, when used, come from user-owned
|
||||
configuration and are not logged.
|
||||
- Groq API keys live in `config.json` or `GROQ_API_KEY` / `JARVIS_GROQ_API_KEY`.
|
||||
Runtime status redacts them. They are never written to the privacy log.
|
||||
- Opt-in Groq chat sends conversation text to `api.groq.com`. Webcam stills are
|
||||
included only when a Groq vision model is selected. Microphone audio is not
|
||||
sent. Do not enable Groq if that cloud path is unacceptable.
|
||||
|
||||
## Filesystem access
|
||||
|
||||
|
||||
+10
-6
@@ -58,10 +58,11 @@ and changes from another window. Invalid JSON is reported and not overwritten.
|
||||
Legacy snake_case keys and explicitly customized voice GSettings are migrated;
|
||||
the JSON file takes precedence over GSettings. New writes use the keys below.
|
||||
|
||||
`JARVIS_TTS_MODEL`, `JARVIS_ASR_MODEL`, `JARVIS_WAKE_COMMAND`, and
|
||||
`JARVIS_QVAC_MODEL` environment variables take precedence over the form. Apply
|
||||
reports active overrides. The custom speech model must match the selected
|
||||
engine; an arbitrary model does not change the engine automatically.
|
||||
`JARVIS_TTS_MODEL`, `JARVIS_ASR_MODEL`, `JARVIS_WAKE_COMMAND`,
|
||||
`JARVIS_QVAC_MODEL`, `GROQ_API_KEY`, and `JARVIS_GROQ_API_KEY` environment
|
||||
variables take precedence over the form. Apply reports active overrides. The
|
||||
custom speech model must match the selected engine; an arbitrary model does
|
||||
not change the engine automatically.
|
||||
|
||||
## Listening and desktop behavior
|
||||
|
||||
@@ -164,8 +165,11 @@ are wiped from `/tmp/jarvis-webcam` on revoke. This is not desktop ScreenCast.
|
||||
|
||||
### Chat model
|
||||
|
||||
- **Chat model profile** — `modelProfile`, default `"laptop-16gb"`. Requires a daemon restart. GPU inference remains required. Choices: `laptop-4gb-mm` (Qwen3.5 0.8B, vision), `laptop-8gb`, `laptop-8gb-mm` (Qwen3.5 2B, vision), `laptop-16gb`, `desktop-gpu`.
|
||||
- **Custom chat model** — `chatModel`, default `""`. Advanced: overrides the profile. Empty uses the profile model. Requires restart.
|
||||
- **Chat model profile** — `modelProfile`, default `"laptop-16gb"`. Local model for files, shell, desktop, camera, and every tool except web-page thinking. Requires a daemon restart. GPU inference remains required. Choices: `laptop-4gb-mm` (Qwen3.5 0.8B, vision), `laptop-8gb`, `laptop-8gb-mm` (Qwen3.5 2B, vision), `laptop-16gb`, `desktop-gpu`.
|
||||
- **Custom chat model** — `chatModel`, default `""`. Advanced: overrides the profile for local tools. Empty uses the profile model. Requires restart.
|
||||
- **Agent inference** — `agentInference`, default `"local"`. Local QVAC runs tools on this GPU. Groq thinks only about pages the local Chrome window has already crawled, via `https://api.groq.com/openai/v1/chat/completions`. The crawl stays local. Files, shell, desktop, camera, speech, and wake stay on the local model. Apply is enough. Choices: `local`, `groq`.
|
||||
- **Groq API key** — `groqApiKey`, default `""`. Shown when Agent inference is Groq. Stored in `config.json`. `GROQ_API_KEY` or `JARVIS_GROQ_API_KEY` overrides this field. Create a key at [console.groq.com](https://console.groq.com). Do not use `groq/compound`; Chrome and every other tool stay on this computer.
|
||||
- **Groq chat model** — `groqModel`, default `"openai/gpt-oss-20b"`. Shown as a dropdown when Agent inference is Groq. Used only to think about a local Chrome crawl. Listed models: `openai/gpt-oss-20b`, `openai/gpt-oss-120b`, `llama-3.1-8b-instant`, `llama-3.3-70b-versatile`, `qwen/qwen3.6-27b`, `qwen/qwen3.8-27b`, `minimaxai/minimax-m2.7`, `openai/gpt-oss-safeguard-20b`. Compound, Whisper, and speech models are omitted. Apply is enough.
|
||||
|
||||
### Agent limits
|
||||
|
||||
|
||||
@@ -106,6 +106,15 @@ visibility, or a stale portal grant. Run the three doctor commands before
|
||||
changing model settings. Restarting `jarvisd` also clears an active desktop
|
||||
grant.
|
||||
|
||||
## Groq agent chat fails immediately
|
||||
|
||||
Settings → Models → Agent inference → Groq requires a key in the Groq API key
|
||||
field, or `GROQ_API_KEY` / `JARVIS_GROQ_API_KEY`. Jarvis does not fall back to
|
||||
local QVAC chat. `401` means the key was rejected; `429` is rate limiting.
|
||||
Speech, wake, tools, desktop, browser, and camera stay local. Webcam stills
|
||||
reach Groq only if you pick a Qwen Groq vision model. Restart `jarvisd` after
|
||||
switching between Local QVAC and Groq.
|
||||
|
||||
## Rolling release did not publish
|
||||
|
||||
Check the Gitea Actions log and repository secret name. It must be exactly
|
||||
|
||||
Reference in New Issue
Block a user