gnome-jarvis/docs/features.md

Jarvis features

Jarvis is a local-first voice and computer-use assistant for Ubuntu GNOME. This document summarizes the product surface and runtime contracts. Related detail lives in Architecture, Capabilities, and the topic guides linked from the documentation index.

Product contract

Jarvis is local-first and session-scoped. The GNOME Shell extension stays thin; all inference, audio, jobs, memory, and computer-use decisions run outside gnome-shell. The copied harness in vendor/agent-harness is the sole cognitive loop and is wrapped rather than rewritten.

Runtime architecture

• One user service, jarvisd, owns the QVAC SDK worker.
• One QVAC_CONFIG_PATH points every Jarvis process at the same qvac.config.json.
• One model profile is active at a time. The QVAC master owns load, unload, suspend, resume, cancellation, and model status.
• Harness sessions call the master; skills never import QVAC independently and never call loadModel() directly.
• qvac serve --openai is an optional diagnostic/API surface and is never started by jarvisd; it must not run beside the master in production.
• GPU is mandatory for Jarvis runtime inference. The bridge requests GPU, enables all GPU layers and GPU multimodal projection, and rejects a load that QVAC reports as CPU. There is no silent CPU fallback.
• Audio capture, TTS, OCR, embeddings, RAG, media jobs, and multimodal work use the same QVAC master scheduler and obey a GPU memory budget.
• A capability may report unavailable: gpu-required without weakening the GPU policy or starting a competing runtime.

Host-dependent pieces such as GPU visibility, wake-word bridge availability, portal consent, and live EIS input must be verified with the doctor commands and acceptance procedures on the target GNOME session.

Repository and dependencies

• The cognitive core lives in vendor/agent-harness (copied, not symlinked). Layout and entrypoints are documented in agent-harness-map.md.
• QVAC API rules are pinned in QVAC.md; the repository locks @qvac/sdk to 0.19.1.
• Node 22.17+ workspaces support tests, packaging, and harness CLI use. Production jarvisd is the packaged Bare process.
• Root qvac.config.json holds localhost server settings and a single model alias.
• Boundaries exist for the daemon, computer-use helpers, GNOME extension, D-Bus, and control center.
• CI covers syntax, unit, schema, and extension checks.

QVAC master

daemon/qvac-master.js is the only Jarvis QVAC owner. It:

• Serializes model loading and shares the resulting harness engine.
• Requests device: "gpu", gpu_layers: 99, and GPU multimodal projection.
• Rejects a CPU result instead of accepting QVAC's internal fallback.
• Prefights QVAC GPU visibility before downloading or loading a model.
• Enforces the vendored @qvac/sdk 0.19.x version at startup.
• Tracks owner counts and exposes a single close path.
• Honors QVAC_CONFIG_PATH, JARVIS_QVAC_MODEL, and GPU policy from the user service.
• Schedules lanes for interactive voice, computer-use vision, background media, and maintenance with single-concurrency admission so background jobs do not run beside the voice lane.
• Owns model profile selection, assessModelFit(), and cancellation by request ID or model ID.

npm run gpu-doctor reports QVAC resources, backend hints, device, VRAM, SDK, and model-profile status. When no usable GPU is visible, the master refuses inference and never falls back to CPU.

Harness bridge and skills

The daemon starts the harness through Agent.create(), routes token events, and uses GPU preflight as the only runtime gate for local chat. It supplies a voice-native system prompt and structured HUD sidecar.

Jarvis registers runtime/status tools and safe local tools through the harness custom-tool registry with permission classes: read, write, dangerous, and computer-use. Confirmation events reach the HUD and spoken confirmation consumers over D-Bus.

Desktop adapters report truthful unavailable states. File search/read, confirmed writes, local memory, and RAG workspace discovery are available; QVAC retrieval stays in the capability adapters. Master-owned wrappers cover lifecycle, resources, model fit, embeddings, translation, OCR, classification, image/video/music jobs, transcription, TTS, LoRA, BCI, VLA, and ABot-World. Wrappers use master status and never load QVAC themselves.

Daemon lifecycle and D-Bus

Session interface io.qvac.Jarvis.Session provides Arm, Sleep, Shutdown, Say, Ask, Cancel, SetMode, and state queries. Signals cover tokens, transcripts, replies, audio level, chips, jobs, computer steps, and errors.

Payloads stay small: stream events through a bounded Unix socket and use file paths for PCM and screenshots. Lock-screen handling cancels work, re-arms, and revokes computer use. Idle sleep uses QVAC suspend() and resumes on wake. Structured JSON logging uses an allowlist that excludes prompt, audio, image, and transcript contents by default.

Live session-bus smoke tests require a normal GNOME user session.

Voice loop

PipeWire captures at 16 kHz mono with a dedicated Jarvis node. A WakeEngine interface backs wake-word detection; VAD segments utterances for the QVAC ASR stream. State flows ARMED → LISTENING → THINKING → SPEAKING → LISTENING.

The loop filters transcripts, applies a TTS anti-feedback gate and playback cooldown, and buffers sentences from streamed harness output into QVAC TTS. Fast paths cover cancel, sleep, privacy, dictate, screen, and computer-use commands, plus push-to-talk and typed fallback. Metrics track wake false-accept / false-reject and feedback drops.

Implementation lives in daemon/audio-pipewire.js, daemon/wake-engine.js, daemon/vad.js, daemon/voice-adapters.js, daemon/voice-loop.js, and daemon/audio-playback.js. Speech models load as auxiliary models through the same QVAC master with GPU settings. Live wake detection defaults to the built-in CPU keyword spotter (jarvis-wake-bridge); a custom openWakeWord or sherpa-onnx command can replace it. Cloud wake is not supported.

Live procedure: voice acceptance.

GNOME HUD

The GNOME 45+ ESM extension in apps/gnome-extension/[email protected]/ owns only Shell UI and an asynchronous session-bus client. It provides a panel indicator, tray HUD with compact conversation menu, Chat/Thinking tabs, hold-to-talk, optional expanded session, transcript rows, chips, local/model status, reduced motion, high contrast, keyboard navigation, and screen-reader labels. Listening/privacy chips, job theater, target highlights, and a visible computer-use cursor are included.

Enabling or disabling the extension never starts inference in the Shell process. SpeakingLevel / ListeningLevel are still emitted on D-Bus; the extension no-ops them (no waveform overlay). Accent color is configurable.

Computer use — observe

npm run cu-doctor checks Wayland/X11, portal, PipeWire, AT-SPI, libei, and optional fallback detection. Observation uses portal ScreenCast/Screenshot with tmpfs frame normalization, GNOME Shell window/focus truth, AT-SPI tree snapshots with per-step stable refs, and OCR plus QVAC multimodal observe bundles.

Tools include cu.observe, cu.zoom, cu.tree, and cu.find. Frames stay local in /tmp/jarvis-cu, downscale to WebP before perception, and return explicit unavailable reasons when portal, Shell, or AT-SPI providers cannot connect. Grant/revoke, expiry, step budget, audit hashes, and no-frame retention by default apply across observe and act.

Computer use — actuation

RemoteDesktop portal consent and restore tokens back libei/EIS pointer, keyboard, scroll, drag, hover, and key input, including typed Unicode and submit. Target preview, agent cursor, and step ticker are shown in the HUD.

Action preference order: domain tools, app D-Bus, AT-SPI, Shell helper, then vision coordinates. Password/PAM roles and lock-screen/greeter actions are refused. Destructive or high-impact actions require confirmation. ydotool and X11 tools stay disabled unless explicitly enabled. State-change self-checks, animation waits, and no-progress aborts apply.

Actuation lives in computer-use/portal-input.js, computer-use/actuator.js, computer-use/safety.js, computer-use/audit.js, skills/computer-act.js, and the local portal/AT-SPI helpers. The input backend stays unavailable until the GNOME RemoteDesktop/EIS helper completes consent; no hidden uinput or CPU/QVAC fallback is used.

Live procedure: computer-use acceptance.

Control center

The GTK4/libadwaita app in apps/control-center/main.py provides General, Voice, Models, Memory, Skills, Computer use, Privacy, Lab, and About pages. It shows single-master status, GPU backend, VRAM, model, fit result, queue, and failure reason. Every capability stays visible even when its model does not fit.

Users can pause/resume model download through the master, enroll and preview voice, manage RAG workspaces, set retention, export/delete audits, and manage computer-use permissions. Preferences persist locally; GPU/model failures stay visible instead of starting another QVAC runtime.

QVAC capability coverage

Adapters in skills/phase9-tools.js cover chat, plan, summarize, rewrite, code, embeddings, RAG, batch prompts, multimodal screenshot/file analysis, OCR, classification, image generation/editing, video and music jobs, ASR, diarized meetings, TTS, voice clone enrollment, translation relay, LoRA training with confirmation and overnight job controls, and labeled Lab slots for BCI, VLA, and ABot-World.

Registry/model manager, checksums, fit assessments, profiler, and job cancellation are included. Optional P2P model fetch and memory sync are opt-in after local mode is stable. Every adapter is permission-tagged and routes through allowlisted QVAC master methods; the harness and UI never import QVAC directly. Media and streaming results return local job/request handles. Capabilities without a configured model report the QVAC error/state rather than falling back to cloud or CPU.

Reliability, privacy, and performance

• Failed media jobs cannot kill the voice lane (OOM isolation).
• GPU telemetry covers utilization, VRAM, queue wait, load time, tokens/sec, ASR latency, TTS latency, and dropped audio.
• Smoke coverage includes wake, feedback, Wayland/X11, accessibility, lock/unlock, crash/restart, and nested Shell behavior.
• No network calls except explicitly enabled model fetch/P2P paths.
• Screenshots, microphone buffers, prompts, and transcripts are not retained unless the user enables retention.
• Computer actions are audited with target metadata and screenshot hashes only.
• Graceful restart and state recovery restore safely to ARMED after an interrupted turn.

Supporting modules include daemon/telemetry.js, daemon/recovery.js, daemon/job-runner.js, daemon/network-policy.js, the privacy log, and the computer audit store. Runtime status exposes scheduler counters, latency samples, GPU resources, and voice metrics.

Packaging and release

Builds produce a .deb, extension bundle, and user-service installer. A first-run wizard covers microphone, wake phrase, GPU/model profile, TTS, and a typed smoke test. Uninstall removes only Jarvis-owned data after an explicit choice. Hardware compatibility is documented for Vulkan/GPU backends.

Release automation lives in .gitea/workflows/rolling-release.yml. A push to main runs verification and package build, then replaces the Gitea rolling prerelease using the repository RELEASE_TOKEN secret. See RELEASE.md, MIGRATIONS.md, and local-demo.md.