gnome-jarvis/docs/features.md

# Jarvis features

Jarvis is a local-first voice and computer-use assistant for Ubuntu GNOME.
This document summarizes the product surface and runtime contracts. Related
detail lives in [Architecture](architecture.md), [Capabilities](capabilities.md),
and the topic guides linked from the [documentation index](README.md).

## Product contract

Jarvis is local-first and session-scoped. The GNOME Shell extension stays thin;
all inference, audio, jobs, memory, and computer-use decisions run outside
gnome-shell. The copied harness in `vendor/agent-harness` is the sole cognitive
loop and is wrapped rather than rewritten.

## Runtime architecture

- One user service, `jarvisd`, owns the QVAC SDK worker.
- One `QVAC_CONFIG_PATH` points every Jarvis process at the same
  `qvac.config.json`.
- One model profile is active at a time. The QVAC master owns load, unload,
  suspend, resume, cancellation, and model status.
- Harness sessions call the master; skills never import QVAC independently and
  never call `loadModel()` directly.
- `qvac serve --openai` is an optional diagnostic/API surface and is never
  started by `jarvisd`; it must not run beside the master in production.
- GPU is mandatory for Jarvis runtime inference. The bridge requests GPU,
  enables all GPU layers and GPU multimodal projection, and rejects a load
  that QVAC reports as CPU. There is no silent CPU fallback.
- Audio capture, TTS, OCR, embeddings, RAG, media jobs, and multimodal work
  use the same QVAC master scheduler and obey a GPU memory budget.
- A capability may report `unavailable: gpu-required` without weakening the
  GPU policy or starting a competing runtime.

Host-dependent pieces such as GPU visibility, wake-word bridge availability,
portal consent, and live EIS input must be verified with the doctor commands
and acceptance procedures on the target GNOME session.

## Repository and dependencies

- The cognitive core lives in `vendor/agent-harness` (copied, not symlinked).
  Layout and entrypoints are documented in [agent-harness-map.md](agent-harness-map.md).
- QVAC API rules are pinned in [QVAC.md](../QVAC.md); the repository locks
  `@qvac/sdk` to 0.19.1.
- Node 22.17+ workspaces support tests, packaging, and harness CLI use.
  Production `jarvisd` is the packaged Bare process.
- Root `qvac.config.json` holds localhost server settings and a single model
  alias.
- Boundaries exist for the daemon, computer-use helpers, GNOME extension,
  D-Bus, and control center.
- CI covers syntax, unit, schema, and extension checks.

## QVAC master

`daemon/qvac-master.js` is the only Jarvis QVAC owner. It:

- Serializes model loading and shares the resulting harness engine.
- Requests `device: "gpu"`, `gpu_layers: 99`, and GPU multimodal projection.
- Rejects a CPU result instead of accepting QVAC's internal fallback.
- Prefights QVAC GPU visibility before downloading or loading a model.
- Enforces the vendored `@qvac/sdk` 0.19.x version at startup.
- Tracks owner counts and exposes a single close path.
- Honors `QVAC_CONFIG_PATH`, `JARVIS_QVAC_MODEL`, and GPU policy from the user
  service.
- Schedules lanes for interactive voice, computer-use vision, background
  media, and maintenance with single-concurrency admission so background jobs
  do not run beside the voice lane.
- Owns model profile selection, `assessModelFit()`, and cancellation by
  request ID or model ID.

`npm run gpu-doctor` reports QVAC resources, backend hints, device, VRAM, SDK,
and model-profile status. When no usable GPU is visible, the master refuses
inference and never falls back to CPU.

## Harness bridge and skills

The daemon starts the harness through `Agent.create()`, routes token events,
and uses GPU preflight as the only runtime gate for local chat. It supplies a
voice-native system prompt and structured HUD sidecar.

Jarvis registers runtime/status tools and safe local tools through the harness
custom-tool registry with permission classes: read, write, dangerous, and
computer-use. Confirmation events reach the HUD and spoken confirmation
consumers over D-Bus.

Desktop adapters report truthful unavailable states. File search/read,
confirmed writes, local memory, and RAG workspace discovery are available;
QVAC retrieval stays in the capability adapters. Master-owned wrappers cover
lifecycle, resources, model fit, embeddings, translation, OCR, classification,
image/video/music jobs, transcription, TTS, LoRA, BCI, VLA, and ABot-World.
Wrappers use master status and never load QVAC themselves.

## Daemon lifecycle and D-Bus

Session interface `io.qvac.Jarvis.Session` provides `Arm`, `Sleep`, `Shutdown`,
`Say`, `Ask`, `Cancel`, `SetMode`, and state queries. Signals cover tokens,
transcripts, replies, audio level, chips, jobs, computer steps, and errors.

Payloads stay small: stream events through a bounded Unix socket and use file
paths for PCM and screenshots. Lock-screen handling cancels work, re-arms, and
revokes computer use. Idle sleep uses QVAC `suspend()` and resumes on wake.
Structured JSON logging uses an allowlist that excludes prompt, audio, image,
and transcript contents by default.

Live session-bus smoke tests require a normal GNOME user session.

## Voice loop

PipeWire captures at 16 kHz mono with a dedicated `Jarvis` node. A `WakeEngine`
interface backs wake-word detection; VAD segments utterances for the QVAC ASR
stream. State flows `ARMED → LISTENING → THINKING → SPEAKING → LISTENING`.

The loop filters transcripts, applies a TTS anti-feedback gate and playback
cooldown, and buffers sentences from streamed harness output into QVAC TTS.
Fast paths cover cancel, sleep, privacy, dictate, screen, and computer-use
commands, plus push-to-talk and typed fallback. Metrics track wake
false-accept / false-reject and feedback drops.

Implementation lives in `daemon/audio-pipewire.js`, `daemon/wake-engine.js`,
`daemon/vad.js`, `daemon/voice-adapters.js`, `daemon/voice-loop.js`, and
`daemon/audio-playback.js`. Speech models load as auxiliary models through the
same QVAC master with GPU settings. Live wake detection defaults to the
built-in CPU keyword spotter (`jarvis-wake-bridge`); a custom openWakeWord or
sherpa-onnx command can replace it. Cloud wake is not supported.

Live procedure: [voice acceptance](voice-acceptance.md).

## GNOME HUD

The GNOME 45+ ESM extension in `apps/gnome-extension/[email protected]/` owns
only Shell UI and an asynchronous session-bus client. It provides a panel
indicator, tray HUD with compact conversation menu, Chat/Thinking tabs,
hold-to-talk, optional expanded session, transcript rows, chips, local/model
status, reduced motion, high contrast, keyboard navigation, and screen-reader
labels. Listening/privacy chips, job theater, target highlights, and a visible
computer-use cursor are included.

Enabling or disabling the extension never starts inference in the Shell
process. `SpeakingLevel` / `ListeningLevel` are still emitted on D-Bus; the
extension no-ops them (no waveform overlay). Accent color is configurable.

## Computer use — observe

`npm run cu-doctor` checks Wayland/X11, portal, PipeWire, AT-SPI, libei, and
optional fallback detection. Observation uses portal ScreenCast/Screenshot with
tmpfs frame normalization, GNOME Shell window/focus truth, AT-SPI tree
snapshots with per-step stable refs, and OCR plus QVAC multimodal observe
bundles.

Tools include `cu.observe`, `cu.zoom`, `cu.tree`, and `cu.find`. Frames stay
local in `/tmp/jarvis-cu`, downscale to WebP before perception, and return
explicit unavailable reasons when portal, Shell, or AT-SPI providers cannot
connect. Grant/revoke, expiry, step budget, audit hashes, and no-frame
retention by default apply across observe and act.

## Computer use — actuation

RemoteDesktop portal consent and restore tokens back libei/EIS pointer,
keyboard, scroll, drag, hover, and key input, including typed Unicode and
submit. Target preview, agent cursor, and step ticker are shown in the HUD.

Action preference order: domain tools, app D-Bus, AT-SPI, Shell helper, then
vision coordinates. Password/PAM roles and lock-screen/greeter actions are
refused. Destructive or high-impact actions require confirmation. ydotool and
X11 tools stay disabled unless explicitly enabled. State-change self-checks,
animation waits, and no-progress aborts apply.

Actuation lives in `computer-use/portal-input.js`, `computer-use/actuator.js`,
`computer-use/safety.js`, `computer-use/audit.js`, `skills/computer-act.js`,
and the local portal/AT-SPI helpers. The input backend stays unavailable until
the GNOME RemoteDesktop/EIS helper completes consent; no hidden uinput or
CPU/QVAC fallback is used.

Live procedure: [computer-use acceptance](cu-acceptance.md).

## Control center

The GTK4/libadwaita app in `apps/control-center/main.py` provides General,
Voice, Models, Memory, Skills, Computer use, Privacy, Lab, and About pages. It
shows single-master status, GPU backend, VRAM, model, fit result, queue, and
failure reason. Every capability stays visible even when its model does not
fit.

Users can pause/resume model download through the master, enroll and preview
voice, manage RAG workspaces, set retention, export/delete audits, and manage
computer-use permissions. Preferences persist locally; GPU/model failures stay
visible instead of starting another QVAC runtime.

## QVAC capability coverage

Adapters in `skills/phase9-tools.js` cover chat, plan, summarize, rewrite,
code, embeddings, RAG, batch prompts, multimodal screenshot/file analysis,
OCR, classification, image generation/editing, video and music jobs, ASR,
diarized meetings, TTS, voice clone enrollment, translation relay, LoRA
training with confirmation and overnight job controls, and labeled Lab slots
for BCI, VLA, and ABot-World.

Registry/model manager, checksums, fit assessments, profiler, and job
cancellation are included. Optional P2P model fetch and memory sync are
opt-in after local mode is stable. Every adapter is permission-tagged and
routes through allowlisted QVAC master methods; the harness and UI never
import QVAC directly. Media and streaming results return local job/request
handles. Capabilities without a configured model report the QVAC error/state
rather than falling back to cloud or CPU.

## Reliability, privacy, and performance

- Failed media jobs cannot kill the voice lane (OOM isolation).
- GPU telemetry covers utilization, VRAM, queue wait, load time, tokens/sec,
  ASR latency, TTS latency, and dropped audio.
- Smoke coverage includes wake, feedback, Wayland/X11, accessibility,
  lock/unlock, crash/restart, and nested Shell behavior.
- No network calls except explicitly enabled model fetch/P2P paths.
- Screenshots, microphone buffers, prompts, and transcripts are not retained
  unless the user enables retention.
- Computer actions are audited with target metadata and screenshot hashes
  only.
- Graceful restart and state recovery restore safely to `ARMED` after an
  interrupted turn.

Supporting modules include `daemon/telemetry.js`, `daemon/recovery.js`,
`daemon/job-runner.js`, `daemon/network-policy.js`, the privacy log, and the
computer audit store. Runtime status exposes scheduler counters, latency
samples, GPU resources, and voice metrics.

## Packaging and release

Builds produce a `.deb`, extension bundle, and user-service installer. A
first-run wizard covers microphone, wake phrase, GPU/model profile, TTS, and a
typed smoke test. Uninstall removes only Jarvis-owned data after an explicit
choice. Hardware compatibility is documented for Vulkan/GPU backends.

Release automation lives in `.gitea/workflows/rolling-release.yml`. A push to
`main` runs verification and package build, then replaces the Gitea `rolling`
prerelease using the repository `RELEASE_TOKEN` secret. See
[RELEASE.md](RELEASE.md), [MIGRATIONS.md](MIGRATIONS.md), and
[local-demo.md](local-demo.md).