gnome-jarvis/docs/architecture.md

System architecture

Jarvis is split into a thin desktop surface, a user daemon, a cognitive harness, and isolated capability adapters. This keeps GNOME Shell responsive, prevents multiple model runtimes from competing for GPU memory, and makes computer use revocable as a session capability.

flowchart TB
  subgraph Desktop[GNOME desktop session]
    EXT[Shell extension\ntray HUD + CU chrome]
    CC[Control Center\nGTK4/libadwaita]
    PORTAL[XDG portals\nScreenCast + RemoteDesktop]
    ATSPI[AT-SPI2\naccessible tree]
    PW[PipeWire\nmicrophone + playback]
  end
  subgraph Service[User services]
    D[ jarvisd\npackaged Bare ]
    CU[Computer-use session\nobserve + act + audit]
  end
  subgraph Cognitive[Cognition]
    H[agent-harness\nsessions + planner + memory]
    S[Skills\nQVAC + desktop + files]
  end
  subgraph Inference[One inference authority]
    M[QVAC master\nload / schedule / cancel]
    Q[QVAC SDK 0.19.x]
    GPU[(GPU backend)]
  end
  EXT <--> |session D-Bus| D
  CC <--> |settings + diagnostics| D
  D --> H --> S
  D --> M --> Q --> GPU
  D <--> CU
  CU <--> PORTAL
  CU <--> ATSPI
  D <--> PW

Process responsibilities

Process	Owns	Must not own
GNOME Shell extension	panel mark, tray popup, optional expanded session, CU target highlight, Shell facts, D-Bus client	model loading, microphone capture, input injection, blocking work
jarvisd	voice state, QVAC master, harness bridge, jobs, D-Bus service	arbitrary Shell evaluation or a second planner
Control Center	settings, model fit, permissions	inference loop or hidden data deletion
Computer-use helpers	portal sessions, frames, AT-SPI, EIS/legacy backend	model planning or lock-screen bypass
Agent harness	session history, planning, memory, tool loop, permissions	direct QVAC ownership or direct desktop hacks
QVAC master	SDK worker, GPU admission, lifecycle, serialization, cancellation	a second QVAC runtime
Groq (opt-in)	Agent-loop chat completions only, host-pinned to api.groq.com	TTS, ASR, tools, or a second planner

Production jarvisd is the packaged Bare process (daemon/bare-entry.js via packaging/jarvisd.service). Node 22 is for tests, packaging, and the harness CLI, not the live daemon.

Data and control boundaries

Control messages and small state changes use session D-Bus. Audio streams and large screenshots use PipeWire, temporary files, or the local IPC channel. The daemon never sends microphone audio to a remote service. Screen frames stay local unless the user opts into Groq and selects a Groq vision model, which may attach webcam stills to that chat request.

sequenceDiagram
  participant Shell as GNOME Shell
  participant Daemon as jarvisd
  participant Harness as Harness
  participant Master as QVAC master
  participant GPU as GPU
  Shell->>Daemon: Ask(text) / wake / PTT
  Daemon->>Master: admit(interactive lane)
  Daemon->>Harness: prompt(text, tools)
  Harness->>Master: completion(request)
  Master->>GPU: one serialized inference
  GPU-->>Master: streamed tokens
  Master-->>Harness: tokens + tool calls
  Harness-->>Daemon: reply + sidecar
  Daemon-->>Shell: Token / Reply / StateChanged

Failure containment

QVAC failures, unavailable models, or failed background jobs are reported as capability errors and do not silently switch to CPU inference. A daemon restart releases the master and restores the user service according to systemd policy. Computer-use revoke is immediate and independent of the language model.