gnome-jarvis/docs/ROADMAP.md

# JARVIS-QVAC roadmap

This is the implementation tracker for the local Ubuntu GNOME assistant. The
roadmap is ordered by dependency: runtime authority first, then cognition,
voice, desktop observation, actuation, UI, capability coverage, hardening, and
packaging.

## Product contract

Jarvis is local-first and session-scoped. The GNOME Shell extension stays thin;
all inference, audio, jobs, memory, and computer-use decisions run outside
gnome-shell. The extracted `/home/raven/dev/agent-harness` remains the sole
cognitive loop and is wrapped rather than rewritten.

## Non-negotiable runtime architecture

- One user service, `jarvisd`, owns the QVAC SDK worker.
- One `QVAC_CONFIG_PATH` points every Jarvis process at the same
  `qvac.config.json`.
- One model profile is active at a time. The QVAC master owns load, unload,
  suspend, resume, cancellation, and model status.
- Harness sessions call the master; skills never import QVAC independently and
  never call `loadModel()` directly.
- `qvac serve --openai` is an optional diagnostic/API surface and is never
  started by `jarvisd`; it must not run beside the master in production.
- GPU is mandatory for Jarvis runtime inference. The bridge requests GPU,
  enables all GPU layers and GPU multimodal projection, and rejects a load
  that QVAC reports as CPU. There is no silent CPU fallback.
- Audio capture, TTS, OCR, embeddings, RAG, media jobs, and multimodal work
  use the same QVAC master scheduler and obey a GPU memory budget.
- A capability may report `unavailable: gpu-required` without weakening the
  GPU policy or starting a competing runtime.

## Status legend

- `[x]` implemented and locally verified
- `[~]` scaffolded or partially implemented
- `[ ]` planned

## Phase 0 — repository and authority bootstrap

- [x] Confirm `/home/raven/dev/agent-harness` exists.
- [x] Copy the extracted harness into `vendor/agent-harness`.
- [x] Document harness layout, entrypoints, sessions, tools, memory, planner,
  and model path in `docs/agent-harness-map.md`.
- [x] Pin the QVAC 0.19.x API rules in `QVAC.md` and lock the repository to
  `@qvac/sdk` 0.19.1.
- [x] Create Node 22.17+ workspaces.
- [x] Add one root `qvac.config.json` with localhost server settings and a
  single model alias.
- [x] Add the initial daemon, computer-use, GNOME extension, D-Bus, and
  control-center boundaries.
- [x] Install and lock dependencies with a successful `npm install`.
- [x] Review runtime dependency audit: `npm audit --omit=dev` reports 10
  transitive findings through `dbus-next` (3 critical, 1 high, 6 moderate;
  upstream reports no available fixes).
- [x] Add repository CI for syntax, unit, schema, and extension checks.

Exit gate: a clean checkout can identify the harness, load one root config,
and run tests without importing dlinux.

## Phase 1 — single GPU QVAC master

- [x] Add `daemon/qvac-master.js` as the only Jarvis QVAC owner.
- [x] Serialize model loading and share the resulting harness engine.
- [x] Request `device: "gpu"`, `gpu_layers: 99`, and GPU multimodal projection.
- [x] Reject a CPU result instead of accepting QVAC's internal fallback.
- [x] Preflight QVAC GPU visibility before downloading or loading a model.
- [x] Enforce the vendored `@qvac/sdk` 0.19.x version at master startup.
- [x] Add owner counting and a single close path.
- [x] Set `QVAC_CONFIG_PATH`, `JARVIS_QVAC_MODEL`, and GPU policy in the user
  service.
- [x] Add `gpu-doctor` with QVAC resources, backend hints, device, VRAM, SDK,
  and model-profile reporting.
- [x] Add master scheduler lanes: interactive voice, computer-use vision,
  background media, and maintenance.
- [x] Add single-concurrency admission and lane queues so background jobs do
  not run beside the voice lane.
- [x] Add model profile selection and master-owned `assessModelFit()` access.
- [x] Add master-level cancellation by request ID or model ID.
- [x] Add ownership/status tests and a single master module boundary.

Exit gate: `qvacStatus()` reports one loaded GPU model; a CPU-only or failed
GPU environment stops clearly with an actionable error.

## Phase 2 — harness bridge and Jarvis skills

- [x] Start the real harness through `Agent.create()` and route token events;
  GPU preflight is the only runtime gate.
- [x] Add the voice-native system prompt and structured HUD sidecar.
- [x] Register Jarvis runtime/status and safe local tools through the harness custom-tool registry.
- [x] Define permission classes: read, write, dangerous, and computer-use.
- [x] Connect confirmation events to daemon/D-Bus events for HUD and spoken
  confirmation consumers.
- [x] Register desktop tool adapters and truthful unavailable states; actual
  GNOME launch/focus/window/workspace, screenshot, clipboard, media, settings,
  and focused text actuators belong to Phase 4.
- [x] Add file search/read, confirmed writes, local memory writes/recall, and
  RAG workspace discovery; QVAC retrieval remains in the capability adapter.
- [x] Add QVAC capability registry/status plus master-owned lifecycle/resource/
  model-fit wrappers; embeddings, translation,
  OCR, classification,
  image/video/music jobs, transcription, TTS, LoRA, BCI, VLA, and ABot-World.
- [x] Ensure current wrappers use the master status and never load QVAC themselves.
- [x] Add fixture tests for current tool schemas and permission metadata.

Exit gate: a typed prompt completes through the real harness, streams tokens,
executes a Jarvis tool, and returns a short local response through the same
GPU-owned worker.

### Phase 1–2 gate status

The implementation gates are complete: the pinned vendored harness, single
GPU-only QVAC master, scheduler, voice prompt/sidecar, permission metadata,
safe local tools, and master-owned QVAC utility wrappers are present and tested.
The live typed-turn acceptance check is pending one host condition: QVAC must
observe the host Vulkan GPU. Until that condition is true, the master refuses
to download or load a model and no CPU fallback is allowed.

## Phase 3 — daemon lifecycle and D-Bus

- [x] Define `io.qvac.Jarvis.Session` XML.
- [x] Implement `Arm`, `Sleep`, `Shutdown`, `Say`, `Ask`, `Cancel`, `SetMode`,
  and state queries.
- [x] Implement token, transcript, reply, audio-level, chip, job, computer
  step, and error signals.
- [x] Add the Node D-Bus service implementation. Live bus smoke testing remains
  pending on a normal user session because this sandbox cannot bind a D-Bus
  session socket.
- [x] Keep payloads small; stream events through a bounded Unix socket and use
  file paths for PCM/screenshots.
- [x] Add lock-screen handling: cancel, arm, and revoke computer use.
- [x] Add idle sleep using QVAC `suspend()` and resume on wake.
- [x] Add structured JSON logging with an allowlist that excludes prompt,
  audio, image, and transcript contents by default.

Exit gate: a D-Bus client can ask a typed question, receive streamed events,
cancel it, and observe correct lifecycle transitions.

### Phase 0–3 review result

Phases 0–3 are complete at the implementation and local verification level.
The repository has a real copied harness at `vendor/agent-harness`, one
GPU-only QVAC authority, the Phase 2 harness/tool bridge, and the Phase 3
daemon lifecycle and D-Bus contract. The Phase 4 handoff is ready.

Two acceptance checks depend on the host session rather than repository code:

- `npm run gpu-doctor` currently reports no GPU visible to the QVAC SDK, even
  though the host exposes an AMD Radeon device and Vulkan packages. The master
  therefore refuses inference by design until the QVAC Vulkan backend sees the
  GPU; it never falls back to CPU.
- Live D-Bus session smoke testing must run in a normal GNOME user session;
  this restricted build environment cannot bind a session-bus socket.

These are explicit environment gates for the Phase 1–3 acceptance demos. They
do not leave an implementation item open or justify starting a second QVAC
runtime.

## Phase 4 — voice loop

- [x] Add PipeWire capture at 16 kHz mono with a dedicated `Jarvis` node.
- [x] Add wake-word engine behind a `WakeEngine` interface.
- [x] Add VAD segmentation and the documented QVAC ASR stream.
- [x] Implement `ARMED → LISTENING → THINKING → SPEAKING → LISTENING`.
- [x] Add transcript filtering, TTS anti-feedback gate, and playback cooldown.
- [x] Add sentence buffering from streamed harness output into QVAC TTS.
- [x] Add fast-path cancel, sleep, privacy, dictate, screen, and computer-use
  commands.
- [x] Add push-to-talk and typed fallback.
- [x] Add wake false-accept/false-reject and feedback measurements.

Implementation is complete in `daemon/audio-pipewire.js`, `daemon/wake-engine.js`,
`daemon/vad.js`, `daemon/voice-adapters.js`, `daemon/voice-loop.js`, and
`daemon/audio-playback.js`. The SDK speech models are loaded as auxiliary
models by the same QVAC master and use GPU settings. Live wake-word detection
uses the built-in CPU keyword spotter by default (`jarvis-wake-bridge`). A
custom openWakeWord or sherpa-onnx command can replace it; cloud wake is not
supported.

Exit gate: implementation complete. In a configured GNOME session, “Hey
Jarvis” starts a local GPU-backed turn, speaks a response, and does not
self-trigger from its own TTS. `docs/voice-acceptance.md` contains the live
hardware/session acceptance procedure.

## Phase 5 — GNOME ARC surface

- [x] Add GNOME 45+ ESM extension metadata, panel indicator, and settings keys.
- [x] Add async D-Bus client and panel state glyphs.
- [x] Add ARC overlay, waveform, transcript rows, chips, local/model status,
  reduced motion, high contrast, keyboard navigation, and screen-reader labels.
- [x] Add listening halo, privacy slash, job theater, target highlights, and
  visible computer-use cursor.
- [x] Keep all extension work nonblocking and free of QVAC/native inference.

The extension implementation is in `apps/gnome-extension/[email protected]/`.
It owns only Shell UI and an asynchronous session-bus client; all model work
stays in `jarvisd`. ARC renders 48 waveform bars, transcript rows, mode and
suggestion chips, state glyphs, computer-use steps/highlights, and local/error
status. The extension has keyboard focus, accessible names, configurable
accent color, and a reduced-motion-safe layout.

Exit gate: enabling/disabling the extension never starts inference in the Shell
process and reflects daemon state without jank.

## Phase 6 — computer-use observe and semantic action

- [x] Complete `cu-doctor`: Wayland/X11, portal, PipeWire, AT-SPI, libei, and
  optional fallback detection.
- [x] Implement portal ScreenCast/Screenshot and tmpfs frame normalization.
- [x] Implement GNOME Shell window truth and focus methods.
- [x] Implement AT-SPI tree snapshots with per-step stable refs.
- [x] Implement OCR and QVAC multimodal observe bundles.
- [x] Implement `cu.observe`, `cu.zoom`, `cu.tree`, and `cu.find`.
- [x] Keep frames local in `/tmp/jarvis-cu`, downscale to WebP before
  perception, and return explicit unavailable reasons when portal, Shell, or
  AT-SPI providers cannot connect.
- [x] Implement `cu.act` and semantic `cu.click`.
- [x] Implement grant/revoke, expiry, step budget, audit hashes, and no-frame
  retention by default.

Exit gate: “what’s on my screen?” returns grounded local observations without
actuation.

## Phase 7 — computer-use portal actuation

- [x] Implement RemoteDesktop portal consent and restore tokens.
- [x] Implement libei/EIS pointer, keyboard, scroll, drag, hover, and key input.
- [x] Implement typed Unicode and submit behavior.
- [x] Add target preview, agent cursor, and step ticker.
- [x] Prefer domain tools, app D-Bus, AT-SPI, Shell helper, then vision
  coordinates in that order.
- [x] Refuse password/PAM roles and lock-screen/greeter actions.
- [x] Require confirmation for destructive or high-impact actions.
- [x] Keep ydotool and X11 tools disabled unless explicitly enabled.
- [x] Add state-change self-checks, animation waits, and no-progress aborts.

Actuation is implemented in `computer-use/portal-input.js`, `computer-use/actuator.js`,
`computer-use/safety.js`, `computer-use/audit.js`, `skills/computer-act.js`, and
the local portal/AT-SPI helpers. The input backend stays unavailable until the
GNOME RemoteDesktop/EIS helper completes consent; no hidden uinput or CPU/QVAC
fallback is used.

Exit gate: Night Light, Text Editor save, Firefox URL entry, hands-off abort,
and lock-screen kill pass `docs/cu-acceptance.md`.

## Phase 8 — GTK4/libadwaita control center

- [x] Implement General, Voice, Models, Memory, Skills, Computer use, Privacy,
  Lab, and About pages.
- [x] Show the single QVAC master status, GPU backend, VRAM, model, fit result,
  queue, and failure reason.
- [x] Make every capability visible even when its model does not fit.
- [x] Add model download pause/resume through the master.
- [x] Add voice enrollment and local preview.
- [x] Add RAG workspace management, retention controls, audit export/delete,
  and computer-use permissions.

The GTK4/libadwaita application lives in `apps/control-center/main.py`. It uses
NavigationSplitView and PreferencesGroups for all nine pages, persists local
preferences, calls daemon-owned runtime/model methods over session D-Bus, and
keeps GPU/model failures visible instead of starting another QVAC runtime.

Exit gate: a user can configure the complete system without editing JSON.

## Phase 9 — full QVAC capability coverage

- [x] Chat, plan, summarize, rewrite, code, embeddings, RAG, batch prompts.
- [x] Multimodal screenshot/file analysis, OCR, classification.
- [x] Image generation/editing, video jobs, music jobs.
- [x] ASR, diarized meetings, TTS, voice clone enrollment, translation relay.
- [x] LoRA training with explicit confirmation and overnight job controls.
- [x] BCI, VLA, and ABot-World as clearly labeled Lab capabilities.
- [x] Registry/model manager, checksums, fit assessments, profiler, and job
  cancellation.
- [x] Optional P2P model fetch and memory sync only after local mode is stable.

Phase 9 adapters are registered in `skills/phase9-tools.js`. Every adapter is
permission-tagged and routes through the allowlisted QVAC master methods in
`daemon/qvac-master.js`; the harness and UI never import QVAC directly. Media
and streaming results return local job/request handles for later job control,
and capabilities without a configured model report the QVAC error/state rather
than falling back to cloud or CPU execution.

Exit gate: every capability in the product inventory has a working skill or a
truthful GPU/model-unavailable state.

## Phase 10 — reliability, privacy, and performance

- [x] OOM isolation: failed media jobs cannot kill the voice lane.
- [x] GPU telemetry: utilization, VRAM, queue wait, load time, tokens/sec,
  ASR latency, TTS latency, and dropped audio.
- [x] Wake, feedback, Wayland/X11, accessibility, lock/unlock, crash/restart,
  and nested Shell smoke tests.
- [x] Verify no network calls except explicitly enabled model fetch/P2P paths.
- [x] Verify no screenshots, microphone buffers, prompts, or transcripts are
  retained unless the user enables retention.
- [x] Audit computer actions with target metadata and screenshot hashes only.
- [x] Add graceful restart and state recovery for daemon crashes.

Phase 10 is implemented through `daemon/telemetry.js`, `daemon/recovery.js`,
`daemon/job-runner.js`, `daemon/network-policy.js`, the privacy log, and the
computer audit store. Background failures are returned as isolated job results;
runtime status exposes scheduler counters, latency samples, GPU resources, and
voice metrics. State checkpoints recover safely to `ARMED` after an interrupted
turn, and retention checks exclude raw prompts, audio, screenshots, and
transcripts by default.

Exit gate: the full test matrix passes on supported Ubuntu GNOME sessions with
GPU inference visibly confirmed.

## Phase 11 — packaging and release

- [x] Build a `.deb`, extension bundle, and user-service installer.
- [x] Add first-run wizard for microphone, wake phrase, GPU/model profile, TTS,
  and a typed smoke test.
- [x] Add uninstall that removes only Jarvis-owned data after explicit choice.
- [x] Publish a hardware compatibility matrix for Vulkan/GPU backends.
- [x] Add release checklist, migration notes, and a reproducible local demo.

Release automation lives in `.gitea/workflows/rolling-release.yml`. A push to
`main` runs the complete verification and package build, then replaces the
Gitea `rolling` prerelease using the repository's `RELEASE_TOKEN` secret.

Definition of done: all product behaviors in the specification work on Ubuntu
GNOME with one GPU-backed QVAC master, one harness planner, explicit computer-
use grants, and truthful capability status.