gnome-jarvis/docs/ROADMAP.md

# JARVIS-QVAC roadmap

This is the implementation tracker for the local Ubuntu GNOME assistant. The
roadmap is ordered by dependency: runtime authority first, then cognition,
voice, desktop observation, actuation, UI, capability coverage, hardening, and
packaging.

## Product contract

Jarvis is local-first and session-scoped. The GNOME Shell extension stays thin;
all inference, audio, jobs, memory, and computer-use decisions run outside
gnome-shell. The extracted `/home/raven/dev/agent-harness` remains the sole
cognitive loop and is wrapped rather than rewritten.

## Non-negotiable runtime architecture

- One user service, `jarvisd`, owns the QVAC SDK worker.
- One `QVAC_CONFIG_PATH` points every Jarvis process at the same
  `qvac.config.json`.
- One model profile is active at a time. The QVAC master owns load, unload,
  suspend, resume, cancellation, and model status.
- Harness sessions call the master; skills never import QVAC independently and
  never call `loadModel()` directly.
- `qvac serve --openai` is an optional diagnostic/API surface and is never
  started by `jarvisd`; it must not run beside the master in production.
- GPU is mandatory for Jarvis runtime inference. The bridge requests GPU,
  enables all GPU layers and GPU multimodal projection, and rejects a load
  that QVAC reports as CPU. There is no silent CPU fallback.
- Audio capture, TTS, OCR, embeddings, RAG, media jobs, and multimodal work
  use the same QVAC master scheduler and obey a GPU memory budget.
- A capability may report `unavailable: gpu-required` without weakening the
  GPU policy or starting a competing runtime.

## Status legend

- `[x]` implemented and locally verified
- `[~]` scaffolded or partially implemented
- `[ ]` planned

## Phase 0 — repository and authority bootstrap

- [x] Confirm `/home/raven/dev/agent-harness` exists.
- [x] Copy the extracted harness into `vendor/agent-harness`.
- [x] Document harness layout, entrypoints, sessions, tools, memory, planner,
  and model path in `docs/agent-harness-map.md`.
- [x] Pin the QVAC 0.19.0 API rules in `QVAC.md`.
- [x] Create Node 22.17+ workspaces.
- [x] Add one root `qvac.config.json` with localhost server settings and a
  single model alias.
- [x] Add the initial daemon, computer-use, GNOME extension, D-Bus, and
  control-center boundaries.
- [x] Install and lock dependencies with a successful `npm install`.
- [~] Review runtime dependency audit: `npm audit --omit=dev` reports 10
  transitive findings through `dbus-next` (3 critical, 1 high, 6 moderate;
  upstream reports no available fixes).
- [ ] Add repository CI for syntax, unit, schema, and extension checks.

Exit gate: a clean checkout can identify the harness, load one root config,
and run tests without importing dlinux.

## Phase 1 — single GPU QVAC master

- [x] Add `daemon/qvac-master.js` as the only Jarvis QVAC owner.
- [x] Serialize model loading and share the resulting harness engine.
- [x] Request `device: "gpu"`, `gpu_layers: 99`, and GPU multimodal projection.
- [x] Reject a CPU result instead of accepting QVAC's internal fallback.
- [x] Preflight QVAC GPU visibility before downloading or loading a model.
- [x] Enforce the vendored `@qvac/sdk` 0.19.x version at master startup.
- [x] Add owner counting and a single close path.
- [x] Set `QVAC_CONFIG_PATH`, `JARVIS_QVAC_MODEL`, and GPU policy in the user
  service.
- [~] Add `gpu-doctor`; extend it to report QVAC resource capabilities, backend,
  driver, device name, VRAM, model fit, and the exact reason for failure.
- [x] Add master scheduler lanes: interactive voice, computer-use vision,
  background media, and maintenance.
- [ ] Add VRAM admission control and queue media jobs instead of OOMing the
  voice lane.
- [ ] Add model profile selection using `assessModelFit()` before downloads.
- [ ] Add master-level cancellation by request ID and job ID.
- [ ] Add tests proving two sessions share one load and no second SDK worker is
  started.

Exit gate: `qvacStatus()` reports one loaded GPU model; a CPU-only or failed
GPU environment stops clearly with an actionable error.

## Phase 2 — harness bridge and Jarvis skills

- [~] Start the real harness through `Agent.create()` and route token events.
- [ ] Add the voice-native system prompt and structured HUD sidecar.
- [~] Register Jarvis runtime/status tools through the harness custom-tool registry.
- [ ] Implement permission classes: read, write, dangerous, and computer-use.
- [ ] Connect confirmation events to spoken confirmation and HUD controls.
- [ ] Add desktop tools: launch/list apps, focus/list windows, workspaces,
  notify, screenshot, clipboard, media, settings, and focused text injection.
- [ ] Add file search/read/write with trash-first destructive handling.
- [ ] Add memory and RAG workspace tools.
- [ ] Add QVAC wrappers for embeddings, translation, OCR, classification,
  image/video/music jobs, transcription, TTS, LoRA, BCI, VLA, and ABot-World.
- [ ] Ensure every wrapper obtains the master lease and never loads QVAC itself.
- [ ] Add fake-QVAC fixture tests for every tool schema and permission gate.

Exit gate: a typed prompt completes through the real harness, streams tokens,
executes a Jarvis tool, and returns a short local response through the same
GPU-owned worker.

## Phase 3 — daemon lifecycle and D-Bus

- [~] Define `io.qvac.Jarvis.Session` XML.
- [~] Implement `Arm`, `Sleep`, `Shutdown`, `Say`, `Ask`, `Cancel`, `SetMode`,
  and state queries.
- [~] Implement token, transcript, reply, audio-level, chip, job, computer
  step, and error signals.
- [~] Add the Node D-Bus service implementation. Live bus smoke testing remains
  pending on a normal user session because this sandbox cannot bind a D-Bus
  session socket.
- [ ] Keep payloads small; stream PCM and screenshots through a Unix socket or
  tmpfs paths.
- [ ] Add lock-screen handling: mute, hide UI, revoke computer use.
- [~] Add idle sleep using QVAC `suspend()` and resume on wake.
- [ ] Add structured JSON logging with no prompt/audio/image contents by default.

Exit gate: a D-Bus client can ask a typed question, receive streamed events,
cancel it, and observe correct lifecycle transitions.

## Phase 4 — voice loop

- [ ] Add PipeWire capture at 16 kHz mono with a dedicated `Jarvis` node.
- [ ] Add wake-word engine behind a `WakeEngine` interface.
- [ ] Add VAD segmentation and the documented QVAC ASR stream.
- [ ] Implement `ARMED → LISTENING → THINKING → SPEAKING → LISTENING`.
- [ ] Add transcript filtering, TTS anti-feedback gate, and playback cooldown.
- [ ] Add sentence buffering from streamed harness output into QVAC TTS.
- [ ] Add fast-path cancel, sleep, privacy, dictate, screen, and computer-use
  commands.
- [ ] Add push-to-talk and typed fallback.
- [ ] Add wake false-accept/false-reject and feedback measurements.

Exit gate: “Hey Jarvis” starts a local GPU-backed turn, speaks a response, and
does not self-trigger from its own TTS.

## Phase 5 — GNOME ARC surface

- [~] Add GNOME 45+ ESM extension metadata, panel indicator, and settings keys.
- [ ] Add async D-Bus client and panel state glyphs.
- [ ] Add ARC overlay, waveform, transcript rows, chips, local/model status,
  reduced motion, high contrast, keyboard navigation, and screen-reader labels.
- [ ] Add listening halo, privacy slash, job theater, target highlights, and
  visible computer-use cursor.
- [ ] Keep all extension work nonblocking and free of QVAC/native inference.

Exit gate: enabling/disabling the extension never starts inference in the Shell
process and reflects daemon state without jank.

## Phase 6 — computer-use observe and semantic action

- [ ] Complete `cu-doctor`: Wayland/X11, portal, PipeWire, AT-SPI, libei, and
  optional fallback detection.
- [ ] Implement portal ScreenCast/Screenshot and tmpfs frame normalization.
- [ ] Implement GNOME Shell window truth and focus methods.
- [ ] Implement AT-SPI tree snapshots with per-step stable refs.
- [ ] Implement OCR and QVAC multimodal observe bundles.
- [ ] Implement `cu.observe`, `cu.zoom`, `cu.tree`, and `cu.find`.
- [ ] Implement `cu.act` and semantic `cu.click`.
- [ ] Implement grant/revoke, expiry, step budget, audit hashes, and no-frame
  retention by default.

Exit gate: “what’s on my screen?” returns grounded local observations without
actuation.

## Phase 7 — computer-use portal actuation

- [ ] Implement RemoteDesktop portal consent and restore tokens.
- [ ] Implement libei/EIS pointer, keyboard, scroll, drag, hover, and key input.
- [ ] Implement typed Unicode and submit behavior.
- [ ] Add target preview, agent cursor, and step ticker.
- [ ] Prefer domain tools, app D-Bus, AT-SPI, Shell helper, then vision
  coordinates in that order.
- [ ] Refuse password/PAM roles and lock-screen/greeter actions.
- [ ] Require confirmation for destructive or high-impact actions.
- [ ] Keep ydotool and X11 tools disabled unless explicitly enabled.
- [ ] Add state-change self-checks, animation waits, and no-progress aborts.

Exit gate: Night Light, Text Editor save, Firefox URL entry, hands-off abort,
and lock-screen kill pass `docs/cu-acceptance.md`.

## Phase 8 — GTK4/libadwaita control center

- [ ] Implement General, Voice, Models, Memory, Skills, Computer use, Privacy,
  Lab, and About pages.
- [ ] Show the single QVAC master status, GPU backend, VRAM, model, fit result,
  queue, and failure reason.
- [ ] Make every capability visible even when its model does not fit.
- [ ] Add model download pause/resume through the master.
- [ ] Add voice enrollment and local preview.
- [ ] Add RAG workspace management, retention controls, audit export/delete,
  and computer-use permissions.

Exit gate: a user can configure the complete system without editing JSON.

## Phase 9 — full QVAC capability coverage

- [ ] Chat, plan, summarize, rewrite, code, embeddings, RAG, batch prompts.
- [ ] Multimodal screenshot/file analysis, OCR, classification.
- [ ] Image generation/editing, video jobs, music jobs.
- [ ] ASR, diarized meetings, TTS, voice clone enrollment, translation relay.
- [ ] LoRA training with explicit confirmation and overnight job controls.
- [ ] BCI, VLA, and ABot-World as clearly labeled Lab capabilities.
- [ ] Registry/model manager, checksums, fit assessments, profiler, and job
  cancellation.
- [ ] Optional P2P model fetch and memory sync only after local mode is stable.

Exit gate: every capability in the product inventory has a working skill or a
truthful GPU/model-unavailable state.

## Phase 10 — reliability, privacy, and performance

- [ ] OOM isolation: failed media jobs cannot kill the voice lane.
- [ ] GPU telemetry: utilization, VRAM, queue wait, load time, tokens/sec,
  ASR latency, TTS latency, and dropped audio.
- [ ] Wake, feedback, Wayland/X11, accessibility, lock/unlock, crash/restart,
  and nested Shell smoke tests.
- [ ] Verify no network calls except explicitly enabled model fetch/P2P paths.
- [ ] Verify no screenshots, microphone buffers, prompts, or transcripts are
  retained unless the user enables retention.
- [ ] Audit computer actions with target metadata and screenshot hashes only.
- [ ] Add graceful restart and state recovery for daemon crashes.

Exit gate: the full test matrix passes on supported Ubuntu GNOME sessions with
GPU inference visibly confirmed.

## Phase 11 — packaging and release

- [ ] Build a `.deb`, extension bundle, and user-service installer.
- [ ] Add first-run wizard for microphone, wake phrase, GPU/model profile, TTS,
  and a typed smoke test.
- [ ] Add uninstall that removes only Jarvis-owned data after explicit choice.
- [ ] Publish a hardware compatibility matrix for Vulkan/GPU backends.
- [ ] Add release checklist, migration notes, and a reproducible local demo.

Definition of done: all product behaviors in the specification work on Ubuntu
GNOME with one GPU-backed QVAC master, one harness planner, explicit computer-
use grants, and truthful capability status.