gnome-jarvis/docs/ROADMAP.md

JARVIS-QVAC roadmap

This is the implementation tracker for the local Ubuntu GNOME assistant. The roadmap is ordered by dependency: runtime authority first, then cognition, voice, desktop observation, actuation, UI, capability coverage, hardening, and packaging.

Product contract

Jarvis is local-first and session-scoped. The GNOME Shell extension stays thin; all inference, audio, jobs, memory, and computer-use decisions run outside gnome-shell. The extracted /home/raven/dev/agent-harness remains the sole cognitive loop and is wrapped rather than rewritten.

Non-negotiable runtime architecture

• One user service, jarvisd, owns the QVAC SDK worker.
• One QVAC_CONFIG_PATH points every Jarvis process at the same qvac.config.json.
• One model profile is active at a time. The QVAC master owns load, unload, suspend, resume, cancellation, and model status.
• Harness sessions call the master; skills never import QVAC independently and never call loadModel() directly.
• qvac serve --openai is an optional diagnostic/API surface and is never started by jarvisd; it must not run beside the master in production.
• GPU is mandatory for Jarvis runtime inference. The bridge requests GPU, enables all GPU layers and GPU multimodal projection, and rejects a load that QVAC reports as CPU. There is no silent CPU fallback.
• Audio capture, TTS, OCR, embeddings, RAG, media jobs, and multimodal work use the same QVAC master scheduler and obey a GPU memory budget.
• A capability may report unavailable: gpu-required without weakening the GPU policy or starting a competing runtime.

Status legend

• [x] implemented and locally verified
• [~] scaffolded or partially implemented
• [ ] planned

Phase 0 — repository and authority bootstrap

• Confirm /home/raven/dev/agent-harness exists.
• Link vendor/agent-harness to the extracted harness.
• Document harness layout, entrypoints, sessions, tools, memory, planner, and model path in docs/agent-harness-map.md.
• Pin the QVAC 0.19.0 API rules in QVAC.md.
• Create Node 22.17+ workspaces.
• Add one root qvac.config.json with localhost server settings and a single model alias.
• Add the initial daemon, computer-use, GNOME extension, D-Bus, and control-center boundaries.
• Install and lock dependencies with a successful npm install.
• Add repository CI for syntax, unit, schema, and extension checks.

Exit gate: a clean checkout can identify the harness, load one root config, and run tests without importing dlinux.

Phase 1 — single GPU QVAC master

• Add daemon/qvac-master.js as the only Jarvis QVAC owner.
• Serialize model loading and share the resulting harness engine.
• Request device: "gpu", gpu_layers: 99, and GPU multimodal projection.
• Reject a CPU result instead of accepting QVAC's internal fallback.
• Add owner counting and a single close path.
• Set QVAC_CONFIG_PATH, JARVIS_QVAC_MODEL, and GPU policy in the user service.
• [~] Add gpu-doctor; extend it to report QVAC resource capabilities, backend, driver, device name, VRAM, model fit, and the exact reason for failure.
• Add master scheduler lanes: interactive voice, computer-use vision, background media, and maintenance.
• Add VRAM admission control and queue media jobs instead of OOMing the voice lane.
• Add model profile selection using assessModelFit() before downloads.
• Add master-level cancellation by request ID and job ID.
• Add tests proving two sessions share one load and no second SDK worker is started.

Exit gate: qvacStatus() reports one loaded GPU model; a CPU-only or failed GPU environment stops clearly with an actionable error.

Phase 2 — harness bridge and Jarvis skills

• [~] Start the real harness through Agent.create() and route token events.
• Add the voice-native system prompt and structured HUD sidecar.
• Register all Jarvis tools through the harness custom-tool registry.
• Implement permission classes: read, write, dangerous, and computer-use.
• Connect confirmation events to spoken confirmation and HUD controls.
• Add desktop tools: launch/list apps, focus/list windows, workspaces, notify, screenshot, clipboard, media, settings, and focused text injection.
• Add file search/read/write with trash-first destructive handling.
• Add memory and RAG workspace tools.
• Add QVAC wrappers for embeddings, translation, OCR, classification, image/video/music jobs, transcription, TTS, LoRA, BCI, VLA, and ABot-World.
• Ensure every wrapper obtains the master lease and never loads QVAC itself.
• Add fake-QVAC fixture tests for every tool schema and permission gate.

Exit gate: a typed prompt completes through the real harness, streams tokens, executes a Jarvis tool, and returns a short local response through the same GPU-owned worker.

Phase 3 — daemon lifecycle and D-Bus

• [~] Define io.qvac.Jarvis.Session XML.
• Implement Arm, Sleep, Shutdown, Say, Ask, Cancel, SetMode, and state queries.
• Implement token, transcript, reply, audio-level, chip, job, computer step, and error signals.
• Keep payloads small; stream PCM and screenshots through a Unix socket or tmpfs paths.
• Add lock-screen handling: mute, hide UI, revoke computer use.
• Add idle sleep using QVAC suspend() and resume on wake.
• Add structured JSON logging with no prompt/audio/image contents by default.

Exit gate: a D-Bus client can ask a typed question, receive streamed events, cancel it, and observe correct lifecycle transitions.

Phase 4 — voice loop

• Add PipeWire capture at 16 kHz mono with a dedicated Jarvis node.
• Add wake-word engine behind a WakeEngine interface.
• Add VAD segmentation and the documented QVAC ASR stream.
• Implement ARMED → LISTENING → THINKING → SPEAKING → LISTENING.
• Add transcript filtering, TTS anti-feedback gate, and playback cooldown.
• Add sentence buffering from streamed harness output into QVAC TTS.
• Add fast-path cancel, sleep, privacy, dictate, screen, and computer-use commands.
• Add push-to-talk and typed fallback.
• Add wake false-accept/false-reject and feedback measurements.

Exit gate: “Hey Jarvis” starts a local GPU-backed turn, speaks a response, and does not self-trigger from its own TTS.

Phase 5 — GNOME ARC surface

• [~] Add GNOME 45+ ESM extension metadata, panel indicator, and settings keys.
• Add async D-Bus client and panel state glyphs.
• Add ARC overlay, waveform, transcript rows, chips, local/model status, reduced motion, high contrast, keyboard navigation, and screen-reader labels.
• Add listening halo, privacy slash, job theater, target highlights, and visible computer-use cursor.
• Keep all extension work nonblocking and free of QVAC/native inference.

Exit gate: enabling/disabling the extension never starts inference in the Shell process and reflects daemon state without jank.

Phase 6 — computer-use observe and semantic action

• Complete cu-doctor: Wayland/X11, portal, PipeWire, AT-SPI, libei, and optional fallback detection.
• Implement portal ScreenCast/Screenshot and tmpfs frame normalization.
• Implement GNOME Shell window truth and focus methods.
• Implement AT-SPI tree snapshots with per-step stable refs.
• Implement OCR and QVAC multimodal observe bundles.
• Implement cu.observe, cu.zoom, cu.tree, and cu.find.
• Implement cu.act and semantic cu.click.
• Implement grant/revoke, expiry, step budget, audit hashes, and no-frame retention by default.

Exit gate: “what’s on my screen?” returns grounded local observations without actuation.

Phase 7 — computer-use portal actuation

• Implement RemoteDesktop portal consent and restore tokens.
• Implement libei/EIS pointer, keyboard, scroll, drag, hover, and key input.
• Implement typed Unicode and submit behavior.
• Add target preview, agent cursor, and step ticker.
• Prefer domain tools, app D-Bus, AT-SPI, Shell helper, then vision coordinates in that order.
• Refuse password/PAM roles and lock-screen/greeter actions.
• Require confirmation for destructive or high-impact actions.
• Keep ydotool and X11 tools disabled unless explicitly enabled.
• Add state-change self-checks, animation waits, and no-progress aborts.

Exit gate: Night Light, Text Editor save, Firefox URL entry, hands-off abort, and lock-screen kill pass docs/cu-acceptance.md.

Phase 8 — GTK4/libadwaita control center

• Implement General, Voice, Models, Memory, Skills, Computer use, Privacy, Lab, and About pages.
• Show the single QVAC master status, GPU backend, VRAM, model, fit result, queue, and failure reason.
• Make every capability visible even when its model does not fit.
• Add model download pause/resume through the master.
• Add voice enrollment and local preview.
• Add RAG workspace management, retention controls, audit export/delete, and computer-use permissions.

Exit gate: a user can configure the complete system without editing JSON.

Phase 9 — full QVAC capability coverage

• Chat, plan, summarize, rewrite, code, embeddings, RAG, batch prompts.
• Multimodal screenshot/file analysis, OCR, classification.
• Image generation/editing, video jobs, music jobs.
• ASR, diarized meetings, TTS, voice clone enrollment, translation relay.
• LoRA training with explicit confirmation and overnight job controls.
• BCI, VLA, and ABot-World as clearly labeled Lab capabilities.
• Registry/model manager, checksums, fit assessments, profiler, and job cancellation.
• Optional P2P model fetch and memory sync only after local mode is stable.

Exit gate: every capability in the product inventory has a working skill or a truthful GPU/model-unavailable state.

Phase 10 — reliability, privacy, and performance

• OOM isolation: failed media jobs cannot kill the voice lane.
• GPU telemetry: utilization, VRAM, queue wait, load time, tokens/sec, ASR latency, TTS latency, and dropped audio.
• Wake, feedback, Wayland/X11, accessibility, lock/unlock, crash/restart, and nested Shell smoke tests.
• Verify no network calls except explicitly enabled model fetch/P2P paths.
• Verify no screenshots, microphone buffers, prompts, or transcripts are retained unless the user enables retention.
• Audit computer actions with target metadata and screenshot hashes only.
• Add graceful restart and state recovery for daemon crashes.

Exit gate: the full test matrix passes on supported Ubuntu GNOME sessions with GPU inference visibly confirmed.

Phase 11 — packaging and release

• Build a .deb, extension bundle, and user-service installer.
• Add first-run wizard for microphone, wake phrase, GPU/model profile, TTS, and a typed smoke test.
• Add uninstall that removes only Jarvis-owned data after explicit choice.
• Publish a hardware compatibility matrix for Vulkan/GPU backends.
• Add release checklist, migration notes, and a reproducible local demo.

Definition of done: all product behaviors in the specification work on Ubuntu GNOME with one GPU-backed QVAC master, one harness planner, explicit computer- use grants, and truthful capability status.