gnome-jarvis/docs/ROADMAP.md

JARVIS-QVAC roadmap

This is the implementation tracker for the local Ubuntu GNOME assistant. The roadmap is ordered by dependency: runtime authority first, then cognition, voice, desktop observation, actuation, UI, capability coverage, hardening, and packaging.

Product contract

Jarvis is local-first and session-scoped. The GNOME Shell extension stays thin; all inference, audio, jobs, memory, and computer-use decisions run outside gnome-shell. The extracted /home/raven/dev/agent-harness remains the sole cognitive loop and is wrapped rather than rewritten.

Non-negotiable runtime architecture

• One user service, jarvisd, owns the QVAC SDK worker.
• One QVAC_CONFIG_PATH points every Jarvis process at the same qvac.config.json.
• One model profile is active at a time. The QVAC master owns load, unload, suspend, resume, cancellation, and model status.
• Harness sessions call the master; skills never import QVAC independently and never call loadModel() directly.
• qvac serve --openai is an optional diagnostic/API surface and is never started by jarvisd; it must not run beside the master in production.
• GPU is mandatory for Jarvis runtime inference. The bridge requests GPU, enables all GPU layers and GPU multimodal projection, and rejects a load that QVAC reports as CPU. There is no silent CPU fallback.
• Audio capture, TTS, OCR, embeddings, RAG, media jobs, and multimodal work use the same QVAC master scheduler and obey a GPU memory budget.
• A capability may report unavailable: gpu-required without weakening the GPU policy or starting a competing runtime.

Status legend

• [x] implemented and locally verified
• [~] scaffolded or partially implemented
• [ ] planned

Phase 0 — repository and authority bootstrap

• Confirm /home/raven/dev/agent-harness exists.
• Copy the extracted harness into vendor/agent-harness.
• Document harness layout, entrypoints, sessions, tools, memory, planner, and model path in docs/agent-harness-map.md.
• Pin the QVAC 0.19.x API rules in QVAC.md and lock the repository to @qvac/sdk 0.19.1.
• Create Node 22.17+ workspaces.
• Add one root qvac.config.json with localhost server settings and a single model alias.
• Add the initial daemon, computer-use, GNOME extension, D-Bus, and control-center boundaries.
• Install and lock dependencies with a successful npm install.
• Review runtime dependency audit: npm audit --omit=dev reports 10 transitive findings through dbus-next (3 critical, 1 high, 6 moderate; upstream reports no available fixes).
• Add repository CI for syntax, unit, schema, and extension checks.

Exit gate: a clean checkout can identify the harness, load one root config, and run tests without importing dlinux.

Phase 1 — single GPU QVAC master

• Add daemon/qvac-master.js as the only Jarvis QVAC owner.
• Serialize model loading and share the resulting harness engine.
• Request device: "gpu", gpu_layers: 99, and GPU multimodal projection.
• Reject a CPU result instead of accepting QVAC's internal fallback.
• Preflight QVAC GPU visibility before downloading or loading a model.
• Enforce the vendored @qvac/sdk 0.19.x version at master startup.
• Add owner counting and a single close path.
• Set QVAC_CONFIG_PATH, JARVIS_QVAC_MODEL, and GPU policy in the user service.
• Add gpu-doctor with QVAC resources, backend hints, device, VRAM, SDK, and model-profile reporting.
• Add master scheduler lanes: interactive voice, computer-use vision, background media, and maintenance.
• Add single-concurrency admission and lane queues so background jobs do not run beside the voice lane.
• Add model profile selection and master-owned assessModelFit() access.
• Add master-level cancellation by request ID or model ID.
• Add ownership/status tests and a single master module boundary.

Exit gate: qvacStatus() reports one loaded GPU model; a CPU-only or failed GPU environment stops clearly with an actionable error.

Phase 2 — harness bridge and Jarvis skills

• Start the real harness through Agent.create() and route token events; GPU preflight is the only runtime gate.
• Add the voice-native system prompt and structured HUD sidecar.
• Register Jarvis runtime/status and safe local tools through the harness custom-tool registry.
• Define permission classes: read, write, dangerous, and computer-use.
• Connect confirmation events to daemon/D-Bus events for HUD and spoken confirmation consumers.
• Register desktop tool adapters and truthful unavailable states; actual GNOME launch/focus/window/workspace, screenshot, clipboard, media, settings, and focused text actuators belong to Phase 4.
• Add file search/read, confirmed writes, local memory writes/recall, and RAG workspace discovery; QVAC retrieval remains in the capability adapter.
• Add QVAC capability registry/status plus master-owned lifecycle/resource/ model-fit wrappers; embeddings, translation, OCR, classification, image/video/music jobs, transcription, TTS, LoRA, BCI, VLA, and ABot-World.
• Ensure current wrappers use the master status and never load QVAC themselves.
• Add fixture tests for current tool schemas and permission metadata.

Exit gate: a typed prompt completes through the real harness, streams tokens, executes a Jarvis tool, and returns a short local response through the same GPU-owned worker.

Phase 1–2 gate status

The implementation gates are complete: the pinned vendored harness, single GPU-only QVAC master, scheduler, voice prompt/sidecar, permission metadata, safe local tools, and master-owned QVAC utility wrappers are present and tested. The live typed-turn acceptance check is pending one host condition: QVAC must observe the host Vulkan GPU. Until that condition is true, the master refuses to download or load a model and no CPU fallback is allowed.

Phase 3 — daemon lifecycle and D-Bus

• Define io.qvac.Jarvis.Session XML.
• Implement Arm, Sleep, Shutdown, Say, Ask, Cancel, SetMode, and state queries.
• Implement token, transcript, reply, audio-level, chip, job, computer step, and error signals.
• Add the Node D-Bus service implementation. Live bus smoke testing remains pending on a normal user session because this sandbox cannot bind a D-Bus session socket.
• Keep payloads small; stream events through a bounded Unix socket and use file paths for PCM/screenshots.
• Add lock-screen handling: cancel, arm, and revoke computer use.
• Add idle sleep using QVAC suspend() and resume on wake.
• Add structured JSON logging with an allowlist that excludes prompt, audio, image, and transcript contents by default.

Exit gate: a D-Bus client can ask a typed question, receive streamed events, cancel it, and observe correct lifecycle transitions.

Phase 0–3 review result

Phases 0–3 are complete at the implementation and local verification level. The repository has a real copied harness at vendor/agent-harness, one GPU-only QVAC authority, the Phase 2 harness/tool bridge, and the Phase 3 daemon lifecycle and D-Bus contract. The Phase 4 handoff is ready.

Two acceptance checks depend on the host session rather than repository code:

• npm run gpu-doctor currently reports no GPU visible to the QVAC SDK, even though the host exposes an AMD Radeon device and Vulkan packages. The master therefore refuses inference by design until the QVAC Vulkan backend sees the GPU; it never falls back to CPU.
• Live D-Bus session smoke testing must run in a normal GNOME user session; this restricted build environment cannot bind a session-bus socket.

These are explicit environment gates for the Phase 1–3 acceptance demos. They do not leave an implementation item open or justify starting a second QVAC runtime.

Phase 4 — voice loop

• Add PipeWire capture at 16 kHz mono with a dedicated Jarvis node.
• Add wake-word engine behind a WakeEngine interface.
• Add VAD segmentation and the documented QVAC ASR stream.
• Implement ARMED → LISTENING → THINKING → SPEAKING → LISTENING.
• Add transcript filtering, TTS anti-feedback gate, and playback cooldown.
• Add sentence buffering from streamed harness output into QVAC TTS.
• Add fast-path cancel, sleep, privacy, dictate, screen, and computer-use commands.
• Add push-to-talk and typed fallback.
• Add wake false-accept/false-reject and feedback measurements.

Implementation is complete in daemon/audio-pipewire.js, daemon/wake-engine.js, daemon/vad.js, daemon/voice-adapters.js, daemon/voice-loop.js, and daemon/audio-playback.js. The SDK speech models are loaded as auxiliary models by the same QVAC master and use GPU settings. Live wake-word accuracy requires a local detector bridge configured with JARVIS_WAKE_COMMAND; the repository does not silently substitute cloud or CPU inference.

Exit gate: implementation complete. In a configured GNOME session, “Hey Jarvis” starts a local GPU-backed turn, speaks a response, and does not self-trigger from its own TTS. docs/voice-acceptance.md contains the live hardware/session acceptance procedure.

Phase 5 — GNOME ARC surface

• Add GNOME 45+ ESM extension metadata, panel indicator, and settings keys.
• Add async D-Bus client and panel state glyphs.
• Add ARC overlay, waveform, transcript rows, chips, local/model status, reduced motion, high contrast, keyboard navigation, and screen-reader labels.
• Add listening halo, privacy slash, job theater, target highlights, and visible computer-use cursor.
• Keep all extension work nonblocking and free of QVAC/native inference.

The extension implementation is in apps/gnome-extension/[email protected]/. It owns only Shell UI and an asynchronous session-bus client; all model work stays in jarvisd. ARC renders 48 waveform bars, transcript rows, mode and suggestion chips, state glyphs, computer-use steps/highlights, and local/error status. The extension has keyboard focus, accessible names, configurable accent color, and a reduced-motion-safe layout.

Exit gate: enabling/disabling the extension never starts inference in the Shell process and reflects daemon state without jank.

Phase 6 — computer-use observe and semantic action

• Complete cu-doctor: Wayland/X11, portal, PipeWire, AT-SPI, libei, and optional fallback detection.
• Implement portal ScreenCast/Screenshot and tmpfs frame normalization.
• Implement GNOME Shell window truth and focus methods.
• Implement AT-SPI tree snapshots with per-step stable refs.
• Implement OCR and QVAC multimodal observe bundles.
• Implement cu.observe, cu.zoom, cu.tree, and cu.find.
• Keep frames local in /tmp/jarvis-cu, downscale to WebP before perception, and return explicit unavailable reasons when portal, Shell, or AT-SPI providers cannot connect.
• Implement cu.act and semantic cu.click.
• Implement grant/revoke, expiry, step budget, audit hashes, and no-frame retention by default.

Exit gate: “what’s on my screen?” returns grounded local observations without actuation.

Phase 7 — computer-use portal actuation

• Implement RemoteDesktop portal consent and restore tokens.
• Implement libei/EIS pointer, keyboard, scroll, drag, hover, and key input.
• Implement typed Unicode and submit behavior.
• Add target preview, agent cursor, and step ticker.
• Prefer domain tools, app D-Bus, AT-SPI, Shell helper, then vision coordinates in that order.
• Refuse password/PAM roles and lock-screen/greeter actions.
• Require confirmation for destructive or high-impact actions.
• Keep ydotool and X11 tools disabled unless explicitly enabled.
• Add state-change self-checks, animation waits, and no-progress aborts.

Actuation is implemented in computer-use/portal-input.js, computer-use/actuator.js, computer-use/safety.js, computer-use/audit.js, skills/computer-act.js, and the local portal/AT-SPI helpers. The input backend stays unavailable until the GNOME RemoteDesktop/EIS helper completes consent; no hidden uinput or CPU/QVAC fallback is used.

Exit gate: Night Light, Text Editor save, Firefox URL entry, hands-off abort, and lock-screen kill pass docs/cu-acceptance.md.

Phase 8 — GTK4/libadwaita control center

• Implement General, Voice, Models, Memory, Skills, Computer use, Privacy, Lab, and About pages.
• Show the single QVAC master status, GPU backend, VRAM, model, fit result, queue, and failure reason.
• Make every capability visible even when its model does not fit.
• Add model download pause/resume through the master.
• Add voice enrollment and local preview.
• Add RAG workspace management, retention controls, audit export/delete, and computer-use permissions.

The GTK4/libadwaita application lives in apps/control-center/main.py. It uses NavigationSplitView and PreferencesGroups for all nine pages, persists local preferences, calls daemon-owned runtime/model methods over session D-Bus, and keeps GPU/model failures visible instead of starting another QVAC runtime.

Exit gate: a user can configure the complete system without editing JSON.

Phase 9 — full QVAC capability coverage

• Chat, plan, summarize, rewrite, code, embeddings, RAG, batch prompts.
• Multimodal screenshot/file analysis, OCR, classification.
• Image generation/editing, video jobs, music jobs.
• ASR, diarized meetings, TTS, voice clone enrollment, translation relay.
• LoRA training with explicit confirmation and overnight job controls.
• BCI, VLA, and ABot-World as clearly labeled Lab capabilities.
• Registry/model manager, checksums, fit assessments, profiler, and job cancellation.
• Optional P2P model fetch and memory sync only after local mode is stable.

Phase 9 adapters are registered in skills/phase9-tools.js. Every adapter is permission-tagged and routes through the allowlisted QVAC master methods in daemon/qvac-master.js; the harness and UI never import QVAC directly. Media and streaming results return local job/request handles for later job control, and capabilities without a configured model report the QVAC error/state rather than falling back to cloud or CPU execution.

Exit gate: every capability in the product inventory has a working skill or a truthful GPU/model-unavailable state.

Phase 10 — reliability, privacy, and performance

• OOM isolation: failed media jobs cannot kill the voice lane.
• GPU telemetry: utilization, VRAM, queue wait, load time, tokens/sec, ASR latency, TTS latency, and dropped audio.
• Wake, feedback, Wayland/X11, accessibility, lock/unlock, crash/restart, and nested Shell smoke tests.
• Verify no network calls except explicitly enabled model fetch/P2P paths.
• Verify no screenshots, microphone buffers, prompts, or transcripts are retained unless the user enables retention.
• Audit computer actions with target metadata and screenshot hashes only.
• Add graceful restart and state recovery for daemon crashes.

Phase 10 is implemented through daemon/telemetry.js, daemon/recovery.js, daemon/job-runner.js, daemon/network-policy.js, the privacy log, and the computer audit store. Background failures are returned as isolated job results; runtime status exposes scheduler counters, latency samples, GPU resources, and voice metrics. State checkpoints recover safely to ARMED after an interrupted turn, and retention checks exclude raw prompts, audio, screenshots, and transcripts by default.

Exit gate: the full test matrix passes on supported Ubuntu GNOME sessions with GPU inference visibly confirmed.

Phase 11 — packaging and release

• Build a .deb, extension bundle, and user-service installer.
• Add first-run wizard for microphone, wake phrase, GPU/model profile, TTS, and a typed smoke test.
• Add uninstall that removes only Jarvis-owned data after explicit choice.
• Publish a hardware compatibility matrix for Vulkan/GPU backends.
• Add release checklist, migration notes, and a reproducible local demo.

Release automation lives in .gitea/workflows/rolling-release.yml. A push to main runs the complete verification and package build, then replaces the Gitea rolling prerelease using the repository's RELEASE_TOKEN secret.

Definition of done: all product behaviors in the specification work on Ubuntu GNOME with one GPU-backed QVAC master, one harness planner, explicit computer- use grants, and truthful capability status.