first commit
This commit is contained in:
+234
@@ -0,0 +1,234 @@
|
||||
# JARVIS-QVAC roadmap
|
||||
|
||||
This is the implementation tracker for the local Ubuntu GNOME assistant. The
|
||||
roadmap is ordered by dependency: runtime authority first, then cognition,
|
||||
voice, desktop observation, actuation, UI, capability coverage, hardening, and
|
||||
packaging.
|
||||
|
||||
## Product contract
|
||||
|
||||
Jarvis is local-first and session-scoped. The GNOME Shell extension stays thin;
|
||||
all inference, audio, jobs, memory, and computer-use decisions run outside
|
||||
gnome-shell. The extracted `/home/raven/dev/agent-harness` remains the sole
|
||||
cognitive loop and is wrapped rather than rewritten.
|
||||
|
||||
## Non-negotiable runtime architecture
|
||||
|
||||
- One user service, `jarvisd`, owns the QVAC SDK worker.
|
||||
- One `QVAC_CONFIG_PATH` points every Jarvis process at the same
|
||||
`qvac.config.json`.
|
||||
- One model profile is active at a time. The QVAC master owns load, unload,
|
||||
suspend, resume, cancellation, and model status.
|
||||
- Harness sessions call the master; skills never import QVAC independently and
|
||||
never call `loadModel()` directly.
|
||||
- `qvac serve --openai` is an optional diagnostic/API surface and is never
|
||||
started by `jarvisd`; it must not run beside the master in production.
|
||||
- GPU is mandatory for Jarvis runtime inference. The bridge requests GPU,
|
||||
enables all GPU layers and GPU multimodal projection, and rejects a load
|
||||
that QVAC reports as CPU. There is no silent CPU fallback.
|
||||
- Audio capture, TTS, OCR, embeddings, RAG, media jobs, and multimodal work
|
||||
use the same QVAC master scheduler and obey a GPU memory budget.
|
||||
- A capability may report `unavailable: gpu-required` without weakening the
|
||||
GPU policy or starting a competing runtime.
|
||||
|
||||
## Status legend
|
||||
|
||||
- `[x]` implemented and locally verified
|
||||
- `[~]` scaffolded or partially implemented
|
||||
- `[ ]` planned
|
||||
|
||||
## Phase 0 — repository and authority bootstrap
|
||||
|
||||
- [x] Confirm `/home/raven/dev/agent-harness` exists.
|
||||
- [x] Link `vendor/agent-harness` to the extracted harness.
|
||||
- [x] Document harness layout, entrypoints, sessions, tools, memory, planner,
|
||||
and model path in `docs/agent-harness-map.md`.
|
||||
- [x] Pin the QVAC 0.19.0 API rules in `QVAC.md`.
|
||||
- [x] Create Node 22.17+ workspaces.
|
||||
- [x] Add one root `qvac.config.json` with localhost server settings and a
|
||||
single model alias.
|
||||
- [x] Add the initial daemon, computer-use, GNOME extension, D-Bus, and
|
||||
control-center boundaries.
|
||||
- [ ] Install and lock dependencies with a successful `npm install`.
|
||||
- [ ] Add repository CI for syntax, unit, schema, and extension checks.
|
||||
|
||||
Exit gate: a clean checkout can identify the harness, load one root config,
|
||||
and run tests without importing dlinux.
|
||||
|
||||
## Phase 1 — single GPU QVAC master
|
||||
|
||||
- [x] Add `daemon/qvac-master.js` as the only Jarvis QVAC owner.
|
||||
- [x] Serialize model loading and share the resulting harness engine.
|
||||
- [x] Request `device: "gpu"`, `gpu_layers: 99`, and GPU multimodal projection.
|
||||
- [x] Reject a CPU result instead of accepting QVAC's internal fallback.
|
||||
- [x] Add owner counting and a single close path.
|
||||
- [x] Set `QVAC_CONFIG_PATH`, `JARVIS_QVAC_MODEL`, and GPU policy in the user
|
||||
service.
|
||||
- [~] Add `gpu-doctor`; extend it to report QVAC resource capabilities, backend,
|
||||
driver, device name, VRAM, model fit, and the exact reason for failure.
|
||||
- [ ] Add master scheduler lanes: interactive voice, computer-use vision,
|
||||
background media, and maintenance.
|
||||
- [ ] Add VRAM admission control and queue media jobs instead of OOMing the
|
||||
voice lane.
|
||||
- [ ] Add model profile selection using `assessModelFit()` before downloads.
|
||||
- [ ] Add master-level cancellation by request ID and job ID.
|
||||
- [ ] Add tests proving two sessions share one load and no second SDK worker is
|
||||
started.
|
||||
|
||||
Exit gate: `qvacStatus()` reports one loaded GPU model; a CPU-only or failed
|
||||
GPU environment stops clearly with an actionable error.
|
||||
|
||||
## Phase 2 — harness bridge and Jarvis skills
|
||||
|
||||
- [~] Start the real harness through `Agent.create()` and route token events.
|
||||
- [ ] Add the voice-native system prompt and structured HUD sidecar.
|
||||
- [ ] Register all Jarvis tools through the harness custom-tool registry.
|
||||
- [ ] Implement permission classes: read, write, dangerous, and computer-use.
|
||||
- [ ] Connect confirmation events to spoken confirmation and HUD controls.
|
||||
- [ ] Add desktop tools: launch/list apps, focus/list windows, workspaces,
|
||||
notify, screenshot, clipboard, media, settings, and focused text injection.
|
||||
- [ ] Add file search/read/write with trash-first destructive handling.
|
||||
- [ ] Add memory and RAG workspace tools.
|
||||
- [ ] Add QVAC wrappers for embeddings, translation, OCR, classification,
|
||||
image/video/music jobs, transcription, TTS, LoRA, BCI, VLA, and ABot-World.
|
||||
- [ ] Ensure every wrapper obtains the master lease and never loads QVAC itself.
|
||||
- [ ] Add fake-QVAC fixture tests for every tool schema and permission gate.
|
||||
|
||||
Exit gate: a typed prompt completes through the real harness, streams tokens,
|
||||
executes a Jarvis tool, and returns a short local response through the same
|
||||
GPU-owned worker.
|
||||
|
||||
## Phase 3 — daemon lifecycle and D-Bus
|
||||
|
||||
- [~] Define `io.qvac.Jarvis.Session` XML.
|
||||
- [ ] Implement `Arm`, `Sleep`, `Shutdown`, `Say`, `Ask`, `Cancel`, `SetMode`,
|
||||
and state queries.
|
||||
- [ ] Implement token, transcript, reply, audio-level, chip, job, computer
|
||||
step, and error signals.
|
||||
- [ ] Keep payloads small; stream PCM and screenshots through a Unix socket or
|
||||
tmpfs paths.
|
||||
- [ ] Add lock-screen handling: mute, hide UI, revoke computer use.
|
||||
- [ ] Add idle sleep using QVAC `suspend()` and resume on wake.
|
||||
- [ ] Add structured JSON logging with no prompt/audio/image contents by default.
|
||||
|
||||
Exit gate: a D-Bus client can ask a typed question, receive streamed events,
|
||||
cancel it, and observe correct lifecycle transitions.
|
||||
|
||||
## Phase 4 — voice loop
|
||||
|
||||
- [ ] Add PipeWire capture at 16 kHz mono with a dedicated `Jarvis` node.
|
||||
- [ ] Add wake-word engine behind a `WakeEngine` interface.
|
||||
- [ ] Add VAD segmentation and the documented QVAC ASR stream.
|
||||
- [ ] Implement `ARMED → LISTENING → THINKING → SPEAKING → LISTENING`.
|
||||
- [ ] Add transcript filtering, TTS anti-feedback gate, and playback cooldown.
|
||||
- [ ] Add sentence buffering from streamed harness output into QVAC TTS.
|
||||
- [ ] Add fast-path cancel, sleep, privacy, dictate, screen, and computer-use
|
||||
commands.
|
||||
- [ ] Add push-to-talk and typed fallback.
|
||||
- [ ] Add wake false-accept/false-reject and feedback measurements.
|
||||
|
||||
Exit gate: “Hey Jarvis” starts a local GPU-backed turn, speaks a response, and
|
||||
does not self-trigger from its own TTS.
|
||||
|
||||
## Phase 5 — GNOME ARC surface
|
||||
|
||||
- [~] Add GNOME 45+ ESM extension metadata, panel indicator, and settings keys.
|
||||
- [ ] Add async D-Bus client and panel state glyphs.
|
||||
- [ ] Add ARC overlay, waveform, transcript rows, chips, local/model status,
|
||||
reduced motion, high contrast, keyboard navigation, and screen-reader labels.
|
||||
- [ ] Add listening halo, privacy slash, job theater, target highlights, and
|
||||
visible computer-use cursor.
|
||||
- [ ] Keep all extension work nonblocking and free of QVAC/native inference.
|
||||
|
||||
Exit gate: enabling/disabling the extension never starts inference in the Shell
|
||||
process and reflects daemon state without jank.
|
||||
|
||||
## Phase 6 — computer-use observe and semantic action
|
||||
|
||||
- [ ] Complete `cu-doctor`: Wayland/X11, portal, PipeWire, AT-SPI, libei, and
|
||||
optional fallback detection.
|
||||
- [ ] Implement portal ScreenCast/Screenshot and tmpfs frame normalization.
|
||||
- [ ] Implement GNOME Shell window truth and focus methods.
|
||||
- [ ] Implement AT-SPI tree snapshots with per-step stable refs.
|
||||
- [ ] Implement OCR and QVAC multimodal observe bundles.
|
||||
- [ ] Implement `cu.observe`, `cu.zoom`, `cu.tree`, and `cu.find`.
|
||||
- [ ] Implement `cu.act` and semantic `cu.click`.
|
||||
- [ ] Implement grant/revoke, expiry, step budget, audit hashes, and no-frame
|
||||
retention by default.
|
||||
|
||||
Exit gate: “what’s on my screen?” returns grounded local observations without
|
||||
actuation.
|
||||
|
||||
## Phase 7 — computer-use portal actuation
|
||||
|
||||
- [ ] Implement RemoteDesktop portal consent and restore tokens.
|
||||
- [ ] Implement libei/EIS pointer, keyboard, scroll, drag, hover, and key input.
|
||||
- [ ] Implement typed Unicode and submit behavior.
|
||||
- [ ] Add target preview, agent cursor, and step ticker.
|
||||
- [ ] Prefer domain tools, app D-Bus, AT-SPI, Shell helper, then vision
|
||||
coordinates in that order.
|
||||
- [ ] Refuse password/PAM roles and lock-screen/greeter actions.
|
||||
- [ ] Require confirmation for destructive or high-impact actions.
|
||||
- [ ] Keep ydotool and X11 tools disabled unless explicitly enabled.
|
||||
- [ ] Add state-change self-checks, animation waits, and no-progress aborts.
|
||||
|
||||
Exit gate: Night Light, Text Editor save, Firefox URL entry, hands-off abort,
|
||||
and lock-screen kill pass `docs/cu-acceptance.md`.
|
||||
|
||||
## Phase 8 — GTK4/libadwaita control center
|
||||
|
||||
- [ ] Implement General, Voice, Models, Memory, Skills, Computer use, Privacy,
|
||||
Lab, and About pages.
|
||||
- [ ] Show the single QVAC master status, GPU backend, VRAM, model, fit result,
|
||||
queue, and failure reason.
|
||||
- [ ] Make every capability visible even when its model does not fit.
|
||||
- [ ] Add model download pause/resume through the master.
|
||||
- [ ] Add voice enrollment and local preview.
|
||||
- [ ] Add RAG workspace management, retention controls, audit export/delete,
|
||||
and computer-use permissions.
|
||||
|
||||
Exit gate: a user can configure the complete system without editing JSON.
|
||||
|
||||
## Phase 9 — full QVAC capability coverage
|
||||
|
||||
- [ ] Chat, plan, summarize, rewrite, code, embeddings, RAG, batch prompts.
|
||||
- [ ] Multimodal screenshot/file analysis, OCR, classification.
|
||||
- [ ] Image generation/editing, video jobs, music jobs.
|
||||
- [ ] ASR, diarized meetings, TTS, voice clone enrollment, translation relay.
|
||||
- [ ] LoRA training with explicit confirmation and overnight job controls.
|
||||
- [ ] BCI, VLA, and ABot-World as clearly labeled Lab capabilities.
|
||||
- [ ] Registry/model manager, checksums, fit assessments, profiler, and job
|
||||
cancellation.
|
||||
- [ ] Optional P2P model fetch and memory sync only after local mode is stable.
|
||||
|
||||
Exit gate: every capability in the product inventory has a working skill or a
|
||||
truthful GPU/model-unavailable state.
|
||||
|
||||
## Phase 10 — reliability, privacy, and performance
|
||||
|
||||
- [ ] OOM isolation: failed media jobs cannot kill the voice lane.
|
||||
- [ ] GPU telemetry: utilization, VRAM, queue wait, load time, tokens/sec,
|
||||
ASR latency, TTS latency, and dropped audio.
|
||||
- [ ] Wake, feedback, Wayland/X11, accessibility, lock/unlock, crash/restart,
|
||||
and nested Shell smoke tests.
|
||||
- [ ] Verify no network calls except explicitly enabled model fetch/P2P paths.
|
||||
- [ ] Verify no screenshots, microphone buffers, prompts, or transcripts are
|
||||
retained unless the user enables retention.
|
||||
- [ ] Audit computer actions with target metadata and screenshot hashes only.
|
||||
- [ ] Add graceful restart and state recovery for daemon crashes.
|
||||
|
||||
Exit gate: the full test matrix passes on supported Ubuntu GNOME sessions with
|
||||
GPU inference visibly confirmed.
|
||||
|
||||
## Phase 11 — packaging and release
|
||||
|
||||
- [ ] Build a `.deb`, extension bundle, and user-service installer.
|
||||
- [ ] Add first-run wizard for microphone, wake phrase, GPU/model profile, TTS,
|
||||
and a typed smoke test.
|
||||
- [ ] Add uninstall that removes only Jarvis-owned data after explicit choice.
|
||||
- [ ] Publish a hardware compatibility matrix for Vulkan/GPU backends.
|
||||
- [ ] Add release checklist, migration notes, and a reproducible local demo.
|
||||
|
||||
Definition of done: all product behaviors in the specification work on Ubuntu
|
||||
GNOME with one GPU-backed QVAC master, one harness planner, explicit computer-
|
||||
use grants, and truthful capability status.
|
||||
Reference in New Issue
Block a user