# JARVIS-QVAC JARVIS-QVAC is a local-first voice assistant and computer-use layer for Ubuntu GNOME. It combines a wake-word loop, QVAC speech and language models, the copied agent-harness planner, GNOME-native UI, local memory, and an explicit Wayland computer-use grant. The implementation covers Phases 0–11: daemon lifecycle, GPU-gated QVAC ownership, voice, ARC UI, computer use, control center, capability adapters, privacy hardening, packaging, and Gitea release automation. Full production acceptance remains host-dependent on GPU visibility, GNOME portals, the local wake bridge, and EIS input. ## Runtime contract JARVIS-QVAC has one cognitive loop and one QVAC authority. - jarvisd owns the QVAC SDK worker, model lifecycle, GPU policy, scheduler, cancellation, voice loop, jobs, and D-Bus service. - vendor/agent-harness is a copied repository dependency. It owns sessions, planning, memory, and the tool loop. - Skills call the QVAC master facade. They do not create workers or load models. - The GNOME Shell extension is presentation-only: no inference, audio capture, or input injection runs inside GNOME Shell. - GPU inference is mandatory. If QVAC cannot see a usable GPU, Jarvis reports an unavailable state and refuses CPU fallback. - Computer use is off until explicitly granted and is revoked on Escape, lock screen, expiry, or a spoken abort command. ~~~mermaid flowchart LR U[User] --> G[GNOME Shell / ARC] G <--> D[jarvisd / packaged Bare] D --> H[agent-harness] D --> M[Single QVAC master] D --> V[Voice + PipeWire] D --> C[Computer-use session] M --> GPU[(GPU backend)] C --> P[XDG portals + AT-SPI + EIS] ~~~ ## Capabilities - Wake word, push-to-talk, typed fallback, VAD, local ASR, streamed replies, TTS playback, anti-feedback gating, and cancellation. - Chat, planning, code, summaries, rewriting, embeddings, RAG workspaces, batch prompts, OCR, image classification, translation, and multimodal input. - Image, video, and music generation through serialized local QVAC jobs. - Voice enrollment, diarized transcription, LoRA training, and labeled BCI, VLA, and ABot-World lab slots. - GNOME app launch/focus/window actions, clipboard, notifications, media, screenshot observation, AT-SPI actions, portal input, and vision fallback. - ARC panel indicator and overlay, live waveform, transcripts, suggestions, job progress, target highlights, and visible agent cursor. Unavailable capabilities remain visible with an honest reason such as missing GPU, insufficient VRAM, missing model, unavailable hardware, or missing desktop permission. ## Requirements Full runtime support requires: - Ubuntu GNOME 24.04 or newer, GNOME Shell 46+, and a logged-in user session. - Bare 1.32.0 or newer, packaged with the application bundle. - PipeWire and WirePlumber. - xdg-desktop-portal with the GNOME backend. - AT-SPI2 and, for primary Wayland actuation, libei/libeis plus a local EIS injector. - A QVAC-visible Vulkan, CUDA, or supported GPU backend. - A local openWakeWord or sherpa-onnx bridge for always-on wake detection. ydotool is an optional legacy fallback and is disabled unless explicitly enabled. X11 fallback is used only in an Xorg session. CPU-only systems can run diagnostics and control surfaces, but Jarvis will not run inference on them. See [hardware compatibility](docs/hardware-compatibility.md) and [troubleshooting](docs/troubleshooting.md). ## Install with the web installer The recommended Gitea installer downloads the source snapshot at the current rolling ref, then runs the reviewed user-local installer. It does not require root access and does not install into system directories. ~~~bash curl -fsSL https://git.ssh.surf/snxraven/gnome-jarvis/raw/branch/main/packaging/web-installer.sh | bash -s -- --enable ~~~ Omit `--enable` to install without starting the service. Run the first-run setup afterward: ~~~bash bash ~/.local/share/jarvis-qvac/packaging/first-run.sh gnome-extensions enable jarvis@qvac.local If GNOME says the extension does not exist immediately after installation, log out and back in once. GNOME Shell only refreshes its user extension catalogue at session startup; the installer persists the enabled state for that login. ~~~ The rolling installer first downloads a self-contained architecture bundle with the Bare ELF runtime and all application dependencies. It therefore needs only curl and tar at runtime. A source fallback needs Node.js/npm when no published bundle exists for the requested ref. Set `JARVIS_SERVER`, `JARVIS_REPO`, or `JARVIS_BUNDLE_URL` for a mirror or a specific bundle. ## Install from a checkout The harness is copied into vendor/agent-harness. It is not a symlink and the dlinux platform is not vendored. ~~~bash git clone jarvis-qvac cd jarvis-qvac npm ci npm test npm run gpu-doctor npm run cu-doctor npm run voice-doctor ~~~ Install the user service, extension, and per-user QVAC configuration: ~~~bash bash packaging/install.sh --enable bash packaging/first-run.sh gnome-extensions enable jarvis@qvac.local ~~~ The first-run flow checks GPU/QVAC and PipeWire, records the wake phrase and model profile, optionally previews TTS, and sends a typed smoke-test prompt through the session bus. Inspect the installed daemon with: ~~~bash systemctl --user status jarvisd.service journalctl --user -u jarvisd.service -f ~~~ The service uses the application directory ~/.local/share/jarvis-qvac, configuration under ~/.config/jarvis, model cache under the user's cache directory, and user data paths described in [Operations](docs/operations.md). ## Development commands ~~~bash npm test # Node tests, including the vendored harness test npm run bare:smoke # Bare runtime and builtin mapping smoke test npm run gpu-doctor # QVAC GPU, VRAM, and profile diagnostics npm run cu-doctor # Wayland/X11, portals, AT-SPI, and libei npm run voice-doctor # PipeWire, wake bridge, and playback diagnostics npm run qvac:doctor # QVAC installation diagnostics npm run qvac:status # Single-master runtime status npm run package:test # Build and inspect release artifacts npm run first-run # Interactive first-run setup npm run start # Start jarvisd in the current session ~~~ The optional QVAC OpenAI-compatible server is diagnostic infrastructure only. If used, it binds to 127.0.0.1 and must not become a second production inference owner beside jarvisd. ## Computer use Computer use is a mode, not default chat behavior. Say “take the wheel” or select Computer in ARC to request a grant. Jarvis then prefers domain actions, app D-Bus/GIO, AT-SPI references, GNOME Shell helpers, and finally vision coordinates. RemoteDesktop plus EIS/libei is the primary Wayland path. Configure the local injector before live actuation: ~~~bash export JARVIS_LIBEI_BRIDGE="" ~~~ The helper will not claim input readiness without this bridge. Password fields, lock screens, dangerous keys, destructive targets, and actions outside the step budget are blocked or require confirmation. See the [computer-use acceptance procedure](docs/cu-acceptance.md). ## Privacy and safety - No cloud API is required and no telemetry is enabled. - Microphone audio is ring-buffered in memory; raw audio is not saved by default. - Computer-use frames stay local in temporary storage and are wiped on revoke unless trace retention is enabled. - Audit records contain action metadata and screenshot hashes, not pixels. - QVAC is localhost-only. Model fetch and optional P2P features are explicit opt-ins. - The daemon runs as the user and must never run as root. Read [Security and privacy](docs/security-privacy.md) before enabling legacy input, retention, synchronization, or computer-use traces. ## Build and release Build the Debian package, extension bundle, and checksums locally: ~~~bash npm ci npm test npm run package cat dist/SHA256SUMS ~~~ Artifacts in dist are: - jarvis-qvac__.deb - jarvis-qvac-extension.zip - jarvis-qvac-bare-linux-.tar.gz (self-contained runtime bundle) - SHA256SUMS Gitea Actions workflows live under .gitea/workflows. CI runs tests, Python checks, and package validation. A push to main additionally builds artifacts and replaces the mutable rolling prerelease using the repository secret RELEASE_TOKEN. Pull requests and non-main pushes do not publish a release. See [Gitea CI and rolling releases](docs/ci-gitea.md) and the [release checklist](docs/RELEASE.md). ## Uninstall and migration The uninstall command requires an explicit flag and preserves user data: ~~~bash bash packaging/uninstall.sh --yes ~~~ To remove Jarvis-owned memory, cache, and generated media as well: ~~~bash bash packaging/uninstall.sh --yes --purge-data ~~~ Re-running the installer upgrades in place: it stops the old daemon, removes the previous application and extension files, then copies the new tree. Config, memory, and model cache are kept. Those steps are written to `~/.local/state/jarvis/install.log`. Read [Migration notes](docs/MIGRATIONS.md) before upgrading or moving an installation. ## Documentation The [documentation index](docs/README.md) is the authoritative guide. - [Architecture](docs/architecture.md) - [QVAC runtime and GPU authority](docs/qvac-runtime.md) - [Agent harness integration](docs/harness-integration.md) - [Voice pipeline](docs/voice-pipeline.md) - [Computer use](docs/computer-use.md) - [API reference](docs/api.md) - [Operations](docs/operations.md) - [Testing](docs/testing.md) - [Troubleshooting](docs/troubleshooting.md) - [Roadmap](docs/ROADMAP.md) ## Project status Automated checks and repository implementation cover Phases 0–11. Before calling the deployment production-ready on a particular Ubuntu host, run the doctor commands and confirm QVAC reports the intended GPU, the wake bridge is configured, and the portal/EIS path is available.