snxraven/gnome-jarvis
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases 1 Wiki Activity

R1

Browse Source
This commit is contained in:
snxraven
2026-09-11 14:55:34 -04:00
parent 660294d0aa
commit 25b2c4be2e
33 changed files with 1293 additions and 34 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+222 -28
View File
@@ -1,38 +1,232 @@
# JARVIS-QVAC
Local-first Ubuntu GNOME voice assistant. The cognitive core is the
already-extracted `/home/raven/dev/agent-harness`, copied into
`vendor/agent-harness`; Jarvis does not copy or depend on `/home/raven/dev/dlinux`.
JARVIS-QVAC is a local-first voice assistant and computer-use layer for Ubuntu
GNOME. It combines a wake-word loop, QVAC speech and language models, the
copied agent-harness planner, GNOME-native UI, local memory, and an explicit
Wayland computer-use grant.
## Bootstrap
The implementation covers Phases 0–11: daemon lifecycle, GPU-gated QVAC
ownership, voice, ARC UI, computer use, control center, capability adapters,
privacy hardening, packaging, and Gitea release automation. Full production
acceptance remains host-dependent on GPU visibility, GNOME portals, the local
wake bridge, and EIS input.
```bash
npm install
## Runtime contract
JARVIS-QVAC has one cognitive loop and one QVAC authority.
- jarvisd owns the QVAC SDK worker, model lifecycle, GPU policy, scheduler,
cancellation, voice loop, jobs, and D-Bus service.
- vendor/agent-harness is a copied repository dependency. It owns sessions,
planning, memory, and the tool loop.
- Skills call the QVAC master facade. They do not create workers or load models.
- The GNOME Shell extension is presentation-only: no inference, audio capture,
or input injection runs inside GNOME Shell.
- GPU inference is mandatory. If QVAC cannot see a usable GPU, Jarvis reports
an unavailable state and refuses CPU fallback.
- Computer use is off until explicitly granted and is revoked on Escape, lock
screen, expiry, or a spoken abort command.
~~~mermaid
flowchart LR
U[User] --> G[GNOME Shell / ARC]
G <--> D[jarvisd / Node 22]
D --> H[agent-harness]
D --> M[Single QVAC master]
D --> V[Voice + PipeWire]
D --> C[Computer-use session]
M --> GPU[(GPU backend)]
C --> P[XDG portals + AT-SPI + EIS]
~~~
## Capabilities
- Wake word, push-to-talk, typed fallback, VAD, local ASR, streamed replies,
TTS playback, anti-feedback gating, and cancellation.
- Chat, planning, code, summaries, rewriting, embeddings, RAG workspaces,
batch prompts, OCR, image classification, translation, and multimodal input.
- Image, video, and music generation through serialized local QVAC jobs.
- Voice enrollment, diarized transcription, LoRA training, and labeled BCI,
VLA, and ABot-World lab slots.
- GNOME app launch/focus/window actions, clipboard, notifications, media,
screenshot observation, AT-SPI actions, portal input, and vision fallback.
- ARC panel indicator and overlay, live waveform, transcripts, suggestions,
job progress, target highlights, and visible agent cursor.
Unavailable capabilities remain visible with an honest reason such as missing
GPU, insufficient VRAM, missing model, unavailable hardware, or missing
desktop permission.
## Requirements
Full runtime support requires:
- Ubuntu GNOME 24.04 or newer, GNOME Shell 46+, and a logged-in user session.
- Node.js 22.17 or newer.
- PipeWire and WirePlumber.
- xdg-desktop-portal with the GNOME backend.
- AT-SPI2 and, for primary Wayland actuation, libei/libeis plus a local EIS
injector.
- A QVAC-visible Vulkan, CUDA, or supported GPU backend.
- A local openWakeWord or sherpa-onnx bridge for always-on wake detection.
ydotool is an optional legacy fallback and is disabled unless explicitly
enabled. X11 fallback is used only in an Xorg session. CPU-only systems can run
diagnostics and control surfaces, but Jarvis will not run inference on them.
See [hardware compatibility](docs/hardware-compatibility.md) and
[troubleshooting](docs/troubleshooting.md).
## Install from a checkout
The harness is copied into vendor/agent-harness. It is not a symlink and the
dlinux platform is not vendored.
~~~bash
git clone <gitea-repository-url> jarvis-qvac
cd jarvis-qvac
npm ci
npm test
npm run cu-doctor
npm run gpu-doctor
npx qvac doctor
npm run qvac:serve:diagnostic # optional; never run beside jarvisd
```
npm run cu-doctor
npm run voice-doctor
~~~
The daemon bridge loads the harness in-process with `Agent.engine.load()` and
creates sessions with `Agent.create()`. The optional QVAC HTTP sibling is
configured for `127.0.0.1:11434`. The GNOME extension is an intentionally thin
ESM panel shell; inference, audio, portals, and computer-use actuation belong
in user services.
Install the user service, extension, and per-user QVAC configuration:
GPU inference is a hard requirement. `gpu-doctor` must report a QVAC-visible
GPU and `jarvisd` refuses to continue if the loaded model reports `device: cpu`.
This prevents a misleading CPU mode and competing QVAC workers.
~~~bash
bash packaging/install.sh --enable
bash packaging/first-run.sh
gnome-extensions enable jarvis@qvac.local
~~~
The repository contains the daemon, single GPU-only QVAC master, copied agent
harness, voice loop, GNOME ARC extension, GTK4/libadwaita control center,
computer-use observation and safety boundaries, and the full QVAC capability
skill registry. See [docs/ROADMAP.md](docs/ROADMAP.md),
[docs/agent-harness-map.md](docs/agent-harness-map.md), and
[docs/cu-acceptance.md](docs/cu-acceptance.md).
The first-run flow checks GPU/QVAC and PipeWire, records the wake phrase and
model profile, optionally previews TTS, and sends a typed smoke-test prompt
through the session bus.
Runtime inference refuses to start until the QVAC SDK reports a usable GPU.
The RemoteDesktop/libei input bridge also remains unavailable until the local
GNOME portal session and a real EIS injector are present; Jarvis never claims
that an action succeeded when the backend is missing.
Inspect the installed daemon with:
~~~bash
systemctl --user status jarvisd.service
journalctl --user -u jarvisd.service -f
~~~
The service uses the application directory
~/.local/share/jarvis-qvac, configuration under ~/.config/jarvis, model cache
under the user's cache directory, and user data paths described in
[Operations](docs/operations.md).
## Development commands
~~~bash
npm test # Node tests, including the vendored harness test
npm run gpu-doctor # QVAC GPU, VRAM, and profile diagnostics
npm run cu-doctor # Wayland/X11, portals, AT-SPI, and libei
npm run voice-doctor # PipeWire, wake bridge, and playback diagnostics
npm run qvac:doctor # QVAC installation diagnostics
npm run qvac:status # Single-master runtime status
npm run package:test # Build and inspect release artifacts
npm run first-run # Interactive first-run setup
npm run start # Start jarvisd in the current session
~~~
The optional QVAC OpenAI-compatible server is diagnostic infrastructure only.
If used, it binds to 127.0.0.1 and must not become a second production
inference owner beside jarvisd.
## Computer use
Computer use is a mode, not default chat behavior. Say “take the wheel” or
select Computer in ARC to request a grant. Jarvis then prefers domain actions,
app D-Bus/GIO, AT-SPI references, GNOME Shell helpers, and finally vision
coordinates. RemoteDesktop plus EIS/libei is the primary Wayland path.
Configure the local injector before live actuation:
~~~bash
export JARVIS_LIBEI_BRIDGE="<local-eis-injector-command>"
~~~
The helper will not claim input readiness without this bridge. Password fields,
lock screens, dangerous keys, destructive targets, and actions outside the
step budget are blocked or require confirmation. See the
[computer-use acceptance procedure](docs/cu-acceptance.md).
## Privacy and safety
- No cloud API is required and no telemetry is enabled.
- Microphone audio is ring-buffered in memory; raw audio is not saved by
default.
- Computer-use frames stay local in temporary storage and are wiped on revoke
unless trace retention is enabled.
- Audit records contain action metadata and screenshot hashes, not pixels.
- QVAC is localhost-only. Model fetch and optional P2P features are explicit
opt-ins.
- The daemon runs as the user and must never run as root.
Read [Security and privacy](docs/security-privacy.md) before enabling legacy
input, retention, synchronization, or computer-use traces.
## Build and release
Build the Debian package, extension bundle, and checksums locally:
~~~bash
npm ci
npm test
npm run package
cat dist/SHA256SUMS
~~~
Artifacts in dist are:
- jarvis-qvac_<version>_<arch>.deb
- jarvis-qvac-extension.zip
- SHA256SUMS
Gitea Actions workflows live under .gitea/workflows. CI runs tests, Python
checks, and package validation. A push to main additionally builds artifacts
and replaces the mutable rolling prerelease using the repository secret
RELEASE_TOKEN. Pull requests and non-main pushes do not publish a release.
See [Gitea CI and rolling releases](docs/ci-gitea.md) and the
[release checklist](docs/RELEASE.md).
## Uninstall and migration
The uninstall command requires an explicit flag and preserves user data:
~~~bash
bash packaging/uninstall.sh --yes
~~~
To remove Jarvis-owned memory, cache, and generated media as well:
~~~bash
bash packaging/uninstall.sh --yes --purge-data
~~~
Read [Migration notes](docs/MIGRATIONS.md) before upgrading or moving an
installation.
## Documentation
The [documentation index](docs/README.md) is the authoritative guide.
- [Architecture](docs/architecture.md)
- [QVAC runtime and GPU authority](docs/qvac-runtime.md)
- [Agent harness integration](docs/harness-integration.md)
- [Voice pipeline](docs/voice-pipeline.md)
- [Computer use](docs/computer-use.md)
- [API reference](docs/api.md)
- [Operations](docs/operations.md)
- [Testing](docs/testing.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Roadmap](docs/ROADMAP.md)
## Project status
Automated checks and repository implementation cover Phases 0–11. Before
calling the deployment production-ready on a particular Ubuntu host, run the
doctor commands and confirm QVAC reports the intended GPU, the wake bridge is
configured, and the portal/EIS path is available.