gnome-jarvis/README.md

<p align="center">
  <img src="apps/gnome-extension/jarvis@qvac.local/brand/icons/jarvis-wordmark.svg" alt="Jarvis" width="320">
</p>

Local voice for your desktop.

Jarvis is a local-first voice assistant and computer-use layer for Ubuntu
GNOME. It combines a wake-word loop, QVAC speech and language models, the
copied agent-harness planner, GNOME-native UI, local memory, and an explicit
Wayland computer-use grant.

It includes daemon lifecycle, GPU-gated QVAC ownership, voice, tray HUD,
computer use, control center, capability adapters, privacy hardening,
packaging, and Gitea release automation. Host readiness depends on GPU
visibility, GNOME portals, the local wake bridge, and EIS input.

## Runtime contract

Jarvis has one cognitive loop and one QVAC authority.

- jarvisd owns the QVAC SDK worker, model lifecycle, GPU policy, scheduler,
  cancellation, voice loop, jobs, and D-Bus service.
- vendor/agent-harness is a copied repository dependency. It owns sessions,
  planning, memory, and the tool loop.
- Skills call the QVAC master facade. They do not create workers or load models.
- The GNOME Shell extension is presentation-only: no inference, audio capture,
  or input injection runs inside GNOME Shell.
- GPU inference is mandatory for local QVAC chat. If QVAC cannot see a usable
  GPU, Jarvis reports an unavailable state and refuses CPU fallback. Opt-in
  Groq agent inference skips the local chat model; speech and tools stay local.
- Computer use is off until explicitly granted and is revoked on Escape, lock
  screen, expiry, or a spoken abort command.

~~~mermaid
flowchart LR
  U[User] --> G[GNOME Shell HUD]
  G <--> D[jarvisd / packaged Bare]
  D --> H[agent-harness]
  D --> M[Single QVAC master]
  D --> V[Voice + PipeWire]
  D --> C[Computer-use session]
  M --> GPU[(GPU backend)]
  C --> P[XDG portals + AT-SPI + EIS]
~~~

## Capabilities

- Wake word, push-to-talk, typed fallback, VAD, local ASR, streamed replies,
  TTS playback, anti-feedback gating, and cancellation.
- Chat, planning, code, summaries, rewriting, embeddings, RAG workspaces,
  batch prompts, OCR, image classification, translation, and multimodal input.
- Image, video, and music generation through serialized local QVAC jobs.
- Voice enrollment, diarized transcription, LoRA training, and labeled BCI,
  VLA, and ABot-World lab slots.
- GNOME app launch/focus/window actions, clipboard, notifications, media,
  screenshot observation, AT-SPI actions, portal input, and vision fallback.
- Tray panel mark, compact conversation menu, Chat/Thinking tabs, hold-to-talk,
  optional expanded session, Settings/Control Center, job progress, target
  highlights, and a visible agent cursor.

Unavailable capabilities remain visible with an honest reason such as missing
GPU, insufficient VRAM, missing model, unavailable hardware, or missing
desktop permission.

## Requirements

Full runtime support requires:

- Ubuntu GNOME 24.04 or newer, GNOME Shell 46+, and a logged-in user session.
- Bare 1.32.0 or newer, packaged with the application bundle.
- PipeWire and WirePlumber.
- xdg-desktop-portal with the GNOME backend.
- AT-SPI2 and, for primary Wayland actuation, libei/libeis plus a local EIS
  injector.
- A QVAC-visible Vulkan, CUDA, or supported GPU backend.
- A local openWakeWord or sherpa-onnx bridge for always-on wake detection.

ydotool is an optional legacy fallback and is disabled unless explicitly
enabled. X11 fallback is used only in an Xorg session. CPU-only systems can run
diagnostics and control surfaces, but Jarvis will not run inference on them.

See [hardware compatibility](docs/hardware-compatibility.md) and
[troubleshooting](docs/troubleshooting.md).

## Install with the web installer

The recommended Gitea installer downloads the source snapshot at the current
rolling ref, then runs the reviewed user-local installer. It does not require
root access and does not install into system directories.

~~~bash
curl -fsSL https://git.ssh.surf/snxraven/gnome-jarvis/raw/branch/main/packaging/web-installer.sh | bash -s -- --enable
~~~

Omit `--enable` to install files without starting the service. With
`--enable`, the installer writes default settings if needed, starts
`jarvisd`, and enables the GNOME extension. The default wake phrase is
**hey jarvis** with a built-in CPU detector; change it in Settings. Spoken
replies are on.

If GNOME says the extension does not exist immediately after installation, log
out and back in once. GNOME Shell only refreshes its user extension catalogue
at session startup; the installer persists the enabled state for that login.

The rolling installer first downloads a self-contained architecture bundle
with the Bare ELF runtime and all application dependencies. It therefore needs
only curl and tar at runtime. A source fallback needs Node.js/npm when no
published bundle exists for the requested ref. Set `JARVIS_SERVER`,
`JARVIS_REPO`, or `JARVIS_BUNDLE_URL` for a mirror or a specific bundle.

## Install from a checkout

The harness is copied into vendor/agent-harness. It is not a symlink and the
dlinux platform is not vendored.

The test suite also runs Python computer-use helpers. On Ubuntu, install their
dependencies with `sudo apt-get install python3-pil python3-gi gir1.2-glib-2.0 libei1`
before running `npm test` (use the distribution Python so it can find these packages).

~~~bash
git clone <gitea-repository-url> jarvis-qvac
cd jarvis-qvac
npm ci
npm test
npm run gpu-doctor
npm run cu-doctor
npm run voice-doctor
~~~

Install the user service, extension, and per-user QVAC configuration:

~~~bash
bash packaging/install.sh --enable
~~~

The installer writes `~/.config/jarvis/config.json` when it is missing, with
wake phrase `hey jarvis`, CPU wake detector `jarvis-wake-bridge`, model profile
`laptop-16gb`, and TTS enabled. Change those later in Settings. Diagnostics stay
available as `npm run gpu-doctor`, `npm run voice-doctor`, and `npm run cu-doctor`.

Inspect the installed daemon with:

~~~bash
systemctl --user status jarvisd.service
journalctl --user -u jarvisd.service -f
~~~

The service uses the application directory
~/.local/share/jarvis-qvac, configuration under ~/.config/jarvis, model cache
under the user's cache directory, and user data paths described in
[Operations](docs/operations.md).

## Development commands

~~~bash
npm test                    # Node tests, including the vendored harness test
npm run bare:smoke          # Bare runtime and builtin mapping smoke test
npm run gpu-doctor          # QVAC GPU, VRAM, and profile diagnostics
npm run cu-doctor           # Wayland/X11, portals, AT-SPI, and libei
npm run voice-doctor        # PipeWire, wake bridge, and playback diagnostics
npm run qvac:doctor         # QVAC installation diagnostics
npm run qvac:status         # Single-master runtime status
npm run package:test        # Build and inspect release artifacts
npm run start               # Start jarvisd in the current session
~~~

The optional QVAC OpenAI-compatible server is diagnostic infrastructure only.
If used, it binds to 127.0.0.1 and must not become a second production
inference owner beside jarvisd.

## Computer use

Computer use is a mode, not default chat behavior. Say “take the wheel” or
open Settings and grant desktop access. Jarvis then prefers domain actions,
app D-Bus/GIO, AT-SPI references, GNOME Shell helpers, and finally vision
coordinates. RemoteDesktop plus EIS/libei is the primary Wayland path.
Jarvis ships a built-in libei injector; `JARVIS_LIBEI_BRIDGE` is optional.

The helper claims input readiness after ConnectToEIS or a portal Notify
fallback. Password fields,
lock screens, dangerous keys, destructive targets, and actions outside the
step budget are blocked or require confirmation. See the
[computer-use acceptance procedure](docs/cu-acceptance.md).

The webcam is a separate Settings → Camera grant. It is off by default. After
**Allow now**, the agent can call `webcam` for one still. That is not desktop
ScreenCast.

## Privacy and safety

- No cloud API is required and no telemetry is enabled.
- Microphone audio is ring-buffered in memory; raw audio is not saved by
  default.
- Computer-use frames stay local in temporary storage and are wiped on revoke
  unless trace retention is enabled.
- Audit records contain action metadata and screenshot hashes, not pixels.
- QVAC is localhost-only. Model fetch and optional P2P features are explicit
  opt-ins.
- The daemon runs as the user and must never run as root.

Read [Security and privacy](docs/security-privacy.md) before enabling legacy
input, retention, synchronization, or computer-use traces.

## Build and release

Build the Debian package, extension bundle, and checksums locally:

~~~bash
npm ci
npm test
npm run package
cat dist/SHA256SUMS
~~~

Artifacts in dist are:

- jarvis-qvac_<version>_<arch>.deb
- jarvis-qvac-extension.zip
- jarvis-qvac-bare-linux-<arch>.tar.gz (self-contained runtime bundle)
- SHA256SUMS

Gitea Actions workflows live under .gitea/workflows. CI runs tests, Python
checks, and package validation. A push to main additionally builds artifacts
and replaces the mutable rolling prerelease using the repository secret
RELEASE_TOKEN. Pull requests and non-main pushes do not publish a release.

See [Gitea CI and rolling releases](docs/ci-gitea.md) and the
[release checklist](docs/RELEASE.md).

## Uninstall and migration

The uninstall command requires an explicit flag and preserves user data:

~~~bash
bash packaging/uninstall.sh --yes
~~~

To remove Jarvis-owned memory, cache, and generated media as well:

~~~bash
bash packaging/uninstall.sh --yes --purge-data
~~~

Re-running the installer upgrades in place: it stops the old daemon, removes
the previous application and extension files, then copies the new tree.
Config, memory, and model cache are kept. Those steps are written to
`~/.local/state/jarvis/install.log`. Read
[Migration notes](docs/MIGRATIONS.md) before upgrading or moving an
installation.

## Documentation

The [documentation index](docs/README.md) is the authoritative guide.

- [Architecture](docs/architecture.md)
- [Brand](docs/brand.md)
- [QVAC runtime and GPU authority](docs/qvac-runtime.md)
- [Agent harness integration](docs/harness-integration.md)
- [Voice pipeline](docs/voice-pipeline.md)
- [Settings](docs/settings.md)
- [Computer use](docs/computer-use.md)
- [API reference](docs/api.md)
- [Operations](docs/operations.md)
- [Testing](docs/testing.md)
- [Troubleshooting](docs/troubleshooting.md)
- [Features](docs/features.md)

## Project status

Before treating a deployment as production-ready on a particular Ubuntu host,
run the doctor commands and confirm QVAC reports the intended GPU, the wake
bridge is configured, and the portal/EIS path is available.

## License

Copyright (C) 2026 HoneyPeer, LLC.

Jarvis is free software under the [GNU Affero General Public License v3](LICENSE)
or later. The copied agent harness in `vendor/agent-harness` keeps its original
MIT license.