gnome-jarvis/docs/voice-pipeline.md

Voice pipeline

Voice is input/output around the harness. The daemon owns capture, wake-word processing, VAD, transcription, completion, TTS playback, barge-in policy, and metrics.

flowchart LR
  MIC[PipeWire mic\n16 kHz mono] --> R[Wake ring buffer]
  R --> W[Wake engine]
  W -->|phrase| V[VAD + QVAC ASR]
  V -->|final utterance| H[Harness bridge]
  H -->|streamed reply| T[Sentence buffer]
  T --> Q[QVAC TTS]
  Q --> OUT[PipeWire playback]
  OUT -. anti-feedback gate .-> V

State machine

stateDiagram-v2
  ARMED --> LISTENING: wake phrase / hotkey
  LISTENING --> THINKING: VAD final utterance
  THINKING --> SPEAKING: first reply audio
  THINKING --> LISTENING: tool work / no audio yet
  SPEAKING --> LISTENING: playback + cooldown
  LISTENING --> ARMED: silence timeout
  SPEAKING --> ARMED: cancel / Escape
  ARMED --> SLEEPING: idle timeout
  SLEEPING --> LISTENING: wake phrase

The wake stage is intentionally separate from QVAC ASR. A built-in CPU detector listens for “hey jarvis” (and aliases) without loading Whisper. You can replace it with a local openWakeWord or sherpa-onnx bridge through wakeCommand / JARVIS_WAKE_COMMAND; no cloud wake service is supported. Microphone audio is still sent to the wake detector while Jarvis is sleeping so the phrase can restore listening. During TTS, capture submission is gated and a 300–500 ms cooldown prevents playback from becoming a new utterance. Trash transcripts and very short fragments are discarded.

Diagnostics and acceptance

Run npm run voice-doctor, then follow voice acceptance. The daemon metrics cover wake accepts/rejects, feedback drops, utterances, and replies. Raw audio is not persisted by default.

Release voice readiness

ASR, TTS, and the chat model stay unloaded until wake, Hold Talk, or a typed question. Idle sleep unloads them again so GPU memory is freed; only PipeWire capture and the CPU wake detector stay resident. Runtime status includes separate asr, tts, capture, and wake flags plus per-component errors. The tray shows WAKE ON when the detector is alive. Hold Talk (mouse, Space, or Enter) still works in the GNOME overlay.

The Control Center's wake command and spoken-reply setting are read from ~/.config/jarvis/config.json on daemon startup. Restart jarvisd.service after changing them. JARVIS_WAKE_COMMAND overrides the saved command. A custom wake command must consume 16 kHz mono signed PCM and emit detected phrases on stdout. The default jarvis-wake-bridge value selects the built-in CPU detector.

QVAC TTS numeric arrays contain signed 16-bit samples, not bytes. Supertonic plays at 44.1 kHz. Captured utterances use the non-streaming transcription API because local VAD already supplies utterance boundaries.

CI installs the Vulkan runtime required by the native addons, pins the native engines, verifies their registration inside the staged release, and synthesizes a sentence then transcribes it with Whisper. Voice models download into a temporary CI cache. Both the archive and Debian package include the validated Bare runtime and native dependencies. The release gate requires model download access; it fails rather than publishing an untested voice bundle. Microphone routing and physical speaker output still require a local PipeWire session and working hardware.

Run the model roundtrip locally with:

bash packaging/bare-launch.sh packaging/bare-run.js packaging/voice-model-smoke.js

PipeWire capture/playback use --properties node.name=Jarvis and - for standard I/O; pw-cat does not accept --name. A local capture check is bash packaging/bare-launch.sh packaging/bare-run.js packaging/pipewire-smoke.js. Add --play to the model roundtrip to exercise speaker playback as well.