gnome-jarvis/docs/voice-pipeline.md

Voice pipeline

Voice is input/output around the harness. The daemon owns capture, wake-word processing, VAD, transcription, completion, TTS playback, barge-in policy, and metrics.

flowchart LR
  MIC[PipeWire mic\n16 kHz mono] --> R[Wake ring buffer]
  R --> W[Wake engine]
  W -->|phrase| V[VAD + QVAC ASR]
  V -->|final utterance| H[Harness bridge]
  H -->|final reply| T[Sentence queue]
  T --> Q[QVAC TTS]
  Q --> OUT[PipeWire playback]
  OUT -. prepare next sentence concurrently .-> Q
  OUT -. anti-feedback gate .-> V

State machine

stateDiagram-v2
  ARMED --> LISTENING: wake phrase / hotkey
  LISTENING --> THINKING: VAD final utterance
  THINKING --> SPEAKING: first reply audio
  THINKING --> LISTENING: tool work / no audio yet
  SPEAKING --> LISTENING: playback + cooldown
  LISTENING --> ARMED: silence timeout
  SPEAKING --> ARMED: cancel / Escape
  ARMED --> SLEEPING: idle timeout
  SLEEPING --> LISTENING: wake phrase

The wake stage is intentionally separate from QVAC ASR. A built-in CPU detector listens for “hey jarvis” (and aliases) without loading Whisper. You can replace it with a local openWakeWord or sherpa-onnx bridge through wakeCommand / JARVIS_WAKE_COMMAND; no cloud wake service is supported. Microphone audio is still sent to the wake detector while Jarvis is sleeping so the phrase can restore listening. During TTS, capture submission is gated and a 300–500 ms cooldown prevents playback from becoming a new utterance. Trash transcripts and very short fragments are discarded.

Diagnostics and acceptance

Run npm run voice-doctor, then follow voice acceptance. The daemon metrics cover wake accepts/rejects, feedback drops, utterances, and replies. Speech metrics also report synthesisCount, synthesisMs, audioMs, and synthesisRealtimeFactor (total synthesis time / generated audio duration; less than 1 means synthesis is faster than playback). Raw audio is not persisted by default.

Inference and playback concurrency

Whisper explicitly requests contextParams.use_gpu: true; it otherwise defaults to CPU independently of the LLM's GPU policy. TTS honors the saved ttsUseGpu setting. These engines use their own configuration fields, not llama.cpp's gpu_layers. Concurrent requests to load the same auxiliary model share the in-flight load, avoiding duplicate allocations. The shared model stays loaded until its last auxiliary user releases it.

Speech synthesis prepares one sentence ahead while PipeWire plays the current sentence. TTS requests and playback remain ordered, with at most one prefetched sentence. The feedback gate stays active across sentence boundaries. Interrupts discard prefetched audio; shutdown drains any in-flight synthesis before unloading models. Synthesis and playback failures release the gate and allow the next reply.

Independent harness tool calls already run concurrently. Conversation inference stays serialized to preserve history ordering and the single active chat model. Increasing llama.cpp parallel also divides its configured context across slots; it is not a free speed increase for a single conversational stream.

Release voice readiness

ASR, TTS, and the chat model stay unloaded until wake, Hold Talk, or a typed question. Idle sleep unloads them again so GPU memory is freed; only PipeWire capture and the CPU wake detector stay resident. Runtime status includes separate asr, tts, capture, and wake flags plus per-component errors. The tray shows WAKE ON when the detector is alive. Hold Talk (mouse, Space, or Enter) still works in the tray HUD.

The Control Center's wake command and spoken-reply setting are read from ~/.config/jarvis/config.json on daemon startup. Restart jarvisd.service after changing them. JARVIS_WAKE_COMMAND overrides the saved command. A custom wake command must consume 16 kHz mono signed PCM and emit detected phrases on stdout. The default jarvis-wake-bridge value selects the built-in CPU detector.

QVAC TTS numeric arrays contain signed 16-bit samples, not bytes. Supertonic plays at 44.1 kHz. Captured utterances use the non-streaming transcription API because local VAD already supplies utterance boundaries.

CI installs the Vulkan runtime required by the native addons, pins the native engines, verifies their registration inside the staged release, and synthesizes a sentence then transcribes it with Whisper. Voice models download into a temporary CI cache. Both the archive and Debian package include the validated Bare runtime and native dependencies. The release gate requires model download access; it fails rather than publishing an untested voice bundle. Microphone routing and physical speaker output still require a local PipeWire session and working hardware.

Run the model roundtrip locally with:

bash packaging/bare-launch.sh packaging/bare-run.js packaging/voice-model-smoke.js

PipeWire capture/playback use --properties node.name=Jarvis and - for standard I/O; pw-cat does not accept --name. A local capture check is bash packaging/bare-launch.sh packaging/bare-run.js packaging/pipewire-smoke.js. Add --play to the model roundtrip to exercise speaker playback as well.