gnome-jarvis/docs/voice-pipeline.md

Voice pipeline

Voice is input/output around the harness. The daemon owns capture, wake-word processing, VAD, transcription, completion, TTS playback, barge-in policy, and metrics.

flowchart LR
  MIC[PipeWire mic\n16 kHz mono] --> R[Wake ring buffer]
  R --> W[Wake engine]
  W -->|phrase| V[VAD + QVAC ASR]
  V -->|final utterance| H[Harness bridge]
  H -->|streamed reply| T[Sentence buffer]
  T --> Q[QVAC TTS]
  Q --> OUT[PipeWire playback]
  OUT -. anti-feedback gate .-> V

State machine

stateDiagram-v2
  ARMED --> LISTENING: wake phrase / hotkey
  LISTENING --> THINKING: VAD final utterance
  THINKING --> SPEAKING: first reply audio
  THINKING --> LISTENING: tool work / no audio yet
  SPEAKING --> LISTENING: playback + cooldown
  LISTENING --> ARMED: silence timeout
  SPEAKING --> ARMED: cancel / Escape
  ARMED --> SLEEPING: idle timeout
  SLEEPING --> LISTENING: wake phrase

The wake stage is intentionally separate from QVAC ASR. Configure a local openWakeWord or sherpa-onnx bridge through JARVIS_WAKE_COMMAND; no cloud wake service is supported. During TTS, capture submission is gated and a 300–500 ms cooldown prevents playback from becoming a new utterance. Trash transcripts and very short fragments are discarded.

Diagnostics and acceptance

Run npm run voice-doctor, then follow voice acceptance. The daemon metrics cover wake accepts/rejects, feedback drops, utterances, and replies. Raw audio is not persisted by default.

Release voice readiness

ASR and TTS initialize independently in the same QVAC runtime. Voice models do not require the chat model to load first. The runtime status includes separate asr, tts, capture, and wake flags plus per-component errors. Without a wake bridge, use Hold Talk (mouse, Space, or Enter) in the GNOME overlay.

The Control Center's wake command and spoken-reply setting are read from ~/.config/jarvis/config.json on daemon startup. Restart jarvisd.service after changing them. JARVIS_WAKE_COMMAND overrides the saved command. A wake command must consume 16 kHz mono signed PCM and emit detected phrases on stdout; simply setting a phrase does not install a wake detector.

QVAC TTS numeric arrays contain signed 16-bit samples, not bytes. Supertonic plays at 44.1 kHz. Captured utterances use the non-streaming transcription API because local VAD already supplies utterance boundaries.

CI pins the native engines, verifies their registration inside the staged release, and synthesizes a sentence then transcribes it with Whisper. Voice models download into a temporary CI cache. Both the archive and Debian package include the validated Bare runtime and native dependencies. The release gate requires model download access; it fails rather than publishing an untested voice bundle. Microphone routing and physical speaker output still require a local PipeWire session and working hardware.

Run the model roundtrip locally with:

bash packaging/bare-launch.sh packaging/bare-run.js packaging/voice-model-smoke.js

PipeWire capture/playback use --properties node.name=Jarvis and - for standard I/O; pw-cat does not accept --name. A local capture check is bash packaging/bare-launch.sh packaging/bare-run.js packaging/pipewire-smoke.js. Add --play to the model roundtrip to exercise speaker playback as well.