86 lines
3.7 KiB
Markdown
86 lines
3.7 KiB
Markdown
# Voice pipeline
|
||
|
||
Voice is input/output around the harness. The daemon owns capture, wake-word
|
||
processing, VAD, transcription, completion, TTS playback, barge-in policy, and
|
||
metrics.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
MIC[PipeWire mic\n16 kHz mono] --> R[Wake ring buffer]
|
||
R --> W[Wake engine]
|
||
W -->|phrase| V[VAD + QVAC ASR]
|
||
V -->|final utterance| H[Harness bridge]
|
||
H -->|streamed reply| T[Sentence buffer]
|
||
T --> Q[QVAC TTS]
|
||
Q --> OUT[PipeWire playback]
|
||
OUT -. anti-feedback gate .-> V
|
||
```
|
||
|
||
## State machine
|
||
|
||
```mermaid
|
||
stateDiagram-v2
|
||
ARMED --> LISTENING: wake phrase / hotkey
|
||
LISTENING --> THINKING: VAD final utterance
|
||
THINKING --> SPEAKING: first reply audio
|
||
THINKING --> LISTENING: tool work / no audio yet
|
||
SPEAKING --> LISTENING: playback + cooldown
|
||
LISTENING --> ARMED: silence timeout
|
||
SPEAKING --> ARMED: cancel / Escape
|
||
ARMED --> SLEEPING: idle timeout
|
||
SLEEPING --> LISTENING: wake phrase
|
||
```
|
||
|
||
The wake stage is intentionally separate from QVAC ASR. A built-in CPU detector
|
||
listens for “hey jarvis” (and aliases) without loading Whisper. You can replace
|
||
it with a local openWakeWord or sherpa-onnx bridge through `wakeCommand` /
|
||
`JARVIS_WAKE_COMMAND`; no cloud wake service is supported. Microphone audio is
|
||
still sent to the wake detector while Jarvis is sleeping so the phrase can
|
||
restore listening. During TTS, capture submission is gated and a 300–500 ms
|
||
cooldown prevents playback from becoming a new utterance. Trash transcripts and
|
||
very short fragments are discarded.
|
||
|
||
## Diagnostics and acceptance
|
||
|
||
Run `npm run voice-doctor`, then follow [voice acceptance](voice-acceptance.md).
|
||
The daemon metrics cover wake accepts/rejects, feedback drops, utterances, and
|
||
replies. Raw audio is not persisted by default.
|
||
|
||
## Release voice readiness
|
||
|
||
ASR, TTS, and the chat model stay unloaded until wake, Hold Talk, or a typed
|
||
question. Idle sleep unloads them again so GPU memory is freed; only PipeWire
|
||
capture and the CPU wake detector stay resident. Runtime status includes
|
||
separate `asr`, `tts`, `capture`, and `wake` flags plus per-component errors.
|
||
The tray shows **WAKE ON** when the detector is alive. Hold Talk (mouse, Space,
|
||
or Enter) still works in the tray HUD.
|
||
|
||
The Control Center's wake command and spoken-reply setting are read from
|
||
`~/.config/jarvis/config.json` on daemon startup. Restart `jarvisd.service` after
|
||
changing them. `JARVIS_WAKE_COMMAND` overrides the saved command. A custom wake
|
||
command must consume 16 kHz mono signed PCM and emit detected phrases on stdout.
|
||
The default `jarvis-wake-bridge` value selects the built-in CPU detector.
|
||
|
||
QVAC TTS numeric arrays contain signed 16-bit samples, not bytes. Supertonic
|
||
plays at 44.1 kHz. Captured utterances use the non-streaming transcription API
|
||
because local VAD already supplies utterance boundaries.
|
||
|
||
CI installs the Vulkan runtime required by the native addons, pins the native
|
||
engines, verifies their registration inside the staged release, and synthesizes a sentence then transcribes it with Whisper. Voice
|
||
models download into a temporary CI cache. Both the archive and Debian package
|
||
include the validated Bare runtime and native dependencies. The release gate
|
||
requires model download access; it fails rather than publishing an untested
|
||
voice bundle. Microphone routing and physical speaker output still require a
|
||
local PipeWire session and working hardware.
|
||
|
||
Run the model roundtrip locally with:
|
||
|
||
```sh
|
||
bash packaging/bare-launch.sh packaging/bare-run.js packaging/voice-model-smoke.js
|
||
```
|
||
|
||
PipeWire capture/playback use `--properties node.name=Jarvis` and `-` for
|
||
standard I/O; `pw-cat` does not accept `--name`. A local capture check is
|
||
`bash packaging/bare-launch.sh packaging/bare-run.js packaging/pipewire-smoke.js`.
|
||
Add `--play` to the model roundtrip to exercise speaker playback as well.
|