81 lines
3.4 KiB
Markdown
81 lines
3.4 KiB
Markdown
# Voice pipeline
|
||
|
||
Voice is input/output around the harness. The daemon owns capture, wake-word
|
||
processing, VAD, transcription, completion, TTS playback, barge-in policy, and
|
||
metrics.
|
||
|
||
```mermaid
|
||
flowchart LR
|
||
MIC[PipeWire mic\n16 kHz mono] --> R[Wake ring buffer]
|
||
R --> W[Wake engine]
|
||
W -->|phrase| V[VAD + QVAC ASR]
|
||
V -->|final utterance| H[Harness bridge]
|
||
H -->|streamed reply| T[Sentence buffer]
|
||
T --> Q[QVAC TTS]
|
||
Q --> OUT[PipeWire playback]
|
||
OUT -. anti-feedback gate .-> V
|
||
```
|
||
|
||
## State machine
|
||
|
||
```mermaid
|
||
stateDiagram-v2
|
||
ARMED --> LISTENING: wake phrase / hotkey
|
||
LISTENING --> THINKING: VAD final utterance
|
||
THINKING --> SPEAKING: first reply audio
|
||
THINKING --> LISTENING: tool work / no audio yet
|
||
SPEAKING --> LISTENING: playback + cooldown
|
||
LISTENING --> ARMED: silence timeout
|
||
SPEAKING --> ARMED: cancel / Escape
|
||
ARMED --> SLEEPING: idle timeout
|
||
SLEEPING --> LISTENING: wake phrase
|
||
```
|
||
|
||
The wake stage is intentionally separate from QVAC ASR. Configure a local
|
||
openWakeWord or sherpa-onnx bridge through `JARVIS_WAKE_COMMAND`; no cloud wake
|
||
service is supported. During TTS, capture submission is gated and a 300–500 ms
|
||
cooldown prevents playback from becoming a new utterance. Trash transcripts and
|
||
very short fragments are discarded.
|
||
|
||
## Diagnostics and acceptance
|
||
|
||
Run `npm run voice-doctor`, then follow [voice acceptance](voice-acceptance.md).
|
||
The daemon metrics cover wake accepts/rejects, feedback drops, utterances, and
|
||
replies. Raw audio is not persisted by default.
|
||
|
||
## Release voice readiness
|
||
|
||
ASR and TTS initialize independently in the same QVAC runtime. Voice models do
|
||
not require the chat model to load first. The runtime status includes separate
|
||
`asr`, `tts`, `capture`, and `wake` flags plus per-component errors. Without a
|
||
wake bridge, use Hold Talk (mouse, Space, or Enter) in the GNOME overlay.
|
||
|
||
The Control Center's wake command and spoken-reply setting are read from
|
||
`~/.config/jarvis/config.json` on daemon startup. Restart `jarvisd.service` after
|
||
changing them. `JARVIS_WAKE_COMMAND` overrides the saved command. A wake command
|
||
must consume 16 kHz mono signed PCM and emit detected phrases on stdout; simply
|
||
setting a phrase does not install a wake detector.
|
||
|
||
QVAC TTS numeric arrays contain signed 16-bit samples, not bytes. Supertonic
|
||
plays at 44.1 kHz. Captured utterances use the non-streaming transcription API
|
||
because local VAD already supplies utterance boundaries.
|
||
|
||
CI installs the Vulkan runtime required by the native addons, pins the native
|
||
engines, verifies their registration inside the staged release, and synthesizes a sentence then transcribes it with Whisper. Voice
|
||
models download into a temporary CI cache. Both the archive and Debian package
|
||
include the validated Bare runtime and native dependencies. The release gate
|
||
requires model download access; it fails rather than publishing an untested
|
||
voice bundle. Microphone routing and physical speaker output still require a
|
||
local PipeWire session and working hardware.
|
||
|
||
Run the model roundtrip locally with:
|
||
|
||
```sh
|
||
bash packaging/bare-launch.sh packaging/bare-run.js packaging/voice-model-smoke.js
|
||
```
|
||
|
||
PipeWire capture/playback use `--properties node.name=Jarvis` and `-` for
|
||
standard I/O; `pw-cat` does not accept `--name`. A local capture check is
|
||
`bash packaging/bare-launch.sh packaging/bare-run.js packaging/pipewire-smoke.js`.
|
||
Add `--play` to the model roundtrip to exercise speaker playback as well.
|