Architecture
One voice turn as the code runs it on the Mac mini: the Rust Voice Gateway as the sole gateway and brain path, with its STT, turn machine, speculative brain dispatch and streamed TTS.
Living doc, checked against the code 2026-11-14
Vulpine follows PiBot’s low-latency worker design, not HTTP microservices. The Rust Voice Gateway is the only
gateway and the only path to the Brain. fox9 and foxline identifiers below are compatibility names. Installing
it is docs/install.md; running and rolling the gateway is
its README.
The parts, and where they run:
Vulpine.app, Vigil and the room speaker’s bridge are clients of the one gateway WebSocket. Vulpine Gateway.app reaches the gateway only through its owner-local Admin socket.
One voice turn:
A turn takes one of two paths after the commit. A verb turn is a request the gateway matches against its own
short list of verbs (what you can say): it starts no brain turn. An agent turn is
everything else, and goes to the brain. Both wait for the commit, the gateway being sure you have finished: 978 ms
p50 over 218 live turns (benchmarks/results/brain-ttft-cloud-vs-local-2026-11-06.json). After it, a verb’s match
costs microseconds (benchmarks/results/page-verb-cost-20261107.md) and the voice worker’s first audio about 140 ms
(benchmarks/results/tts-idle-ttfa-2026-11-07.json). An agent turn waits 1.4 s p50 (3.8 s p90) for a hosted
model’s first token and 441 ms more for first audio; speech end to audible is 3.0 s p50 for an agent turn.
The STT box is the ANE authority lane, which the installed config turns on ([shadow_stt] mode = "authority").
When mode is not authority, the loadout’s own STT adapter takes its place. Without speculation, the committed
text goes to Pi as an ordinary prompt.
The gateway never calls the Brain’s LLM servers; Pi does. The decision model is optional and off by default
([decision_shadow], below): the turn never waits for it.
The gateway binds each warm Brain process to a BrainIdentity of eleven fields
(crates/voice_gateway/src/brain/launch.rs): the owner, agent, workspace, loadout, model, persona and persona
prompt, the frontend capability profile’s hash, the tool-policy hash (the loadout’s allowed Pi tools plus the
negotiated frontend tools), the admitted project root’s id, and the launch hash (command, arguments and working
directory). The warm-Brain pool is keyed by the whole identity plus the launch (brain/pool.rs). A Brain is reused
only when every field matches. The Pi session is named
foxline-voice-<agent>-<loadout>-<model hash>-<capability hash>-<conversation>. The capability hash covers the
owner, the frontend and tool-policy hashes, the persona and its prompt, the project root and the launch.
Runtime defaults
- Brain: Pi over RPC. Tools, voice-only extensions and MCP servers are Pi’s. The operator loadout loads MCP through
local:mcpfromcatalog/agents/operator/.vulpine/mcp/default.json(moraine,fox9docsandhomeha). A server entry’stoolOverridesreplaces a tool’s description and narrows its schema in the voice prompt (catalog/pi/extensions/mcp/README.md). - TTS: Qwen3-TTS through the
qwen3-workerbackend. WithVULPINE_TTS_HTTP_URLset, the qwentts.cpptts-server(127.0.0.1:8850, installed byjust models) does the synthesis (TTS adapter, below). - STT: the loadouts name
parakeet-silero(a parakeet.cpp worker with Silero VAD behind a local WebSocket). The installed config sets[shadow_stt] mode = "authority", which makes the CoreML Parakeet v3 lane (coreml-parakeet-v3, on the ANE) the session STT instead. - Either one can be your own: a TTS or STT implementation plugs in as a hardpoint (hardpoint.v1).
The system map is docs/architecture/fox9-system.html, with a static docs/architecture/fox9-system.svg.
Product boundaries
- Voice Gateway: the Rust runtime for voice sessions, frame routing, adapters, the Pi RPC lifecycle, frontend tools, Avatar Actions and tracing.
- Vigil: the browser skin. It maps the gateway protocol to a radio-codec call. Its screens are Vigil Frames over a presentation layer shared with Glyph.
- Vulpine.app: the Mac client (
apps/glyph, a Tauri app whose surfaces are moving to GPUI). Glyph is its look. - Room speaker: the Voice PE puck, bridged by
apps/satellite-esphomeas an ordinary gateway client. - The Admin: the GPUI control plane (
apps/admin-shadow, shipped as Vulpine Gateway.app) over the gateway’s owner-local Admin socket, not the voice WebSocket. - Integrations: home, room and fleet adapters run outside the gateway and speak Vulpine’s vocabulary over two contracts (integrations).
- Brain: the Pi RPC process. It owns reasoning, model and provider access, tool use, skills, extensions, prompt templates and session behavior.
- Agent: a Pi-backed working identity rooted in a workspace and its Pi configuration.
- Persona: the voice and UI identity a frontend presents. A bundled demo persona may carry metadata and instructions.
- Media and runtime outputs: generated or copyrighted media (voice references, fillers, avatar assets, game sound) and runtime outputs (traces, recordings) stay out of git, on the machine that made them.
Rust Voice Gateway protocol
Frontends talk to the gateway over WebSocket:
- JSON text messages carry control and event frames such as
hello,start_session,end_session,vad_hint,audio_playback_observed,audio_echo_risk_observed,herdr_pane_focus_confirmation,interrupt, and errors. - Binary messages carry PCM audio. Audio is never base64 in JSON.
- A frontend declares its capabilities before the session starts. The profile binds the warm-Brain identity, frontend tool negotiation, Avatar Actions and confirmation capabilities.
- Debug trace streaming is off unless configured; server-side JSONL traces are the default.
Trace event names include mic_frame_received, stt_partial, stt_final, brain_request_start,
brain_first_token, tts_request_start, tts_audio_start, frontend_audio_play_observed, barge_in_received and
tts_cancel_sent.
Turn management
The gateway owns turn state. Frontend VAD and noise-gate messages are hints; they never commit a turn by
themselves. STT and VAD evidence drives the turn.user_started, turn.user_committed and turn.interrupted
frames.
The turn machine (TurnState in crates/voice_gateway/src/turn.rs) has three states:
[turn] in the gateway config sets the turn strategy:
max_utterance_duration_msbarge_in_confirmation_window_ms- the tiered verb resolver keys (
verb_tier_shadow,verb_tier_route_relaxed,verb_confirm_*) commit_guards(shadowby default: a guard writes acommit_guardtrace row and changes no turn)- duplicate-commit arbitration (
arbitrate_duplicate_commits,arbitration_window_ms,arbitration_takeover_ms,arbitration_cohorts)
Utterance-ending silence is not a [turn] key: the STT’s end-of-utterance evidence decides it.
With [speculation] enabled, a stable interim transcript goes to Pi before the final commits. The commit then
promotes that generation or reforks the Pi session with the committed text. A final that resolves to a tier-1/2
gateway verb cancels it (docs/speculation.md).
Barge-in stops gateway output at once with a turn.interrupted frame; later STT and VAD evidence confirms where your
speech began. Client Core plays assistant audio in receive order on one timeline and binds each span to its turn.
Only audio actually playing can authorize an interruption. Implausible render or capture timing fails closed to
echo-risk quarantine. Each STT cohort carries an id, so a late final from a retired cohort never changes a newer
turn, and a transport without cohort ids is refused.
The decision front door
The Decision model is optional and off by default. It is meant to settle more spoken turns before the Brain does;
today it settles one kind. With [decision_shadow] on, the gateway asks it on the first partials, beside the turn,
and never waits for it. A verb is still matched only by the gateway’s own grammar at the commit. The one answer that
acts is the decision gate ([decision_gate]): in the follow-up window after a reply, a line said without the
callsign that the model reads as not addressed to Vulpine is dropped instead of going to the Brain. A turn started
with the callsign is never gated. Every other answer is traced only (docs/contracts/decision.v1.md,
crates/voice_gateway/src/decision_shadow.rs, crates/voice_gateway/src/decision_gate.rs,
crates/decidectl/src/lib.rs).
Each entry sits behind its own breaker. The whole cascade answers under one deadline, the session’s follow-up window; running out of it abstains. The gate abstains (the line goes to the Brain as before) whenever no answer was in by the final, or the answering model is not the one its lock pins, and it turns itself off after three Missed labels on its drops in seven days (said as “that was for you”, or the Admin’s Missed button).
Approvals
An agent that wants to act asks the Voice Gateway first, on the owner-local approval socket. Nothing on the socket
runs anything (crates/voice_gateway/src/approval_queue.rs, ADR 0040).
create, update and delete are never pre-authorized, and delete never becomes trusted. A DANGER command is the operator’s every time.
Configuration
The gateway reads ~/.config/vulpine/config.toml on every platform and writes it with defaults on first run
(default_config_path in crates/voice_gateway/src/config.rs). --config (or VULPINE_GATEWAY_CONFIG) names
another file. Command-line flags override the config file. --bind, --trace-dir and --debug-traces also read
VULPINE_GATEWAY_* variables. --env-file exports a KEY=VALUE file before anything else runs. The environment
variables are named VULPINE_*; vulpine-env.toml lists each one.
Generate the JSON schema with:
just gateway --print-config-schema
.vulpine loadouts
Project-local voice configuration lives under .vulpine/; Pi’s own defaults stay under .pi/:
.vulpine/is the gateway’s voice-session configuration: loadouts, voice-only extensions, adapter choices, lifecycle policy, frontend tool requirements, and any project-root coordinate a closed gateway tool needs ([tools] herdr_attention_project_root, below)..pi/is Pi’s Brain configuration: Pi defaults, provider and model behavior, tools, skills, prompt templates and session semantics.
The gateway resolves the named loadout from .vulpine/loadouts/<name>.toml, or the default from
.vulpine/loadout.toml. It then resolves voice-only extensions from bundled builtin: references or project-local
.vulpine/extensions/ paths. A workspace with only .foxline/ is refused at bind; rename it:
mv .foxline .vulpine (refuse_legacy_only_workspace in crates/voice_gateway/src/loadout.rs).
docs/foxline-loadouts.md has the schema.
Bundled loadouts are examples and shared defaults. A project-local .vulpine loadout overrides them for that
workspace: voice constraints, frontend tool requirements, adapter selection and voice-only extensions. Personas are
not per workspace: they live in the Catalog (see Agent and Persona switching).
Pi RPC Brain adapter
The Brain adapter launches only pi --mode rpc, or a process named by the loadout’s [pi] command /
[brain] background_command that speaks Pi’s RPC lines (runtime/k-brain, runtime/fm-brain). The selected
.vulpine loadout sets the launch arguments:
- Pi profile, config and session directory fields become explicit Pi arguments.
- Allowed Pi tools become
--tools. - Resolved voice-only extensions become explicit
--extensionflags. --no-context-filesis on by default, from the gateway config.
The adapter maps Pi RPC JSONL events into canonical Brain frames:
- prompt send ->
brain.request_start - the first text delta is traced as
brain_first_token - text deltas ->
brain.text_delta - tool start/end ->
brain.tool_callandbrain.tool_result - agent done ->
brain.done - Pi errors or rejected prompts ->
brain.error
A warm Brain can be prewarmed, shut down when idle, restarted and shut down fully.
The Herdr attention tool’s project root comes from the loadout, never from Pi or a frontend. A loadout that allows
the brain_herdr_attention Pi tool must name [tools] herdr_attention_project_root; either one without the other
fails the bind (brain/pool.rs). The root is a relative path: non-empty, at most 1,024 bytes, no control
characters, never absolute. It resolves from the selected workspace and is refused when it passes through a symlink
(ProjectRootAuthority::admit, crates/voice_gateway/src/herdr_attention.rs). Its opaque id is the Brain
identity’s project root id.
TTS adapter
The gateway exposes TTS through an adapter trait. The built-in backend is qwen3-worker; a TTS of your own binds as
a hardpoint instead (hardpoint.tts.v1: a Wyoming server on loopback, checked at
bind, with qwen3-worker as its fallback). qwen3-worker speaks the
runtime/qwen3_tts_worker/qwen3_tts_worker.py binary protocol:
- input
speak,cancelandshutdownframes use a 9-byte<type,u32 request_id,u32 payload_len>header. - worker
audio_start,audio_chunk,audio_doneanderrorframes map to canonical TTS frames. - PCM chunks also emit binary audio frames for the frontend.
- request start, first audio and cancel events are traced.
The gateway hands the worker its voice through inherited descriptors, never a path. The persona WAV and transcript
are /dev/fd/3 and /dev/fd/4; the Python worker’s source, embedded in the gateway binary, is /dev/fd/5. Every
session’s TTS setup requires VULPINE_QWEN_UV: a root-owned, non-writable absolute executable whose parent folders are root-owned and
non-writable too. PATH cannot choose the worker’s bytes.
The worker has two synthesis modes behind the same frame protocol. With VULPINE_TTS_HTTP_URL set (the setting
for the TTS server just models installs), it health-checks the qwentts.cpp tts-server, registers the sealed
reference as a named voice (VULPINE_TTS_HTTP_VOICE, default operator) plus the mood bank, and streams PCM from
the server over HTTP. It loads no model itself. In this mode the worker is the gateway binary itself
(--vulpine-tts-worker-v1, the same frames, no Python) at the server’s 24 kHz; VULPINE_TTS_NATIVE_WORKER=0 keeps
the Python worker. Without VULPINE_TTS_HTTP_URL, the Python worker loads Qwen3-TTS through MLX on macOS.
docs/voice-operator-reference.md covers the live voice.
rust-candle, cpp-ggml, elevenlabs and openai are reserved backend names for future adapters; none is active.
STT adapter
The gateway exposes STT through an adapter trait. Loadouts can name parakeet-silero, native-parakeet-silero or
in-process-parakeet, or an STT of your own as a hardpoint
(hardpoint.stt.v1). The installed session STT is the CoreML authority lane
below. parakeet-silero speaks the
Parakeet + Silero WebSocket protocol; its server is not in this repo:
- binary messages carry PCM audio to the STT service.
wordandinterimmessages map to canonical STT partial frames.finalmessages map to VAD speech-stop evidence plus canonical STT final frames.- status messages such as speech start/end map to VAD evidence frames.
- errors map to canonical STT error frames.
native-parakeet-silero is a fail-closed path for a deployment that supplies VULPINE_STT_NATIVE_WORKER,
VULPINE_STT_NATIVE_MODEL, VULPINE_STT_NATIVE_VAD and VULPINE_STT_ARTIFACT_MANIFEST. The gateway checks the
manifest against the files’ bytes and sends nothing until the service’s first frame, within five seconds, is a
matching ready. This repo supplies none of them, so the path is inactive.
[shadow_stt] runs a second recognizer beside the session STT. With mode = "authority" and
backend = "coreml-parakeet-v3" (the installed setting), the CoreML Parakeet v3 lane in crates/coreml_stt becomes
the session STT. One recognizer thread serves every session. A segmenting adapter per session closes a final after
authority_eou_hold_ms of silence, and emits partials only when authority_partial_interval_ms is non-zero. The
loadout’s relay adapter then runs as the shadow only when authority_relay_shadow is on.
The turn manager takes these VAD and STT frames as its turn evidence.
When a session’s recognizer is lost, the client gets the typed stt_error {reason, recoverable}
(docs/contracts/adapter-degradation.v1.md), and [stt] on_fault decides the rest:
end_session(the default) ends the session on the next microphone frame.text_onlykeeps it. Typed turns and replies carry on, and the microphone is refused with one spoken line. The gateway rebuilds the recognizer on a 2 s to 30 s backoff until one takes a frame, then sends the clientstt_recovered(ws/stt_fault.rs; trace rowsstt_mic_refused,stt_reconnect,stt_recovered,stt_fault_pcm_refused).
Frontend clients
Vulpine.app, Vigil, the room speaker’s bridge, the terminal skin and any new client connect through the same gateway WebSocket protocol. The gateway emits only semantic events and Avatar Actions; each client maps them to its own presentation. The LED scanner is a Glyph Avatar, not a separate client.
Vigil advertises the narrow herdr.pane.focus Confirmation Capability:
- A Pi extension submits only its live tool-call ID and a typed
pane_id, through a private, token-authenticated gateway Unix socket. The gateway matches both against the active Brain operation. - Client Core shows the gateway’s challenge without answering it. Vigil renders Cancel and Focus buttons.
- Only the gateway validates the action, capability, pane, session, nonce and 30-second expiry, refuses a replay,
and calls Herdr
pane.focusover the deployment’s Herdr socket. Pi and the frontend never run that mutation. - No other Herdr command, key, close, worktree or layout operation is part of this surface.
Agent and Persona switching
A gateway session names its Pi-backed agent and its voice and presentation persona separately. Both come in
start_session, beside workspace, loadout, voice and model (crates/protocol/src/lib.rs).
- The persona resolves in one place (
resolve_session_persona,crates/voice_gateway/src/ws/persona.rs): an explicitpersonafirst (empty or blank counts as unset), then[frontend] default_personain the gateway config, then the agent’s name. - A persona’s package lives in the runtime root’s Catalog,
catalog/personas/<id>, and supplies the voice reference and the persona prompt. A workspace that holds a persona folder is refused at bind, with the move named (refuse_workspace_local_personasincrates/voice_gateway/src/loadout.rs). - The persona and its prompt are part of the warm-Brain identity
(
.with_persona_context(persona, persona_prompt)incrates/voice_gateway/src/ws/connection/text/start_session/bind.rs). A different persona or prompt binds a different warm Pi session for the same agent, workspace and loadout. A voice change alone keeps the identity. - A different agent changes more: the workspace with its
.piand.vulpineconfiguration, the loadout, the tool policy and the extensions. - No frame switches the agent or the persona inside a session. A second
start_sessionis refused withsession_already_started(crates/voice_gateway/src/ws/wire.rs). Client Core switches by ending the session, reconnecting and starting a new one (switchPersonaandswitchPersonaVoiceinpackages/voice-client/src/index.ts).switchCharacteralso moves the agent when the character names an agent workspace and no workspace override is set.switch_modelis the in-session control; the model is part of the identity too.
Avatar Actions
Avatar Actions are semantic presentation requests, not sprite commands. A frontend advertises the action names it
supports in its capability profile. The gateway routes only those, as avatar_action server events:
set_stateset_expressionfocusplay_animationclear
Each client maps these actions to its own visual language.
Frontend tools
Frontend tools are negotiated when the session starts. The effective tool surface is:
(loadout.required_frontend + loadout.optional_frontend) ∩ frontend.advertised_tools
A missing required frontend tool fails the session start with a clear error. An optional tool is available only when the frontend advertises it. The gateway rejects a tool call outside the negotiated surface before it reaches the frontend.
The gateway emits frontend_tools_negotiated, frontend_tool_call and frontend_tool_rejected, and accepts
frontend_tool_result control messages. Pi reaches them through the voice-only frontend-tools extension, which is a
placeholder today.
Why worker processes instead of HTTP services
- Lower latency: PCM can stream as soon as the worker emits it.
- Better interruption/barge-in: cancellation is a direct worker frame.
- Fewer ports and health checks.
- One runtime server owns character state, session state, brain, and TTS lifecycle.
Python boundary
Python is a compatibility and development boundary. It remains for:
runtime/qwen3_tts_worker/qwen3_tts_worker.py, the MLX path and the fallback; with a TTS server the gateway runs its own Rust worker instead;- installer and extraction scripts under
tools/; - the transcript helper service during asset installation.
Model provisioning
Models are fetched by an explicit provisioning step, never at gateway start. Release binaries do not embed large
model weights. Each model is pinned by sha256 in tools/install/models.toml; what a fresh install downloads, and
its size, is in docs/install.md.
Important models:
- Qwen3-TTS (live): the 0.6B Base Q8_0 GGUF served by the qwentts.cpp
tts-server. - Qwen3-TTS (MLX fallback):
mlx-community/Qwen3-TTS-12Hz-0.6B-Base-6bit, the worker’s default withoutVULPINE_TTS_HTTP_URL. - CoreML Parakeet v3: the bundle named by
[shadow_stt] bundle_dirfor the authority lane. - Parakeet:
mudler/parakeet-cpp-gguf. - VAD:
ggml-org/whisper-vadSilero GGML.
Repo layout
crates/protocol/- Rust source of truth for the gateway’s protocol types.crates/voice_gateway/- Rust Voice Gateway runtime core.crates/overlay_stdio/- the roll, run as the gateway binary’soverlay-stdiosubcommand.crates/vulpine_cli/- thevulpinecommand (vulpine models, the pack verifier).crates/coreml_stt/- CoreML Parakeet v3 recognizer for the ANE STT lane.crates/decidectl/- decision.v1 client the gateway asks the decision model through.crates/head_mlxc/- head_r9 run from Rust over mlx-c; an adapter outside the gateway’s Cargo graph.crates/aec_daf/- Rust port of LocalVQE’s DAF front-end (bulk-delay prealignment, Kalman filter, GRU step-size controller): subtractive echo cancellation on the wake-worker fanout.crates/evidence_harness/-evidence-replay, offline replay of recorded capture streams for acoustic experiments; reports are aggregate-only.crates/provider_publisher/- bounded native provider admission and journaled publication, with no execution or host acquisition.crates/vulpine_client/- the shell-neutral Rust client of the gateway WebSocket.crates/vulpine_env/- the environment-variable registry, generated fromvulpine-env.toml.crates/admin_core/- the Admin’s UI-free core: the owner-socket client, projections and host-local reads.crates/admin_markers/- the owner-socket rules the gateway and the Admin share (markers, socket path, frame names, bounds), and the config and state roots.crates/vulpine_fleet/,crates/fleet_probe/- the fleet adapter, and the probe binary it runs on each fleet host.crates/vulpine_provider/- whether an upstream model provider is degraded.crates/vulpine_room/- one interface for making a room say or play something.crates/vulpine_wechat/- the WeChat send channel’s control plane.crates/vigil_serve/- Vigil’s installed build served as a static page on loopback.crates/host_footprint/,crates/glyph_shader_check/- the Glyph’s memory footprint read through libproc, and its WGSL shaders checked with naga.packages/protocol/- generated TypeScript protocol package (@voidfab/vulpine-protocol).packages/voice-client/- browser-clean TypeScript Client Core (@voidfab/vulpine-voice-client).packages/vulpine-voice-ui/- shared React voice chrome (connect control, device rows, captions, confirmations).packages/vulpine-voice-ui-state/- the voice chrome’s framework-free view-model.apps/vigil/- Vigil, Vulpine in a browser (a web app).apps/glyph/- Vulpine.app, the Mac client, in its Glyph look (a Tauri ambient overlay: swappable chassis around Orb, Cast, LED and SVG Visual Themes).apps/admin-shadow/- the Admin, shipped as Vulpine Gateway.app.apps/satellite-esphome/- the room speaker bridge for the Voice PE (ESPHome native API to the gateway WebSocket).apps/tui/- the terminal skin: pi-tui over Client Core, text only.apps/phone/- Vulpine on iPhone, in development (GPUI).apps/site/- Vulpine’s public site (static Astro pages).runtime/- the runtime helpers the gateway, a launchd job or a roll spawns: the worker adapters, the adapter CLIs (homectl, roomctl, visionctl, callctl, screenctl), the brain tools’ scripts, the launchd jobs and the roll;runtime-tools.tomllists them.services/- the remaining Parakeet STT helpers.tools/- operator and repo tooling by seam: install and extraction utilities, evaluation, docs and verification, meetctl.scripts/- user-facing shell entrypoints.catalog/agents/- bundled demo agent and persona metadata only; generated voice and avatar assets stay local.docs/- architecture, benchmark, extraction and loadout notes.
See also
- CONTEXT.md - the glossary behind the names above.
- docs/architecture/README.md - the system map.
- docs/key-decisions.md - the decisions that shape it, in plain words.
- docs/speculation.md - the speculative turn in full.
- docs/contracts/decision.v1.md - the decision wire.