Avatar rigs
The rig side of the avatar wire: how every rig (VRM, Live2D, the 2D fox) draws typed picks, with the VRM 1.0 override table, the gaze split over spine, chest, neck, head and eyes, the VRMA rest-pose retarget, the face director (1.1 s dwell, 0.85 apex cap, per-family onset and release, blinks) and the gated five-question decision.v1 set.
Living doc, checked against the code 2026-11-14
avatar-puppet.v1 — what a rig draws from typed picks: overrides, gaze, clips, the director and the gate
- Status: live in the native kit. No wire frame. The VRM scene, the Live2D scene and the puppet2d fox all consume
it (
apps/native-kit/src/surface3d/avatar.rs, featuresurface3d). The kit holds the wire and theExpressionDriverseam and links no model client; the real client (Decidectl) is the app’s, inapps/native-kit-edge. - Where it is seen: the fox window’s Avatar look (a Live2D portrait) and its VRM look (a
.vrmread in place) take the same commands from one driver (apps/fox-gpui/src/avatar.rs), so one stream of relayed avatar actions directs either. - Scope: the rig side of
avatar-action.v1, which carries closed verbs and names from the gateway to a frontend. This one says how a rig turns typed picks (an expression’s weight, a face, a focus target, a clip) into what it draws, the same way on every rig, so a model’s picks never flicker or fight. The model’s wire isdecision.v1. The brain never sets a weight. - Doctrine: a decision-like model gives typed expression, gaze and gesture picks at sub-second latency. The avatar need not be humanoid. The rules are plain data, so a rig that is not a VRM keeps the semantics and maps them onto its own parameters by a recipe.
Expressions and picks
Eleven expressions, named by their VRM 1.0 presets: the emotions happy, angry, sad, relaxed, surprised;
blink; the lip vowels aa, ih, ou, ee, oh. A pick is one expression’s weight, 0..=1, held for a time or
until the next pick (Command::Pick). decision.v1 noul answers become picks through picks_from_nouls and the
expression choice through picks_from_answer, both behind the reply guard. A rig maps each expression onto what
it has: a VRM onto its preset or the ARKit channels its preparation composed (vrm/presets.rs), a Live2D model and
the puppet2d fox by a recipe (a share of each parameter’s range per expression).
Layers meet by max, never by sum. A frame’s raw weight for an expression is the largest of its eased pick, the
director’s layer (its face, the blink schedule) and, for a vowel, the lip track and the played mouth (Ease::drawn).
The played mouth. Command::Mouth(lips) is the mouth from the audio as it is played: the five lip weights a host
read off its player’s level, one reading at a time. The ease follows the newest reading quickly open
(MOUTH_OPEN_TAU_S, 35 ms) and more slowly shut (MOUTH_CLOSE_TAU_S, 90 ms). It shuts the mouth on its own once no
reading came for MOUTH_STALE_S (0.4 s). An open played mouth engages the director as a spoken line does (the blinks
run). A lip track (Command::Speak, the words’ syllable estimate) plays beside it by max. A host that has the played
level sends the track’s shape inside its readings instead (Vulpine.app’s Avatar portrait, fox-gpui’s
avatar_mouth.rs).
Overrides
Every rig gets one table (face.rs). It follows VRM 1.0’s rule (overrideBlink, overrideLookAt,
overrideMouth): each expression adds a rate to each category it overrides (block: 1 once its weight is above 0;
blend: its weight), and each category is drawn at 1 - min(rate, 1) of its weight. The categories are the blink,
the eyes’ look-at and the five vowels. The emotions are never overridden.
| expression | blink | look-at | mouth |
|---|---|---|---|
| happy, angry, sad, relaxed | blend | none | none |
| surprised | block | none | none |
| blink, aa, ih, ou, ee, oh | none | none | none |
So a smile and a blink on one frame do not stack past shut, wide eyes do not blink, and a smiling fox still talks. A
VRM’s own declared overrides apply inside its preset on top (compose_with takes another table).
Gaze
One aim, degrees in the character’s frame (yaw positive toward the character’s left, the viewer’s right; pitch
positive up), shared out over the body and the eyes (gaze.rs):
| bone | share | yaw limit | pitch limit |
|---|---|---|---|
| spine | 0.15 | 15 | 0 |
| chest | 0.20 | 20 | 0 |
| neck | 0.25 | 25 | 20 |
| head | 0.40 | 40 | 30 |
The body follows the aim at 10 a second (exponential) and each bone its share at 25 a second. The eyes aim at the raw
target at once and take what the body has not turned yet, within 12° yaw and 10° pitch: the eyes lead, the head
catches up, the eyes re-center. A look-at override scales the eyes’ share. A pick is avatar-action.v1’s focus
target: user (ahead), away (30° aside), up / down (18°), point x, y in -1..=1 (45° and 30° at the edges).
Where it lands:
- A VRM turns the spine, chest, neck and head on its animator’s base pose (normalized space) and the eyes on the
eyeLook*channels. - A Live2D model takes the body on
ParamBodyAngleX, the head onParamAngleX/ParamAngleYand the eyes onParamEyeBallX/ParamEyeBallY. - The puppet2d fox has no eyes to lead with, so it turns
angle_xby the whole aim.
Clips
One clip library plays on every rig by the VRM animation spec’s rest-pose formula (retarget.rs): a source bone’s
local rotation is normalized against the source’s local and global rests, then denormalized onto the destination’s:
dst = dst_local * dst_global^-1 * (src_global * src_local^-1 * src * src_global^-1) * dst_global. A clip authored on
one VRM means the same motion on another VRM or the 3D fox rig. A clip is picked by the gate’s what question (below)
or avatar-action.v1’s play_animation.
The VRM scene plays clips from .vrma files. The loader’s reader (apps/gpui-scene3d-gltf-vrm/src/vrma.rs) applies
the same formula onto VRM’s normalized space and gives the hips’ motion as an offset the scene scales by its model’s
hips height. The scene’s clip player (apps/native-kit/src/surface3d/vrm/clips.rs) loops a clip on its clock and
crossfades every start, switch and stop over 0.4 s. A clip replaces the rotations of the bones it names. The gaze,
the breath and the springs are laid on after.
The director
A face pick (Command::Face: a name, a family, a mix of emotion presets, an intensity) goes through the director
(director.rs):
- Dwell 1.1 s: no other face replaces the one on show sooner. A later offer waits and starts when the dwell ends
(a still later one replaces it). An
alertface (surprised, shocked, afraid, curious) cuts in at once. - Apex cap 0.85: no emotion preset is driven past it.
- Per-family onset and release (seconds): alert 0.12 / 0.30, warm 0.30 / 0.42, down 0.45 / 0.60, hot and calm 0.25 / 0.42. A face rises over its onset (the old face blends out under it), holds its apex 0.6 to 1.2 s, settles to a sustain of 0.55 to 0.85 over 0.5 s, and lets go 4.5 s after its last renewal. A renewal (the same face again) extends the hold, never past 13.5 s from the face’s start. The variation is seeded, so a scripted stream replays.
- Neutral lets the face on show go and is over once its onset has blended it out.
- Blinks: every 2.4 to 6 s (closing 60 ms, shut 30 ms, opening 90 ms), one in seven a double, while a face is
held or a line is spoken, or while the scene keeps its avatar alive at rest (
Ease::alive, a scene’s idle life). Otherwise at rest the eyes stay open and the view draws nothing. The override table then thins a blink under an emotion.
The decision model’s expression choice reaches the director through face_from_answer (the reply guard, then
p >= 0.6). A caller that wants weights uses the raw picks (picks_from_answer).
Picks over decision.v1
Five questions about a reply (instinct.rs; benchmarks/verbs/avatar-instinct.v1.json holds a sample request for
the fox’s clips):
| id | type | options |
|---|---|---|
performs |
noul | is the avatar committing to an action in its own current words |
what |
choice | the rig’s declared clips, each an action or a gesture, and none |
reaction |
choice | agree, disagree, empathize, celebrate, recoil, ponder, bashful, scoff, none |
face |
choice | neutral and 21 faces over the five emotion presets (FACES) |
energy |
choice | gentle, moderate, high (gentle shows a face at 0.75) |
The gate (Gate::gate):
| what | acts when |
|---|---|
| any answer | it passes decision.v1’s own checks (typed as asked, a choice among its options, probabilities summing to 1, a noul in 0..=1); a missing or refused answer abstains for its question alone, never a no |
| a face | p >= 0.6 |
| a reaction | the reply’s words are final, p >= 0.75, and 5 s since the last reaction |
| a clip | p >= 0.6 on final words, 0.8 (gesture) or 0.85 (action) on a partial; an action also needs performs >= 0.5 (final) or 0.6 (partial); one clip a turn on partials |
| a veto | performs <= 0.2 with what at none >= 0.7 vetoes every clip |
A choice’s p is its confidence, else its own probability, else 0 (no probability, no act).
Errors
Nothing here is on a wire, so nothing is refused to a peer. A reply the guard refuses moves nothing (PickError). A
gated answer under its threshold is counted (GateCounts: faces_under, reactions_under, reactions_cooling,
vetoes, abstains) and does not act.
Compatibility
Additive. surface3d::vrm re-exports the picks, the ease, the lips, the expression question and the
ExpressionDriver seam; Decidectl is apps/native-kit-edge’s.
Command ends with Face, Gaze and then Mouth. A rig that ignores the director and the gaze draws its picks
under the override table.
See also
avatar-action.v1— the verbs and names a rig declares and the gateway relays.decision.v1— the typed questions’ wire and its reply guard.