---
title: "Release and rollback"
description: "How to cut a gateway release, roll it to the gateway host and roll it back. just release folds the changelog, tags, builds and writes a receipt. just rollback <N> prints how to go back."
type: "living"
source: "docs/release.md"
verified: "2026-11-14"
group: "Install and operate"
---

> **Living doc**, checked against the code 2026-11-14

Neither recipe rolls anything: the roll
is its own step.

## How it works

**The version.** A release is a tag `v<N>`. N is one more than the highest `v<N>` tag. While no tag exists, N is one
more than the live stamp: the `VULPINE_OVERLAY_VERSION` the last roll wrote. `release.py` reads it, read-only, with the
installed gateway binary's `overlay-stdio status` over ssh, or you pass it with `--live N`. The receipt's
`version_source` says which. The gateway host's overlay number is the deploy counter: tag `v504` is the binary that
`just overlay-roll-internal 504 503` ships. `--version N` overrides the derived number; the tool refuses a number at
or behind the last tag. Other tags (`fox9-<sha>-<slug>-<n>`, the Admin deploys) are ignored.

**The two deploy mechanisms.** The gateway host's agent is `com.voidfab.fox9.gateway`. Two mechanisms can own it, and
both use the same version numbers:

| Mechanism | Who writes the LaunchAgent | How a roll restarts it | Used today |
|---|---|---|---|
| overlay (`just overlay-roll` / `overlay-roll-internal`: the gateway binary's `overlay-stdio` subcommand, crates/overlay_stdio) | the roll writes the plist from `overlay.env` (`write_agent_plist`) and stamps `VULPINE_OVERLAY_VERSION` in it | bootout, a bounded wait for the old pid and the Admin socket, then bootstrap (five tries) | **yes** |
| nix (`modules/foxline.nix` in the nix host configuration) | nix-darwin, read-only (0444), marked `VULPINE_AGENT_OWNER=<nix owner>`; it execs the stable binary with `--env-file gateway.env` | the roll stages the binary, config and `gateway.env`, then `kickstart -k` (`start_nix_owned`) | no: the module declares no gateway |

The first line of `just overlay-status` names the owner: the nix owner, or
`agent owner: overlay_stdio (plist written by the roll)`. The roll chooses the same way (`agent_is_nix_owned`), so the
release and roll commands are the same for both; only the restart differs. The version number does not identify the
binary. The build identity does: `foxline-voice-gateway@0.1.0+<sha>`, printed when the roll is ready and written as
the first `gateway_build` row of every session trace.

**The roll runs from the binary it ships.** `just overlay-roll-internal` copies the signed binary to the gateway host
and runs its own `overlay-stdio roll --internal` there (`just/steps/overlay-roll-internal-rust.steps`). `just overlay-status`,
`just overlay` and `just overlay-roll` also run the installed binary's subcommand; a binary without it exits 78.

Between the copy and the roll, the steps file runs the shipped binary's `roll-notice`. On an idle session the live
gateway says "Updating myself, back in a minute." (at most 8 s, never a refusal). The marker it leaves makes the new
gateway say "All systems back." once its Health rows are green (`[announcements] roll`,
`crates/voice_gateway/src/roll_announce.rs`).

**The roll is the commit you build.** `build.rs` stamps the binary from `git status`: `<sha>-dirty` when a tracked
file differs. `mm item update` writes `.memento/**`, which is tracked, so a tracker append during the build stamps
`-dirty`. The order is commit → build → roll, then tracker appends, docs and memory.

**Check the commit before the roll.** `build.rs` re-stamps only when HEAD moves, so a refused commit followed by a
build ships the previous commit's stamp. `release.py` checks for this:

- It refuses a dirty tree before folding.
- It checks the tree is clean again after the release commit.
- After the build, it refuses an artifact that lacks the tagged commit's short sha or holds `<sha>-dirty`
  (`stamp_matches_commit` in the receipt).


## Run it

### Release

1. Start with the tree clean and on the commit you mean to ship: `git status --short | wc -l` prints 0. Read
   `git log --oneline -1`.
2. `just release --dry-run`. It prints the version and where it came from, the fold (on a scratch copy), the tag and
   build commands it would run, and the receipt it would write (under a scratch prefix). It writes nothing in the
   checkout: `git status --porcelain` and `git tag -l` are hashed before and after, and the receipt shows both
   (`checkout_unchanged: true`).
3. `just release`. In order:
   - `tools/docs/changelog_fold.py` folds `changelog.d/`. This is the same fold as `just changelog-fold`, called rather
     than copied.
   - Everything under `## Unreleased` becomes `## v<N> (<date>)`, and `## Unreleased` stays, empty.
   - It commits `chore(release): v<N> - changelog folded`. A release with no fragments adds only the heading, so
     this commit runs with `HK_SKIP_STEPS=changelog-fragments`.
   - It tags `v<N>` (annotated).
   - It runs `just build-gateway`.
   - It checks the stamp.
   - It writes `benchmarks/results/rolls/roll-v<N>-<date>.json`.
   - It prints the roll command and the archive command.
4. Roll: `just overlay-roll-internal <N> <N-1>`. Compare the roll's `binary: identity=` line with the tag's commit.
   - **Signing.** The machine you roll from does not sign, unless it is the signing Mac (below). The build host signs
     with its Developer ID (`tools/repo/remote_release.py`: the host's `VULPINE_CODESIGN_IDENTITY`, else its one Developer ID Application identity of the team in
     `apps/glyph/src-tauri/bundle-identity.json`; `--identifier com.voidfab.fox9.gateway --timestamp=none`). Never
     write the identity into the repo.
   - **Attended.** The host's key is behind Touch ID. The roll prints one `attended-sign:` line naming the command to
     run in Terminal at the signing Mac, and waits `VULPINE_SIGN_WAIT_MIN` (20) minutes. No tool unlocks the keychain
     or asks for a password. When nothing is signed, the roll exits 5 before anything ships.
   - To check a binary first:
     `python3 tools/repo/remote_release.py sign --check --binary "${CARGO_TARGET_DIR:-target}/release/vulpine-voice-gateway"`.
   - A roll on the signing Mac signs there only on an explicit signal: `VULPINE_SIGN_HOST=local` (`=remote` forbids
     it), else the marker file `~/.config/vulpine/sign-host` holding that Mac's `paths.py` host label, else a
     configured host name equal to the Mac's hostname, LocalHostName or ComputerName.
   - Every ssh and rsync to the host has a 10 s connect timeout. `just overlay-dry-run <N> <N-1>` rehearses the roll
     on the host and changes nothing.
5. Archive the binary so a later rollback can use it; the internal roll skips the volume's versioned archive. The
   receipt's `archive_command` is the line to run:
   `scp target/release/vulpine-voice-gateway <gateway host>:<archive-dir>/foxline-voice-gateway-stdio-v<N>`.
6. After the roll, commit the receipt and the tracker appends. Push the tag with the branch:
   `git push origin v<N>`, with the output logged, never piped.


### Rollback

1. `just rollback <N>`, where N is the live version (`just overlay-status`). It names the highest tag below N and
   that tag's commit. It prints:
   - `just overlay-roll <prev> <N>`, which rolls the archived `foxline-voice-gateway-stdio-v<prev>` from the volume;
   - the `shasum` to run on the gateway host first, with the expected sha256 from `roll-v<prev>-*.json` when that receipt
     exists.
2. If that binary was never archived, rebuild the tag in a worktree and roll it under a new deploy number,
   `just overlay-roll-internal <N+1> <N>`, never as `v<prev>` again. One number must never name two binaries.
3. **The config does not roll back.** The staged config and `overlay.env` stay as the v<N> roll left them, and a key
   v<N> added can stop v<prev> at parse. Before rolling, read `git diff v<prev> v<N> --
   crates/voice_gateway/src/config.rs` and `just overlay-config-check`. If a roll's bootstrap fails, read
   `~/.local/state/foxline/overlay-agent.log` on the gateway host for a parse error before retrying.
4. The gateway host's checkout (loadouts, pi-config, the brain's tools) does not roll back with the binary. After a
   revert, run `just mini-sync-checkout`.
5. **The brain's Pi follows nix, not the roll.** Where `/run/current-system/sw/bin` exists, the roll stages a
   `/nix/store/<hash>-<pkg>/bin/<exe>` `[brain] pi_command` as `/run/current-system/sw/bin/<exe>` when the system
   profile has that command, and collapses the `PATH` entries the profile covers. Any other `overlay.env` value of that
   shape, such as `FOXLINE_QWEN_UV`, becomes the resolved store path, because the gateway refuses a symlinked
   `VULPINE_QWEN_UV` (`tts_adapter_failed`). A `darwin-rebuild switch` that moves `pi` changes the brain's Pi at its next
   start; a gateway rollback does not. To pin a version, set `VULPINE_FOLLOW_SYSTEM_PROFILE=0` in the volume
   `overlay.env` and do a normal roll. Step 5 of the roll's dry run lists what moves:
   `just overlay-dry-run <N> <N-1> --normal`.

## Config

| Flag | Default | Meaning |
|---|---|---|
| `--dry-run` | off | Write nothing in the checkout. The fold, the receipt and the folded changelog go under `--scratch` (default: a new temp dir). |
| `--version N` | derived | Release number. Refused when it is at or behind the last tag or the live stamp. |
| `--live N` | read over ssh | The live stamp, used only while no `v<N>` tag exists. |
| `--host ALIAS` | `tools/repo/paths.py host mini` | The gateway host, used in the roll commands and the ssh read. |
| `--date YYYY-MM-DD` | today | Section heading and receipt name. |
| `--build-cmd`, `--artefact` | `just build-gateway`, `target/release/vulpine-voice-gateway` | Overridable for tests. |

The receipt (`fox9.roll-receipt.v1`) holds:

- `version`, `tag`, `version_source`, `previous`, `commit`, `dry_run`;
- `changelog.fragments_folded` and `changelog.section`;
- `build`: the command, the artifact, its `sha256`, its `bytes`, `stamp_matches_commit`;
- `host`, `roll_command`, `archive_command`, `rolled: false`;
- `who`, `build_host`, `when` (UTC);
- `checkout_before`, `checkout_after`, `checkout_unchanged`.

## Every client refused after a roll from master: protocol lockstep

**Symptom.** The roll is up (`listener ready on :8780`, the identity line names your commit), but every client fails
at hello: Vulpine.app, Vigil in a browser and the room speakers. The gateway answers a hello whose `protocol_version` is too low
with `error {code: unsupported_protocol_version}` (`validate_frontend_capabilities` in
`crates/voice_gateway/src/ws/wire.rs`). Client Core closes with `protocol_handshake_failed: incompatible gateway
hello` when the gateway's hello is not its own version. The room speaker's bridge closes the same way.

**Cause.** The client wire protocol is pinned in both directions. Master is protocol 5: `PROTOCOL_VERSION = 5` and
`MIN_PROTOCOL_VERSION = 5` in `crates/protocol/src/lib.rs`. Protocol 5 requires owner admission (`hello.owner`; a
bind without an owner fails with `owner_required`), so a protocol-4 client cannot be admitted. Each client carries
the version it was built with: Vulpine.app and Vigil through Client Core (`packages/voice-client`, an
exact-equality check on the gateway's hello), and the room speaker's bridge through
`apps/satellite-esphome/src/gateway-link.ts`, which takes `PROTOCOL_VERSION = 5` from `packages/protocol/src/wire.ts`.

**What ships with the roll.** A roll whose binary changes the protocol version is a lockstep deploy. Rebuild every
client from the same tree and ship it in the same sitting as the gateway:

- Vulpine.app on every Mac that runs it (launch a new bundle on the gateway host from Finder the first time; a
  launchd first launch is killed by the launch constraint);
- Vigil;
- the bridge for each room speaker (`apps/satellite-esphome`).

The Admin in Vulpine Gateway.app speaks its own version over its socket (`ADMIN_PROTOCOL_VERSION`, 6) and is not part
of this check. Inside protocol 5, every wire addition is an optional field or variant that older protocol-5 clients
ignore. A roll that keeps the version needs no client rebuild. After the roll, `vulpine-glyphctl owner` on each Mac
that runs Vulpine.app shows `attached`. To back out instead, roll the gateway back; a rollback across a version change needs the old clients too.

## Failure modes

- **No `v<N>` tag and no ssh to the gateway host.** The tool refuses. Pass `--live N` from `just overlay-status`, or
  `--version N`.
- **A dirty tree.** The tool refuses before anything moves. Commit or remove the stray change. Untracked files do not
  count, because `build.rs` ignores them too.
- **The release commit is refused by a hook.** Nothing is tagged. Fix the hook's complaint, then reset the fold with
  `git checkout -- CHANGELOG.md changelog.d`.
- **The build fails, or the stamp does not match.** The tag exists but nothing was built or receipted. Do not roll.
  Fix the build, then either rebuild and run the stamp check by hand, or delete the tag (`git tag -d v<N>`) and
  release again.
- **Rollback finds no earlier tag.** This holds until two releases exist. Read the gateway host's volume archive
  (`foxline-voice-gateway-stdio-v<N>`) and roll by hand with `just overlay-roll <old> <live>`.
- **The old gateway does not let go (exit 3, `roll refused (old_gateway_lingered)`).** After the bootout the roll
  waits up to 30 s (`OLD_GATEWAY_EXIT_BOUND_S`) for the old pid to exit and for nothing to answer on the Admin socket.
  Past that it replaces nothing, bootstraps the old plist again (the old binary keeps serving) and exits 3. Find why
  the old process hung (a stalled disk, `sample <pid>`) before rolling again. The gateway's own Admin bind retries a
  lost race for 10 s (`ADMIN_SELF_PROBE_RETRY_WINDOW`). It writes `admin_listener_selfprobe_retry` rows on the Admin
  timeline, then one `admin_listener_selfprobe_failed` if it gives up.


