Voice input and Web Push

Both features are disabled by default. You can enable a microphone button in the composer, and browser notifications that trigger when an agent is waiting for input.

Voice input (optional)

A microphone button in the composer, and a hands-free switch in Settings. Tap the button, speak, and the transcript lands in the message box for you to read and send. With hands-free on it is sent for you — down the same guarded reply path a typed message takes, never around it. The microphone is the round button at the end of the row, for as long as the box is empty; the first character you type turns it back into Send. You dictate a message or you type one, so there is one primary action rather than two competing for the width of the field.

It does not exist until you run collie stt setup. No button is drawn, no audio leaves the phone, no credential is held, no child process runs. Absent, not disabled. Three providers:

Setup is a CLI act for the reason pairing is one: this surface accepts a credential, so it belongs on the host's keyboard. There is no web setup form.

$ bin/collie stt setup
Which speech-to-text provider?
  openai-compatible  any endpoint that speaks POST /audio/transcriptions —
                     the public OpenAI API, or a local whisper.cpp / parakeet.cpp
                     server, which is the zero-egress choice and the one to prefer.
  codex              borrow your own `codex` sign-in. No new key, no new account —
                     and a private endpoint that may break without notice.
  local-cli          run a transcription command already on this machine, such as
                     whisper-cli or muesli-cli. Its stdout is the transcript.
provider [openai-compatible]:
The API base, INCLUDING its version prefix — the provider appends /audio/transcriptions.
  local  http://127.0.0.1:8080/v1     (whisper.cpp / parakeet.cpp — nothing leaves the host)
  cloud  https://api.openai.com/v1    (room audio leaves this machine)
base URL: http://127.0.0.1:8080/v1
The model the endpoint understands. Empty takes Collie's default, gpt-transcribe.
model [gpt-transcribe]: whisper-1
API key [none]:
The language you speak, as a two-letter ISO-639-1 code — en, de, tr, ja.
LEAVE IT EMPTY to let the model detect it, which is what you want if you mix languages in one
sentence. Name one only if short clips keep coming back in a language you did not speak: a few
seconds of accented audio is too little for the model to detect from, and it guesses.
spoken language [auto-detect]: en
✓ speech-to-text configured — /home/you/.local/state/collie/stt.json (owner-only)
  Live immediately — no restart needed. The bridge re-reads this file per request.
  Check it end to end with `collie stt test`.

Every question above has a flag (--provider · --url · --model · --key · --lang), so a provisioning run needs no terminal. Leaving the key empty is a supported mode — a keyless endpoint is dialled with no Authorization header at all, rather than an empty one.

The spoken language is worth setting only for one failure. Left blank — the default — the model detects the language itself, which is what someone who mixes two languages in a sentence needs. Set it if short clips keep coming back in a language you did not speak: a few seconds of accented audio is too little to detect from, and the model guesses. A two-letter code, or a regional tag Collie narrows for you (en-GB → en). It rides on the openai-compatible provider only; the codex endpoint takes no language, and collie stt status says so rather than letting you believe otherwise.

A long recording gets a long deadline. The browser's budget for one clip is a function of that clip's size, not a flat number — it assumes a sustained 256 kb/s uplink and adds the bridge's own provider deadline on top, so the 8 MiB maximum is allowed a little under six minutes. A clip Collie was willing to record is a clip it is willing to wait for. While the upload is in flight Collie stops polling and stops escalating the connection banner: your own audio saturating a phone's uplink is not an outage, and it must not be reported as one.

Did it work? stt test sends a fifth of a second of generated silence through the real provider, once per container a phone can record:

$ bin/collie stt test
provider: openai-compatible (http://127.0.0.1:8080/v1, model whisper-1, language en)
sending:  0.2 s of generated silence as audio/wav (the setup probe) … ✓ 214 ms
  transcript: (empty) — expected from silence, and the empty answer still proves the pipeline.
sending:  0.2 s of generated silence as audio/webm;codecs=opus (Chrome, Android, Firefox) … ✓ 198 ms
sending:  0.2 s of generated silence as audio/mp4 (Safari, iOS) … ✓ 190 ms

An empty transcript is a pass — silence transcribes to nothing, and the round trip is what was being proved. If it fails, the error names its kind (auth, endpoint, response shape). Then reload Collie on the phone: a microphone sits beside the message box. collie stt status says what is configured and where each setting came from (the file, or an environment variable that outranks it); collie stt off removes stt.json and the button is gone again, no restart either way.

Container support is provider-specific

The phone never records WAV. It records Opus in a WebM container on Chrome, Android and Firefox, or AAC in an MP4 container on Safari and iOS, and it sends those bytes as they are. A provider that transcribes WAV can still answer 400 to both of them, and every dictation then fails with "refused" while stt test looks healthy. That is why stt test sends all three clips: the refusal is found at setup, not on the phone.

One case, verified on 2026-09-01 against OpenRouter's POST /v1/audio/transcriptions:

The workaround is the model, not the key: point the same OpenRouter key at openai/whisper-large-v3-turbo, which takes all four.

bin/collie stt setup --provider openai-compatible \
  --url https://openrouter.ai/api/v1 --model openai/whisper-large-v3-turbo --key <key>

A refused transcription now names the upstream status and the container it was sent as, so the error on the phone says which format the provider rejected. (#148, thanks @drewbitt)

Zero-egress — point it at your own engine

The reason openai-compatible is the provider to reach for: give it a local base URL and no room audio ever leaves the host. Two engines serve an OpenAI-compatible transcription endpoint — whisper.cpp's bundled server, and mudler/parakeet.cpp (MIT). Build or install either by its own instructions, run it on loopback, and point --url at it:

bin/collie stt setup --provider openai-compatible --url http://127.0.0.1:8080/v1

That is the whole integration — Collie has no opinion about which engine answers.

Mistral's Voxtral needs no support of its own, and neither does anything else that speaks this contract — that is the point of the seam. vLLM serves the open-weights Voxtral models on /v1/audio/transcriptions, so a local one is the same --url as any other engine. The hosted models are the same request at Mistral's own base:

bin/collie stt setup --provider openai-compatible \
  --url https://api.mistral.ai/v1 --model voxtral-mini-latest --key <key> --lang en

Voxtral Mini Transcribe covers 13 languages and takes the same ISO-639-1 language field Collie already sends. Prove it with collie stt test before you trust it — "OpenAI-compatible" is a claim each endpoint makes for itself, and that verb exists to check it.

The codex provider — what you are accepting

collie stt setup --provider codex prints a consent block and stops until you type yes, because the honest sentence is this: recordings go to an undocumented, unsupported ChatGPT endpoint authorised by your sign-in, so your ChatGPT account carries the rate-limit and ban exposure, and it may break without notice.

Collie asks that endpoint under its own name first. Only if the honest identity is refused does it fall back to the Codex CLI's headers — and that fallback is written into the config, in a word collie stt status reads back to you. Collie never reads or stores ~/.codex/auth.json; the binary you already trust stays the only thing that touches it.

Run your own command: the local-cli provider

local-cli runs a transcription command you already have, once per recording, with no HTTP server in between:

bin/collie stt setup --provider local-cli \
  --command /Applications/Muesli.app/Contents/MacOS/muesli-cli --args transcribe
bin/collie stt test

Collie writes the recording to a private temp file and runs <command> [args…] <file>. The file's path is always the last argument. The command's stdout, trimmed, is the transcript.

A bare name is looked up when you run setup, and the full path is saved. The service runs with a short PATH, so a name your shell finds may not resolve there. A relative path such as ./whisper-cli is refused.

--args takes words separated by spaces. No shell reads them, so quotes are kept as typed. An argument with a space in it needs the JSON form instead:

bin/collie stt setup --provider local-cli --command whisper-cli \
  --args '["-m", "/srv/my models/ggml-base.bin", "-nt"]'

What the command must do:

• —
  Be a regular file your user can execute. A symlink to one is fine, because Collie follows it. collie stt setup refuses anything else, and collie stt status says what is wrong.
• —
  Exit with status 0 and print the transcript to stdout. Anything else fails the dictation.
• —
  Finish within 60 seconds. Collie then kills it, and every process it started, with SIGKILL.
• —
  Print at most 256 KiB to stdout. Collie kills it as soon as it prints more.
• —
  Leave nothing running. When the command exits, Collie kills any process it started that is still there. A helper that must stay up has to start its own session, for example with setsid.

On Windows, Collie ends the command's whole process tree at the same moments, and after a clean exit it ends any process the command started that is still running. One case is left: a process started by a helper that has already exited can outlive the command.

At most two dictations run the command at the same time. A third gets the same "busy" answer the phone already knows, and Collie starts nothing for it.

An empty stdout is a failure on the phone, because there is nothing to put in the box. collie stt test counts it as a pass, since its clips are silence.

Note. The command runs as the bridge's own user, with that user's rights, once per dictation. It gets no shell, no stdin, and no COLLIE_* environment variable. Its stderr is never read, and the phone never sees the command line or a path, only that the command failed and its exit status.

The temp file sits in a folder only your user can open, and Collie deletes it after every run. If the bridge itself was killed mid-run, the next bridge start removes such folders once they are an hour old. Where the audio goes next is up to the command: a local engine keeps it on the host.

The reasoning for all of the above — why this was declined twice, what changed, and why the seam looks like this — is ADR 0029.

Web Push (optional)

Disabled by default. Setup requires three steps. The sender library (web-push) is included as an optional dependency during the build:

collie push-keys     # 1. generate + write the VAPID keys
collie restart       # 2. Collie reads them at start
#                      3. on your phone: Settings → Alerts

The push-keys command generates the keypair and writes COLLIE_VAPID_PUBLIC and COLLIE_VAPID_PRIVATE with file mode 600 to the active .env, which is ~/.config/collie/.env on a binary install or the one in Herdr's plugin config dir on a Herdr install.

Pass a contact URI as an argument to set the RFC 8292 subject claim:

collie push-keys mailto:you@example.com

On Herdr-managed installs, both steps are available as actions (herdr plugin action invoke push-keys --plugin herdr.collie, and restart). Herdr actions do not accept positional arguments, so setting a subject requires running the command directly in the shell.

Key handling details:

The command refuses to overwrite existing keys unless you pass --force. Replacing keys invalidates all current subscriptions, requiring every device to resubscribe before it can receive notifications again. Providing a subject argument on an existing configuration updates only the contact address and preserves the current keys.

Note. On Herdr versions before 0.8.0, actions remain fixed to the set cached during initial plugin installation (ADR 0006). The push-keys and push-test actions will not appear until you run herdr plugin install. Run bash scripts/collie-ctl.sh push-keys directly instead. The wrapper script passes the command directly to the binary.

Test the delivery path across all subscribed devices:

collie push-test                     # or: push-test "Title" "Body"

Delivery takes one to two seconds. If the command reports push is disabled, restart the service so it loads the generated keys. If it reports no subscribed devices, complete step 3 in the phone browser.

Web Push requires a secure context (HTTPS). This is provided by tailscale serve (MagicDNS certificates) or an external reverse proxy terminating TLS (Variant C). Plain HTTP setups (COLLIE_SERVE_MODE=http) lack a secure context, and the browser disables the subscription controls in Settings.

Collie sends notifications when an agent enters the blocked or done state, placing the agent message in the body. Selecting the notification navigates directly to that agent in the web UI.

Which alerts Collie sends

Four kinds, each with its own switch under Settings → Alerts → Notify when.

Finished fires when an agent goes from working to idle, so an agent you interrupt also counts as finished.

Each device shows an alert's title in its own language, and English on a device that has not opened Collie since it was updated. The body stays as the bridge wrote it.

Every switch is bridge-wide. A push fans out to every subscribed device, so there is nothing per-device to set.

The cache warning can also be switched on for one pane at a time, from that pane's own settings: open the pane, tap the ⋮, then Pane settings. Those panes are listed under the global switch in Settings, with a Remove button each.

The two states add up rather than override. A pane is warned when the global switch is on or the pane is on that list, so turning the global switch off does not silence a pane you watched by hand.

A watched pane is warned once per warm cycle. The agent's next request moves the deadline, and only then can the same pane warn again. The quiet hours apply: a snoozed bridge sends nothing, and a snooze that ends inside the window still warns. A pane that is already cold is never warned, and that includes one a /model switch made cold with time still left.

COLLIE_CACHE_WARN_SECONDS moves the window, default 300 seconds (configure.md).

Note. This alert needs the prompt-cache reading, so it covers the panes that show a cache countdown and no others: the harness has to keep a transcript Collie can read, and the agent has to have taken at least one turn. The window a pane actually warns at is the configured seconds, or half that pane's own cache lifetime, whichever is shorter. A one-hour Claude cache keeps the full 300 second window. A five-minute cache, Codex, OpenCode, pi and omp, warns about two and a half minutes before it goes cold instead.

Stale subscriptions can accumulate over time because home-screen reinstalls and service-worker resets create new endpoints without always returning an HTTP 410. Collie updates the record when a device re-registers. You can view and delete stored endpoints directly:

# one line per device: service, since, user agent, endpoint tail
bin/collie push list
bin/collie push forget <substring>   # or: push forget --all