Troubleshooting a durable agent

Experimental. The wire, the SDK and this page may change, or the feature may be removed, in a later minor release. Durable agents says what decides whether it stays.

A durable agent that does not show up, shows Unreachable, or does not answer a tap: three commands find the cause, and a table of reasons and a list of cases give the fix. Troubleshooting covers Collie as a whole.

First checks

Run these three, in this order:

collie durable list
collie doctor
collie logs

The phone says less. A durable agent row reads Starting or Unreachable and gives no reason. A wire version refusal has its own word, Version mismatch, and a caution box on the durable agent's home. A refused token has its own words in the connection bar. collie doctor and collie logs name every other reason.

The reasons table

When Collie cannot reach a durable agent, the rest of Collie starts anyway, and the durable agent shows as unreachable with one reason.

A reason names the environment variable even when the durable agent is an entry of sources.json. Read COLLIE_DURABLE_SOCKET as the entry's socket, COLLIE_DURABLE_URL as its url and COLLIE_DURABLE_TOKEN as its token. The fix for an entry is in sources.json. The fix for the durable-only form is in .env, then collie restart.

The setting breaks a rule

Collie reports these without dialling the durable agent. Rules that sit in sources.json entries are refused at the start of the bridge as well (A typo in sources.json).

The durable agent does not answer, or answers wrongly

Doctor rows for durable agents

collie doctor asks each durable agent the same questions the bridge asks, in the order the bridge asks them. It sends nothing but hello.

Each entry of sources.json gets one block of six rows, written durable-socket:desk for a durable agent named desk. A sources row before the blocks checks the file, and one durable-lead row follows them. A durable-only Collie prints the same six rows with no suffix, and its mux row says durable, see the durable-* lines.

A row that fails stops the rows after it. They print as skipped with the reason fix the row above first, because Collie never asks a durable agent over a socket that fails the privacy check or with a token that breaks its rule. A skipped row is not a fault.

Cases

Each case says what you see, why, and the fix.

A typo in sources.json

You see Collie down, not only one durable agent: the phone cannot connect. collie logs ends with [bridge] FATAL:, the path of the file, the entry and the field.

The bridge reads sources.json at start and stops on a field it does not know, a kind that is not durable, a name that breaks the rule, a name used twice, socket and url together, neither, a url with no token, or a token with a socket. Fix the file, then run collie restart. collie doctor shows the same message in its sources row.

Where the bridge cannot name the entry (it is not an object, it has an unknown field, or the name breaks the rule), it counts from 0 and writes entry 1 for the second one. collie durable attach counts from 1 and writes entry 2 for the same entry. Both say which field. Once the name is valid, the bridge writes the name: entry "desk".

attach and detach refuse to write a file they cannot read, so a typo never costs your other entries. They stop with exit code 1.

A durable agent does not appear after attach

You see no durable agent on the phone, and collie durable list says not loaded (restart to apply).

attach and detach only write sources.json. The bridge reads it at start. Run collie restart, or add --restart to the verb next time. When not loaded survives a restart, read collie logs: list words an HTTP 400 from the bridge the same way as a 404.

The privacy check fails

You see socket-not-private: the socket's folder /tmp/hello has mode 0755, not 0700, and the doctor row durable-socket is red.

Collie dials a socket only when the whole path is private. The usual causes are a folder with mode 0755, a socket directly in /tmp or another shared folder, and a symlink in the path. A symlink is easy to miss: /var/run links to /run on many Linux hosts, so a path that starts /var/run/ fails with /var/run is a symlink. Use the real path.

Run chmod 700 on the socket's own folder, or move the socket under $XDG_RUNTIME_DIR. The rules are in Security of a durable agent. A durable agent built with the SDK refuses to start in an existing folder whose mode is not 0700, and says so on its own.

A stale socket file

You see connection refused on a socket, and the file is there.

A durable agent that crashed leaves its socket file behind, and nothing listens on it. Start the durable agent again. The SDK removes a dead socket file before it listens, and stops with another instance is already listening when the socket is alive. Delete the file by hand only when your durable agent does not clear it.

A name that is also a terminal session

You see the bridge stop at start with the name is the terminal session's own; pick another, or a terminal session missing on the phone.

A durable agent cannot take the name of the terminal Collie runs from. That name is default on most installs, and on a durable-only Collie default is its durable agent's. The bridge stops at start and names the entry. A Herdr session that has the name of a durable agent is another case: the durable agent keeps the name, the phone does not show that session, and the log says so once. Rename one of them.

A durable-ready warning that stays

You see durable-ready as a warning on every run of collie doctor.

The durable agent answers hello and says ready: false. The phone shows it as Starting. A durable agent says it is ready only after it laid out its conversations and raised its open questions again. Read the durable agent's own log for what it waits on.

The durable agent stops at start on the descendant check

You see the durable agent stop at start with the descendant check needs Bun on Linux to read the peer of a socket; pass callerCheck: "off" to run without it.

The check needs Bun on Linux. On macOS, run the durable agent with Node, or pass callerCheck: "off" to openCollie and read what the check protects first. The error comes from the durable agent. Collie is not involved.

A question comes back after a restart

You see one question again after the durable agent restarted, or a card that says it was already answered.

An answer to a question from before the restart gets too-late, and the card says so and goes. A durable agent that derives the question's id from its tool call raises the same question again, so an answer sent across the restart lands and the tool runs once. A durable agent that makes a new id each time raises a new question for the same tool call. The card shows it, and Collie sends no second push, because the push guard keys on the tool call. The guard also allows 3 pushes per conversation a minute, 10 per host a minute and 30 per host an hour. The push that would be the 31st in an hour becomes one summary push for that host, and the host sends no more until an hour after it. A card is always shown.

403 caller-refused

You see the phone say the answer or the message was refused, and the durable agent's reason is 403 caller-refused.

The durable agent's descendant check refused the caller. It refuses the durable agent itself, any process the durable agent started, and a caller it cannot tell apart from them. Collie falls on the durable agent's side when the durable agent, or a tool it ran, started the Collie process and the process is still its child. Start Collie from outside the durable agent, then try again.

Try again, and an answer that may be applied

You see "Not sure the answer arrived" with a Try again button, and after the tap "May already be applied. Check the chat."

A timeout, a dropped connection or a 5xx after the answer left means it may have landed. Try again sends the same request id, so the durable agent answers replayed and never resolves the question twice. When the question is already closed, the first try may have closed it, so the phone says to check the chat. A send, a stop and a button tap say the same with their own words.

The durable agent refused the token

You see the connection bar say "The durable agent refused the token. Check COLLIE_DURABLE_TOKEN on this peer.", and the reason is token-refused.

The durable agent answered 401 or 403 to the token. The most common cause is a rotation where one side changed first. The words on the phone name COLLIE_DURABLE_TOKEN even for an entry of sources.json: read it as the entry's token. Give the durable agent the new token before you restart Collie. Tokens has the rotation steps. A 403 can also come from an access proxy in front of the durable agent.

A flip to Unreachable that takes two reads

You see a durable agent stay Ready for one failed read, then flip to Unreachable.

One failed read after a good one keeps the last layout for that poll. The second failure in a row flips the durable agent. A refused token follows the same rule. A socket path that fails the privacy check is the one exception: it shows at once.

Over https, a durable agent that is down or refuses the token is asked at most once every 10 s, because each try is a billed request. Over a socket the least gap between two reads of a down durable agent is 250 ms.

In a crew

A durable agent on a member of a crew shows on the phone of the lead, so the lead needs to know what a durable agent is.

A lead older than 1.21.0 lists the durable agent's conversations without the question card, and its Terminal tab polls a conversation that has no terminal. Nothing on the member fixes that. The member says so itself: collie doctor warns in durable-lead, and the log gets one line, [durable] the lead is 1.20.1; it shows durable panes without the question card and its Terminal tab polls; update the lead, once for each release it hears. Update the lead.

A member that is older than the lead has the opposite fault. The phone says "This machine runs an older Collie and cannot take answers here. Update it, or answer in the durable agent." Update that member.