otter service enable: generated systemd unit lacks SSH_AUTH_SOCK, breaking all git-over-SSH workflows on GNOME

[loocars]

Summary

otter service enable generates a systemd user unit that does not provide SSH_AUTH_SOCK. On a stock GNOME desktop the daemon then inherits SSH_AUTH_SOCK=$XDG_RUNTIME_DIR/gnupg/S.gpg-agent.ssh (gpg-agent's ssh emulation, which holds no identities) instead of the gnome-keyring agent that actually has the user's keys. Every workflow whose steps talk to git over SSH fails at its first step, forever and silently.

Impact

This breaks the shipped Gerrit workflows on GNOME. gerrit-auto-review, gogootter and gerrit-auto-rebase-on-ci-error all fetch from Gerrit over SSH as their first step. Observed on otter 0.1.9, Ubuntu/GNOME:

INFO otter_core::engine: Executing step step=0 step_type=shell command=Some(["gerrit-auto-review-fetch-patchset.sh"])
ERROR otter_core::engine: Step failed step=0 error=step execution failed:
  'gerrit-auto-review-fetch-patchset.sh' exited with code 128
stdout: Fetching refs/changes/69/151269/1 (change 151269, patchset 1)...
stderr: <user>@review.example.com: Permission denied (publickey).
fatal: Could not read from remote repository.
INFO otter_core::triggers::polling: run failed for hash 151269; hash not marked seen and will be retried on next poll

Two things make this worse than a one-off failure:

• It never terminates. Because a failed run does not mark the hash seen, the poller retries the same change every interval and fails identically each time. Mine burned ~7 hours of 5-minute retries on one change.
• It is silent. Nothing surfaces outside daemon.log. The only visible symptom is that reviews stop appearing, which is easy to attribute to a quiet Gerrit.

Root cause

The daemon's own SSH_AUTH_SOCK is load-bearing, but nothing arranges for it to be correct.

inject_isolated_env deliberately forwards SSH_AUTH_SOCK from the daemon process into steps that need host resources (crates/otter-core/src/process.rs):

/// Variables that grant access to privileged host resources.
const UNSAFE_ENV_VARS: &[&str] = &["SSH_AUTH_SOCK", "SSH_AGENT_PID"];

    if include_unsafe {
        for &key in UNSAFE_ENV_VARS {
            if let Some(val) = std::env::var_os(key) {
                cmd.env(key, val);
            }
        }
    }

So git steps get whatever SSH_AUTH_SOCK the daemon happens to hold. But the generated unit only ever sets SHELL (crates/otter-cli/src/service/systemd.rs:41-60):

"[Unit]\n\
 Description=Otter workflow automation daemon\n\
 \n\
 [Service]\n\
 Environment=SHELL=/bin/bash\n\
 ExecStart={binary_str} _daemon\n\
 Restart=on-failure\n\
 \n\
 [Install]\n\
 WantedBy=default.target\n"

Under systemd --user on GNOME, SSH_AUTH_SOCK comes from gpg-agent-ssh.socket, not from gnome-keyring. The consumer side expects a working agent socket; the provider side never supplies one.

Reproduced directly on my machine:

$ SSH_AUTH_SOCK=/run/user/1000/gnupg/S.gpg-agent.ssh ssh-add -l
The agent has no identities.
$ SSH_AUTH_SOCK=/run/user/1000/gnupg/S.gpg-agent.ssh ssh -p 29418 <user>@review.example.com gerrit version
<user>@review.example.com: Permission denied (publickey).

$ SSH_AUTH_SOCK=/run/user/1000/keyring/ssh ssh-add -l
256 SHA256:... (ED25519)   # 5 keys
$ SSH_AUTH_SOCK=/run/user/1000/keyring/ssh ssh -p 29418 <user>@review.example.com gerrit version
gerrit version 3.14.2

Note this only bites once the daemon runs under systemd. Started from a terminal it inherits the desktop session's correct socket and everything works — which is why it presents as "worked yesterday, broken today" right after running otter service enable.

Reproduction

1. On a GNOME desktop with SSH keys in gnome-keyring, run otter service enable.
2. Log out and back in (or systemctl --user restart otter.service), so the daemon starts from systemd rather than inheriting a terminal environment.
3. Trigger any workflow with a git-over-SSH step.
4. Step 0 fails with Permission denied (publickey) and retries indefinitely.

Suggested fix

There is already a precedent for exactly this problem in the codebase: PATH under systemd is also wrong, and process.rs solves it by capturing the value from a login shell at daemon startup:

let output = std::process::Command::new(&shell).args(["-li", "-c", "echo $PATH"])

Environment=SHELL=/bin/bash is in the unit specifically to make that capture work. Extending the same treatment to SSH_AUTH_SOCK seems like the natural fix, and handles gnome-keyring, gpg-agent and plain ssh-agent alike without hardcoding any agent's socket path.

Alternatives, both weaker:

• Snapshot the enabling session's SSH_AUTH_SOCK into the unit at enable time. Simple, but bakes in a value that may not survive a re-login on setups where the socket path is not stable.
• Emit Environment=SSH_AUTH_SOCK=%t/keyring/ssh. This is what I applied locally and it works, but it hardcodes gnome-keyring and would be wrong for gpg-agent users.

Whatever the mechanism, it would also help to fail loudly: a startup check that SSH_AUTH_SOCK points at an agent with at least one identity, logged at WARN, would have turned seven hours of silent retries into one obvious line.

Workaround

# ~/.config/systemd/user/otter.service
[Service]
Environment=SHELL=/bin/bash
Environment=SSH_AUTH_SOCK=%t/keyring/ssh

systemctl --user daemon-reload && systemctl --user restart otter.service

Confirmed fixed after the restart: the previously-failing change ran through all steps to completion and was marked seen.

Environment

• otter 0.1.9
• Linux 7.0.0-29-generic, GNOME, systemd user session
• Keys held by gnome-keyring ($XDG_RUNTIME_DIR/keyring/ssh)