feat: add wake phrase voice assistant preview

[calebeden]

Additional instructions

MUST: Keep Allow edits from maintainers enabled for this PR so maintainers can help update the branch when needed.

What Problem This Solves

Windows users do not have an always-on, wake-phrase voice experience comparable to the macOS app. Existing speech input, gateway chat, and speech output work independently, but users must manually start each interaction.

Why This Change Was Made

This adds a default-off Assistant preview that reuses the existing local Whisper STT, energy VAD, gateway chat path, and configured TTS provider. V0 deliberately supports one flow: say the wake phrase and request in one utterance, receive one spoken reply in the currently selected native Chat session, then return to listening.

Each Assistant request owns exactly one gateway-accepted run. Response claiming, cancellation, terminal handling, and late-response suppression are scoped to that run so unrelated activity in the same session is not spoken or aborted. Turning the feature off disposes its coordinator, subscriptions, and microphone capture machinery.

V0 adds no wake-word model, protocol command, persistent conversation mode, barge-in, or overlay.

User Impact

Users can configure a custom one-to-three-word wake phrase on Voice & Audio after configuring speech input and Companion Voice. When enabled, a matching utterance sends only the request after the wake phrase, capitalizes its first character, routes it through the visible selected Chat session, speaks the matching reply once, and resumes listening.

While Assistant owns the microphone for wake listening, native Chat visibly explains why single-message voice input is unavailable. Switching sessions does not restart wake capture. Turning Assistant off releases capture and restores normal voice transcription.

Evidence

• Full repository build passed on Windows ARM64 with .NET SDK 10.0.301.
• Shared tests: 3,399 passed, 32 skipped.
• Tray tests: 2,107 passed, 0 skipped.
• Focused voice and run-correlation tests: 34 passed.
• The contributor manually retested the complete user-visible wake-to-selected-session-to-spoken-reply flow on current head 31c90691 and confirmed that it matches the existing recording.
• Automated coverage includes wake matching, settings persistence, readiness, exclusive microphone ownership, selected-session routing, accepted-run binding, pre-binding response buffering, cross-run isolation, exact cancellation, UI Stop ordering, terminal/final event reordering, deduplication, timeout, TTS recovery, Mode Off disposal, localization, and accessibility feedback.
• Current-head CI passed the revocation-recovery, setup-connect, and network-recovery E2E jobs. The aggregate test job is currently red because the unrelated SystemRun_Where_ReturnsExpectedOutput Tray integration test returned exit code -1 instead of 0.

Change Type

• Bug fix
• Feature
• Refactor
• Docs or instructions
• Tests or validation
• Security hardening
• Chore or infrastructure

Scope

• Tray or WinUI UX
• Windows node capability
• Local MCP or winnode
• Gateway, connection, or pairing
• Setup or onboarding
• Permissions, privacy, or security
• Tests, CI, or docs

Validation

• ./build.ps1: passed
• dotnet test ./tests/OpenClaw.Shared.Tests/OpenClaw.Shared.Tests.csproj --no-restore: 3,399 passed, 32 skipped
• dotnet test ./tests/OpenClaw.Tray.Tests/OpenClaw.Tray.Tests.csproj --no-restore: 2,107 passed, 0 skipped
• Focused voice and run-correlation selection from OpenClaw.Tray.Tests: 34 passed
• GitHub Actions e2etests (revocation-recovery, ...): passed on 31c90691
• GitHub Actions e2etests (setup-connect, ...): passed on 31c90691
• GitHub Actions e2etests (network-recovery, ...): passed on 31c90691
• GitHub Actions aggregate test: failed in unrelated McpHttpServerIntegrationTests.SystemRun_Where_ReturnsExpectedOutput; Shared, Tray, Connection, and WinNode CLI test steps passed before that failure

Real Behavior Proof

• Environment tested: Windows 11 ARM64, .NET SDK 10.0.301, Windows SDK 10.0.26100.0
• PR head tested: 31c906917f00b369bde04b928e29d721be7dfd02
• Exact current-head steps: Enabled Assistant with configured STT and TTS, spoke the configured wake phrase and request, switched native Chat sessions, and confirmed that the request routed to the visible selected session with one matching TTS playback. Capture resumed normally, and switching sessions did not cause a microphone restart loop.
• Existing recording of the same wake-to-request-to-spoken-reply user experience:

OpenClawAssistantV0.mp4

• Recording applicability: The recording was captured on an earlier commit, but the complete user-visible flow was manually repeated on 31c90691 and produced the same behavior. Changes after the recording harden internal run correlation, cancellation, event ordering, and disposal without changing the demonstrated interaction.
• Current-head automated proof: exact-run regressions cover unrelated same-session finals, final/terminal ordering before and after accepted-run binding, cancellation while another run is active, UI Stop before a late final, readiness changes during the owned run, and Mode Off disposal.
• Observed result: Wake requests route through the selected Chat session, only the owned response is eligible for speech, and capture resumes after the turn.
• Screenshot or artifact links verified? (Yes/No/N/A): Yes, the recording resolves as an MP4 and remains representative of the manually retested current-head UX
• Not verified or blocked: No user-visible flow is blocked or unverified on current head. Concurrent-run isolation, exact cancellation isolation, and Mode Off disposal are internal invariants covered by current-head automated tests rather than separately visible in the recording.

Security Impact

• New permissions or capabilities? (Yes/No): No
• Secrets or tokens handling changed? (Yes/No): No
• New or changed network calls? (Yes/No): No, requests use the existing configured gateway chat path
• Command or tool execution surface changed? (Yes/No): No
• Data access scope changed? (Yes/No): Yes
• If any answer is Yes, explain the risk and mitigation: When explicitly enabled, the microphone remains active and local Whisper transcribes silence-bounded speech bursts to detect the configured phrase. The feature is off by default, requires existing STT and TTS capabilities, does not log audio or transcript content, and sends only the extracted request through the existing configured gateway path.

Compatibility and Migration

• Backward compatible? (Yes/No): Yes
• Config or environment changes? (Yes/No): Yes, two optional Assistant settings are persisted with safe defaults
• Migration needed? (Yes/No): No
• If yes, list the exact upgrade steps: N/A

Review Conversations

• I replied to or resolved every bot review conversation addressed by this PR.
• I left unresolved only conversations that still need maintainer judgment.

[clawsweeper]

Codex review: needs real behavior proof before merge. Reviewed August 24, 2026, 7:03 AM ET / 11:03 UTC.

ClawSweeper review

What this changes

The PR adds a default-off wake-phrase assistant that transcribes one request, sends it through the selected native chat session, and speaks the correlated reply.

Merge readiness

⛔ Blocked until stronger real behavior proof is added - 9 items remain

Keep open, but do not merge the current branch. It is conflicted with current main and does not carry forward the released no-data capture recovery needed for persistent wake listening; maintainer sponsorship is also needed for this microphone-on preview.

Priority: P1
Reviewed head: 31c906917f00b369bde04b928e29d721be7dfd02
Owner decision: Required. See Decision needed.

Review scores

Measure	Result	What it means
Overall readiness	🦪 silver shellfish (2/6)	The branch has substantial feature and test coverage, but a current-main availability blocker and insufficient current-head proof prevent merge readiness.
Proof confidence	🦐 gold shrimp (3/6)	Needs stronger real behavior proof before merge: The linked recording is from an earlier revision and the current-head retest is only asserted, so add redacted current-head Windows evidence showing enable, wake request, matched reply, and recovery before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Patch quality	🦪 silver shellfish (2/6)	1 actionable review finding remain.

Verification

Check	Result	Evidence
Real behavior	Needs proof	Needs stronger real behavior proof before merge: The linked recording is from an earlier revision and the current-head retest is only asserted, so add redacted current-head Windows evidence showing enable, wake request, matched reply, and recovery before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
Evidence reviewed	5 items	PR capture path lacks the released watchdog: The branch starts WASAPI capture directly and has no first-audio timeout or recovery path for a device that opens but emits no data. Released main baseline has no-data recovery: The current released AudioPipeline arms a first-audio watchdog and tears down capture on timeout, a behavior absent from this branch baseline. Release provenance: The watchdog-bearing commit is contained by v2026.7.1, so rebasing must preserve shipped capture recovery rather than restore the older pipeline.
Findings	1 actionable finding	[P1] Release wake ownership after watchdog failure
Security	None	None.

Live Verification

Command: powershell -ExecutionPolicy Bypass -File .\run-app-local.ps1 -NoBuild -Isolated -AllowNonMain

Result: FAIL (failed) — execution before step 1 run: sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.23.0.tgz

sh -lc pnpm install --ignore-scripts --frozen-lockfile failed: ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-11.23.0.tgz

Assertions:

• FAIL expect_output: Launching OpenClaw Tray

How this fits together

The Windows tray voice stack captures microphone audio, transcribes it locally, sends a request through the existing chat provider, and plays the reply through configured text-to-speech. This PR adds a coordinator spanning those existing capture, chat, and playback paths.

flowchart LR
  A[Microphone audio] --> B[Local transcription]
  B --> C[Wake phrase gate]
  C --> D[Assistant coordinator]
  D --> E[Selected chat session]
  E --> F[Correlated final reply]
  F --> G[Text to speech]

Loading

Decision needed

Question	Recommendation
Should the project sponsor a default-off, persistent-microphone wake-phrase preview after it is rebased onto the current voice and chat architecture?	Defer the preview: Keep this PR unmerged until a maintainer explicitly sponsors the persistent-microphone direction and a current-main implementation is proposed.

Why: This adds a new persisted microphone behavior and its current branch conflicts with the post-release capture and chat refactors; code review cannot establish whether this product surface is desired.

Before merge

• Add real behavior proof - Needs stronger real behavior proof before merge: The linked recording is from an earlier revision and the current-head retest is only asserted, so add redacted current-head Windows evidence showing enable, wake request, matched reply, and recovery before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• Release wake ownership after watchdog failure (P1) - Current main's first-audio watchdog tears down a no-data capture, but this added wake path only forwards pipeline state and retains its capture lease. After rebasing, that leaves the assistant in WakeListening and blocks normal voice input after a no-data device failure. Release the lease and transition to a recoverable state, with the watchdog regression covered.
• Resolve merge risk (P1) - A rebase across the current AudioPipeline and native-chat ownership changes can reintroduce the shipped no-data capture failure unless the watchdog and wake-state recovery are integrated together.
• Resolve merge risk (P1) - The only linked recording predates the current head, so the persistent microphone flow and recovery state lack directly inspectable current-head proof.
• Complete next step (P2) - Maintainers must first choose whether to sponsor this persistent-microphone preview; the conflicted branch also needs a non-mechanical rebase across current voice and chat owners.
• Improve patch quality - Rebase onto current main and preserve the released no-data capture watchdog through the wake path.
• Improve patch quality - Add a regression proving watchdog failure releases wake ownership and restores normal voice availability.
• Improve patch quality - Post redacted current-head Windows evidence for the wake-to-reply flow and recovery state.

Findings

• [P1] Release wake ownership after watchdog failure — src/OpenClaw.Tray.WinUI/Services/VoiceService.cs:431-449

Agent review details

Security

None.

Review metrics

Metric	Value	Why it matters
Change surface	54 files affected	The feature spans voice capture, chat routing, settings, UI, localization, and tests, while its base is now conflicted with main.
Production versus tests	production +2678/-89, tests +1696/-1	The broad production addition needs a rebase that preserves current ownership and recovery invariants.

Merge-risk options

Maintainer options:

1. Rebase onto capture recovery (recommended)
   Rebase onto current main, retain the released first-audio watchdog, and make its error transition release wake ownership before merge.
2. Pause the preview
   Leave the branch unmerged if maintainers do not want to own persistent microphone behavior in the current tray roadmap.

Technical review

Best possible solution:

If maintainers sponsor the preview, reimplement it on current voice and chat seams so watchdog failure releases wake ownership and enters a recoverable state, then provide redacted current-head Windows proof.

Do we have a high-confidence way to reproduce the issue?

Not applicable: this PR proposes a new feature. The relevant failure mode is source-proven from the branch's older capture path and current main's released watchdog.

Is this the best way to solve the issue?

No. The proposed feature may be viable, but this stale branch is not the maintainable implementation because it must first adopt current capture and chat ownership boundaries.

Full review comments:

• [P1] Release wake ownership after watchdog failure — src/OpenClaw.Tray.WinUI/Services/VoiceService.cs:431-449
  Current main's first-audio watchdog tears down a no-data capture, but this added wake path only forwards pipeline state and retains its capture lease. After rebasing, that leaves the assistant in WakeListening and blocks normal voice input after a no-data device failure. Release the lease and transition to a recoverable state, with the watchdog regression covered.
  Confidence: 0.96

Overall correctness: patch is incorrect
Overall confidence: 0.96

AGENTS.md: found and applied where relevant.

Codex review notes: model internal, reasoning high; reviewed against ccd64bbb7d68.

Labels

Label justifications:

• P1: Without watchdog-aware wake recovery, a no-data microphone can leave the voice workflow unavailable until user intervention.
• merge-risk: 🚨 availability: The patch changes persistent microphone ownership and can bypass the currently released no-data recovery behavior.
• rating: 🦪 silver shellfish: Overall readiness is 🦪 silver shellfish; proof is 🦐 gold shrimp and patch quality is 🦪 silver shellfish.
• status: 📣 needs proof: The PR needs real behavior proof before ClawSweeper can clear the contributor ask. Needs stronger real behavior proof before merge: The linked recording is from an earlier revision and the current-head retest is only asserted, so add redacted current-head Windows evidence showing enable, wake request, matched reply, and recovery before merge. After adding proof, update the PR body; ClawSweeper should re-review automatically. If it does not, the PR author or someone with repository write access can comment @clawsweeper re-review.
• proof: 🎥 video: Contributor real behavior proof includes video or recording evidence. The linked recording is from an earlier revision and the current-head retest is only asserted, so add redacted current-head Windows evidence showing enable, wake request, matched reply, and recovery before merge.

Evidence

What I checked:

• PR capture path lacks the released watchdog: The branch starts WASAPI capture directly and has no first-audio timeout or recovery path for a device that opens but emits no data. (src/OpenClaw.Tray.WinUI/Services/AudioPipeline.cs:124, 31c906917f00)
• Released main baseline has no-data recovery: The current released AudioPipeline arms a first-audio watchdog and tears down capture on timeout, a behavior absent from this branch baseline. (src/OpenClaw.Tray.WinUI/Services/AudioPipeline.cs:802, 4206611f2b2e)
• Release provenance: The watchdog-bearing commit is contained by v2026.7.1, so rebasing must preserve shipped capture recovery rather than restore the older pipeline. (src/OpenClaw.Tray.WinUI/Services/AudioPipeline.cs:36, 4206611f2b2e)
• Prior blocker remains unchanged: The voice service code implicated by the prior watchdog finding is unchanged since the last reviewed head. (src/OpenClaw.Tray.WinUI/Services/VoiceService.cs:431, 31c906917f00)
• Current merge state: GitHub reports this PR as dirty against main, whose voice and native-chat owners have been substantially refactored since the branch base.

Likely related people:

• shanselman: The collaborator review established the run and capture invariants, and current-main blame attributes the released first-audio watchdog to Scott Hanselman. (role: prior reviewer and current audio-path contributor; confidence: medium; commits: 4206611f2b2e; files: src/OpenClaw.Tray.WinUI/Services/AudioPipeline.cs)
• Barbara Kudiess: Recent main history shows ownership work on the chat provider that this branch extends for turn routing and response correlation. (role: recent native-chat refactor contributor; confidence: high; commits: f8a8825ccbb4; files: src/OpenClaw.Tray.WinUI/Chat/OpenClawChatDataProvider.cs)
• Caleb Eden: Has a merged current-main native-chat contribution and authored the proposed voice-assistant implementation. (role: recent native-chat contributor; confidence: medium; commits: a76c85218c7d, 31c906917f00; files: src/OpenClaw.Tray.WinUI/Chat/OpenClawChatDataProvider.cs, src/OpenClaw.Tray.WinUI/Services/VoiceAssistant/VoiceAssistantCoordinator.cs)

Rating scale

Score	Internal tier	Crab rank	Meaning
6/6	S	🦀 challenger crab	Exceptional readiness
5/6	A	🦞 diamond lobster	Very strong readiness
4/6	B	🐚 platinum hermit	Good normal PR; ordinary maintainer review
3/6	C	🦐 gold shrimp	Useful, but confidence is limited
2/6	D	🦪 silver shellfish	Proof or implementation needs work
1/6	F	🧂 unranked krab	Not merge-ready
N/A	NA	🌊 off-meta tidepool	Rating does not apply

Overall follows the weaker of proof and patch quality.
Shiny media proof means a screenshot, video, or linked artifact directly shows the changed behavior. Runtime, network, CSP, and security claims still need visible diagnostics.

Workflow

• ClawSweeper keeps one durable marker-backed review comment per issue or PR.
• Re-runs edit this comment so the latest verdict, findings, and automation markers stay together instead of adding duplicate bot comments.
• A fresh review can be triggered by eligible @clawsweeper re-review comments, exact-item GitHub events, scheduled/background review runs, or manual workflow dispatch.
• PR/issue authors and users with repository write access can comment @clawsweeper re-review or @clawsweeper re-run on an open PR or issue to request a fresh review only.
• Maintainers can also comment @clawsweeper review to request a fresh review only.
• Fresh-review commands do not start repair, autofix, rebase, CI repair, or automerge.
• Maintainer-only repair and merge flows require explicit commands such as @clawsweeper autofix, @clawsweeper automerge, @clawsweeper fix ci, or @clawsweeper address review.
• Maintainers can comment @clawsweeper explain to ask for more context, or @clawsweeper stop to stop active automation.

History

Review history (46 earlier review cycles; latest 8 shown)

• reviewed 2026-08-07T18:41:41.902Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Guard wake listening against zero-data capture
• reviewed 2026-08-07T21:25:25.824Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Guard wake listening against zero-data capture
• reviewed 2026-08-08T14:24:37.371Z sha 31c9069 :: needs real behavior proof before merge. :: none
• reviewed 2026-08-08T15:39:57.567Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Rebase wake capture onto the silent-capture watchdog
• reviewed 2026-08-09T00:06:24.377Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Rebase capture lifecycle onto the silent-capture watchdog
• reviewed 2026-08-09T03:16:26.018Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Rebase wake capture onto the silent-capture watchdog
• reviewed 2026-08-09T14:51:40.695Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Rebase wake capture onto the no-data watchdog
• reviewed 2026-08-09T22:31:09.935Z sha 31c9069 :: needs real behavior proof before merge. :: [P1] Propagate the no-data watchdog to wake recovery

[shanselman]

Adversarial maintainer review found blocking run-correlation defects, so this is not ready to land yet:

• The assistant can claim and speak an unrelated final message from the same thread because correlation is thread-scoped rather than gateway-run-scoped.
• Timeout, TTS failure, disable, or recovery can call the thread-scoped abort path and terminate another active run, or queue an abort against the next run.
• Readiness changes can cancel the assistant's own in-flight turn, and the UI Stop path is not coordinated before a late response is claimed/spoken.
• Turning the feature off reconciles to Off but leaves the coordinator, subscriptions, and capture machinery alive.

Please bind the turn to the gateway run id, make cancellation no-op unless that exact run is still active, move abort suppression ahead of response claiming, and dispose the coordinator when disabled. The current revocation-recovery check is also red. Current merge confidence is 45%.

[calebeden]

Caleb Eden + GitHub Copilot pair-programming follow-up

@shanselman Thanks for the detailed adversarial review. I addressed each of the run-ownership and lifecycle concerns:

• Each Assistant turn is now bound to the authoritative gateway run ID returned by chat.send. Final-response claiming requires the matching session, accepted run, message identity, and sequence boundary, including responses that arrive before run binding completes.
• Assistant cancellation targets only that accepted run. It becomes a no-op if another run is active or the owned run is already terminal, and it does not queue a deferred thread abort.
• Readiness changes preserve the Assistant's own in-flight run. UI Stop invalidates matching Assistant ownership before late responses can be claimed, and cancelled-run suppression is checked before response claiming.
• Switching the mode Off now disposes the coordinator, chat adapter subscriptions, and wake-input capture machinery.

The revocation-recovery E2E check now passes on the current head. The aggregate workflow is still red because of an unrelated Tray integration failure in SystemRun_Where_ReturnsExpectedOutput, so I am not claiming the entire CI workflow is green.

[calebeden]

@clawsweeper re-review

[clawsweeper]

🦞🧹
ClawSweeper re-review requested.

I asked ClawSweeper to review this item again.
Action: item re-review queued (workflow sweep.yml, event repository_dispatch).
Result: when the review finishes, ClawSweeper will create the durable review comment if needed or update the existing comment in place.

Re-review progress:

• State: Complete
• Detail: The durable review result and its route handoff completed.
• Run: https://github.com/openclaw/clawsweeper/actions/runs/30962126123
• Updated: 2026-08-05T00:05:05.744Z

[calebeden]

Note that this should not be merged until microphone bugs are resolved. #1097 describes a UI freeze when attempting to capture audio input. Merging this with that bug present could lead to a bricked installation where a user enables voice assistant, it starts persistently listening, and then the UI freezes and the user is unable to turn off the voice assistant.

[github-actions]

This pull request has been automatically marked as stale due to inactivity.
Please update it or it will be closed.