Implement MediaDevices microphone and AudioWorklet PCM for Code OSS Chat

[wieslawsoltes]

Current microphone and AudioWorklet checkpoint — 20 September 2026

Exact merged heads: WebScene a46b62e14eb197603f7e227c1558afb9088633b0; AppScene 97702daeb9ad71341ced7226f0aa966dc77ea53c; unchanged Code OSS 645f29cc3176500b4b5762ba887cf2a7f0ffdf2c; local vscode-demo e4b98e5470cd7856aff3dfd6843c8e125d7db933 (clean, intentionally unpushed). Open consolidations remain WebScene #76 085bba760fa571c8f08ea866dc5dd1ec27c4d376, AppScene #65 ce66ef6ceb776b3f14e129f853bfc08b5c5e608c, and vscode-demo #1; do not merge them. One-agent mode is active.

WebScene media implementation is merged through #780 and AppScene transport plus macOS, Windows and Linux providers through #297. This includes bounded permission/device/capture transport, browser-shaped MediaDevices/MediaStreamTrack behavior, realtime PCM graph delivery and the narrow AudioWorklet/MessagePort path required by unchanged Code OSS. No media implementation branch is active.

Physical microphone, permission UI, device switching/loss, deterministic PCM, realtime allocation/overrun, VM, package, performance and unchanged Code OSS Chat dictation/PTT acceptance remain deferred and are not claimed.

Parents: #261 and #227
Capability ledger: #263
Related media foundations: #55 and #58
Native host owner: SceneTech/AppScene#289

Proven unchanged-Code-OSS gap

Code OSS 1.137 browser Chat and voice paths directly use APIs that WebScene does not expose:

• navigator.mediaDevices.enumerateDevices() in voice onboarding and microphone selection;
• navigator.mediaDevices.getUserMedia({ audio }) with deviceId, channelCount: 1, sampleRate: 16000, echo cancellation, and noise suppression;
• MediaStream.getAudioTracks()/getTracks() plus track muted, mute/unmute, stop(), and lifecycle;
• AudioContext.createMediaStreamSource() feeding AnalyserNode and an AudioWorkletNode loaded by the unchanged PCM capture worklet.

Without this chain, Chat speech-to-text and push-to-talk cannot capture PCM. Existing WebScene media work implements playback, an output audio graph, and createMediaStreamDestination; it explicitly excludes arbitrary AudioWorklet processing and has no native microphone input.

Representative unchanged sources are chatSpeechToTextService.ts, micCaptureService.ts, dictationOnboarding.ts, voiceModeOnboarding.ts, chatSpeechToTextActions.ts, and pcmCaptureWorklet.ts.

Ordered implementation stack

1. Define a size-versioned host contract for permission, privacy-preserving device enumeration, open/reconfigure/close, and bounded timestamped PCM delivery; AppScene owns the native provider and permission UI.
2. Add browser-shaped MediaDevices, MediaDeviceInfo, devicechange, constraint validation, secure-origin/permission errors, opaque origin/profile device IDs, and one live MediaStreamTrack per capture.
3. Feed native PCM into the existing realtime audio graph through MediaStreamAudioSourceNode; no JS, allocation, locks, or I/O may run on the device callback.
4. Implement the narrow AudioWorklet module/node/MessagePort subset reached by the unchanged mono PCM capture processor, with bounded quantum queues and owner-task delivery.
5. Preserve microphone switching, stored-device fallback, mute/unmute, cancellation, navigation, device loss, context close, and runtime shutdown.

Do not expose successful placeholders, synthesize microphone data, patch Code OSS, or add Electron/Chromium/CEF/WebView.

Quality and performance gates

• WPT-derived Media Capture/Streams and Web Audio shape, constraints, permissions, errors, device-change, track, realm, and teardown contracts.
• Deterministic injected PCM verifies 16 kHz mono conversion, analyser values, worklet chunks, timestamps, ordering, switching, mute and cancellation.
• Permission denial, no device, stale exact device ID, unplug, host failure, overrun, navigation and shutdown settle once with browser-compatible errors.
• Explicit caps for devices, live captures, buffered frames/bytes, worklet modules/nodes/messages, callbacks and retained roots.
• No realtime allocation/lock/JS; publish callback time, overrun/drop counts, end-to-first-PCM latency, CPU/RSS and 100-cycle resource plateaus.
• Exact-package Chat dictation/PTT and onboarding acceptance on macOS arm64, Windows x64, and Linux x64.

Under the active throughput directive the implementation may merge with authored gates; execution remains deferred to the cumulative Chat/release lanes.