[Frameforge 2] Implement Web Audio mixing, media sources and waveform decoding

[danwalmsley]

Outcome and dependencies

Depends on the media service in #54. AudioContext is currently undefined. Provide the real mix graph Frameforge uses for transport audio, fades, monitor controls, meters, waveform jobs and recording input.

Specification and implementation

• Implement AudioContext lifecycle/state/currentTime/sampleRate/destination, resume/suspend/close and statechange, with defined host user-activation policy. Expose constructors/methods through AOT-safe bindings, not reflection-only activation.
• Implement createGain, createAnalyser, createMediaElementSource, createMediaStreamDestination and decodeAudioData; node connect/disconnect and valid routing/ownership; AudioParam.value and setTargetAtTime with audio-clock scheduling. Add validation/errors for invalid graph operations and duplicate media-element source creation.
• Implement gain/mixing, channel conversion/resampling, analyser fftSize/smoothingTimeConstant and getFloatTimeDomainData sufficient for real RMS meters. decodeAudioData must return real AudioBuffer sample data, numberOfChannels, duration/sampleRate/length and getChannelData; failures reject correctly.
• MediaElementAudioSourceNode consumes the media decode service without duplicate audible output, observes mute/rate/seek and synchronizes clock changes. Keep context graph and device callbacks independent of V8/UI; no JS invocation, blocking file work or unbounded allocation on the realtime callback.
• MediaStreamAudioDestinationNode exposes a real audio track for the recording issue. Track cloning/stopping must not inadvertently stop monitor playback or sibling consumers.
• Frameforge copies decoded PCM into transferable worker buffers for waveform reduction. Reuse the existing worker/clone implementation and preserve transferred backing-store lifetime after either realm terminates.

Tests and verification

Generate sine, silence, impulse and stereo fixtures; verify gain/fades and mix numerically, channel handling, sample-rate conversion, analyser output and decode failures. Test scheduling across suspend/resume, seek/rate changes, source/context disposal, track cloning and queue bounds. Compare waveform worker envelopes with an independent reduction. Measure A/V drift using a timestamped flash/click fixture during playback and recording; record tolerance and results, not merely audible sound.

Acceptance

• Original playback, volume/mute and clip gain/fades work; no doubled audio.
• Waveforms reflect real decoded media; meters respond correctly to known signals.
• A real mixed audio track is consumable by recording, with bounded latency and clean shutdown.
• Focused Web Audio WPT/project contracts and realtime/lifetime tests pass.

References

Web Audio specification.

Original application references

• src/media.js
• src/peaks.worker.js
• src/export.js

Shared delivery requirements

Parent: #53. Baseline: PR #43 / ce56d49; sample experiments/WebScene.Frameforge.
Reference application: https://github.com/wieslawsoltes/Frameforge (pin f414cd44896b659a5c6da2e4fc596a2ded055f2a, includes demo media and THIRD_PARTY_NOTICES.md).
Implement in WebScene, leaving original application JavaScript/WGSL unchanged. No fake capability objects, synthetic successful events or swallowed errors. Keep public contracts cross-platform with isolated backend implementations; qualify macOS arm64 first and explicitly track Windows/Linux follow-up. Production Avalonia 11 must remain supported; only samples opt into Avalonia 12. Native AOT/trim safety is mandatory.
Add focused native/managed regression coverage and a pinned WPT subset or clearly labeled project-owned contract for non-WebGPU behavior. Applicable WebGPU CTS cases belong with GPU work. Record exact revisions, test commands, negative cases and reproducible evidence in the implementation PR. Unsupported functionality cannot be marked passed; distinguish app coverage from standards conformance. Review existing implementation before adding overlapping infrastructure.

Cross-platform and airspace requirements

Use common services and lifecycle contracts across Windows, macOS and Linux. Prefer reusable cross-platform libraries; native OS service providers are acceptable where appropriate. No native child video window, AVPlayerLayer, HWND overlay or separate presentation surface: video must compose as a retained WebScene texture with normal HTML stacking, clipping, transforms, opacity and scrolling. Share frame ownership with WebGPU external textures and the existing vsync/mailbox/versioned-image architecture. GPU consumers retain immutable frame leases through completion, including resize and replacement.

Foundation progress (not issue completion)

PR #43 commit 42c2e34 adds a bounded asynchronous decoder service, statically compiled pinned miniaudio PCM decoding and a macOS AVFoundation native-frame provider. Original Frameforge score decodes to 576000 stereo frames at 24000 Hz; actual 960x540 H.264 frames at 0 and 1 second differ and retain IOSurfaces after decoder destruction. Local native suite: 17/17 passed; Windows/Linux audio CI coverage added but not yet qualified.

This is opt-in native foundation work. HTML media bindings/playback scheduling, Web Audio realtime graph/output and GPU scene consumption are still pending. No fake browser capability is exposed, and Frameforge remains a compatibility preview. Windows/Linux video currently rejects as unsupported; evaluate FFmpeg versus native providers before selecting those implementations.

Design, library comparison and next slices: https://github.com/wieslawsoltes/WebScene/blob/42c2e349/experiments/WebScene.NativeEngine.Probe/native/media/README.md

[danwalmsley]

Implemented in PR #43, commits ea6b3aa and af02cc8, for the issue's macOS-first Frameforge media milestone.

Delivered

• Actual HTML audio/video lifecycle, markup/source selection, generation cancellation, async seek/play readiness, playback clock and retained native frame leases.
• macOS video is composed through WebScene's versioned IOSurface/GPU scene path, including ordinary DOM clipping and stacking. No native video window/overlay and no video pixel readback/upload in this path. GPU completion wakes retry pending frames after backpressure.
• Statically compiled miniaudio output/PCM decode; macOS container/AAC decoding; native mixing, gain automation, resampling, meters, real stereo capture track leases, independent clone/stop cursors and bounded queues.
• Original Frameforge MediaEngine and waveform worker run unchanged in a Native AOT host with reflection JSON disabled.
• Binary resource fixes cover Avalonia, Uno, archives, Response/Blob/object URLs. The sample server supports real MIME/HEAD/range/416 behavior.
• Normal native package builds enable media and include the pinned miniaudio license without adding a miniaudio shared library.

Verification at af02cc8

• Full native CTest suite: 18/18 passed, including real H.264/AAC decode, seek/cancellation, frame lifetime through consumer completion, backpressure, audio DSP/capture/rate/seek/queue tests.
• Required shared media/audio project contracts: 9/9 passed; required macOS video contract: 1/1 passed. These are explicitly project-owned contracts, not full standards conformance.
• Avalonia managed suite: 305 passed / 15 existing skips on each of net8.0 and net10.0. Uno Release build passed.
• verify-media.py against the Native AOT executable: passed. Nine original cut positions prepared; playback/pause/nonzero native meter and native track checks passed; original transferable waveform worker maximum error against independent PCM reduction 1.49e-8. Actual server MIME, HEAD, closed/open/suffix ranges and 416 checks passed. Native host rendered 30 scenes with no JS/runtime/resource errors in this run.
• Flash/click decoded-video versus actual mixed/recorded PCM timestamp difference: maximum 0.0417ms against 10ms tolerance. This excludes physical display/speaker latency; full device/long-duration qualification remains [Frameforge 6] Qualify unchanged AOT editor, 60 Hz interaction and distributable media runtime #59.
• Ubuntu 22.04 linux/amd64 container: native_document footprint 368 bytes (384-byte budget), native audio graph/capture and PCM decode/session tests passed. The first pushed revision hit the Linux footprint guard; af02cc8 fixes it with shared lazy auxiliary storage rather than increasing the budget.
• Local NuGet inventory: engine/Dawn/ICU/snapshot payloads and exact miniaudio license present; no extra miniaudio dylib.

Scope and follow-up

Windows/Linux media/video hardware parity is explicitly tracked in #61 and #62, using the common service/graph/lease design. Those video backends still report unsupported, not a passing capability. This closes the macOS-first media-engine milestone specified here, not the complete Frameforge editor epic.

Full editor rendering still needs external image/video WebGPU APIs in #40, Canvas2D export in #34, and the remaining #53 issues. The real mixed audio track is now available to the recording adapter; MediaRecorder/encoding remains #58. Broader codecs, rotated/color-managed video, full HTML/Web Audio conformance and physical/long-duration A/V/performance qualification are not claimed by these subset tests.

Design, limits, exact commands and reproducible AOT harness:
https://github.com/wieslawsoltes/WebScene/blob/af02cc83/experiments/WebScene.NativeEngine.Probe/native/media/README.md

Original application reference remains pinned at f414cd44896b659a5c6da2e4fc596a2ded055f2a:
https://github.com/wieslawsoltes/Frameforge

CI will validate the pushed revision separately; no cross-platform CI pass is claimed here.