Native WebGPU on macOS and Windows with AOT application and media support

[danwalmsley]

Deliver native WebGPU rendering for the unchanged Kestrel CAD application on macOS and Windows using Avalonia, plus AOT-safe application compatibility and macOS-first native media/Web Audio support for Frameforge. This is a bounded milestone within epic #22; the epic remains open.

Rendering and application support

Pinned Dawn/Tint integrates with Metal/IOSurface on macOS and D3D12/shared DXGI images through ANGLE/D3D11 on Windows. Vsync-driven composition, bounded immutable image ownership, producer waits and consumer retirement avoid per-frame CPU pixel readback during normal GPU presentation. Retained images remain visible while resized images are pending. Shared font/text optimizations and CSS fixes include regression coverage.

HTML audio/video now uses a shared asynchronous media service, cancellation generations, bounded queues and real playback/seek clocks. macOS decoded video composes through the retained IOSurface image path with ordinary DOM clipping/stacking, without native overlays. Statically compiled miniaudio supplies audio output/PCM decoding and native mixing, gain, meters and capture tracks. Original Frameforge MediaEngine and waveform worker modules pass a separate Native AOT test. --media-demo opens an interactive MP4 player. The full Frameforge editor is still unqualified.

Binary resources, Blob/Response handling, workers and application compatibility changes support the original AureonStudio/Frameforge assets. Production Avalonia 11 remains supported; samples may opt into Avalonia 12 for cooperative macOS resizing. Native AOT/trim safety is mandatory.

Packaging

This PR ships multiple native libraries. V8 and miniaudio are statically linked into the native engine; Dawn remains shared, and Windows composition also requires ANGLE EGL/GLES. macOS does not require ANGLE. Runtime data and host native dependencies remain packaged assets. Single-native-binary linking is deferred. Native packages include the pinned miniaudio license.

Final stabilization

Commits c65ab896 and 01f88e65 complete stabilization. 01f88e65 converts pan/sidebar/zoom diagnostic serialization to generated metadata after the AOT smoke test exposed reflection-only tracing; the full AOT interaction/resize rerun then passed. c65ab896 fixes the Windows package compiler failures: the browser bootstrap is assembled from bounded string parts to respect MSVC's literal limit, and worker resource paths are explicitly converted at the runtime boundary. The earlier cross-RID evidence failure resulted from the missing Windows build artifact. After compilation succeeded, Windows exposed a cold-start race in the resource single-flight integration test; its worker realms are now warmed before the simultaneous-load barrier, retaining the existing waiter/shared-byte assertions.

Local verification of the updated runtime:

• Native CTest: 18/18 passed.
• Required shared media/audio contracts: 9/9 passed; macOS video contract: 1/1 passed. These are project contracts, not full standards conformance.
• Packaged Frameforge Native AOT test passes using the bundle's own engine/dependencies/assets: nine cut positions, playback/pause, real RMS, capture-track lifetime and original waveform worker; maximum waveform error 1.49e-8. Actual MIME/HEAD/range/416 server checks pass; 30 rendered scenes.
• Kestrel original archive: continuous and four-size resize checks passed with zero uncaught JS errors using the updated runtime. The rebuilt reflection-free AOT probe also passed editing/undo/redo, application exercise, pan and sidebar validation. Evidence: docs/graphics/evidence/2026-09-10-merge-stabilization/.
• Live TradingView desktop startup reached chart readiness (approximately 3.26 seconds wall time). This is a startup regression check, not a panning or physical frame-rate measurement.
• Earlier managed verification: Avalonia 305 passed / 15 existing skips on each of net8/net10; Uno Release build passed.

Final-revision CI and package-consumer checks are required before merge. Keep draft while those checks are unresolved. Prior Windows hardware evidence is recorded in the existing integration reports; new Windows/Linux video hardware support is not claimed.

Deferred work and qualification limits

Linux GPU parity (#46), browser WebGL/fallback, Uno GPU qualification, full CTS/WPT conformance, exhaustive device-loss testing and single-binary packaging remain deferred. Windows/Linux media parity is #61/#62. Full Frameforge editor rendering still needs external image/video WebGPU APIs (#40), Canvas2D export (#34), MediaRecorder (#58) and remaining #53 work; physical/long-duration A/V qualification remains #59. Select popups remain #44.

Automated resize/input checks do not establish physical 60fps or identical transient Chrome resizing. The existing milestone document is docs/graphics/pr43-milestone-scope.md; media design, limits and reproducible commands are in experiments/WebScene.NativeEngine.Probe/native/media/README.md. Issues #54/#55 close the documented macOS-first media milestones, not complete browser conformance or the full editor.

[danwalmsley]

Implemented in PR #43, commit ea6b3aa, for the issue's macOS-first Frameforge media milestone.

Delivered

• Actual HTML audio/video lifecycle, markup/source selection, generation cancellation, async seek/play readiness, playback clock and retained native frame leases.
• macOS video is composed through WebScene's versioned IOSurface/GPU scene path, including ordinary DOM clipping and stacking. No native video window/overlay and no video pixel readback/upload in this path. GPU completion wakes retry pending frames after backpressure.
• Statically compiled miniaudio output/PCM decode; macOS container/AAC decoding; native mixing, gain automation, resampling, meters, real stereo capture track leases, independent clone/stop cursors and bounded queues.
• Original Frameforge MediaEngine and waveform worker run unchanged in a Native AOT host with reflection JSON disabled.
• Binary resource fixes cover Avalonia, Uno, archives, Response/Blob/object URLs. The sample server supports real MIME/HEAD/range/416 behavior.
• Normal native package builds enable media and include the pinned miniaudio license without adding a miniaudio shared library.

Verification at ea6b3aa

• Full native CTest suite: 18/18 passed, including real H.264/AAC decode, seek/cancellation, frame lifetime through consumer completion, backpressure, audio DSP/capture/rate/seek/queue tests.
• Required shared media/audio project contracts: 8/8 passed; required macOS video contract: 1/1 passed. These are explicitly project-owned contracts, not full standards conformance.
• Avalonia managed suite: 305 passed / 15 existing skips on each of net8.0 and net10.0. Uno Release build passed.
• verify-media.py against the Native AOT executable: passed. Nine original cut positions prepared; playback/pause/nonzero native meter and native track checks passed; original transferable waveform worker maximum error against independent PCM reduction 1.49e-8. Actual server MIME, HEAD, closed/open/suffix ranges and 416 checks passed. Native host rendered 32 scenes with no JS/runtime/resource errors in this run.
• Flash/click decoded-video versus actual mixed/recorded PCM timestamp difference: maximum 0.0417ms against 10ms tolerance. This excludes physical display/speaker latency; full device/long-duration qualification remains [Frameforge 6] Qualify unchanged AOT editor, 60 Hz interaction and distributable media runtime #59.
• Local NuGet inventory: engine/Dawn/ICU/snapshot payloads and exact miniaudio license present; no extra miniaudio dylib.

Scope and follow-up

Windows/Linux media/video hardware parity is explicitly tracked in #61 and #62, using the common service/graph/lease design. Those video backends still report unsupported, not a passing capability. This closes the macOS-first media-engine milestone specified here, not the complete Frameforge editor epic.

Full editor rendering still needs external image/video WebGPU APIs in #40, Canvas2D export in #34, and the remaining #53 issues. The real mixed audio track is now available to the recording adapter; MediaRecorder/encoding remains #58. Broader codecs, rotated/color-managed video, full HTML/Web Audio conformance and physical/long-duration A/V/performance qualification are not claimed by these subset tests.

Design, limits, exact commands and reproducible AOT harness:
https://github.com/wieslawsoltes/WebScene/blob/ea6b3aa9/experiments/WebScene.NativeEngine.Probe/native/media/README.md

Original application reference remains pinned at f414cd44896b659a5c6da2e4fc596a2ded055f2a:
https://github.com/wieslawsoltes/Frameforge

CI will validate the pushed revision separately; no cross-platform CI pass is claimed here.

[danwalmsley]

Implemented in PR #43, commits ea6b3aa and af02cc8, for the issue's macOS-first Frameforge media milestone.

Delivered

• Actual HTML audio/video lifecycle, markup/source selection, generation cancellation, async seek/play readiness, playback clock and retained native frame leases.
• macOS video is composed through WebScene's versioned IOSurface/GPU scene path, including ordinary DOM clipping and stacking. No native video window/overlay and no video pixel readback/upload in this path. GPU completion wakes retry pending frames after backpressure.
• Statically compiled miniaudio output/PCM decode; macOS container/AAC decoding; native mixing, gain automation, resampling, meters, real stereo capture track leases, independent clone/stop cursors and bounded queues.
• Original Frameforge MediaEngine and waveform worker run unchanged in a Native AOT host with reflection JSON disabled.
• Binary resource fixes cover Avalonia, Uno, archives, Response/Blob/object URLs. The sample server supports real MIME/HEAD/range/416 behavior.
• Normal native package builds enable media and include the pinned miniaudio license without adding a miniaudio shared library.

Verification at af02cc8

• Full native CTest suite: 18/18 passed, including real H.264/AAC decode, seek/cancellation, frame lifetime through consumer completion, backpressure, audio DSP/capture/rate/seek/queue tests.
• Required shared media/audio project contracts: 9/9 passed; required macOS video contract: 1/1 passed. These are explicitly project-owned contracts, not full standards conformance.
• Avalonia managed suite: 305 passed / 15 existing skips on each of net8.0 and net10.0. Uno Release build passed.
• verify-media.py against the Native AOT executable: passed. Nine original cut positions prepared; playback/pause/nonzero native meter and native track checks passed; original transferable waveform worker maximum error against independent PCM reduction 1.49e-8. Actual server MIME, HEAD, closed/open/suffix ranges and 416 checks passed. Native host rendered 30 scenes with no JS/runtime/resource errors in this run.
• Flash/click decoded-video versus actual mixed/recorded PCM timestamp difference: maximum 0.0417ms against 10ms tolerance. This excludes physical display/speaker latency; full device/long-duration qualification remains [Frameforge 6] Qualify unchanged AOT editor, 60 Hz interaction and distributable media runtime #59.
• Ubuntu 22.04 linux/amd64 container: native_document footprint 368 bytes (384-byte budget), native audio graph/capture and PCM decode/session tests passed. The first pushed revision hit the Linux footprint guard; af02cc8 fixes it with shared lazy auxiliary storage rather than increasing the budget.
• Local NuGet inventory: engine/Dawn/ICU/snapshot payloads and exact miniaudio license present; no extra miniaudio dylib.

Scope and follow-up

Windows/Linux media/video hardware parity is explicitly tracked in #61 and #62, using the common service/graph/lease design. Those video backends still report unsupported, not a passing capability. This closes the macOS-first media-engine milestone specified here, not the complete Frameforge editor epic.

Full editor rendering still needs external image/video WebGPU APIs in #40, Canvas2D export in #34, and the remaining #53 issues. The real mixed audio track is now available to the recording adapter; MediaRecorder/encoding remains #58. Broader codecs, rotated/color-managed video, full HTML/Web Audio conformance and physical/long-duration A/V/performance qualification are not claimed by these subset tests.

Design, limits, exact commands and reproducible AOT harness:
https://github.com/wieslawsoltes/WebScene/blob/af02cc83/experiments/WebScene.NativeEngine.Probe/native/media/README.md

Original application reference remains pinned at f414cd44896b659a5c6da2e4fc596a2ded055f2a:
https://github.com/wieslawsoltes/Frameforge

CI will validate the pushed revision separately; no cross-platform CI pass is claimed here.

[danwalmsley]

Merge stabilization is committed and pushed through 185694f.

• c65ab89 fixes both Windows compiler errors and adds the interactive video sample.
• 01f88e6 replaces reflection-only AOT pan/sidebar diagnostics with typed generated serialization; the combined original Kestrel edit/undo/redo, pan, sidebar and continuous/stepped resize run passes.
• 185694f warms resource-test realms before the load barrier and waits for the existing five-second memory telemetry sampling window. Three consecutive native-suite runs pass without dropping the waiter or mapped-memory assertions.
• Native suite 18/18, media/video contracts 10/10, packaged Frameforge AOT media/server checks, live TradingView startup and six graphics tooling tests pass locally. Actual packaged-player UI pause, backward seek and clip switching were also checked.
• The preceding Windows package run has passed its build/pack/test step and reached candidate discovery, confirming the compiler fixes. This is not a final-head CI pass.

The working tree is clean. PR scope/title/evidence are updated; there are no unresolved review threads. Keep draft until the latest revision's CI, native packages and package-consumer checks pass. No merge has been performed.

[danwalmsley]

The latest macOS package failure was repeated TradingView switch activation did not request host frames (demand=0); it was not a compiler/media-test failure. Windows native packaging on the same revision passed.

Input dispatch now imports the latest observed host timestamp immediately before handling the event. The worker's loop-entry timestamp can otherwise precede a newly dequeued event even when the host observed its compositor boundary before enqueueing the input. A transition can then start on the stale timeline and be advanced past completion at the next loop. The additional acquire load updates only the monotonic document clock: it does not release RAF, add frame production, or change the host-vsync architecture.

The original failure was not reproduced in 100 local pre-fix runs. The existing repeated-switch regression remains unchanged; post-fix repetition and the complete native suite are used for local validation. Final CI remains required to verify the affected runner.

Post-fix validation: 100/100 repeated-switch runs and full native CTest 18/18 passed. The fix is committed and pushed; final-revision CI is pending.