Skip to content

[Media parity] Qualify Windows decoding, retained video composition and native audio #61

Description

@danwalmsley

Follow-up qualification for #54 and #55 under #53 and PR #43. Reference: https://github.com/wieslawsoltes/Frameforge at f414cd44896b659a5c6da2e4fc596a2ded055f2a. Do not change the application JavaScript/WGSL.

Implementation

Reuse native/media/media_session, audio_graph, audio_capture, browser bindings, resource-byte contracts and the existing versioned scene-image ABI. The common audio implementation statically links pinned miniaudio; qualify actual device output on Windows. Replace video_decode_unsupported.cpp with Media Foundation with D3D11/DXGI shared textures, or a reviewed portable FFmpeg provider. Document the choice, pinned versions/licenses, static-library packaging and system prerequisites. Avoid bringing both codec stacks without need.

Implement open_video/read/audio/duration with admitted bytes only, persistent demux/decoder state, timestamped immutable frame leases, cancellation, source replacement, rapid/end/backward seeks, memory limits and bounded decode/mailbox queues. Never fetch arbitrary URLs inside the codec. Preserve real AAC/container soundtrack decoding or report unsupported codecs honestly.

Use Windows's existing GPU image provider and completion/wake contract. No native child video windows or overlays. DOM video and WebGPU external sources must share the decoded lease, normal stacking/opacity/transforms/clipping/resize and vsync. No steady-state CPU GPU readback; retain underlying decoder allocations until every GPU consumer completes. Preserve pending latest-frame publication under backpressure; do not spin or substitute mutable/newer content for retained scenes.

Tests and acceptance

  • Run portable media decode and audio graph/capture CTests; run the shared media/audio browser contract. Add platform video profile equivalent to webscene-macos-video-runtime-profile.json.
  • Decode numbered-motion.mp4, flash-click.mp4 and the original Frameforge assets. Test corrupt/truncated/unsupported files, cancellation, replacement, loop/end/replay, many seeks and repeated close. Preserve exact bytes through Avalonia/Uno loaders.
  • Run Frameforge Native AOT --media-verify against original assets; all cut preparation, transport, nonzero meter and independent waveform comparison assertions must pass without swallowed errors. Keep the full-editor external-texture dependency tracked in [GPU 18/20] Complete external image/video sources and GPU texture interoperability #40/[Frameforge 6] Qualify unchanged AOT editor, 60 Hz interaction and distributable media runtime #59.
  • Record actual mixed stereo PCM, track clone/stop independence, overrun reporting and context close. Measure flash/click pipeline drift below 10ms and report physical output/display latency separately.
  • Visually verify video behind/in front of normal DOM, clipped rounded corners, opacity, scrolling, scaling and live resize. Stress delayed GPU completion and device loss.
  • Publish runtime NuGets, inspect the complete dependency/license inventory, consume from a clean Avalonia 11 app and Uno host, and run Native AOT with reflection JSON disabled. Samples may opt into Avalonia 12.
  • Attach exact revision/OS/GPU, commands, results and screenshots. Unsupported tests are not passes; do not advertise support before hardware qualification.

Current macOS implementation and reproducible commands: experiments/WebScene.NativeEngine.Probe/native/media/README.md. The macOS AOT verification harness is experiments/WebScene.Frameforge/media-verify.html. The original assets are obtained by cloning the pinned public Frameforge repository; do not rely on an agent-local Downloads path.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions