[GPU 18/20] Complete external image/video sources and GPU texture interoperability

[danwalmsley]

Parent epic: #22
Implementation order: 18 of 20.
Depends on: #39.

Outcome

Implement the image-source and external-texture requirements that Kestrel does not exercise, without weakening full-support claims.

Implementation

• Inventory every source type required by the pinned WebGPU/WebGL specifications and enabled browser interfaces: ImageBitmap, ImageData, HTML image/canvas, OffscreenCanvas, video and VideoFrame where required.
• Complete copyExternalImageToTexture and applicable WebGL texImage/texSubImage source overloads, with crop/flip/premultiplication/color-space/origin-clean semantics.
• Implement GPU external textures and their source refresh/expiry/lifetime rules. Add the minimal actual media/source infrastructure needed for the exposed source APIs; a fabricated video wrapper/no-op source is not acceptable.
• Use native shared allocations when source and destination permit them; separately account for required conversion or upload. Do not claim all source ingestion is zero-copy.
• Keep source decode/media lifetimes separate from JS wrappers and GPU completion. Correctly handle closed/detached/expired sources and device/context loss.
• Ensure browser security/taint behavior and source dimensions/format validation are performed before exposing pixel contents.
• Reuse the common canvas snapshot/image contract so source reads cannot observe a later unrelated frame.
• Document which optional source interfaces are enabled and tie support claims to applicable conformance; mandatory requirements cannot be excluded merely because infrastructure is missing.

Tests

• Official source-overload/external-texture CTS/WPT/WebGL cases.
• Deterministic image/video frames with crop, flip, alpha/color conversion, odd dimensions and changing frame content.
• Origin-clean violations, closed VideoFrame/ImageBitmap, detached source and expired external texture.
• Source destroyed while GPU reads are queued; repeated import/refresh/expiry; worker source paths where specified.
• Pixel/readback comparisons against browser reference and residency/copy-cost measurements.

Verification evidence

Attach a source-type coverage table, conformance results, pixel comparisons, source/texture lifetime traces and classified copy counters.

Acceptance

• Every source operation required for the final claimed profile is implemented and tested.
• Color/alpha/origin-clean/expiry behavior matches the pinned standard.
• Media/source objects remain alive exactly as required by queued GPU operations.
• Required conversions/uploads are measured, not hidden as zero-copy.
• Missing mandatory infrastructure blocks full-support qualification.

Completion requirements

Link the implementation PR, exact test commands, native/runtime/framework revisions and verification artifacts before closing. Follow the epic's architecture and copy/lifetime invariants. Any unavailable mandatory hardware or missing required functionality remains incomplete, not a pass.

Frameforge workload specification (epic #53)

This existing issue owns external source/texture implementation; do not duplicate it in a separate Frameforge GPU implementation. For the macOS Frameforge milestone, the required path depends on actual media decoding in #54 and existing canvas/worker infrastructure. It need not wait for every optional source type or WebGL branch in the broader #22 scope; those broader acceptance criteria remain open.

Reference: Frameforge repository, pinned source renderer.js and media.js. Baseline PR #43/ce56d49b initializes both WGSL pipelines but device.importExternalTexture is undefined. Each render task imports each active HTMLVideoElement, builds a bind group, and samples texture_external via textureSampleBaseClampToEdge. Missing imports are caught by the app and can silently skip a layer: tests must inspect output, not just absence of console errors.

Required implementation

• Implement GPUExternalTexture native ownership/binding and importExternalTexture for real HTMLVideoElement frames. Snapshot the selected decoded frame with source generation/timestamp and retain native plane/surface leases until queued GPU consumers finish. Implement specification refresh/expiry and reject invalid/expired/cross-device sources; do not bind a mutable latest-frame pointer.
• Route decoder CVPixelBuffer/IOSurface-backed frames through Dawn/Metal where supported. Handle plane formats, chroma subsampling, range/matrix conversion, display dimensions, orientation and crop. Document when GPU conversion/copy is necessary; shared decoder surfaces are preferred, but never claim universal zero-copy.
• Implement/qualify queue.copyExternalImageToTexture for the app's cached Canvas2D title textures and imported images, including source dimensions/origin/flip, destination bounds, alpha/color-space and origin-clean validation. Reuse canvas frame snapshots rather than rasterizing the entire UI or polling readback.
• Handle simultaneous clips during dissolve, source replacement/seek, frame reuse, source removal, device loss and destruction during pending GPU work. Device callbacks rejoin the existing owner-thread completion/wake path; do not invoke V8 on decode/GPU callback threads.
• Wire decoded-frame invalidation into existing vsync/mailbox presentation. Latest ready presentation can replace stale queued presentation, while resources already referenced by consumers remain immutable and leased.

Frameforge-specific tests and acceptance

• Run original tests/gpu-smoke.html (audit browser assumptions) plus CTS/project contracts for external texture expiry, bindings and source validation.
• Known changing video frames, two-source dissolve, seek, crop/rotation, title overlay and still-image pixels match browser reference within documented color tolerance.
• Decoder source destruction before GPU completion does not corrupt pixels or crash; frame generations survive delayed consumers and resize.
• Original demo viewer shows actual decoded video, not black/skipped layers or placeholder textures.
• Trace native surface/texture ownership and classify uploads, GPU copies, conversions and CPU readbacks. Normal preview must not read back decoded frames to CPU per frame.

Specification: WebGPU external textures. Chromium reference: GPU external texture implementation; pin any source revision used in implementation notes.

Cross-platform and airspace requirements

Use common services and lifecycle contracts across Windows, macOS and Linux. Prefer reusable cross-platform libraries; native OS service providers are acceptable where appropriate. No native child video window, AVPlayerLayer, HWND overlay or separate presentation surface: video must compose as a retained WebScene texture with normal HTML stacking, clipping, transforms, opacity and scrolling. Share frame ownership with WebGPU external textures and the existing vsync/mailbox/versioned-image architecture. GPU consumers retain immutable frame leases through completion, including resize and replacement.