Skip to content

feat(native): add direct image pixel updates - #62

Open
monotykamary wants to merge 2 commits into
remorses:mainfrom
monotykamary:feat/dynamic-image-primitives
Open

feat(native): add direct image pixel updates#62
monotykamary wants to merge 2 commits into
remorses:mainfrom
monotykamary:feat/dynamic-image-primitives

Conversation

@monotykamary

@monotykamary monotykamary commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Summary

Add two desktop primitives for ordinary retained <img> nodes:

  • updateImage(elementId, width, height, bgra) stages copied, tightly packed BGRA pixels without encoding them into the mutation JSON or committing React per frame.
  • clearImage(elementId) restores the normal source; source changes, resize, and destruction release the previous override/atlas identity.

Same-sized updates preserve RenderImage::id and use GPUI's in-place atlas update. Normal GPUI image rendering still owns object-fit, clipping, radii, and layout. A root-frame snapshot keeps deferred virtual-list rows consistent; direct changes invalidate native measurements when needed.

Scope and dependency

Refs #40. Advance only the Zed submodule to merged remorses/zed#4 (81c99f816b4a5f69d3c014774068034c24d1d7af), reachable on remorses/zed's gpuix branch. No new GPUI source patch or toolchain change.

No terminal/PTY/VT code, CEF, new image element, shared-GPU-texture import, global image registry, public cached-frame presentation, or custom painter. Sampling stays linear: GPUI's ordinary Img has no nearest-sampling builder at this revision. That belongs in a separate GPUI change rather than a GPUix painting workaround.

Ownership and failure contract

The API uses an explicit pixel layout and copy-on-ingress boundary, informed by wgpu texture uploads and Flutter's pixel-buffer lifetime contract, without importing their broader texture APIs.

Dimensions are positive integers, capped at 4096 per axis / 64 MiB before copying. Invalid input preserves the previous image. Storage and staged/uploaded/failed versions belong to each renderer. GPU upload failures fall back to src (or the normal empty placeholder), do not block other images, and are not retried on unrelated paints; a newly submitted version can retry. Failed versions never reach Img's implicit upload path.

README, generated declarations, optional renderer methods, a changeset, and a tiny animated color-field example document the contract. Producers and their scheduling remain application code.

Validation

On macOS Apple Silicon, against the freshly rebuilt release addon:

  • Native library: 233 passed, including seven focused data/input and real GPU lifecycle/failure regressions.
  • React: 409 passed; package build passed.
  • Examples: 48 passed, including producer cleanup and no animated React commits.
  • Real background sample retained one image ID/bounds, changed captured frames, and closed normally.
  • Tests cover copied bytes, actual atlas retention, resize/drop, source precedence, upload bounds/failure/recovery, effect ordering, decoded pixels, and virtual-list intrinsic sizing.
  • git diff --check passed. New/changed Rust regions are formatted. Full cargo fmt --check reports four verified unchanged baseline regions (accessibility.rs, custom_elements/img.rs, custom_elements/input.rs, lib.rs); unrelated formatting is deliberately excluded.

The private real-GPU Rust tests use a dev-only, environment-gated subprocess adapter because libtest runs tests on worker threads while AppKit requires main-thread window creation. It is excluded from production builds. PNG decoding dependencies are test-only. Windows/Linux runtime behavior was not exercised locally.

Grok 4.6 reviewed the design, implementation, upload-safety fix, and test harness. Final production and test reviews: APPROVE.

Agent provenance

Harness: Pi coding agent with Fabric
Agent: Main, with isolated dynamic-image-pr / embedding-pr implementation agents
Model: openai-codex/gpt-6-astra
Design and code review: xai/grok-4.6

User prompts driving these primitive PRs (verbatim)
  1. Perfect. Recall what PRs we wanted to do once the zed PR we made was merged.

  2. Does GPUIx need them baked in? Or leave it up to the user? Our colleague wants to make sure we expose primitives and create small scoped PRs.

  3. Help me add (1) and (2) while considering carefully best practices of related primitives out in the wild (feel free to search the web and check). Make sure grok 4.6 reviews your work and code; it needs to be tight and small in scope and PR, follow code conventions and most importantly good code style. Create those PRs for me.

  4. gh watch the CI and resolve the issues.

  5. I've converted them to a draft. Open them back up once all fixed. Proceed.

CI follow-up

Commit 5375b6b fixes Windows subprocess file-URL conversion and samples pixels inside actual image bounds rather than relative to the entire native window. Additional larger-window/offset regressions retain the strict color checks. Targeted local tests: 76 passed; package build passed. Grok 4.6 approved the test-only changes. All platform builds, macOS/Windows test jobs, and the publish job passed in CI run 34314125957. Marked ready only after the exact updated head was green.

Stage owned BGRA overrides on existing img nodes using merged GPUI updates.
Preserve image identity and bound upload memory and failed-version retries.
Add lifecycle, atlas, pixel, React ordering and animated example regressions.
@monotykamary
monotykamary marked this pull request as draft September 9, 2026 05:14
Decode file URLs for Windows subprocess imports. Sample decoded screenshots
inside actual image bounds and cover larger windows with offset images.
@monotykamary
monotykamary marked this pull request as ready for review September 9, 2026 05:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant