feat(native): add direct image pixel updates - #62
Open
monotykamary wants to merge 2 commits into
Open
Conversation
Stage owned BGRA overrides on existing img nodes using merged GPUI updates. Preserve image identity and bound upload memory and failed-version retries. Add lifecycle, atlas, pixel, React ordering and animated example regressions.
monotykamary
marked this pull request as draft
September 9, 2026 05:14
Decode file URLs for Windows subprocess imports. Sample decoded screenshots inside actual image bounds and cover larger windows with offset images.
monotykamary
marked this pull request as ready for review
September 9, 2026 05:24
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Add two desktop primitives for ordinary retained
<img>nodes:updateImage(elementId, width, height, bgra)stages copied, tightly packed BGRA pixels without encoding them into the mutation JSON or committing React per frame.clearImage(elementId)restores the normal source; source changes, resize, and destruction release the previous override/atlas identity.Same-sized updates preserve
RenderImage::idand use GPUI's in-place atlas update. Normal GPUI image rendering still owns object-fit, clipping, radii, and layout. A root-frame snapshot keeps deferred virtual-list rows consistent; direct changes invalidate native measurements when needed.Scope and dependency
Refs #40. Advance only the Zed submodule to merged remorses/zed#4 (
81c99f816b4a5f69d3c014774068034c24d1d7af), reachable onremorses/zed'sgpuixbranch. No new GPUI source patch or toolchain change.No terminal/PTY/VT code, CEF, new image element, shared-GPU-texture import, global image registry, public cached-frame presentation, or custom painter. Sampling stays linear: GPUI's ordinary
Imghas no nearest-sampling builder at this revision. That belongs in a separate GPUI change rather than a GPUix painting workaround.Ownership and failure contract
The API uses an explicit pixel layout and copy-on-ingress boundary, informed by wgpu texture uploads and Flutter's pixel-buffer lifetime contract, without importing their broader texture APIs.
Dimensions are positive integers, capped at 4096 per axis / 64 MiB before copying. Invalid input preserves the previous image. Storage and staged/uploaded/failed versions belong to each renderer. GPU upload failures fall back to
src(or the normal empty placeholder), do not block other images, and are not retried on unrelated paints; a newly submitted version can retry. Failed versions never reachImg's implicit upload path.README, generated declarations, optional renderer methods, a changeset, and a tiny animated color-field example document the contract. Producers and their scheduling remain application code.
Validation
On macOS Apple Silicon, against the freshly rebuilt release addon:
git diff --checkpassed. New/changed Rust regions are formatted. Fullcargo fmt --checkreports four verified unchanged baseline regions (accessibility.rs,custom_elements/img.rs,custom_elements/input.rs,lib.rs); unrelated formatting is deliberately excluded.The private real-GPU Rust tests use a dev-only, environment-gated subprocess adapter because libtest runs tests on worker threads while AppKit requires main-thread window creation. It is excluded from production builds. PNG decoding dependencies are test-only. Windows/Linux runtime behavior was not exercised locally.
Grok 4.6 reviewed the design, implementation, upload-safety fix, and test harness. Final production and test reviews: APPROVE.
Agent provenance
Harness: Pi coding agent with Fabric
Agent: Main, with isolated
dynamic-image-pr/embedding-primplementation agentsModel:
openai-codex/gpt-6-astraDesign and code review:
xai/grok-4.6User prompts driving these primitive PRs (verbatim)
Perfect. Recall what PRs we wanted to do once the zed PR we made was merged.
Does GPUIx need them baked in? Or leave it up to the user? Our colleague wants to make sure we expose primitives and create small scoped PRs.
Help me add (1) and (2) while considering carefully best practices of related primitives out in the wild (feel free to search the web and check). Make sure grok 4.6 reviews your work and code; it needs to be tight and small in scope and PR, follow code conventions and most importantly good code style. Create those PRs for me.
gh watch the CI and resolve the issues.
I've converted them to a draft. Open them back up once all fixed. Proceed.
CI follow-up
Commit
5375b6bfixes Windows subprocess file-URL conversion and samples pixels inside actual image bounds rather than relative to the entire native window. Additional larger-window/offset regressions retain the strict color checks. Targeted local tests: 76 passed; package build passed. Grok 4.6 approved the test-only changes. All platform builds, macOS/Windows test jobs, and the publish job passed in CI run 34314125957. Marked ready only after the exact updated head was green.