refactor(architectures): drive latent-space facts from the registry - #28
Draft
Pfannkuchensack wants to merge 2 commits into
Draft
refactor(architectures): drive latent-space facts from the registry#28Pfannkuchensack wants to merge 2 commits into
Pfannkuchensack wants to merge 2 commits into
Conversation
The preview chain was never really keyed on architecture. The branch comments in
the dispatch chain this removes said so by hand -- "Krea-2 decodes with the
Qwen-Image VAE", "Z-Image uses FLUX-compatible VAE", "Anima uses Wan 2.1 VAE".
Fifteen architectures share eight latent spaces, and the five most recent
introduced no new preview data at all.
That one fact was written in six partial, drifting copies. This replaces four of
them with LatentSpaceFacet; the remaining two are named below as follow-ups.
No behaviour changes. Every projection matrix moves byte-identically, comments
included -- read the move with `git diff --color-moved`.
Both Wan dispatches collapse into one. step_callback.py used to pick the factors
by `sample.shape[-3] == 48` and then, forty lines later, pick the spatial scale
by the same test. They are two halves of one fact and could drift; now resolving
one LatentSpace settles both. LatentSpaceFacet.resolve matches on channel count,
which is literally what the old comment said it was doing ("the latent channel
count uniquely identifies the variant"). An architecture with a single latent
space short-circuits without touching the sample at all, so an unusually shaped
tensor cannot start raising IndexError for the other fourteen.
Ideogram 4 keeps its own preview loop -- its callback signature is
step/total/packed_latents and it must unpatchify and denormalize first -- but it
now reads the same declaration instead of inlining the FLUX.2 factors and a
hardcoded x8. Its try/except fallback and its missing is_canceled check are left
alone; the latter is a real bug, in that Ideogram generations do not stop
promptly on cancel, but fixing it would change behaviour, which this PR does not.
max_unet_downscale was duplicated verbatim across denoise_latents and
T2IAdapterExt, comment and error string included. It is a UNet property, not VAE
geometry, so it gets its own optional facet; the accessor rather than require
carries the error, reproducing the old message exactly, enum repr included.
The three Wan 2.1 matrices -- QWEN_IMAGE_, ANIMA_ and WAN_LATENT_RGB_FACTORS --
were byte-identical, as were their three biases. They are now one WAN21_16,
asserted by object identity so the duplication cannot creep back.
Two known copies stay: constants.LATENT_SCALE_FACTOR, whose blast radius is
every latent node, and pid/decode.py, which is the same shape in different units
(packed latents at 128ch/16x, not VAE latents at 32ch/8x). Both are follow-ups
now that a correct source exists.
invocation_context.py is untouched. The spec proposed collapsing
flux_step_callback and flux2_step_callback into sd_step_callback, but they are
already thin wrappers over the same function, so the change would be pure docs
churn in docs/src/generated/invocation-context.json.
Tests. test_latent_space.py carries a reference table read off the old chain, one
row per architecture, plus first/last rows and column sums as exact fingerprints
-- IEEE-754 addition in fixed order, so platform-independent, and the rows catch
a reordering a sum is blind to. test_step_callback.py now covers
diffusion_step_callback end to end for all 15 bases and both Wan cases, which is
the first automated coverage the spatial-scale path has ever had.
The relocated projection tests hardcode their expected pixel. The versions they
replace recomputed it by summing the very matrix under test, so they would have
passed against a corrupted one -- and had in fact drifted: that test's docstring
claimed column sums of 0.3677/0.4577/0.9101 where the real ones are
0.3887/0.8771/1.3152.
The registry fixture now empties Facet.FACET_TYPES as well as the registry. The
first REQUIRED facet exposed that isolating only one of the two globals let
validate() hold dummy architectures to real facets.
denoise_latents.py keeps a third, divergent copy of the SDXL BGR rule: it reads
the UNet's base where the other paths read each adapter's. The two disagree for
an SD1 adapter on an SDXL UNet. Left as-is with a NOTE, since fixing it changes
behaviour; the natural home is a bgr_input field on UNetDownscaleFacet.
openapi.json, schema.ts and invocation-context.json are unchanged.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This was referenced Aug 2, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Second of the series from
.ideas/Backend Modularization Plan.md. No behaviour changes.Why
The preview chain was never really keyed on architecture — the branch comments in the dispatch chain
this removes said so by hand: "Krea-2 decodes with the Qwen-Image VAE", "Z-Image uses
FLUX-compatible VAE", "Anima uses Wan 2.1 VAE". Fifteen architectures share eight latent
spaces, and the five most recent introduced no new preview data at all.
That one fact lived in six partial, drifting copies. This replaces four of them.
Reading the diff
Every projection matrix moves byte-identically, comments included — use
git diff --color-moved.step_callback.pygoes from 396 lines to 60.The two Wan dispatches collapse into one
step_callback.pypicked the factors bysample.shape[-3] == 48, then forty lines later picked thespatial scale by the same test. Two halves of one fact, in two places, free to drift. Resolving one
LatentSpacenow settles both, because it carriesspatial_compression.LatentSpaceFacet.resolve()matches on channel count — literally what the old comment claimed it wasdoing ("the latent channel count uniquely identifies the variant"). An architecture with a single
latent space short-circuits without touching the sample at all, so an unusually shaped tensor
cannot start raising
IndexErrorfor the other fourteen.Deliberately not changed
(step, total, packed_latents)and it must unpatchify and denormalize first, so it cannot route through
diffusion_step_callbackwithout a fabricated
PipelineIntermediateState. It now reads the same declaration instead ofinlining the FLUX.2 factors and a hardcoded
× 8. Itstry/exceptfallback stays, and so does itsmissing
is_canceled()check — that is a real bug (Ideogram generations do not stop promptly oncancel), but fixing it would change behaviour. Follow-up.
invocation_context.pyis untouched. The spec proposed collapsingflux_step_callback/flux2_step_callbackintosd_step_callback, but they are already thin wrappers over the samefunction — the change would be pure docs churn in
docs/src/generated/invocation-context.json.constants.LATENT_SCALE_FACTOR(blast radius: every latent node) andpid/decode.py(sameshape, different units — packed latents at 128ch/16×, not VAE latents at 32ch/8×). Both follow-ups
now that a correct source exists.
denoise_latents.pykeeps a third, divergent copy of the SDXL BGR rule. It reads the UNet'sbase where the other paths read each adapter's; the two disagree for an SD1 adapter on an SDXL
UNet. Flagged with a
NOTErather than fixed — the natural home is abgr_inputfield onUNetDownscaleFacet, together with untangling the two parallel T2I-Adapter paths.max_unet_downscaleDuplicated verbatim across
denoise_latentsandT2IAdapterExt, comment and error string included.It is a UNet property, not VAE geometry, so it gets its own optional facet rather than a field on
LatentSpaceFacet. The accessor — notrequire()— carries the error, reproducing the old messageexactly, including that
BaseModelTypeis astr, Enummixin and so renders asBaseModelType.Fluxrather than
flux.The three identical Wan 2.1 matrices
QWEN_IMAGE_,ANIMA_andWAN_LATENT_RGB_FACTORSwere byte-identical (verified programmatically),as were their three biases. Now one
WAN21_16, asserted by object identity so the duplicationcannot creep back.
Two things worth a reviewer's attention
The relocated projection tests now hardcode their expected pixel. The versions they replace
computed the expectation by summing the very matrix under test — a tautology that would have passed
against a corrupted matrix. It had already drifted: that test's docstring claimed column sums of
0.3677 / 0.4577 / 0.9101where the real ones are0.3887 / 0.8771 / 1.3152.The registry fixture now empties
Facet.FACET_TYPEStoo. The firstREQUIREDfacet exposed thatisolating only one of the two globals let
validate()hold the dummy architectures to real facets.Tests
test_latent_space.py— a reference table read off the oldelifchain, one row per architecture,plus first/last rows and column sums as exact fingerprints (IEEE-754 addition in fixed order, so
platform-independent; the rows catch a reordering a sum is blind to). Plus the closed-set check and
the relocated projection tests.
test_step_callback.py—diffusion_step_callbackend to end for all 15 bases and both Wan cases.The first automated coverage the spatial-scale path has ever had.
test_unet_downscale.py— both values, and the verbatim error message for the other 13.Verification
pytest tests/backend/architectures tests/app/util tests/test_imports.py— 218 passedmypy invokeai/backend/architectures(strict) — cleanruff@0.11.2 check .+format --check— cleanopenapi.jsonregenerated and compared normalized — identical. No Pydantic class, invocationfield or enum is touched, so
schema.tsandinvocation-context.jsonare unchanged too.Not done here, and worth doing before merge: a manual generation on SD1, FLUX and Wan TI2V-5B to
see previews render at the right dimensions on hardware. The reference table covers the data; it
cannot cover the screen.
🤖 Generated with Claude Code