Newer/bigger tokenizer model possibilities

[MrDrMcCoy]

The Qwen3 and VL 4B models have been doing an admirable job, but in LLM years are practically ancient. Stable-diffusion.cpp doesn't appear to support any Qwen3.5+ models, nor am I able to get Qwen3 8B models to work. Is there anything intrinsic to Krea2, Zimage, etc that ties them to Qwen3 4B? If not, what would it take to get them working with any of the newer and/or bigger models?

[stduhpf]

Is there anything intrinsic to Krea2, Zimage, etc that ties them to Qwen3 4B?

Yes. The diffusion models are trained to expect text embeddings generated by one model, embeddings generated by any other model will most likely not even have the same shape, and would be uncorrelated to Qwen3 4B.

What could work is using a distilled model that is trained to match the embeddings of Qwen3, or using a small adapter model that transforms the embeddings from another text encoder into embeddings that the diffusion model can understand, or continue pretraining the diffusion model with the new text encoder.

[MrDrMcCoy]

I see, thanks. Out of curiosity though, why do larger models of the same architecture and generation also give unusable results? I would have thought Qwen3-VL-8B would work as well as the 4B.

I'll see about trying my hand at doing some post-training. I have the hardware, in theory... (shakes fist at AMD)

[stduhpf]

Because even though they are from the ssame familly with a similar architecture, they aren't trained to generate matching embeddings. In the case of Qwen3, the 4B and 8B models don't even have the same embedding dimension (2560 vs 4096), so there's no way they would match.

You could try projecting down the Qwen3 8B embeddings into the smaller 2560 dim space, maybe even a linear projection would be enough to roughly match the embeddngs from the 4B, but I don't believe it would actually imporve the performance of the diffusion model, projecting down would erase the extra information from the embeddings anyways... It often makes more sense to do it the other way around, projecting up the embeddings of the smaller model to match the bigger model, to save on time and memry requirements, at the expanse of prompt adherence.

[MrDrMcCoy]

I appreciate the explanation, thanks!