Does low bit quantization affect convergence?

[micklexqg]

I tried training a diffusion model from scratch using 8 bit quantization, its construction is from a pretrained one but it did not converge, so does low bit quantization affect convergence?

[gyanu2507]

Yes. 8-bit (and 4-bit) weights are a fine way to fine-tune a model that already converged in higher precision. They are a bad way to train from scratch.

The 8-bit Linear used for LLM.int8() / QLoRA keeps a higher-precision copy for the update (or updates LoRA in bf16/fp16 while the base stays quantized). A diffusion UNet trained from random init with the frozen 8-bit weight path has quantization noise in every residual, and the optimizer is not seeing a faithful gradient of the loss you think you are minimizing. People who "train 8-bit from scratch" and get a flat loss are hitting that, not a mysterious bitsandbytes bug.

If the UNet came from a pretrained fp32/fp16 checkpoint, load it in fp16/bf16 (or LoRA on top of 8-bit base weights) and train the adapters. Keep the 8-bit flag for inferencing a finished model, not for the from-scratch loop.