Skip to content

[Quantization] Fix ModelOpt pre-quantized loading #14188

Open
yzhautouskay wants to merge 1 commit into
huggingface:mainfrom
yzhautouskay:yzhautouskay/modelopt_prequant_fix
Open

[Quantization] Fix ModelOpt pre-quantized loading #14188
yzhautouskay wants to merge 1 commit into
huggingface:mainfrom
yzhautouskay:yzhautouskay/modelopt_prequant_fix

Conversation

@yzhautouskay

Copy link
Copy Markdown
Contributor

What does this PR do?

Fixes loading of pre-quantized ModelOpt models via from_pretrained(..., device_map=...),
which currently crashes with:

NotImplementedError: Cannot copy out of meta tensor; no data! ...

Note: this is required for the pre-quantized path to work at all with HF checkpointing enabled —
no combination of device_map / .to(device) load options works around it without this change.

Root cause

For pre-quantized models, NVIDIAModelOptQuantizer.check_if_quantized_param returned True
for every tensor. That routes the quantizer buffers ModelOpt recreates on restore
(_amax, _scale) through create_quantized_param, which assigns them into
module._parameters. Those buffers are then never materialized by the low-memory loader and
stay on the meta device, so the later device move fails.

Fix

  • check_if_quantized_param: for pre-quantized models, only claim real parameters
    (return tensor_name in module._parameters); buffers fall back to the standard loader, which
    materializes _amax/_scale on their target device.
  • create_quantized_param: guard the pre-quantized branch so a non-parameter reaching it
    raises clearly instead of silently mis-assigning.

TODO (follow-up)

Add a real regression test: enable enable_huggingface_checkpointing() before save+load (current tests don't do that), load
the round-tripped model with device_map, and assert no meta params/buffers remain (fails on
main, passes with this fix). Not included here.

Environment

Single ModelOpt version on both export and load (ModelOpt state is version-sensitive, so
producer and consumer must match):

  • diffusers 0.40.0.dev0 (this branch)
  • nvidia-modelopt 0.44.0 (quantize + load)
  • torch 2.9.0a0+145a3a7bda.nv25.10 (NVIDIA NGC container build)

Who can review?

<>

@github-actions github-actions Bot added quantization size/S PR with diff < 50 LOC labels Jul 14, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

quantization size/S PR with diff < 50 LOC

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant