fix(kimi_k25): honour the quantization flag - #3633
Conversation
…ig in sub-configs KimiK25VLStateDictAdapter.convert_single_tensor_to_hf read its `quantization` kwarg and then unconditionally overwrote it with True. The checkpointer passes the flag deliberately (True only for the base-checkpoint load when checkpoint.dequantize_base_checkpoint is set, False for saves) so the override made the INT4 layout the only loadable K2.5 checkpoint and re-packed experts to INT4 on every consolidated save (while the saved config.json has its quantization_config stripped). The override masks a second problem: dequantize_base_checkpoint is auto-resolved from hasattr(model.config, "quantization_config"), but composite models keep it on the language sub-config (K2.5: text_config), so the auto-detect returned False for exactly the checkpoint that needs it. Resolve it through the sub-configs as well. Net effect: INT4 K2.5 still loads with no config change (auto-detected), BF16 K2.5 checkpoints become loadable, and saves honour the flag like every other adapter (like Kimi-K3). Co-authored-by: Claude Fable 5 <noreply@anthropic.com> Signed-off-by: Aarni Koskela <akx@iki.fi>
|
/claude review |
|
LGTM |
|
Removing the hardcoded if getattr(self.text_config, "quantization_config", None) is not None:
self.quantization_config = self.text_config.quantization_configCan we fix the Kimi config instead of changing common infrastructure? Also, BF16 export must remove the nested |
svcnvidia-nemo-ci
added
the
waiting-on-customer
I'm all ears for a better fix!
Unfortunately I don't currently have access to infrastructure to test this with right now (this was found as a side-effect of LoRA training on pre-emptible 2x 8xB200 nodes). I also found out the hard way that vLLM doesn't support LoRAs on Kimi K2.5 yet. 😅 |
svcnvidia-nemo-ci
removed
the
waiting-on-customer
|
What about fixing it entirely in elif isinstance(text_config, dict):
+ text_config = text_config.copy()
+ quantization_config = text_config.pop("quantization_config", None)
+ if quantization_config is not None:
+ kwargs.setdefault("quantization_config", quantization_config)
text_config = DeepseekV3Config(**text_config) |
svcnvidia-nemo-ci
added
the
waiting-on-customer
3 participants
What does this PR do ?
Removes a hardcoded
quantization = Truein the Kimi-K2.5 state-dict adapter that ignored the checkpointer's flag, and fixes thedequantize_base_checkpointauto-detection it was compensating for (composite models keepquantization_configontext_config).Changelog
kimi_k25_vl/state_dict_adapter.py: drop the override;convert_single_tensor_to_hfnow honours
quantization=like the other adapters (Kimi-K3 is the in-tree reference)._transformers/infrastructure.py:config_has_quantization()checks the top-level config and its sub-configs (sub_configs,text_config,language_config,llm_config)Before your PR is "Ready for review"
Pre checks:
Additional Information
Missing key ... weight_packed); reproduced on 8e6f854 and d800383.quantization = Trueline that has been there ever since K2.5 support was introduced in feat: add support for kimi K2.5 VL #1132. cc @HuiyingLi :)