Read rope_theta from rope_parameters across Inference V2 - #8345
Open
alanhuangyoo wants to merge 1 commit into
Open
Read rope_theta from rope_parameters across Inference V2#8345alanhuangyoo wants to merge 1 commit into
alanhuangyoo wants to merge 1 commit into
Conversation
transformers 5.0 folded the rotary settings into config.rope_parameters
and dropped the rope_theta attribute. Eight V2 models still read the
attribute:
llama_v2, mistral, mixtral, phi, phi3, qwen_v2, qwen_v2_moe
self._config.rope_theta -> AttributeError on 5.x
exaone4
getattr(self._config, "rope_theta", 1000000.0)
-> no error, but 1e6 instead of the 1e4 the config carries
exaone4 is the worse of the two: a 100x rotary base is a silent
numerical error, not a startup failure.
exaone4_5 already reads both spellings, added with the model in deepspeedai#8121.
Hoist the same lookup onto DSTransformerModelBase so every model that
reaches it through the base gets it, and raise instead of guessing a
base when neither spelling carries one.
Same drift as deepspeedai#8341, which covers the v1 kernel-injection policy.
Signed-off-by: alanhuangyoo <alanhuangyoo@gmail.com>
alanhuangyoo
requested review from
hwchen2017,
loadams,
tjruwase and
tohtana
as code owners
August 28, 2026 13:56
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #8341, which covers the same drift in the v1 kernel-injection policy. Separate engine, separate change; #8341 noted this exposure but did not touch it.
What breaks
transformers 5.0 folded the rotary settings into
config.rope_parametersand dropped therope_thetaattribute. Eight V2 models still read the attribute.Seven of them read it directly and raise;
exaone4reads it through agetattrdefault and quietly uses that default instead:llama_v2AttributeError: 'LlamaConfig' object has no attribute 'rope_theta'mistralAttributeError: 'MistralConfig' …mixtralAttributeError: 'MixtralConfig' …phiAttributeError: 'PhiConfig' …phi3AttributeError: 'Phi3Config' …qwen_v2AttributeError: 'Qwen2Config' …qwen_v2_moeAttributeError: 'Qwen2MoeConfig' …exaone4exaone4is the worse of the two.Exaone4Configcarriesrope_parameters['rope_theta'] == 10000.0, so thegetattrdefault puts a 100x rotary base intoRotateHalfConfigwith nothing raised — wrong frequencies rather than a startup failure.requirements-inf.txtasks fortransformers>=4.32.1with no upper bound, so 5.x is in range.The fix
exaone4_5already reads both spellings — added with the model in #8121:Hoist the same lookup onto
DSTransformerModelBaseas arope_thetaproperty.DSMoETransformerModelBaseextends it, so all eight models are covered by one place and each call site becomestheta_base=self.rope_theta.The attribute is tried first, so pre-5.0 installs take exactly the path they take today. When neither spelling carries a base it raises instead of guessing one — that is the only behaviour change beyond the fix, and it replaces
exaone4's silent 1e6.exaone4_5is left alone; its own helper also handles the nested per-layersliding_attentiondict, which is specific to that model.Test
tests/unit/inference/v2/model_implementations/test_rope_theta.py— the three layouts, the precedence between them, the raise, plus a parametrization over the eight real configs from the installed transformers:I exercised the property and the configs, not a full V2 engine run against downloaded checkpoints.