Skip to content

Read rope_theta from rope_parameters across Inference V2 - #8345

Open
alanhuangyoo wants to merge 1 commit into
deepspeedai:masterfrom
alanhuangyoo:fix/v2-rope-theta-from-rope-parameters
Open

Read rope_theta from rope_parameters across Inference V2#8345
alanhuangyoo wants to merge 1 commit into
deepspeedai:masterfrom
alanhuangyoo:fix/v2-rope-theta-from-rope-parameters

Conversation

@alanhuangyoo

Copy link
Copy Markdown
Contributor

Follow-up to #8341, which covers the same drift in the v1 kernel-injection policy. Separate engine, separate change; #8341 noted this exposure but did not touch it.

What breaks

transformers 5.0 folded the rotary settings into config.rope_parameters and dropped the rope_theta attribute. Eight V2 models still read the attribute.

Seven of them read it directly and raise; exaone4 reads it through a getattr default and quietly uses that default instead:

V2 model on master (transformers 5.8.0) with this PR
llama_v2 AttributeError: 'LlamaConfig' object has no attribute 'rope_theta' 10000.0
mistral AttributeError: 'MistralConfig' … 10000.0
mixtral AttributeError: 'MixtralConfig' … 1000000.0
phi AttributeError: 'PhiConfig' … 10000.0
phi3 AttributeError: 'Phi3Config' … 10000.0
qwen_v2 AttributeError: 'Qwen2Config' … 10000.0
qwen_v2_moe AttributeError: 'Qwen2MoeConfig' … 10000.0
exaone4 silently 1000000.0 10000.0

exaone4 is the worse of the two. Exaone4Config carries rope_parameters['rope_theta'] == 10000.0, so the getattr default puts a 100x rotary base into RotateHalfConfig with nothing raised — wrong frequencies rather than a startup failure.

requirements-inf.txt asks for transformers>=4.32.1 with no upper bound, so 5.x is in range.

The fix

exaone4_5 already reads both spellings — added with the model in #8121:

theta = rope_parameters.get("rope_theta", getattr(config, "rope_theta", None))

Hoist the same lookup onto DSTransformerModelBase as a rope_theta property. DSMoETransformerModelBase extends it, so all eight models are covered by one place and each call site becomes theta_base=self.rope_theta.

The attribute is tried first, so pre-5.0 installs take exactly the path they take today. When neither spelling carries a base it raises instead of guessing one — that is the only behaviour change beyond the fix, and it replaces exaone4's silent 1e6.

exaone4_5 is left alone; its own helper also handles the nested per-layer sliding_attention dict, which is specific to that model.

Test

tests/unit/inference/v2/model_implementations/test_rope_theta.py — the three layouts, the precedence between them, the raise, plus a parametrization over the eight real configs from the installed transformers:

13 passed
tests/unit/inference/v2/model_implementations/   19 passed
yapf --diff / flake8                             clean

I exercised the property and the configs, not a full V2 engine run against downloaded checkpoints.

transformers 5.0 folded the rotary settings into config.rope_parameters
and dropped the rope_theta attribute. Eight V2 models still read the
attribute:

  llama_v2, mistral, mixtral, phi, phi3, qwen_v2, qwen_v2_moe
      self._config.rope_theta       -> AttributeError on 5.x
  exaone4
      getattr(self._config, "rope_theta", 1000000.0)
      -> no error, but 1e6 instead of the 1e4 the config carries

exaone4 is the worse of the two: a 100x rotary base is a silent
numerical error, not a startup failure.

exaone4_5 already reads both spellings, added with the model in deepspeedai#8121.
Hoist the same lookup onto DSTransformerModelBase so every model that
reaches it through the base gets it, and raise instead of guessing a
base when neither spelling carries one.

Same drift as deepspeedai#8341, which covers the v1 kernel-injection policy.

Signed-off-by: alanhuangyoo <alanhuangyoo@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant