fix test fakes drift and MoE subset hook-order flake - #3012
Merged
Conversation
Port of GPT-QModel-Ultra PR #224 to upstream, limited to what applies here: - test_calibration_data_device.py: StageInputsCapture.cache_inputs calls get_modules_with_direct_meta_tensors on the model, but the three FakeGPTQModel stubs predate that API, so the stage-capture tests failed with AttributeError on multi-GPU hosts. Add the classmethod stub returning an empty list. - test_subset.py: test_qwen3_5_moe_subset_early_stop_follows_module_tree_ execution_order asserts the last forward hook is mlp.experts.3.up_proj, but with num_experts_per_tok=2 the executed expert set comes from the random router weights, whose RNG state shifts with prior test files, so the test flakes. Route all experts (num_experts_per_tok=4) to make the executed set and hook order deterministic. The HRM module_path and get_input_embeddings/offload_to_disk fake updates from the ultra PR do not apply upstream (those APIs do not exist here).
Qubitium
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
1.
tests/test_calibration_data_device.py— 3 failingtest_stage_capture_*tests (deterministic)StageInputsCapture.cache_inputscallsget_modules_with_direct_meta_tensorson the model, but the threeFakeGPTQModelstubs predate that API →AttributeError: 'FakeGPTQModel' object has no attribute 'get_modules_with_direct_meta_tensors'. Added the classmethod stub (returns[]).2.
tests/module_tree/test_subset.py—test_qwen3_5_moe_subset_early_stop_follows_module_tree_execution_order(intermittent flake)The test asserts the last forward hook is
mlp.experts.3.up_proj, but withnum_experts_per_tok=2the executed expert set is chosen by the random router weights. Those weights depend on the RNG state at model construction, which shifts with how many earlier test files consumed randomness — so the test passes in isolation and fails after other suites. Routing all experts (num_experts_per_tok=4) makes the executed set and the asserted hook order fully deterministic (verified experts execute in module-tree index order for multiple seeds).Not ported (ultra-specific, do not exist upstream)
module_pathkwarg inshell_direct_meta_materialize/ turtle materialize fakes (upstream signature has nomodule_path)get_input_embeddings/get_input_embeddings_name/offload_to_diskfake additions (upstream capture path doesn't call them)Validation (Python 3.15.0rc1t free-threaded, torch 2.14 nightly cu130; single-GPU = upstream CI env)
tests/test_calibration_data_device.py(single GPU)tests/module_tree/test_subset.pyruff checkon both filesAll changes are test-only.
Pre-existing (unrelated, fails with and without this change):
tests/module_tree/test_moe_flag_parsing.py::test_get_moe_module_name_none_tree— left untouched.