Skip to content

Add reported reproducer config test suite#1173

Open
anujgupt-github wants to merge 2 commits into
mainfrom
qeff-reproducer-config-tests
Open

Add reported reproducer config test suite#1173
anujgupt-github wants to merge 2 commits into
mainfrom
qeff-reproducer-config-tests

Conversation

@anujgupt-github

@anujgupt-github anujgupt-github commented Jul 10, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Add a pre-merge reported reproducer config suite under tests/reproducer_configs/.
  • Capture reported import/download/export/compile/inference cases as explicit RegressionScenario entries.
  • Run the real QEfficient stage for active scenarios instead of placeholder option checks.
  • Use QEFF_REPRODUCER_SCENARIO for targeted local runs.
  • Add tests/reproducer_configs/README.md with the common run commands.

Notes

  • Compile-stage scenarios either run as reduced tests or stay gated behind official-model/full-model settings.
  • MDP scenarios generate local compiler dump/intersection artifacts instead of relying on reporter-machine paths.
  • Compile kwargs are checked before long compile jobs start.
  • The runtime report records the Hugging Face cache selected for the run.
  • The reduced run still emits an existing non-fatal qconfig warning: Object of type PosixPath is not JSON serializable; QPC programqpc.bin artifacts were still produced and validated.

Validation

  • Ran lint/format checks for the reproducer test file.
  • Checked full test collection for the reproducer suite.
  • Checked targeted collection with QEFF_REPRODUCER_SCENARIO=hf-transfer-import-order.
  • Ran catalog guard tests for scenario count, uniqueness, and active scenario coverage.
  • Earlier full reproducer-suite run passed with reduced scenarios: 18 passed, 34 skipped.
  • Earlier model quickcheck gate passed: 185 passed, 3 skipped.

@anujgupt-github anujgupt-github force-pushed the qeff-reproducer-config-tests branch 2 times, most recently from d4d40a8 to 5b29d4f Compare July 10, 2026 13:02
@anujgupt-github anujgupt-github force-pushed the qeff-reproducer-config-tests branch 2 times, most recently from 0a52553 to b85110b Compare July 10, 2026 17:01
Signed-off-by: Anuj Gupta <anujgupt@qti.qualcomm.com>
@anujgupt-github anujgupt-github force-pushed the qeff-reproducer-config-tests branch from b85110b to a886801 Compare July 11, 2026 05:40
@anujgupt-github anujgupt-github self-assigned this Jul 13, 2026
@anujgupt-github anujgupt-github added the enhancement New feature or request label Jul 13, 2026
Signed-off-by: Anuj Gupta <anujgupt@qti.qualcomm.com>

@quic-hemagnih quic-hemagnih left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Overall the framework looks very nice and informatime, few suggestions -
Can we add following in the scenario details -

  • JIRA# - would be good for future reference
  • PR# - would be good for furture reference

Currently we are categorizing issues based on stages but there are many issues which are reported let say due to

  • Accuracy degradation,
  • Memory issues (RAM increase, QPC bloat up) etc.

how are we planning to categorize these issue?

@anujgupt-github

anujgupt-github commented Jul 13, 2026

Copy link
Copy Markdown
Contributor Author

@quic-hemagnih thanks!.

Regarding internal JIRA#, I intentionally didn't want to add those. We can have a JIRA to PR mapping in internal database.

Issues related to accuracy, ONNX Export RAM bloat will require full model execution. By default, those tests are disabled but can be added.
Check the test scenario here
name="qwen3vl-moe-memory-after-prefill

We can enable couple of envvars to avoid skipping those models. There is a skip_reason which can be used to set which models require full model and which don't
See here: https://github.com/quic/efficient-transformers/pull/1173/changes#diff-ec050fc650d48b6106c7c5c19d99f2c4f5622bc51d0e531c9110c60ac728919bR961

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

enhancement New feature or request

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants