Add reported reproducer config test suite#1173
Conversation
d4d40a8 to
5b29d4f
Compare
0a52553 to
b85110b
Compare
Signed-off-by: Anuj Gupta <anujgupt@qti.qualcomm.com>
b85110b to
a886801
Compare
Signed-off-by: Anuj Gupta <anujgupt@qti.qualcomm.com>
quic-hemagnih
left a comment
There was a problem hiding this comment.
Overall the framework looks very nice and informatime, few suggestions -
Can we add following in the scenario details -
- JIRA# - would be good for future reference
- PR# - would be good for furture reference
Currently we are categorizing issues based on stages but there are many issues which are reported let say due to
- Accuracy degradation,
- Memory issues (RAM increase, QPC bloat up) etc.
how are we planning to categorize these issue?
|
@quic-hemagnih thanks!. Regarding internal JIRA#, I intentionally didn't want to add those. We can have a JIRA to PR mapping in internal database. Issues related to accuracy, ONNX Export RAM bloat will require full model execution. By default, those tests are disabled but can be added. We can enable couple of envvars to avoid skipping those models. There is a skip_reason which can be used to set which models require full model and which don't |
Summary
tests/reproducer_configs/.RegressionScenarioentries.QEFF_REPRODUCER_SCENARIOfor targeted local runs.tests/reproducer_configs/README.mdwith the common run commands.Notes
Object of type PosixPath is not JSON serializable; QPCprogramqpc.binartifacts were still produced and validated.Validation
QEFF_REPRODUCER_SCENARIO=hf-transfer-import-order.18 passed, 34 skipped.185 passed, 3 skipped.