Implemented executorch's multi-optimization profile - #4441
Conversation
github-actions
Bot
added
component: tests
shoumikhin
left a comment
shoumikhin
left a comment
There was a problem hiding this comment.
I reviewed the multi-profile runtime, policy tests, and reference documentation. I left one runtime correctness concern and two nonblocking test/documentation suggestions inline.
| } | ||
| } | ||
| engine->profiles.active = profile; | ||
| ET_LOG(Info, "TensorRTBackend::execute: switched to optimization profile %d", profile); |
There was a problem hiding this comment.
Once setOptimizationProfileAsync(profile, stream) succeeds, several later failures can return before enqueueV3() and before the existing completion event is recorded. The profile switch may therefore remain in flight while the next execute() or destroy() reconfigures or destroys the same IExecutionContext. Could you add cleanup for every post-switch error path, either by synchronizing the stream or recording completion that the next execution and teardown will wait for? Delaying the profiles.active assignment alone would not address the context-lifetime race.
| int32_t selected = -1; | ||
|
|
||
| EXPECT_EQ(select_profile(table, ProfileRequest::kAuto, 0, prefill_input(), selected), ProfileSelection::kOk); | ||
| EXPECT_EQ(selected, 1); |
There was a problem hiding this comment.
Nonblocking test suggestion: could we add (1) a two-input table where profile 0 fits input 0 but not input 1, verifying auto-selection skips it, and (2) a three-profile table where the active profile no longer fits and two lower profiles do, verifying the rescan selects the lowest matching index? The implementation handles both today, but the current tests cover only one-input tables and a rescan with one fitting alternative.
| } | ||
| { | ||
| OptimizationProfileGuard profile_guard(kDecodeProfile); | ||
| auto result = module.forward(decode_inputs); |
There was a problem hiding this comment.
Nit: kPrefillProfile and kDecodeProfile are example-local constants and are not defined or exported by the public API, so this copied snippet does not compile as written. Could you define them in the snippet, or use profile indices 1 and 0 with a note that the indices follow the export-time profile order?
2 participants
Description
Please include a summary of the change and which issue is fixed. Please also include relevant motivation and context. List any dependencies that are required for this change.
Fixes # (issue)
Type of change
Please delete options that are not relevant and/or add your own.
Checklist: