Force fp32 fallback for CPU-assigned fp16 nodes with no matching kernel - #32197
Open
Arne H Juul (arnej27959) wants to merge 4 commits into
Open
Force fp32 fallback for CPU-assigned fp16 nodes with no matching kernel#32197Arne H Juul (arnej27959) wants to merge 4 commits into
Arne H Juul (arnej27959) wants to merge 4 commits into
Conversation
Level2+ fusion transformers assign a fused node the execution provider of the nodes it replaces without checking that a kernel exists for the fused op. For example, an fp16 Add and an fp16 Gelu both have CPU kernels, but fusing them produces a com.microsoft.BiasGelu node that the CPU EP only implements for float, so session initialization fails when its kernel is looked up. Detect these nodes (IsFp16NodeOnCpuWithoutKernel) and route them through the existing isolated-fp16-node fp32 fallback in InsertCastTransformer, regardless of whether they're otherwise "isolated" or produce a graph output, since running them in fp16 isn't an option to begin with. Track which nodes had their CPU assignment already recorded by the partitioner so the partition-assignment callback isn't fired twice. Also fixes two gaps in the isolated-node check: the no-fp16-input bailout wasn't skipped for these no-kernel nodes (unlike its output-side twin), and the kernel-less check only looked at input types, missing nodes whose fp16-ness is only on the output.
|
Azure Pipelines: There may be pipelines that require an authorized user to comment /azp run to run. |
Contributor
There was a problem hiding this comment.
Pull request overview
Addresses ARM64 FP16 model initialization failures by falling back to FP32 when CPU-assigned fused nodes lack matching FP16 kernels.
Changes:
- Detects kernel-less CPU FP16 nodes and adds regression tests.
- Updates Softsign ARM behavior and MLAS test naming.
- Includes unrelated Java, CMake, ONNX patch, and provider-wrapper changes.
Reviewed changes
Copilot reviewed 11 out of 11 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
onnxruntime/core/optimizer/insert_cast_transformer.cc |
Adds kernel-aware FP32 fallback. |
onnxruntime/test/framework/insert_cast_transformer_test.cc |
Tests fallback and FP16-kernel preservation. |
onnxruntime/core/providers/cpu/activation/activations.h |
Changes Softsign division implementation. |
onnxruntime/test/providers/cpu/activation/activation_op_test.cc |
Updates Softsign expectations. |
onnxruntime/test/mlas/unittest/test_sqnbitgemm_neon_fp16.cpp |
Renames SQNBit test class. |
onnxruntime/test/mlas/unittest/test_hqnbitgemm_neon.cpp |
Renames HQNBit test class. |
onnxruntime/core/providers/shared_library/provider_wrappedtypes.h |
Reorders wrapped type declarations. |
cmake/CMakeLists.txt |
Adds broad compiler-warning controls. |
cmake/patches/onnx/onnx.patch |
Suppresses ONNX warnings. |
cmake/onnxruntime_java.cmake |
Enables origin-relative JNI build RPATH. |
java/gradlew |
Redirects wrapper invocation to system Gradle. |
💡 Configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
Regression test for dd3a91c: builds a com.microsoft.BiasGelu node (fp16 in/out, only a float CPU kernel) that consumes only graph inputs and produces a graph output, matching the fusion scenario the fix targets, and verifies it still gets forced to fp32. A contrast test with Round (a real fp16 CPU kernel) confirms the same graph shape is left running in fp16 when a kernel actually exists. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Arne H Juul (arnej27959)
force-pushed
the
arnej/handle-missing-fp16-cpu-kernel
branch
from
August 21, 2026 15:01
4bbc4e9 to
b585de8
Compare
… is an output ForceSingleNodeCPUFloat16ToFloat32 clears the EP of a kernel-less CPU-assigned fp16 node so ApplyImpl's cast-insertion loop treats it as unassigned, but that loop only rewrites a node when NeedInsertCast finds an fp16 *input*. A node whose only fp16 value is an output (e.g. a generator op such as RandomNormal with dtype=float16, which has no inputs at all) was left with no EP, an unconverted dtype attribute, and no output cast. Track which nodes ForceSingleNodeCPUFloat16ToFloat32 forces to fp32 and make ApplyImpl run the CPU EP assignment and output/dtype fixups for them even when no input needs a cast. This closes the transformer-level gap only. A node is still forced to fp32 just when IsIsolatedFp16NodeOnCpu can confirm an fp32 CPU kernel exists for it; where that confirmation fails the node is left CPU-assigned in fp16, and session initialization still fails later on kernel lookup as before. That case now logs a warning naming the node and op type, since the eventual failure reports only the op type and is hard to trace back to a specific node. Two known causes are an op whose CPU kernel is registered for an older opset than the model uses, and one whose kernel declares fewer type-constraint names than its schema. Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Co-authored-by: Copilot Autofix powered by AI <175728472+Copilot@users.noreply.github.com>
Arne H Juul (arnej27959)
added a commit
to vespa-engine/vespa-3rdparty-deps
that referenced
this pull request
Aug 21, 2026
Update patches.fp16-cpu-kernel-fallback.diff to track the
PR we have submitted, see:
microsoft/onnxruntime#32197
Drop patches.no-fp16-gelu-fusions.diff (handled by above patch).
Also regenerate patches.use-gradle.diff with "git diff".
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
Level2+ fusion transformers assign a fused node the execution provider of the nodes it replaces without checking that a kernel exists for the fused op. For example, an fp16 Add and an fp16 Gelu both have CPU kernels, but fusing them produces a com.microsoft.BiasGelu node that the CPU EP only implements for float, so session initialization fails when its kernel is looked up.
Detect these nodes (IsFp16NodeOnCpuWithoutKernel) and route them through the existing isolated-fp16-node fp32 fallback in InsertCastTransformer, regardless of whether they're otherwise "isolated" or produce a graph output, since running them in fp16 isn't an option to begin with. Track which nodes had their CPU assignment already recorded by the partitioner so the partition-assignment callback isn't fired twice.
Also fixes two gaps in the isolated-node check: the no-fp16-input bailout wasn't skipped for these no-kernel nodes (unlike its output-side twin), and the kernel-less check only looked at input types, missing nodes whose fp16-ness is only on the output.
Motivation and Context
Proposed fix for this issue:
#32186
the unit test tried using "Abs" pretending it has a FP16 kernel, but the new check detects a missing kernel. Replacing with "Round" which actually has FP16 generally preserves the original intent of the check.