Goal
Extend MatrixResultOracle in LinAlgTests.cpp beyond bit-exact comparison, so operations whose results the specification does not pin down uniquely can be verified without either weakening the test or enumerating an impractical number of candidate matrices.
Why
The oracle's three modes — Exact, PermittedResults and Excluded — all compare encoded component bits exactly. PermittedResults expresses "the spec permits this result or that result" as an enumerated set of complete candidate matrices; it is not a tolerance band.
That is correct and deliberate for the cases covered today, where every value is exactly representable in its component type and bit-exactness is both stronger than a tolerance and reproducible across implementations. It does not extend to operations where the specification leaves a range: matrix multiply accumulation order over a large K, conversion rounding, and results produced through the optimal layouts. Enumerating candidates there is not viable.
The harness already has the vocabulary for this. ValidationType::Epsilon and ValidationType::Ulp in HlslTestDataTypes.h are used by the pre-existing verifyFloatBuffer and verifyHalfBuffer paths in this same file. The oracle should adopt those rather than introduce a fourth comparison vocabulary.
Required work
- Add tolerance-based and ULP-based comparison to
MatrixResultOracle, reusing ValidationType and the existing doValuesMatch overloads.
- Keep
Exact the default; a case must opt in to a looser comparison and state why.
- Require that any tolerance is derived from the specification or from an independent error analysis, and record that derivation at the call site.
- Report the worst observed deviation on failure, not just the first mismatch.
Acceptance criteria
- At least one operation whose result the spec does not uniquely determine is covered with a justified tolerance.
- No existing case is relaxed from
Exact as part of this work.
- Tolerances are traceable to a public specification statement or a written derivation; none is fitted to an implementation's observed output.
Blocked on
Nothing, but it should land with the first case that genuinely needs it so the tolerance is chosen against a real result rather than guessed.
Public references
Out of scope
- Relaxing any currently exact expectation.
- Templating the oracle, and the checked-arithmetic redesign (tracked separately).
Assisted-by: GitHub Copilot
Goal
Extend
MatrixResultOracleinLinAlgTests.cppbeyond bit-exact comparison, so operations whose results the specification does not pin down uniquely can be verified without either weakening the test or enumerating an impractical number of candidate matrices.Why
The oracle's three modes —
Exact,PermittedResultsandExcluded— all compare encoded component bits exactly.PermittedResultsexpresses "the spec permits this result or that result" as an enumerated set of complete candidate matrices; it is not a tolerance band.That is correct and deliberate for the cases covered today, where every value is exactly representable in its component type and bit-exactness is both stronger than a tolerance and reproducible across implementations. It does not extend to operations where the specification leaves a range: matrix multiply accumulation order over a large K, conversion rounding, and results produced through the optimal layouts. Enumerating candidates there is not viable.
The harness already has the vocabulary for this.
ValidationType::EpsilonandValidationType::UlpinHlslTestDataTypes.hare used by the pre-existingverifyFloatBufferandverifyHalfBufferpaths in this same file. The oracle should adopt those rather than introduce a fourth comparison vocabulary.Required work
MatrixResultOracle, reusingValidationTypeand the existingdoValuesMatchoverloads.Exactthe default; a case must opt in to a looser comparison and state why.Acceptance criteria
Exactas part of this work.Blocked on
Nothing, but it should land with the first case that genuinely needs it so the tolerance is chosen against a real result rather than guessed.
Public references
hlsl-specs), Linear Algebra matrix operations.Out of scope
Assisted-by: GitHub Copilot