Support converting MINUS_MULTISET and INTERSECTION_PRIMARY SetOps to Calcite

[nielspardon]

Background

#303 (implemented in #998) removed the incorrect direct SetOp → Calcite relation mappings for MINUS_MULTISET and INTERSECTION_PRIMARY. Converting a Substrait plan that uses either op to Calcite now throws UnsupportedOperationException, because neither has a direct single-relation Calcite equivalent and the previous mappings were the wrong operation entirely.

This issue tracks actually supporting the conversion, since both ops can be expressed as a composition of Calcite relations rather than a single Minus/Intersect.

Spec semantics

From the set operation types table (primary input p, secondary inputs s1…sn):

• INTERSECTION_PRIMARY: primary rows present in any secondary input, deduplicated.
  Example: p:{1,2,2,3,3,3,4}, s1:{1,2,3,5}, s2:{2,3,6} → {1,2,3}.
• MINUS_MULTISET: primary rows excluding those present in all secondary inputs.
  Example: p:{1,2,3,4}, s1:{1,2}, s2:{1,2,3} → {3,4}.

Both are documented as having no direct SQL mapping.

Proposed composed mappings

• INTERSECTION_PRIMARY = Intersect(all=false)( primary, Union(all=true)(s1…sn) ) — unambiguous; verifiable against the example above. This one could be implemented on its own.
• MINUS_MULTISET = Minus( primary, Intersect(s1…sn) ) — blocked: the spec gives no duplicate/multiplicity rule for MINUS_MULTISET (unlike MINUS_PRIMARY_ALL's explicit max(0, m − Σn) formula), so the all flags on the composed relations can't be chosen faithfully. See the companion spec issue: Clarify multiplicity/duplicate semantics of SET_OP_MINUS_MULTISET substrait#1123.

Scope

• Implement the composed conversion in SubstraitRelNodeConverter (getRelBuilder / visit(Set, …)).
• Add behavior tests (extend SubstraitRelNodeConverterTest); the minusMultisetUnsupported / intersectionPrimaryUnsupported tests added in feat(isthmus)!: remove invalid SetOp to Calcite relation mappings #998 would be replaced.
• Update the exclusions/comment in SetUtils.setTestConfig.

Suggest starting with INTERSECTION_PRIMARY and deferring MINUS_MULTISET until the spec clarifies its duplicate semantics.

🤖 Generated with AI