starts_with, ends_with and match_substring have no Substrait mapping, so expressions using them cannot be serialized.
import pyarrow as pa
import pyarrow.compute as pc
from pyarrow.substrait import serialize_expressions
schema = pa.schema([pa.field("cat", pa.string())])
serialize_expressions([pc.starts_with(pc.field("cat"), "al")], ["f"], schema)pyarrow.lib.ArrowNotImplementedError: No conversion function exists to convert
the Arrow function starts_with to a Substrait call
Same for pc.ends_with and pc.match_substring. Comparisons, isin and arithmetic all serialize fine, so this looks like a per-function gap rather than a structural one.
Substrait defines starts_with, ends_with and contains in functions_string.yaml, which cover all three.
The signatures differ: the Substrait functions take the pattern as a second argument, while the Arrow kernels are unary and carry it in MatchSubstringOptions. So the mapping needs the second argument to be a string literal. Substrait's case_sensitivity option maps onto ignore_case, except for CASE_INSENSITIVE_ASCII which has no Arrow equivalent.
This matters for engines that ingest a PyArrow filter through Substrait, where an unmappable function turns into a hard failure rather than a fallback.
Component(s)
C++, Python
starts_with,ends_withandmatch_substringhave no Substrait mapping, so expressions using them cannot be serialized.Same for
pc.ends_withandpc.match_substring. Comparisons,isinand arithmetic all serialize fine, so this looks like a per-function gap rather than a structural one.Substrait defines
starts_with,ends_withandcontainsinfunctions_string.yaml, which cover all three.The signatures differ: the Substrait functions take the pattern as a second argument, while the Arrow kernels are unary and carry it in
MatchSubstringOptions. So the mapping needs the second argument to be a string literal. Substrait'scase_sensitivityoption maps ontoignore_case, except forCASE_INSENSITIVE_ASCIIwhich has no Arrow equivalent.This matters for engines that ingest a PyArrow filter through Substrait, where an unmappable function turns into a hard failure rather than a fallback.
Component(s)
C++, Python