Generate the MobilityDuck UDF surface from the MEOS-API catalog#208
Open
estebanzimanyi wants to merge 77 commits into
Open
Generate the MobilityDuck UDF surface from the MEOS-API catalog#208estebanzimanyi wants to merge 77 commits into
estebanzimanyi wants to merge 77 commits into
Conversation
Register the quadbin / tquadbin cell-index types and their cast, accessor, and operator surface, with the rtree index module and build wiring.
… 1.4.4 DuckDB runs scalar, cast and table-function bodies on worker threads whose MEOS thread-local state is uninitialised; initialize MEOS once per worker thread at the registration chokepoints and table-function Init bodies, and consume the DuckDB 1.4.4 unique_ptr_cast Copy signature.
The restrict argument was dropped from the spatiotemporal relationships in the 1.4 orthogonalization; drop the non-canonical 3-arg overloads and use the 2-arg form in the tests.
Refresh expecteds that drifted with the pin: share-inclusive span adjacency, ln/log10/exp densification, and right-open step segment-duration bounds, all matching MobilityDB canonical output.
Pin the MEOS vcpkg port to ecosystem-pin-2026-06-22a (043d5e72), carrying the GEOS CMake CONFIG-target fix for the macOS build. Consume the pgtypes public leaf headers for date/timestamp arithmetic and drop the obsolete source patches the clean MEOS-only surface makes unnecessary.
The MEOS type-name catalog (meos/src/temporal/meos_catalog.c) defines the canonical type names in lowercase (tint, intspan, stbox, tgeompoint, ...). Name the DuckDB type accessors and SQL aliases to match that canonical for the temporal, span, box, and temporal-geo families, consistent with the set and spanset accessors that already use it. The third-party duckdb-spatial GEOMETRY type is left unchanged.
The MEOS catalog provides the tbigint type (T_TBIGINT, tbigint_in/out) in the pinned library. Register it in the binding the same way as tint: a BASE_TYPES entry, a DEFINE_TEMPORAL_TYPE accessor, a RegisterType call, membership in AllTypes(), and an alias-to-meosType catalog entry. This wires tbigint into the generic temporal cast and function loops and the catalog-driven generator.
Add a catalog-driven generator (tools/codegen_duck_udfs.py) that emits DuckDB scalar UDFs from the vendored MEOS-API catalog (tools/catalog/meos-idl-22a.json), organized by doxygen @InGroup group into one RegisterGenerated_<group> function each, mirroring the structure of the other ecosystem binding generators. The generated surface (src/generated/generated_temporal_udfs.cpp) is wired into the extension sources and registered last in LoadInternal, after every type accessor exists. Generated names carry a coexistence prefix so they run alongside the hand-written surface; a function is gated against the installed MEOS headers so only declared symbols are emitted.
…t guard MEOS keeps locale and collation, the session timezone, the PROJ context, and random-number state in thread-local storage, so every generated UDF body must run EnsureMeosThreadInitialized() before any MEOS call. After writing the output, scan every emitted body and exit non-zero if the guard is missing, so a future emit path cannot silently drop it.
…rations The frechetDistance, dynTimeWarpDistance, and hausdorffDistance functions are emitted by the catalog-driven generator (group meos_temporal_analytics_similarity). Add that group to RETIRED_GROUPS so the generated surface owns the canonical names (no coexistence prefix), and remove the corresponding hand registrations; the discreteFrechet and dynTimeWarp aliases and the *Path variants, which the generator does not emit, are kept.
Two ways a RETIRED_GROUPS retirement can silently lose coverage, now caught at generation time by reading the actual emitted output: - Trap 1 (split name): a name emitted both bare and g_-prefixed means a retired group and a non-retired group share it, so a query resolves only one spelling. Shared bare/operator dialect must be retired as a coherent wave. - Trap 2 (uncovered): every @sqlfn of a retired group must be emitted, or retiring it drops the function. A documented generator-shape gap kept by hand is allowlisted in RETIRE_UNCOVERED_OK with a reason; anything else is fatal. The checks are output-neutral (read-only over the generated file) and mirror the existing per-thread-init guard. RETIRE_UNCOVERED_OK currently holds dynTimeWarpPath/frechetDistancePath (LIST(STRUCT) path returns the generator cannot yet emit).
The vcpkg meos port linked only a partial family set (H3 off; cbuffer, pose, rgeo, pointcloud and the Arrow export absent), so the standalone libmeos the extension loads was a strict subset of the catalog surface the generator targets, and a faithful regen emitted unlinkable symbols. Enable every family at the pin (which already folds all family PRs and carries the MEOS_OBJECTS shared-link fix, proven by quadbin linking): CBUFFER, H3, POSE, POINTCLOUD and ARROW alongside JSON, NPOINT, QUADBIN. RGEO follows POSE automatically; RASTER stays off (PostgreSQL-only). H3 uses the system libh3 (auto-found); Arrow is header-only (vendored arrow and nanoarrow headers). POINTCLOUD links pointcloud-pg/lib/libpc.a, a build artifact not shipped in the tarball, so build it first from the vendored autotools tree (autogen, configure, make -C lib; the PGXS pgsql subdir is skipped). Port-version bumped so vcpkg rebuilds the port. The libmeos now loads with every family plus the 18 Arrow roundtrip helpers, a faithful regen from the catalog has zero of its 858 called symbols missing, and the release suite passes (1331 assertions, 60 cases).
With the vendored libmeos now built with every family, the generator emits the cross-family conversion UDFs that were previously dropped (tcbuffer_to_tfloat, ttext_to_tjsonb, tbigint_to_tquadbin, the h3/npoint conversions, ...). Make the generated registrations a faithful, buildable projection of the catalog again: - Regenerate src/generated/generated_temporal_udfs.cpp: the conversion registrations return; the file is once more a clean regen of the catalog. - meos_wrapper_simple.hpp: include the per-family public headers (alphabetical) so the family type and conversion declarations are visible to the generated translation unit; they live in the family headers, not the meos.h umbrella. - CMakeLists.txt: add the libh3 include path (meos_h3.h includes <h3api.h>), discovered with NO_CMAKE_FIND_ROOT_PATH since the vcpkg toolchain confines find_path to the vcpkg root and never sees the system /usr/include/h3.
…talog
The catalog carries function-shape metadata (shape.arrayReturn, from the
MEOS-API shapeinfer pass) plus the struct layouts (warp/Match = {i, j}).
Use it to generate the SETOF-returning similarity path functions instead
of hand-writing them:
- codegen: a new array-return shape (poc_path + emit_path_table) emits a
DuckDB TABLE function (bind/init/exec) whose output schema and per-row
marshalling are driven entirely by the catalog struct fields. The
registration goes through loader.RegisterFunction(TableFunction(...)),
registered over every temporal type. The retire-safety guard now counts
TableFunction registrations too.
- vendored catalog: enriched with the per-function shape and the struct
table (same 4465-function surface, shape added).
- retire the hand surface: the SimilarityPath table functions, the scalar
RunSimilarityPath path executors, and the non-canonical discreteFrechet /
dynTimeWarp aliases (MobilityDB-canonical names are frechetDistance /
dynTimeWarpDistance, both generated). frechetDistancePath / dynTimeWarpPath
RETURNS SETOF warp, so the canonical form is the table function.
- tests: the path tests use the canonical SETOF/table form
(SELECT i, j FROM ...Path(...)); 038 uses the canonical dynTimeWarpDistance.
Release suite: 1339 assertions in 60 cases pass.
A MEOS span/box function that can produce an empty result returns a NULL pointer (e.g. intersection_span_span on disjoint spans). The generated pointer-returning bodies marshalled that pointer straight through SpanToBlob, which reads sizeof(Span) from it -> a NULL dereference and a crash on the empty case (latent while the names carried the coexist prefix and were not exercised by the suite). Make the generator emit null-safe bodies for the pointer-returning span shapes (unary X->X and binary X,X->X, shared by span/spanset/stbox/tbox): use ExecuteWithNulls and, when MEOS returns NULL, set the row invalid and return an empty string_t -- mirroring the hand SpanFunctions handlers. Proven by exercising the generated function directly: the disjoint-span intersection now returns SQL NULL, the overlapping case returns the intersection. Release suite: 1339 assertions in 60 cases pass.
Extends the span/box null-safety (previous commit) to the rest of the generated scalar surface: any MEOS function that returns a NULL pointer for an empty/undefined result (set intersection/minus on disjoint inputs, temporal at/minus restrictions with no match, etc.) now maps to SQL NULL instead of dereferencing NULL in *ToBlob and crashing. Add centralized null-aware marshalling helpers TemporalToBlobN / SetToBlobN (if the pointer is NULL, set the row invalid via the mask and return an empty string_t) and route the pointer-returning emitter shapes through them with ExecuteWithNulls: emit_set (unary/binary set), emit_body, emit_body_binary, emit_body_ternary, emit_binary_tt, emit_scalar_first. Mirrors the hand binding's ExecuteWithNulls handlers; bool/scalar returns keep the simple Execute path. Proven by exercising the generated functions directly: set intersection on disjoint sets returns SQL NULL, the overlapping case returns the result. Release suite: 1339 assertions in 60 cases pass.
Drop the g_ coexist prefix so the generated UDFs own the bare canonical SQL names, and correct the generated geo return types: ever/always spatial relationships (group *_rel_ever) return BOOLEAN, temporal spatial relationships (group *_rel_temp) return tbool, the temporal distance tDistance returns tfloat, and geo conversions (X_to_t{geometry,geography,geompoint,geogpoint}) return the target type.
Scan the extended-family headers (cbuffer, h3, json, npoint, pointcloud, pose, quadbin, rgeo) in the pin/ABI gate, matching the per-family headers the generated translation unit includes, so the cross-family conversions are emitted.
Drop the hand tint value-accessor, trend, and temporal-distance registrations that the generated surface now owns with the canonical return types.
…ames 030_temporal_compops called the retired hand names temporal_teq/tne/tlt/tle/tgt/tge; the comparison surface is now generated as tEq/tNe/tLt/tLe/tGt/tGe (RETIRED_GROUPS meos_temporal_comp_temp). Update the test to the generated names.
…he composed catalog Vendor the complete MEOS-API catalog (4492 fns; the composed all-open-PR surface) and bump the pinned MEOS base to upstream master bb5f9e70 (all families + Arrow/Raster), regenerating the scalar UDFs. PIN and portfile REF both bb5f9e70; vcpkg port-version 7. Note: bb5f9e70 trails current upstream master (b529843ae8) by the arc-clip + cmake-ALL commits; rebasing the base to current master is a follow-up.
The pointcloud-pg autotools configure step fails to build on arm64-linux, and the binding surfaces no pointcloud type or function. Drop the pointcloud-pg build and the POINTCLOUD option from the meos port, and the meos_pointcloud.h include from the family header block.
…neration - Drop the g_ coexistence prefix for good (remove COEXIST_PREFIX and the --prefix flag); generated UDFs always register under the canonical @sqlfn name. Hand code is retired via RETIRED_GROUPS (generate-then-retire), never a transition prefix. - Add geometry-argument marshalling (poc_geo_temporal/emit_geo_temporal): generate the geo spatial-relationship surface (eContains/eIntersects/ eDwithin/tIntersects/tDwithin/... over tgeo x geometry, both argument orders) via GeometryToGSerialized(blob, tspatial_srid(t)); ever/always int -> nullable BOOLEAN, temporal -> tbool. - Restrict conversion UDFs to registered type families (REGISTERED_FAMILIES) so no UDF is emitted for an unregistered family. - Generate temporal_lcss_distance via the (Temporal, Temporal, scalar) shape. Build + full test suite green (1339 assertions / 60 cases).
Per the per-binding-canonical generator policy, each binding owns the executable form of its GENERATION.md. The script bumps the vcpkg MEOS portfile REF + SHA512 and vcpkg.json port-version, writes tools/catalog/PIN, vendors the catalog produced by MEOS-API run.py, and (given the freshly installed headers) runs the generator pin-gated and builds the extension.
- portfile REF -> b71198726a (current upstream master), new SHA512, port-version 10; PIN + vendored catalog meos-idl-b71198726a.json (4519 functions / 2555 @sqlfn), replacing the stale meos-idl-26a.json. - Enable RASTER=ON: the raquet tile type (T_RAQUET, MobilityDB #1332) is a GDAL-free varlena value type that links standalone (15 raquet symbols in libmeos), so the standalone MEOS surface matches the full catalog. - Regenerate src/generated/generated_temporal_udfs.cpp (pure add-only, +36 registrations): picks up the completed geo-first spatial-relationship grid (eDisjoint/eIntersects/eTouches/eDwithin/tDwithin with geometry as the first argument), now that the MEOS API exposes those symbols. Full test suite green (1339 assertions in 60 test cases).
reg_scope maps the abstract tpoint_ supertype to its two point subtypes, and ret_temporal_type takes the output temporal type from the catalog's single unambiguous SQL return subtype (falling back to the name heuristics only for input-polymorphic returns, whose result preserves the input type). This generates getX/getY/getZ, azimuth, speed, cumulativeLength, angularDifference, isSimple, length and the atGeometry/atValue/minusGeometry/ minusValue restrictions on tgeompoint and tgeogpoint, and gives trend a tint result and derivative a tfloat result, matching the MobilityDB SQL surface. Full test suite green (1339 assertions in 60 test cases).
…talog reg_scope maps the abstract tspatial_ supertype to all four geo temporal types. This generates setSRID, transform and transformPipeline (each preserving the operand type) and asText/asEWKT (returning text) on tgeometry, tgeography, tgeompoint and tgeogpoint. Full test suite green (1339 assertions in 60 test cases).
estebanzimanyi
force-pushed
the
feat/duck-generator-canonical
branch
from
e47ba3c to
5f6f745
Compare
Pins are abandoned: the vcpkg meos portfile REF is a raw upstream MobilityDB/MobilityDB commit that tracks master. Bump REF + SHA512 to master 4bf009b1 (port-version 11) and vendor the catalog regenerated from that checkout (meos-idl-4bf009b120.json, 4526 functions / 2555 @sqlfn). Remove the legacy pin-model tooling — tools/catalog/PIN and tools/regen-from-pin.sh — now that freshness is the base REF versus upstream master. Full test suite green (1339 assertions in 60 test cases).
The span and spanset headers each declared a text-based type accessor -- SpanTypes::textspan() and SpansetTypes::textspanset() -- with no definition, no RegisterType call, and no member in AllTypes(). These are not MEOS types: a span is an interval over an ordered base type with arithmetic, which text is not, so MEOS defines neither T_TEXTSPAN nor T_TEXTSPANSET (the meos_catalog.c type array carries textset but no text span or spanset). They were binding-only declarations that break the base-value x container orthogonality -- text has a set but not a span/spanset. The generator's SPANSET_TYPES map also listed a textspanset accessor, so a hypothetical textspanset_* catalog function would have been routed to the undefined accessor. No such function exists, so nothing referenced these today. Drop the two declarations and the generator's textspanset map entry and _to_ return regex. No generated output changes (nothing emitted them).
The set, span and spanset families each hand-listed their type accessors, RegisterTypes, AllTypes, alias->MeosType and GetChildType/GetSetType/ GetBaseType mappings. Every entry is uniform macro boilerplate whose only per-family variation is the (base-value x container) grid -- which the catalog MeosType enum already encodes exactly: T_<BASE><SUFFIX> exists for int/bigint/float/text/date/tstz sets and for int/bigint/float/date/tstz spans and spansets (text has a set but no span/spanset). Derive that grid from the enum in codegen_duck_udfs.py and emit the whole registration into src/generated/generated_type_registration.cpp, retiring the three hand blocks. The generated lists are identical to the retired ones, so the type set, cast wiring and suite are unchanged; deriving them from the enum makes the base-value x container orthogonality mechanical -- a base with no span simply produces no accessor, so a phantom span type can no longer be fabricated by hand.
The set-family generator treated the second argument of a (Set, scalar)->Set function as a set element, inferring the set type from it. That is correct for setUnion/setMinus/setIntersection, whose scalar co-varies with the set base ((geomset, geometry), (intset, integer)), but wrong for round/setSRID/transform, whose integer argument is a precision or SRID that stays fixed while the set type varies. round therefore registered on intset (from the integer element) instead of floatset, and setSRID/transform registered on intset instead of their geo set types. Read the set type from the catalog sqlSignatures instead: when a two-argument overload set keeps the same second-argument type across differing first-argument set types, the second argument is a fixed parameter, so register over the declared set types (round -> floatset) rather than the element scalar. The extended geo sets (geomset/geogset/...) are gated to their own family files, so setSRID/transform now emit nothing here instead of a spurious intset overload.
The two-argument round(floatset, integer) is now generated from the catalog set_round signature, so its hand registration was a duplicate. Drop it and keep only the one-argument round(floatset) default-arg form: the catalog declares sqlArity 1..2 but carries no default value, so the generator cannot yet emit the shorter overload. The generated two-argument overload and the remaining one-argument hand overload together preserve both call forms.
The catalog now carries the SQL DEFAULT of an optional argument as argDefaults on each sqlSignatures entry (round(floatset, integer DEFAULT 0) -> argDefaults [null, "0"]). The set generator reads it: a scalar-param set function with an optional trailing argument also emits the shorter overload, substituting the declared default (round(floatset) -> set_round(s, 0)), alongside the existing full-arity round(floatset, integer). Both arities are now generated, so the hand SetFunctions::Floatset_round registration, body and declaration are removed. The vendored catalog is regenerated at the same MobilityDB REF (de560130); its only change is the additive argDefaults field. Consumes MobilityDB/MEOS-API#40.
…m the catalog round/floor/ceil/degrees/radians are float-base-value scalar transforms. They are now generated from the MEOS-API catalog at both arities across every container that carries them (floatset, floatspan, floatspanset, tfloat, plus tgeompoint/tgeogpoint/tgeometry/tgeography for round), including the shorter DEFAULT-arg overload - round(x, integer DEFAULT 0) and degrees(x, boolean DEFAULT FALSE) - folded from the sqlSignatures argDefaults metadata. The generator gains a set-track scalar-param shape (setcsc) alongside the existing span-track (csc) shape and the temporal defaulted-arity hook, so a (Container, fixed-scalar) transform is name-scoped to its float base value and registered at both the full and the defaulted arity. All corresponding hand registrations, bodies, and declarations are retired. Only round(DOUBLE) - the scalar base helper with no catalog signature - stays hand-written in span_functions.cpp.
Bump the vcpkg meos portfile REF (+ SHA512, port-version) to the current MobilityDB master and re-vendor the generated catalog from that same commit. The generated UDF surface is unchanged (regeneration produces an identical src/generated), and the full suite stays green (1372 assertions / 62 cases); this only moves the MEOS base forward off a stale pin.
The poc_ (proof-of-concept) prefix on the shape-dispatch functions is a relic name; these are the canonical shape classifiers that pair with the catalog shape field and the emit_ emitters. Rename poc_* -> shape_* uniformly. This is generator-internal only: regeneration produces byte-identical src/generated output, so the built extension and the suite are unchanged.
Bump the vcpkg meos port (REF + SHA512, port-version 4) to the current MobilityDB master and re-vendor tools/catalog/meos-idl.json regenerated from it. The refreshed catalog carries the array-return shape metadata now resolved at the source: element type on all 165 array-returning accessors and the trailing int* count length parameter on set_spans, spanset_spans, spanset_spanarr, spanset_sps and the jsonb/pointcloud getValues accessors. It also picks up the upstream fix that tags ttext_initcap with its own @csqlfn, so the generated temporal text function is initcap instead of the mistagged lower.
…and implementations Drive the trailing-count array-return convention (shape.arrayReturn with element + lengthFrom.name=count) through codegen_duck_udfs.py so the LIST-returning accessors are emitted directly from meos-idl.json, gated on each overload's canonical sqlSignatures return type: getValues(<set>) / getValues(tbool) -> LIST(<element>) timestamps(<temporal>) (9 types) -> LIST(TIMESTAMP_TZ) spans(<set> | <spanset> | <temporal>) -> LIST(<span type>) tboxes(tint | tfloat) -> LIST(tbox) Element marshalling picks the DuckDB child type from the catalog element (scalars, epoch-corrected TIMESTAMP_TZ/DATE, text, and Span/STBox/TBox value structs as BLOB-backed named LIST types) with no name heuristics, and the SQL element type comes from the per-overload sqlSignatures so tint/ tfloat/ttext getValues correctly stay as their spanset/set forms rather than being mis-emitted as LIST. Retire the now-superseded hand registrations and their implementations: Set_values, Set_spans, Spanset_spans, Temporal_timestamps (incl. the five geo-family regs) and Tnumber_tboxes. The generated accessors are the sole implementation; the full suite passes (1372 assertions / 62 cases).
…al ports The meos vcpkg port enables H3=ON (find_library h3 + find_path h3api.h) and RASTER=ON (find_package GDAL), but declared neither dependency, so it relied on a system libh3-dev / libgdal-dev. The DuckDB extension CI builds each triplet inside a manylinux container that ships neither, so the meos configure aborted on arm64 with "H3=ON was requested but libh3 was not found" (the x64 job only appeared to pass because a developer's host happens to have libh3 installed). Declare h3 and gdal as vcpkg dependencies of the meos port so both families are provisioned by vcpkg on every triplet (x64/arm64/osx/wasm) — the cross-arch equivalent of the provision-meos action's apt libh3-dev / libgdal-dev. This is the "all families except POINTCLOUD" tier: POINTCLOUD FORCE-requires the vendored pgPointCloud PGXS build (a live pg_config) which has no vcpkg port, so it stays off until a vcpkg pointcloud port lets the port flip to true -DALL.
Enable every optional MEOS family in the vcpkg meos port via -DALL=ON, replacing the explicit family list that held POINTCLOUD OFF. POINTCLOUD now builds without a live pg_config: its pgPointCloud core (libpc) is compiled directly from lib/*.c as a CMake static library (MobilityDB #1370, on master 79f24b7948), so -DALL builds in the vcpkg/manylinux model with no system PostgreSQL. Its only new external dependencies are libxml2 and zlib, added to the port's vcpkg.json alongside h3 (H3) and gdal (RASTER); port-version bumped to 6. Bump the port REF/SHA512 to master 79f24b7948 and re-vendor tools/catalog/meos-idl.json from the same commit so the catalog stays a subset of the installed libmeos surface (portfile REF == catalog pin). The generated UDF surface is unchanged and the suite is green (1372 assertions in 62 test cases).
Retire the hand-registered timeSpan/valueSpan/tbox and tint/tfloat temporal-conversion scalar functions (@InGroup meos_temporal_conversion) in favour of catalog-driven generation. A new sqlSignatures-driven shape in tools/codegen_duck_udfs.py emits the unary Temporal -> Span/SpanSet/TBox/STBox conversions (temporal_to_tstzspan -> timeSpan, tnumber_to_span -> valueSpan, tnumber_to_tbox -> tbox); each overload's container return type comes straight from the catalog sqlSignatures (tint -> intspan, tbigint -> bigintspan, tfloat -> floatspan), replacing the previous name heuristic. The group is added to RETIRED_GROUPS and the retire-safety check verifies every @sqlfn of the group is generated. The generated surface also adds the tbigint overloads the hand layer was missing. The temporal-number cast helpers and their cast registrations are unchanged; only the scalar function registrations are retired. Full suite green (1372 assertions / 62 cases).
…enerated surface
The before/after/overbefore/overafter (time axis) and left/right/overleft/
overright (numeric axis on tnumber x {numspan, tbox, tnumber}) positional
predicates, plus the tspatial x {stbox, tspatial} spatial-axis variants, are
already emitted from the catalog sqlSignatures in the generated UDF file,
including the <<, >>, &<, &> operator forms. Delete the three redundant hand
registration blocks in temporal.cpp.
The generated surface follows the catalog exactly: positional overloads are
same-base-family only (before_temporal_temporal is tbool/ttext, numbers go
through _tnumber_tnumber, spatial through _tspatial_tspatial), so the old hand
loop's spurious mixed pairs (e.g. before(tint, tfloat)) are dropped, aligning
the DuckDB surface with MobilityDB. The non-canonical temporal_* snake aliases
are dropped in favour of the bare names.
Migrate the tests that used the retired snake aliases to the bare names
(034_temporal_posops, 040/041/042 parity).
…generated surface The before/after/overbefore/overafter time-axis predicates on tgeometry, tgeography and tgeogpoint against tstzspan (both directions) are already emitted from the catalog sqlSignatures in the generated UDF file — their backing MEOS exports (before_temporal_tstzspan etc., @InGroup meos_temporal_bbox_pos) carry the bare @sqlfn names and dispatch generically over every temporal subtype. Delete the redundant TIME_POS_REG macro blocks in the three geo ops files. These only ever registered the non-canonical temporal_* snake aliases; the bare names come from the generated surface. This removes the last hand-registered snake positional aliases in the binding.
Bump the meos vcpkg port REF to the current MobilityDB master (5c5b3cd25b) and re-vendor tools/catalog/meos-idl.json from the same SHA (portfile REF == catalog pin). -DALL stays on, so libmeos carries every family. This pulls in the time-axis span/set positional tag fix (before/after/overbefore/ overafter on tstzspan/dateset now resolve to their own name/operator instead of the value name left/right), so the generated surface emits before(tstzspan, ...) rather than left(tstzspan, ...); and the pointcloud gated-declaration catalog entries now captured by the header parser. Generated code regenerates byte-identical against the fresh installed headers; suite green (1372 assertions / 62 cases).
…rations The set/span/spanset relative-position operators (left/right/overleft/overright + the time-axis before/after/overbefore/overafter, with the <<//>>/&</&> operators) are now fully generated from the catalog. Close the one remaining shape gap: the mixed span<->spanset pair (left_span_spanset / left_spanset_span, ...), which the same-container (X,X) case skipped because the two operands are different containers. A new shape marshals each operand as its own container and registers the concrete type pairs (intspan x intspanset, ...) from the catalog sqlSignatures. Add meos_setspan_pos to RETIRED_GROUPS (retire-safety now covers every @sqlfn of the group) and delete the hand span_left/set_left snake registrations and their operator forms in span.cpp/set.cpp. duckdb_functions() shows 0 snake positional names remain, the bare left/right/before/after surface intact, and the mixed span<->spanset overloads present; suite green (1372 assertions / 62 cases).
…trations The set/span/spanset topological operators (contains/contained/overlaps/ adjacent and the @>/<@/&&/-|- operators) are generated from the catalog span and set shapes, under their bare portable names in both argument orders, including the symmetric value-first adjacent(value, span). The hand set_contains/span_contains snake-case functions and operator registrations are deleted; meos_setspan_topo joins RETIRED_GROUPS so the retire-safety ledger fails the build if the catalog ever drops coverage. The value-first adjacent(value, span) overload is backed by the adjacent_<type>_span MEOS-C functions, so the catalog is re-vendored and the MEOS port advanced to pick them up.
The meos port pulls h3 as a dependency (via -DALL → H3). The h3 v4.3.0 port at the builtin-baseline builds with ENABLE_LINTING ON, which sets the h3 target's C_CLANG_TIDY property. Under the wasm32-emscripten toolchain the runner's clang-tidy then runs on h3's own sources with warnings-as-errors and the vendored h3 code trips readability-braces-around-statements and non-const-global findings, failing the DuckDB-Wasm build. Add a version-identical h3 overlay port (same v4.3.0 REF and SHA512) that passes -DENABLE_DOCS=OFF -DENABLE_FORMAT=OFF -DENABLE_LINTING=OFF, so the build no longer lints upstream h3 sources. Newer upstream vcpkg h3 ports carry these same flags. This mirrors the existing gdal/proj/sqlite3 overlays under vcpkg_ports.
The temporal bounding-box topological operators (contains/contained/
overlaps/same/adjacent and @> <@ && ~= -|-) for temporal × temporal,
temporal × tstzspan and tnumber × {numspan, tbox} are generated from the
catalog sqlSignatures. Delete the hand registrations in temporal.cpp,
which duplicated the generated bare surface and additionally exposed the
non-canonical temporal_* snake aliases and a spurious mixed-family
cross-product, and add meos_temporal_bbox_topo to RETIRED_GROUPS.
Migrate the affected parity tests to the canonical bare names. Adjacent
uses TBox (value × time) adjacency, which holds when the operands are
adjacent in at least one dimension; the deleted hand temporal × temporal
handler computed time-only adjacency, so adjacent on two tints whose
integer value spans share a boundary is now correctly true.
Mirror MobilityDB's tools/codegen/inherited/ for DuckDB: a manifest, a per-behaviour template and a driver that emit the DuckDB scalar-function registrations every Temporal<T> subtype inherits (topological, as the first behaviour) — the same inherited surface PostgreSQL generates from its .sql templates, with only the output form changed. The live DuckDB surface is generated by tools/codegen_duck_udfs.py from the MEOS-API catalog; this generator is a design reference for the template-driven approach, kept honest by --validate, which proves it reproduces the catalog-generated topological surface byte-for-byte.
Add the circular-buffer temporal family to the DuckDB binding: the cbuffer value type and the tcbuffer temporal type, plus their catalog-generated operation surface. The type + text-I/O boundary is hand-registered in src/cbuffer/tcbuffer.cpp (cbuffer and tcbuffer are surfaced as BLOB with the canonical alias; the four VARCHAR in/out casts marshal through cbuffer_in/out and tcbuffer_in / temporal_out). Everything else is generated from the MEOS-API catalog by tools/codegen_duck_udfs.py: registering the family adds the Cbuffer boundary marshaller (a 4-byte-header varlena) and resolves the tcbuffer_* / _tcbuffer names to the tcbuffer type, yielding 77 functions — accessors, conversions (tfloat radius, tgeompoint centre), temporal distance, and the ever/always/ temporal spatial relationships. tcbuffer_make is the one function left ungenerated: its C signature is two polymorphic Temporal* operands (a tgeompoint and a tfloat) that the catalog cannot yet type per argument, so the generator excludes it (hetero-temporal- args) rather than register wrong argument types; it stays reachable through the text-I/O cast.
tcbuffer is a TSpatial<T> subtype, so it inherits the spatial behaviour previously confined to the geometry/geography temporal types, as well as the generic Temporal<T> behaviour of every temporal type. The first registration scoped tcbuffer only through its own tcbuffer_* / _tcbuffer names, which gave it the native operations but not the inherited surface: asText/asEWKT/SRID/ setSRID/transform, the bounding-box topological and positional operators, and the generic temporal duration/atTime/minusTime/deleteTime were all missing (a few appeared to work only through incidental BLOB-alias coercion). Introduce SPATIAL_ALLTYPES (the geo types plus tcbuffer, extensible to the other spatial families) and use it wherever the surface is "all spatial subtypes": the abstract tspatial_* scope, the generic Temporal<T> registration loop, and the time-restriction type set (ALL_TEMPORAL_ACCS). The geo supertype tgeo stays geometry+geography only. Also generate the shorter-arity overload of scalar-returning functions with a trailing default (duration(temporal), asText/asEWKT(tspatial)), which was previously hand-written for the geo types only. tcbuffer now inherits the full surface (asText, asEWKT, SRID, atTime with interpolation, the topological operators, etc.). This is the pattern to replicate to the remaining spatial families.
startValue and endValue return the family base value through the per-type
MEOS symbol (tcbuffer_start_value / tcbuffer_end_value -> Cbuffer *).
The generator gains a base-value pointer-return shape in shape_emittable:
BASEVAL_PTR_RET is the set of PTR_RET entries that are neither a temporal
handle nor a container (currently Cbuffer). The owned pointer marshals
through {Base}ToBlobN, and reg_scope keys the per-type symbol to its own
family, so startValue/endValue register on tcbuffer returning cbuffer.
Add the GSERIALIZED-return shape to the generator: an owned geometry return marshals to the DuckDB GEOMETRY type via GSerializedToGeometry (the inverse of GeometryToGSerialized). The shape is scoped to the non-geo spatial families so it does not double-register against the hand-written geometry accessors on the geo temporal types. For tcbuffer this generates convexHull(tcbuffer) -> geometry. convexHull is a linear (GEOS-polygonized) hull of the traversed area, not a curved geometry; the test uses a zero-radius input for a stable, arc-free result.
The committed tools/catalog/meos-idl.json was a point-in-time snapshot that drifts from the libmeos the extension is built against. Drop it and derive the catalog in CI from MobilityDB via the shared MEOS-API provision-meos action — the same model the JVM bindings (JMEOS/Spark/Flink/Kafka) use. A new generate workflow reads the MobilityDB commit from the vcpkg MEOS port REF (the single source of truth for which MobilityDB the port builds), derives meos-idl.json from that commit, regenerates src/generated, and fails if the committed generated code has drifted. The committed src/generated/*.cpp remains the buildable snapshot the distribution pipeline compiles. The freshly derived catalog is byte-identical in effect: regenerating src/generated from it leaves the committed files unchanged.
…ued bases Extend shape_binary to accept a base-value passed by pointer as the 2nd operand (BASEVAL_PTR_IN, the input mirror of BASEVAL_PTR_RET: PTR_IN minus the temporal handles and collection/box pointers). The operand is marshalled via PTR_IN's BlobTo<Base> and freed after the MEOS call, reusing the existing owned-arg pattern. Making the base value marshallable as a binary operand generates the whole (tcbuffer, cbuffer) surface uniformly from the catalog: the value comparisons (eEq/aEq/tEq and the ?=/%= operators), tDistance/nearestApproachDistance/ nearestApproachInstant, the spatial relationships (eIntersects/eContains/ eCovers/tTouches/...), and the range-point restrictions atValue/minusValue. The set is derived, so Pose/Npoint/Jsonb join automatically once their PTR_IN entry and DuckDB type land. 30 net-new registrations, no hand collisions.
The (Temporal<T>, set-of-T) value restrictions atValues/minusValues were hand-registered for tint/tfloat/ttext. Generate them from the catalog instead: temporal_at_values / temporal_minus_values carry explicit sqlSignatures pairing each temporal type with its element-set type, so a new shape_temporal_restrict_sig registers one overload per pairing whose accessors are both registered Duck types (int/bigint/float/text sets; cbufferset/geomset/etc. are skipped until their set type lands). The pairing is taken straight from the catalog — no name heuristic — and the dispatch runs only for restrictions shape_temporal_span did not claim (atTime), so each function is registered by exactly one path. The generated surface is behaviour-equivalent to the retired hand regs (both call the MEOS set restriction) and adds the previously-missing tbigint overload, now covered by a test.
Extend the sqlSignatures-driven restriction shape to the joint value x time box: tnumber_at_tbox / tnumber_minus_tbox pair each number temporal type with a tbox in their catalog sqlSignatures, so shape_temporal_restrict_sig now accepts a TBox second operand (marshalled via the already-emitted BlobToTbox) and registers one atTbox/minusTbox overload per pairing. The 3-operand stbox restrictions (atStbox, with a border bool) are a separate shape and stay hand for now. Retire the hand atTbox/minusTbox regs (tint/tfloat) and their now-dead Tnumber_at_tbox / Tnumber_minus_tbox wrappers. Behaviour-equivalent — the wrappers called the same MEOS tnumber_at_tbox / tnumber_minus_tbox — and the generated surface adds the previously-missing tbigint overload, now covered by a test.
Bump the MEOS pin to MobilityDB f26ad75164, which names the temporal number span and span set value restrictions atSpan/atSpanset/minusSpan/ minusSpanset. The catalog attaches sqlSignatures to those wrappers, so the generator emits atSpan/atSpanset/minusSpan/minusSpanset for tint, tbigint and tfloat. Point the tint and tfloat span-restriction tests at the canonical names.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What
Generates the MobilityDuck UDF surface from the MEOS-API catalog, replacing the hand-written scalar registrations.
tools/codegen_duck_udfs.py— the catalog-driven generator, organized by doxygen@ingroupgroup (oneRegisterGenerated_<group>per group + aRegisterGeneratedTemporalUdfsaggregator), mirroring the JMEOS and Spark generators. Every emitted body carries the per-threadEnsureMeosThreadInitializedguard, enforced by the generator.src/generated/generated_temporal_udfs.cpp— the generated surface (all families), wired into the extension build.tools/catalog/— the vendored catalog the generator reads.eq/ne/cmp,eEq/aEq/tEq,contains/overlaps,setUnion/tAdd, …).Covered by the full 60-case sqllogictest suite, which CI runs across the Linux / macOS / wasm matrix.
Scope notes
eIntersects/eDwithin,tEq(geometry, tgeo)) keep their hand geo registrations — the generator auto-excludesGSERIALIZED/geo-argument functions (codegen_duck_udfs.py:240) because it does not yet marshal aGSERIALIZEDargument. Teaching it that marshalling is the next generator step; type registration, casts, aggregates, and table functions also stay hand-written by design.